Computing device, computing method and board

Abstract

The present invention provides a computing device, computing method, and board, relating to the field of data processing technology. The computing device, through a scheduling unit, determines the number of times any row in a second matrix participates in a multiplication operation, and indicates the number of multiplications for each row to a cache unit, allowing the cache unit to know in advance the number of times each row in the second matrix participates in the operation. Consequently, the cache unit can cache data more efficiently, thereby improving the computing efficiency of the computing device.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of data processing technology, and in particular to a computing device, a computing method, and a board. Background Art

[0002] With the development of computer vision algorithms, machine learning algorithms, and emerging AI large-scale models, large-scale matrix operations are required in many tasks. In particular, there is an increasing demand for higher computational efficiency in matrix multiplication operations.

[0003] Existing computing devices have problems with slow computing speed and low computing efficiency when performing matrix multiplication operations, making it difficult for existing computing devices to meet the computing requirements of some task scenarios. Summary of the Invention

[0004] The embodiments of the present application provide a computing device, a computing method, and a board for quickly computing multiplication between matrices, thereby improving computing speed and efficiency.

[0005] In a first aspect, an embodiment of the present application provides a computing device, which is applied to a multiplication operation of a first matrix by a second matrix, and includes a scheduling unit, a cache unit, and at least one computing unit;

[0006] The scheduling unit is configured to determine, based on the non-zero elements in the first matrix, the number of multiplications of any row in the second matrix that participates in the multiplication operation, and indicate the number of multiplications of each row in the second matrix to the cache unit;

[0007] The cache unit is configured to read and cache non-zero elements of each row in the second matrix according to the number of multiplications of each row;

[0008] The scheduling unit is further configured to input each non-zero element of any row in the first matrix into one of the operation units;

[0009] The operation unit is used to read the non-zero elements corresponding to each row of the second matrix participating in the multiplication operation in the cache unit according to each non-zero element in any row of the first matrix, and perform matrix multiplication and merging operations to obtain corresponding elements in the result matrix.

[0010] In a second aspect, an embodiment of the present application provides a computing method, which is applied to the computing device according to the first aspect, wherein the computing device includes a scheduling unit, a cache unit, and at least one computing unit. The method is used for multiplying a first matrix by a second matrix, including:

[0011] determining, based on non-zero elements in the first matrix, the number of multiplications of any row in the second matrix that participates in the multiplication operation, and indicating the number of multiplications of each row in the second matrix to the cache unit;

[0012] Reading and caching non-zero elements of each row in the second matrix according to the number of multiplications of each row;

[0013] Inputting each non-zero element of any row in the first matrix into one of the operation units;

[0014] According to each non-zero element in any row of the first matrix, the non-zero elements corresponding to each row of the second matrix participating in the multiplication operation are read in the cache unit, and a matrix multiplication and merging operation is performed to obtain the corresponding elements in the result matrix.

[0015] In a third aspect, an embodiment of the present application provides a board card, which includes the computing device as described in the first aspect.

[0016] The present application provides an arithmetic device, arithmetic method, and a board. The arithmetic device can be used for multiplying a first matrix by a second matrix. The arithmetic device includes a scheduling unit, a cache unit, and at least one arithmetic unit. During the operation, the scheduling unit can determine in advance the number of multiplications of each row in the second matrix based on the non-zero elements in the first matrix, and can indicate the number of multiplications of each row to the cache unit, so that the cache unit can accurately know in advance the number of times the elements in each row of the multiplier matrix participate in the multiplication. In this way, when reading and caching the non-zero elements of each row of the second matrix, the cache unit can accurately determine which elements cached in the cache unit should be retained and which should be deleted as soon as possible based on the number of multiplications of each row. This allows the elements of the rows that will subsequently participate in the operation to be retained in the cache unit for as long as possible, avoiding the increase in the number of repeated element reads due to their early deletion. It also allows the elements of the rows that will no longer participate in the operation to be released as soon as possible, avoiding their excessive occupancy of cache space. Therefore, making a high reuse rate decision on the elements in the cache unit based on the number of multiplications can reduce the number of repeated data reads and the time consumed by data reading. Based on this, through the close cooperation of the scheduling unit and the cache unit, on-chip data with a higher hit rate can be prepared as early as possible for the calculation process of the calculation unit, so that the calculation unit can calculate quickly, thereby improving the overall calculation speed and efficiency of the calculation device. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0018] Figure 1 Schematic diagram of matrix multiplication for inner product operation;

[0019] Figure 2 Schematic diagram of matrix multiplication for outer product operation;

[0020] Figure 3 Schematic diagram of matrix multiplication of Gustafson operation;

[0021] Figure 4 This is one of the structural diagrams of the computing device provided in the embodiment of the present application;

[0022] Figure 5 The second structural diagram of the computing device provided in the embodiment of the present application;

[0023] Figure 6 A schematic diagram of the operation flow of the computing device provided in an embodiment of the present application;

[0024] Figure 7 A schematic diagram of the rearrangement of element positions provided in an embodiment of the present application;

[0025] Figure 8 A schematic diagram of a read / write strategy based on a frequency lock provided in an embodiment of the present application;

[0026] Figure 9 A schematic diagram of a multi-stage combiner provided in an embodiment of the present application;

[0027] Figure 10 A schematic diagram of a merge tree provided in an embodiment of the present application;

[0028] Figure 11 A schematic diagram of multiple cache slices provided in an embodiment of the present application;

[0029] Figure 12 A schematic diagram of the PFCSR compression method provided in an embodiment of the present application;

[0030] Figure 13 A schematic diagram of the calculation method provided in the embodiment of the present application;

[0031] Figure 14 A schematic diagram of a board provided in an embodiment of the present application.

[0032] The above drawings illustrate specific embodiments of the present disclosure, which will be described in more detail below. These drawings and textual descriptions are not intended to limit the scope of the present disclosure in any way, but rather to illustrate the concepts of the present disclosure to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION

[0033] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all possible embodiments consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.

[0034] In the technical solutions of the embodiments of this application, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0035] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and corresponding operation entrances must be provided for users to choose to authorize or refuse.

[0036] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described in this application as "exemplary" or "for example" should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0037] In the embodiments of this application, if words such as "first" and "second" are used, they are intended to distinguish between identical or similar items with substantially the same functions and effects. For example, the first electronic device and the second electronic device are merely used to distinguish between different electronic devices and do not limit the order of precedence. Those skilled in the art will understand that words such as "first" and "second" do not limit the quantity or execution order, and words such as "first" and "second" do not necessarily mean that they are different.

[0038] In the embodiments of this application, "at least one" refers to one or more, and "more" refers to two or more. "And / or" describes the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0039] In order to clearly understand the technical solution of this application, the prior art is first introduced in detail.

[0040] Matrices have a wide range of applications in fields such as computer science and engineering. For example, sparse matrix multiplication, which involves a large number of nonzero elements, is widely used in fields such as statistical mathematics, computer vision algorithms, machine learning algorithms, and emerging large models. The computational efficiency of sparse matrix multiplication significantly impacts task performance.

[0041] The multiplication between matrices can be referred to as matrix multiplication, which includes various operations such as inner product, outer product and Gustavson operation. Figures 1 to 3 Introducing different forms of matrix multiplication. Figures 1 to 3 The blank squares represent zero elements in the matrix, and the squares filled with slashes represent non-zero elements in the matrix.

[0042] Figure 1 This is a schematic diagram of matrix multiplication of inner product operation, such as Figure 1 As shown, during the inner product operation, the rows of the first matrix (MatA) are inner-producted with the columns of the second matrix (MatB), yielding an element at the corresponding position in the resulting matrix (MatC). By traversing all rows of MatA and performing the inner product operation with the corresponding columns of MatB, the element at each position in MatC is calculated. After calculating all the elements at each position in MatC, the elements are concatenated (concat) according to their positions to yield the complete resulting matrix MatC. The word "concat" can be understood as concatenation or combination.

[0043] Matrix multiplication based on the inner product operation has certain advantages, such as good output locality, which allows a single operation to fully compute an output point. Furthermore, the rows and columns of MatA or MatB are reused in a clear format, making scheduling simple and enabling easy task splitting and parallelization. However, matrix multiplication based on the inner product operation also has certain disadvantages. For example, because the rows of MatA and the columns of MatB are required as input, the compressed dense vectors of MatA or MatB need to be converted into sparse rows with zeros during the operation in the case of sparse representation. Low sparsity results in extremely low utilization of the operator and poor performance.

[0044] Figure 2 This is a schematic diagram of matrix multiplication of outer product operation, such as Figure 2As shown in the figure, during the outer product operation, the columns of the first matrix (MatA) are outer-producted with the rows of the second matrix (MatB), resulting in the partial sums in the result matrix (MatC). By traversing and calculating all rows and columns in MatA and MatB, the partial sums at all positions in MatC are obtained. The partial sums at each position are further merged to obtain the complete result matrix MatC. Merge can be understood as accumulating the partial sums at each position at their respective positions.

[0045] Matrix multiplication based on the outer product operation has certain advantages. For example, the outer product is based on point-to-point multiplication, which does not require the positional relationship between the input and output. In sparse representation formats, all multiplication operations are valid, resulting in high processor utilization. Furthermore, row-column splitting allows for parallel computation and input reuse. However, matrix multiplication based on the outer product operation also has certain disadvantages. For example, the parallel splitting of the outer product operation can lead to significant load imbalance, reducing overall computational utilization. Furthermore, the outer product operation requires the accumulation of MatC partial sums. For MatC partial sum accumulation in sparse representation, hash mapping or heap sorting methods require high storage space and may not be computationally feasible for large matrices. Using various comparison and sorting methods results in the partial sum accumulation process being positively correlated with the output size, making the merging process difficult to parallelize, resulting in a performance bottleneck shifting to merging efficiency. For large matrices, the MatC partial sums require frequent read and write communications to main memory, amplifying the task input / output (IO) load and becoming an IO bottleneck.

[0046] Figure 3 The diagram below shows the matrix multiplication of Gustafson operation, which can also be called row-to-row operation. Figure 3 As shown, an element in any row of the first matrix (MatA) is multiplied with an entire row of the second matrix (MatB), where the entire row is the row corresponding to the column of the element in MatA. For example, the element in row 0 and column 0 of MatA is multiplied with every element in row 0 of MatB. For another example, the element in row 0 and column 1 of MatA is multiplied with every element in row 1 of MatB. For another example, the element in row 1 and column 2 of MatA is multiplied with every element in row 2 of MatB.

[0047] An element in MatA is multiplied by an entire row of elements in MatB, resulting in a partial sum for that row. The position of each partial sum corresponds to the position of the element in MatB at the time the partial sum was obtained. These partial sums can be considered intermediate results; they must be merged to form the elements in the final result matrix (MatC). After merging, all partial sums of elements in the same row of MatA are combined into a single row, and the resulting row in MatC is the same row as the elements in that row in MatA before the multiplication. Similarly, by traversing each element in MatA and multiplying it with the corresponding row of MatB, the corresponding partial sum is obtained. Merging the partial sums of all rows row by row yields the element at each position in MatC. Concatenating these elements yields the complete result matrix MatC. It should be understood that since 0 multiplied by any number equals 0, the Gustafson operation can be performed only on non-zero elements.

[0048] Matrix multiplication based on the Gustafson operation has certain advantages. For example, like the outer product algorithm, all multiplication operations performed are valid, resulting in high computing device utilization. Compared to the outer product algorithm, by replacing row-by-column multiplication with row-by-row multiplication, the entire task operation process is split, reducing the size of the partial sums from the matrix to the row. This algorithm also has stronger locality in the output matrix, avoiding bandwidth pressure and upper cache pressure caused by large-scale data swapping. The task splitting method is also relatively clear, and the reduced computational granularity leads to a simpler computational load balancing strategy. Matrix multiplication based on the Gustafson operation also has certain disadvantages. For example, although the data size is reduced, a merging process is still required to accumulate and merge the partial sums of different rows. This process still consumes a lot of hardware resources and is difficult to parallelize and accelerate. When the matrix size is large or the number of rows is large, the partial sum data still needs to be stored in the main memory for caching, which increases I / O bandwidth pressure and affects performance. The input data has poor locality due to its sparse representation, making it difficult to reuse input data.

[0049] Matrix multiplication can be performed using general-purpose or specialized computing devices. For example, it can be performed using a general-purpose computing device such as a central processing unit (CPU) or a graphics processing unit (GPU), or using a matrix multiplication accelerator specifically designed for matrix multiplication.

[0050] When processing matrix multiplication tasks through existing CPUs, due to the relatively small number of computing units in the CPU and the low integration of computing power, the concurrency of the calculations is low and the total amount of calculations per unit time is low, which in turn leads to low computational efficiency when processing matrix multiplication tasks.

[0051] When processing matrix multiplication tasks through existing GPUs, although the GUP has more computing units and a high degree of computing power integration, it is limited by the GPU's memory capacity and vectorized computing power units. Some computing units will be idle, making it difficult to achieve large-scale matrix storage and calculations, and the calculation efficiency is also low.

[0052] Although existing matrix multiplication accelerators are designed or improved specifically according to the different forms of matrix multiplication operations, they are limited by the bandwidth bottleneck of data reading. The long data reading and writing operations will affect the computing speed, resulting in lower computing speed and low computing efficiency.

[0053] For example, an existing matrix multiplication accelerator includes a computing unit, an on-chip cache (such as a cache memory), and a memory (such as main memory). The computing unit is used to perform multiplication and / or addition operations on the elements involved in the matrix multiplication, and the memory is used to store the matrices involved in the matrix multiplication and the intermediate results during the calculation process. The on-chip cache can be understood as a buffer storage device between the computing unit and the memory, which is used to read and cache the matrix elements required for the calculation from the memory, so that the computing unit can quickly call the elements for calculation.

[0054] Memory typically has a large storage capacity and can store all matrix elements. However, on-chip cache typically has a smaller storage capacity and cannot store all matrix elements simultaneously. Therefore, the on-chip cache needs to frequently release cached data as the computation progresses to facilitate reading the data required for the current computation from memory. Because the bandwidth between the on-chip cache and memory is limited, reading data takes time. Therefore, the more frequently the on-chip cache reads data from memory, the more time it consumes. This causes the computation to be limited by bandwidth bottlenecks, resulting in decreased computation speed and efficiency.

[0055] When multiplying two matrices, such as MatA by MatB, MatA can be understood as the matrix being multiplied, and MatB as the multiplier matrix. For example, in existing matrix multiplication accelerators, the on-chip cache typically determines which rows of the multiplier matrix elements to retain on-chip and which to delete as soon as possible to free up space for caching other elements that need to participate in the calculation, based on the historical access frequency of each row of the multiplier matrix.

[0056] This decision-making strategy is based on historical access frequencies, which cannot accurately indicate the subsequent operations during the entire multiplication process. This means that based on the data accesses in the early stages of the operation, the cache cannot accurately determine which rows of elements will be involved in the subsequent operations, how many times they will be involved, or which rows of elements will not be involved in the operation.

[0057] Because the on-chip cache cannot accurately perceive and determine in advance the number of times that elements in each row of the matrix will participate in calculations, during calculations, the on-chip cache may mistakenly delete elements of rows that will frequently participate in subsequent calculations, or may retain elements of rows that no longer participate in calculations on the chip for an excessively long period of time. If elements of rows that will frequently participate in subsequent calculations are released prematurely, they will need to be read from memory again or multiple times later, increasing data read overhead and slowing down the overall calculation speed. If elements of rows that no longer participate in calculations are retained for a long time, the storage space of the on-chip cache will be pointlessly occupied for a long time, increasing storage pressure. Furthermore, due to the storage of row elements that no longer participate in calculations and the pressure on storage space, the on-chip cache may mistakenly release elements of rows that will still participate in calculations in the later stages prematurely, resulting in the need to read these rows again later, increasing data read overhead.

[0058] As can be seen, because existing matrix multiplication accelerators cannot accurately perceive and determine in advance the number of times each row of the multiplier matrix participates in a multiplication operation, the on-chip cache cannot make optimal decisions when retaining and releasing row elements, resulting in a large number of repeated data reads, which in turn reduces the overall computing speed and efficiency. This situation can also be understood as the overall computing speed and efficiency of the multiplication accelerator being reduced due to the low on-chip data reuse efficiency.

[0059] In view of this, an embodiment of the present application provides a computing device, which includes a scheduling unit, a cache unit, and a computing unit. During the operation, the scheduling unit can determine in advance the number of times the elements of each row in the multiplier matrix participate in the multiplication operation, and can indicate the number of multiplications of each row to the cache unit, so that the cache unit can accurately know in advance the number of times the elements of each row in the multiplier matrix participate in the multiplication operation. In this way, when the cache unit reads and caches the row elements of the multiplier matrix, it can accurately decide whether to retain or remove on-chip data based on the number of multiplications of each row. It can be understood that during the operation, the row elements that need to participate in the operation frequently can be retained in the cache unit for as long as possible to avoid increasing the read overhead of reading the elements from the outside due to their early release. It can also be decided based on the number of multiplications to release the row elements that no longer participate in the operation in the later stage as soon as possible to avoid them occupying cache space. Therefore, the computing device provided by the embodiment of the present application can improve the reuse efficiency of on-chip data during the operation, reduce the number of data reads, reduce the time delayed by frequent data reading and handling, speed up the operation, and thus improve the efficiency of the operation.

[0060] The technical solutions of the present application are described in detail below with reference to specific embodiments. The specific embodiments below may be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments. The embodiments of the present application will be described below with reference to the accompanying drawings.

[0061] Figure 4 This is one of the structural diagrams of the computing device provided in the embodiment of the present application. The computing device can be applied to the multiplication operation of the first matrix by the second matrix, such as Figure 4 As shown, the computing device includes a scheduling unit 401 , a cache unit 402 and at least one computing unit 403 .

[0062] In an embodiment of the present application, the computing device may be a physical hardware device, such as an accelerator or acceleration chip for matrix multiplication operations; or, the computing device may be a virtual software device, such as a virtual device that implements matrix multiplication operations through computer programs and instructions; or, the computing device may be a device composed of a physical part and a virtual part, which is not limited in the embodiment of the present application.

[0063] The first matrix and the second matrix are two matrices participating in the multiplication operation, and these two matrices can be any matrices that can be multiplied. Any element in the first matrix and the second matrix can be a zero element or a non-zero element. If at least one of the first matrix and the second matrix is ​​a sparse matrix, the multiplication operation of the first matrix and the second matrix can be understood as a sparse matrix multiplication operation. When the first matrix is ​​multiplied by the second matrix, the first matrix can be understood as the multiplied matrix, and the second matrix can be understood as the multiplier matrix. The computing device of the embodiment of the present application can perform a multiplication operation on the first matrix and the second matrix according to an operation form such as the Gustafson operation.

[0064] The scheduling unit 401 is configured to determine the number of multiplications of any row in the second matrix according to the non-zero elements in the first matrix, and indicate the number of multiplications of each row in the second matrix to the cache unit.

[0065] Exemplarily, the scheduling unit 401 may be a component in a computing device that is responsible for managing and allocating computing resources, such as a Bubble Sort Scheduler (BSS).

[0066] In an embodiment of the present application, the computing device is configured to process a multiplication operation of a first matrix by a second matrix. Therefore, any non-zero element in the first matrix can be used to determine the corresponding element in the second matrix to be multiplied based on its row pointer and column index and matrix multiplication rules. After the corresponding element is determined, the number of times the row is multiplied based on the row in which the corresponding element is located can be determined.

[0067] Among them, the row pointer can be the row identifier of the row where the element is located in the matrix, for example, the row pointer can be understood as the row number of the row where the element is located; the column index can be the column identifier of the column where the element is located in the matrix, for example, the column index can be understood as the column number of the column where the element is located.

[0068] It should be understood that since the result of multiplying a zero element by any element is zero, the computing device in the embodiment of the present application can only perform corresponding computing processing on the non-zero elements in the matrix, which can improve computing efficiency.

[0069] Since the corresponding elements in the second matrix that are multiplied by any non-zero element in the first matrix can be determined based on the non-zero elements in the first matrix, the scheduling unit 401 can count the number of multiplications that any row in the second matrix participates in through a statistical computer program or instruction. The number of multiplications can be understood as the number of times each element in an entire row participates in the multiplication operation. For example, if the number of multiplications for a row is a, it means that each element in the row participates in a multiplication operation.

[0070] In one possible implementation, the scheduling unit 401 is specifically used to: count the number of any column index based on the column index of the non-zero elements in the first matrix; and determine the number of any column index as the number of times the rows in the second matrix whose row pointer is the index value of any column index participate in the multiplication operation based on the index value of any column index.

[0071] Exemplarily, since the second matrix is ​​a multiplier matrix, taking the Gustafson operation as an example, if the elements in the i column and the j column of the x row in the first matrix are all non-zero elements, then the elements in the i column of the x row in the first matrix should be multiplied with each element in the i row of the second matrix respectively to obtain the partial sum of the elements in the corresponding column of the x row in the result matrix; similarly, the elements in the j column of the x row in the first matrix should be multiplied with each element in the j row of the second matrix respectively to obtain the partial sum of the elements in the corresponding column of the x row in the result matrix.

[0072] A partial sum can be understood as the intermediate result of gradually adding products to any element during a multiplication operation. For example, by adding all the partial sums in row x and column i of the result matrix during the operation, we can obtain the value of the element in row x and column i of the result matrix.

[0073] It can be seen that the number of column indices of non-zero elements in the first matrix can represent the number of multiplication operations performed on the non-zero elements of the column index in the second matrix. Therefore, based on the index value of any column index, the number of any column index can be determined as the number of multiplication operations performed on the rows in the second matrix whose row pointer is the index value of any column index.

[0074] For example, assuming the first matrix is ​​a 4×4 matrix, the row numbers are 0 to 3, e.g., the row pointer for row 0 is 0, the row pointer for row 1 is 1, and so on. Similarly, the columns of the first matrix are numbered 0 to 3, e.g., the index value of column 0 is 0, the index value of column 1 is 1, and so on. Assuming the second matrix is ​​also a 4×4 matrix, the row numbers, column numbers, row pointers, and column index values ​​are similar to those of the first matrix and are not further described.

[0075] Based on the column indices of the non-zero elements in the first matrix, for example, assuming there are 2 non-zero elements in column 0 of the first matrix, then the number of columns 0 is 2; based on the index value of column 0, the number of columns 0, 2, can be determined as the number of times the row with row pointer 2 in the second matrix participates in the multiplication operation, that is, the number of multiplications of 2 rows in the second matrix is ​​2. For another example, assuming there are 4 non-zero elements in column 3 of the first matrix, then the number of columns 3 is 4. Based on the index value of column 3, the number of columns 3, 4, can be determined as the number of times the row with row pointer 3 in the second matrix participates in the multiplication operation, that is, the number of multiplications of 3 rows in the second matrix is ​​4.

[0076] It should be understood that the examples in the above embodiments only describe the processing of any one or two rows, and in actual applications, the elements of each row in the matrix will be traversed and processed in the same or similar manner.

[0077] In an embodiment of the present application, since in a matrix multiplication operation, each non-zero element of the first matrix will be multiplied by each non-zero element in the row of the row pointer of the second matrix having the same index value as the column index of the non-zero element, therefore, by counting the number of column indices of each non-zero element in the first matrix, the number of multiplications of each row in the second matrix can be quickly and accurately determined, thereby improving the efficiency and accuracy of determining the number of multiplications.

[0078] Illustratively, after determining the number of multiplications of each row in the second matrix, the scheduling unit 401 may send an instruction or information to the cache unit 402 to indicate the number of multiplications to the cache unit 402 through the instruction or information.

[0079] The cache unit 402 is configured to read and cache the non-zero elements of each row in the second matrix according to the number of multiplications of each row.

[0080] Exemplarily, cache unit 402 can be understood as an on-chip cache. Cache unit 402 can interact with computing unit 403 and input elements in the second matrix that need to participate in the computation to computing unit 403. Cache unit 402 can be any of a variety of types of caches, such as a distributed ring buffer (DRB). A DRB can include multiple cache slices (e.g., cache slices), which can also be referred to as cache lines or slices. Cache unit 402 can temporarily store data and instructions that need to be accessed to reduce the latency of data access by computing unit 403.

[0081] Cache unit 402 can interact with main memory and read elements of the second matrix from the main memory. Main memory can be understood as a storage unit for long-term data storage, such as High Bandwidth Memory (HBM). Before performing operations on the first and second matrices, the first and second matrices can be stored in the main memory. During the operation, the elements required for the operation can be read from the main memory via cache unit 402.

[0082] After interacting with the scheduling unit 401, the cache unit 402 may receive the number of multiplications of each row in the second matrix indicated by the scheduling unit 401. The cache unit 402 may adopt different read and write strategies according to the number of multiplications of each row to read and cache the non-zero elements of each row in the second matrix.

[0083] Since the greater the number of multiplications, the more times the elements of the corresponding row need to participate in the operation, the read and write strategy can be, for example: compare the size of each multiplication number, give priority to reading the elements of a portion of the rows with a larger number of multiplications and cache them in the cache unit 402 for access and call by the operation unit 403. During the operation, if the storage space of the cache unit 402 allows, the elements of the rows with a relatively large number of multiplications can be retained first, and the elements of the rows with a relatively small number of multiplications can be released first. In this way, the cache unit 402 can keep the elements that need to participate in the operation frequently on the chip as much as possible, reduce the data reading overhead between the cache unit and the main memory, improve the reuse efficiency of the on-chip data, and thus improve the operation speed and efficiency.

[0084] During the operation, if it is determined that the number of times the operation unit 403 has called a row of elements has reached the number of multiplications, it indicates that the elements in the row will no longer participate in the operation, and the cache unit 402 can delete them as soon as possible to free up storage space. The cache unit 402 can also use other read and write strategies to read and cache the non-zero elements of each row in the second matrix based on the number of multiplications of each row, which is not limited in this embodiment of the present application.

[0085] The scheduling unit 401 is further configured to input each non-zero element in any row of the first matrix into an operation unit 403 .

[0086] Exemplarily, during the multiplication operation, the scheduling unit 401 may also interact with the main memory, read the elements of the first matrix stored in the main memory, and assign the non-zero elements of each row in the first matrix to at least one operation unit 403 to perform operations on the elements.

[0087] For example, the scheduling unit 401 can read the non-zero elements of n rows of the first matrix from the main memory and assign these n rows to n computing units 403, with each computing unit 403 responsible for calculating the non-zero elements of one row. After a computing unit 403 completes processing a row, the scheduling unit 401 can assign the non-zero elements of other rows to continue calculating. After all non-zero elements in the first matrix are calculated, the scheduling unit 401 can stop inputting elements to the computing units 403.

[0088] The operation unit 403 is used to read the non-zero elements of each row of the second matrix participating in the multiplication operation in the cache unit 402 according to the non-zero elements of any row in the first matrix, and perform matrix multiplication and merging operations to obtain the corresponding elements in the result matrix.

[0089] Exemplarily, the operation unit 403 may be any type of computing unit capable of performing multiplication and merging operations on matrix elements. If the operation device is a hardware device, the operation unit 403 may be, for example, a unit composed of a multiplication circuit, an addition circuit, and a merger. The operation unit 403 may be, for example, a multiplication merge unit (MMU).

[0090] The MMU integrates the multipliers and adders required for matrix multiplication operations. It can implement concurrent comparisons through time-division multiplexing and frontier comparison concepts, avoiding the repeated data download / store (load / store) and repeated comparison operations caused by recursive comparisons, thus saving hardware overhead.

[0091] Taking the Gustafson operation as an example, the operation unit 403 can call the element of the row with the same index value as the column index in the cache unit 402 according to the column index of each non-zero element in any row of the first matrix input by the scheduling unit 401, multiply each non-zero element in the first matrix with the corresponding row, and merge the results of the multiplication operation to obtain the element at the corresponding position in the result matrix.

[0092] The computing device provided in an embodiment of the present application can be used for multiplying a first matrix by a second matrix. The computing device includes a scheduling unit, a cache unit, and at least one computing unit. During the operation, the scheduling unit can determine in advance the number of multiplications of each row in the second matrix based on the non-zero elements in the first matrix, and can indicate the number of multiplications of each row to the cache unit, so that the cache unit can accurately know in advance the number of times the elements in each row of the multiplier matrix participate in the multiplication. In this way, when reading and caching the non-zero elements of each row of the second matrix, the cache unit can accurately determine which elements cached in the cache unit should be retained and which should be deleted as soon as possible based on the number of multiplications of each row. This allows the elements of the rows that will subsequently be required to participate in the operation to be retained in the cache unit for as long as possible, avoiding the increase in the number of repeated element reads due to their early deletion. It also allows the elements of the rows that will no longer participate in the operation to be released as soon as possible, avoiding their excessive occupancy of cache space. Therefore, making a high reuse rate decision on the elements in the cache unit based on the number of multiplications can reduce the number of repeated data reads and the time consumed by data reading. Based on this, through the close cooperation of the scheduling unit and the cache unit, on-chip data with a higher hit rate can be prepared as early as possible for the calculation process of the calculation unit, so that the calculation unit can calculate quickly, thereby improving the overall calculation speed and efficiency of the calculation device.

[0093] Figure 5 The second structural diagram of the computing device provided in the embodiment of the present application is as follows: Figure 5 As shown, the BSS is a scheduling unit; the DRB is a cache unit, which includes multiple cache slices, for example, 16, 32, or more cache slices, each of which can cache matrix elements and other data; the MMU is an arithmetic unit, and the arithmetic device can include multiple MMUs, for example, 16, 32, or more MMUs; the HBM is the main memory, which can be understood as the first storage unit and can be used to store the first matrix (MatA) and the second matrix (MatB); the write back buffer (WBB) can be understood as the second storage unit and is used to store the result matrix (MatC). In some implementations, if storage space permits, the first and second storage units can also be set to the same storage unit.

[0094] Figure 6 A schematic diagram of the operation flow of the operation device provided in the embodiment of the present application is shown in FIG. Figure 5 and Figure 6 As shown, BSS can read the elements of any row of MatA. For the non-zero elements in a row, for example, the non-zero element a in row i and column m of MatA im and the non-zero element a in row i and column n of MatA in , the row pointer of the row in the second matrix to be multiplied can be determined according to the column index of the non-zero element.

[0095] For example, the BSS may be configured according to a im Column index index a im Determine the non-zero element a in MatB that is consistent with MatA through the stream prefetcher (StreamPrefetcher) im The elements to be multiplied are all non-zero elements in row m. The stream prefetcher can be understood as a unit that can read multiple data streams simultaneously, for example, can read 64 data streams simultaneously.

[0096] For example, a stream prefetcher can determine a im The element value value a im and with a im A non-zero element bm of row m in the multiplied MatB is stored at address addr b in the HBM. m For example, the non-zero element a of MatA can also be determined by the stream prefetcher. in The element value value a in and with a in A non-zero element b in row n of the multiplied MatB n The address addr b stored in HBM n .

[0097] The BSS can indicate the addresses of non-zero elements in MatB to the DRB, letting it know where to read data in the HBM. The DRB retrieves and caches the corresponding non-zero elements in MatB based on the addresses. The BSS can input multiple rows of MatA into multiple MMUs, with each MMU responsible for calculating the elements of a row. This allows multiple MMUs to calculate elements of multiple rows in parallel.

[0098] The MMU can read the non-zero elements of MatB that need to participate in the calculation from DRB. Figure 6 As shown, BSS sets the value a of the non-zero element aik in row i and column k of MatA to ik After inputting the multiplier of the MMU, the MMU can calculate the value of a ik Get all non-zero elements of the kth row of MatB in DRB (using b k ), and these non-zero elements are sequentially combined with a ik Multiply them to get products, which are the partial sums of the corresponding columns in row i of the result matrix.

[0099] For example, value a ik A non-zero element b in the kth row of MatB kj The element value b kj Multiply them to get value c ij , the value c ij Represents the element c in row i and column j of MatC ij After all the partial sums of the elements in row i and column j in MatC are calculated, the element c in row i and column j in MatC can be obtained by adding up all the partial sums. ij The element value of .

[0100] like Figure 3 As shown in the Gustafson operation, assuming that an MMU is responsible for processing row 0 of MatA, the BSS inputs all non-zero elements in row 0 to the MMU. Assuming that row 0 includes two non-zero elements, the element in row 0 and column 0, and the element in row 0 and column 1, the MMU can read all non-zero elements in row 0 and row 1 of MatB from the DRB based on the column index of the non-zero element, and sequentially multiply the element in row 0 and column 0 of MatA with all non-zero elements in row 0 of MatB, and then sequentially multiply the element in row 0 and column 1 of MatA with all non-zero elements in row 1 of MatB.

[0101] Assuming that row 0 of MatB contains two non-zero elements, namely the element at row 0 and column 0 and the element at row 0 and column 3, then after multiplying them with the element at row 0 and column 0 of MatA, we can obtain the partial sum of row 0 and column 0 and the partial sum of row 0 and column 3 of MatC. Assuming that row 1 of MatB contains one non-zero element, namely the element at row 0 and column 2, then after multiplying them with the element at row 0 and column 1 of MatA, we can obtain the partial sum of row 0 and column 2 of MatC. It can be seen that if a row of MatA contains n non-zero elements, then these n non-zero elements will be multiplied with n rows of MatB, and at most the partial sum of n rows can be obtained. These partial sums of n rows will be merged by the merger in the MMU to form one row in MatC.

[0102] When merging, the combiner sorts the partial sums belonging to the same row in MatC according to their column index values ​​to merge them into a row of partial sums. During the merging process, if some partial sums have the same column index, indicating that these partial sums are partial sums of the same element in MatC, the combiner will add them together.

[0103] like Figure 6 As shown, after the MMU multiplier calculates the value cij, the MMU can store the column index of each non-zero element in the k rows of MatB into the cache, for example, km Column index b km and non-zero elements b kn Column index b kn Stored in the ping buffer and pang buffer.

[0104] After the multiplier calculates multiple partial sums, these partial sums can be cached for merging. For example, the element c in row i and column m of MatC im Part and value c im and element c in row i and column n of MatC in Part and value c in The combiner can be used with a selector to combine the parts.

[0105] For example, if the calculated partial sum contains element c in row i and column j of MatC ij The combiner can indicate the column index c of column j to the selector when combining. kj So that the selector can select the element c in row i and column j in MatC ijThe selector can input these partial sums into the adder for summing. When summing, the combiner can also indicate the flag bit (such as acc-flag c) to the adder. kj ) to continuously instruct the adder to sum these partial sums until the sum is completed and the flag bit can be reset to zero. ij After summing all the parts, we can get the element c ij The element value of the combiner can be element c ij The element value and element c ij Column index c ij Deposit into WBB.

[0106] For example, in the data flow of data reading and calling, the following can be used: Figure 6 The arbiter (ARB) in the system manages access requests to shared resources to respond to each data flow in a balanced manner.

[0107] For example, after a row of MatA is input into the MMU, the MMU will usually perform multiplication operations in sequence according to the order of the elements in the row, and the order in which the elements are calculated will not affect the calculation results. Therefore, before allocating MMUs for multiple rows of elements, if the elements in each row are first rearranged and the elements corresponding to the same column index are placed at the front of their respective rows, at least some of the MMUs can read the elements of the rows corresponding to the same column index of MatB from the DRB as early as possible for calculation, and the elements of the rows with more multiplication times in the DRB can be called as soon as possible to meet the multiplication times, and the DRB can release them as early as possible, so as to achieve the purpose of using up the reusable elements in the shortest possible time, thereby improving the efficiency of data reuse and improving the overall computing speed and efficiency of the computing device.

[0108] In one possible implementation, the scheduling unit is specifically used to: for the non-zero elements of multiple rows in the first matrix, compare the sizes according to the number of column indexes of each non-zero element, and adjust the position of the non-zero element with a larger number of column indexes to the front column of the row where the non-zero element is located, so as to give priority to multiplication operations on the non-zero element.

[0109] Exemplarily, the non-zero elements of the multiple rows may be non-zero elements of some or all rows in the first matrix. If the scheduling unit has high reading and processing capabilities, the non-zero elements of all rows in the first matrix may be read and processed simultaneously.

[0110] For the non-zero elements of multiple rows in the first matrix currently processed by the scheduling unit, when comparing the numbers of column indexes of the non-zero elements, it can also be understood as comparing the numbers of non-zero elements corresponding to the column indexes.

[0111] Figure 7 A schematic diagram of the rearrangement of element positions provided in the embodiment of the present application, such as Figure 7 As shown, the scheduling unit is currently processing the non-zero elements in rows 0, 1, 2, and 3 of the first matrix. The number in the box in each row represents the column index of a non-zero element. For example, if the column index of the third non-zero element in row 0 (counting from right to left) is 8, then this non-zero element will be multiplied by all non-zero elements in the eight rows of the second matrix. For another example, if the column index of the second non-zero element in row 1 is also 8, then this non-zero element will also be multiplied by all non-zero elements in the eight rows of the second matrix. For another example, if the column index of the first non-zero element in row 3 is also 8, then this non-zero element will also be multiplied by all non-zero elements in the eight rows of the second matrix. Therefore, in the four rows of the first matrix, from 0 to 3, there are three non-zero elements with a column index of 8, indicating that the number of column indices 8 is three, which is larger than the number of other column indices.

[0112] If the scheduling unit does not rearrange the positions of the non-zero elements in each row and directly inputs each row into the corresponding calculation unit, each calculation unit will calculate each element in order of the element's position. In this case, the calculation unit in row 3 will first calculate the non-zero element with column index 8 and then call all non-zero elements in the eight rows of the second matrix from the cache unit. The calculation unit in row 1 will first calculate the non-zero element with column index 3 and, after completing this calculation, will begin calculating the non-zero element with column index 8. The calculation unit in row 0 will first calculate the non-zero element with column index 0, then the non-zero element with column index 5, and finally the non-zero element with column index 8. Because these calculation units cannot process elements with the same column index synchronously, they cannot call all non-zero elements in the eight rows of the second matrix from the cache unit in a short period of time. As a result, the non-zero elements in the eight rows of the second matrix will remain in the cache unit for a long time, waiting to be called by the calculation units corresponding to rows 0 and 1. This will cause the non-zero elements in the eight rows to remain in the cache unit for a long time, affecting the efficiency of on-chip data reuse.

[0113] Therefore, in order to reuse the non-zero elements of the 8 rows of the second matrix in a shorter time and improve the data reuse efficiency on the chip, the scheduling unit can count and compare the number of non-zero elements corresponding to the same column index for each non-zero element in rows 0, 1 and 3 of the first matrix according to the number of column indexes of each non-zero element, and adjust the position of the non-zero elements with a large number to the front column of the row where the non-zero element is located. This allows multiple computing units to call the reusable row elements in the cache unit more synchronously, so that the elements of these reusable rows can be called in a shorter time, thereby improving the data reuse efficiency on the chip.

[0114] For example, when rearranging the positions of the elements, a comprehensive rearrangement can be performed based on the original positions of the elements in the rows and the number of column indexes of the elements. For example, the number of elements with column index 8 in rows 0, 1, and 3 is 3; assuming that rows 0, 1, and 3 all include elements with column index 104, the number of column index 104 is also 3, and the number of column index 8 and column index 104 is the same, both greater than the number of other column indexes. However, since the original positions of the elements with column index 104 are further back than the positions of the elements with column index 8, when adjusting and rearranging the positions of the elements, the elements with column index 8 can be prioritized to be placed at the first position of their respective rows, and the elements with column index 104 can be rearranged to the second position of each row. This makes it easier to rearrange the positions of the elements in each row.

[0115] In an embodiment of the present application, by comparing the number of column indexes of non-zero elements in multiple rows of the first matrix, and adjusting the position of the non-zero element with a larger number of column indexes to the front column of the row where the non-zero element is located, so as to give priority to multiplication operations on the non-zero element, the cache unit can release reusable data as early as possible, and there is enough space to read and cache other non-zero elements in the main memory, which can reduce the time for data preparation, improve the data reuse efficiency, and also improve the hit rate of on-chip data, thereby improving the overall computing speed and computing efficiency of the computing device.

[0116] In one possible implementation, the scheduling unit may be a BSS. The BSS can be understood as a balanced scheduler based on sliding window statistics, responsible for balancing and efficiently multiplexing matrix multiplication tasks in units of MatA rows to various computing units. The BSS is used to reorder and merge the index accesses of MatA rows to MatB rows based on frequency through sliding window statistics and continuous bubble sorting during scheduling, thereby avoiding the problem of read cache misses (cache misses) caused by accesses to MatB rows during subsequent operations, resulting in reduced computing efficiency. At the same time, in conjunction with DRB, it can also effectively reduce the demand for off-chip memory access, thereby alleviating the off-chip memory access bottleneck and improving the overall accelerator performance ceiling of the computing device.

[0117] The present application embodiment takes three sliding windows, sliding window 1, sliding window 2 and sliding window 3, as an example. Figure 7 For example, the number of sliding windows may not be limited in specific applications. Figure 7 As shown, sliding window 1, sliding window 2 and sliding window 3 in the dotted box may be three sliding windows generated successively at different times, and each sliding window performs column index quantity statistics on a preset number of non-zero elements in multiple rows within its corresponding time period.

[0118] For example, each sliding window uses a statistic buffer to store a preset number of data (e.g. Figure 7 The number of column indices of non-zero elements (4 in the example) is counted, and the positions of the non-zero elements in the sliding window are rearranged according to the rule that the larger the number of non-zero elements corresponding to the same column index, the closer they are to the front row. This rule can also be understood as sliding window statistics and continuous bubble sort.

[0119] like Figure 7 As shown, sliding window 1 can count the number of 4 non-zero elements in each row from row 0 to row 3 (a total of 16 non-zero elements), and obtain the number of elements corresponding to each column index in these rows. When counting, a list of each column index can be formed through the column index data stream (stream index). Figure 7 In the example, the frequency data stream (stream freq) of window 1 counts the number of columns corresponding to the index within the statistical period of window 1. Windows 2 and 3 are similar. It can be understood that subsequent windows are generated by sliding backward on the previous window. These windows can each count the number of frequency data streams for the 16 non-zero elements within the window.

[0120] like Figure 7 As shown, window 2 is shifted backward by one position relative to window 1, and window 3 is shifted backward by one position relative to window 2. The frequency data streams for window 2 and window 3 each count the number of column indices within the statistical period of their respective windows. It will be appreciated that since the positions of elements may be rearranged, this rearrangement can be achieved through column index remapping.

[0121] like Figure 7 As shown, within each sliding window, elements requiring reordering can be promptly reordered, forming an issue queue. The issue queue can be understood as the new positional ordering of non-zero elements after the sliding window has been reordered. After all sliding windows are completed, the elements of each row can be input to multiple computation units according to the issue queue of the last sliding window. These computation units can then perform operations on each element sequentially based on the final reordered elements.

[0122] like Figure 7 As shown, in the transmit queue corresponding to sliding window 3, rows 0, 1, and 3 all place the element with column index 8 first; rows 1, 2, and 3 all place the element with column index 23 first, and so on. Compared to before the reordering, elements with column index 8 and 23 can be operated on almost synchronously, allowing the cache unit to release the used elements sooner, reducing storage space pressure and improving data reuse efficiency.

[0123] In this embodiment, the positions of non-zero elements in multiple rows can be rearranged in a sliding manner through multiple sliding windows, which can moderately reduce the computational overhead of the scheduling unit during position rearrangement, making the bubble sort algorithm applicable to multiplication operations of large-scale matrices and improving the universality of position rearrangement.

[0124] For example, if the rows input into multiple operation units are in an order rearranged according to the number of column indices, but the cache unit does not read and cache data from the main memory according to the number of column indices, then after each operation unit starts the operation, it may be that the cache unit does not cache the elements of each row in the second matrix in the rearranged order, which may cause the operation unit to fail to read the non-zero elements required to participate in the operation in the cache unit, resulting in a read data miss or a low hit rate, and then it is necessary to wait for the cache unit to read the elements required to participate in the operation from the main memory, which will increase time consumption and lead to a decrease in operation efficiency.

[0125] In one possible implementation, the scheduling unit is further used to: sort in descending order according to the number of column indices of each non-zero element in the first matrix, obtain the sorting results of each column index, and indicate the sorting results to the cache unit; the cache unit is specifically used to: read and cache the non-zero elements of each row in the second matrix in sequence according to the sorting results.

[0126] For example, the scheduling unit may count the number of column indices of non-zero elements in some or all rows of the first matrix, and sort the column indices in descending order based on the number of columns. It is understood that sorting the number of column indices in the first matrix is ​​equivalent to sorting the number of row pointers in each row of the second matrix, and can also be understood as sorting the number of multiplications in which each row in the second matrix participates in the operation.

[0127] For example, the number of column indices of non-zero elements in four rows from row 0 to row 3 in the first matrix is ​​counted, where the number of column index 8 is 3, the number of column index 5 is 2, the number of column index 23 is 2, the number of column index 2 is 1, and so on; then the sorting results of the column indexes are 8, 5, 23, 2, etc. Among them, for multiple column indices of the same number, they can be sorted randomly, or in descending order according to the size of the index value of the column index, or they can be sorted in other ways.

[0128] After obtaining the sorting results of the column indexes, the scheduling unit may indicate the sorting results to the cache unit through data interaction with the cache unit.

[0129] After obtaining the sorting result, the cache unit may sequentially read and cache the non-zero elements of each row of the row pointer corresponding to the column index in the second matrix according to the order of the column indexes in the sorting result.

[0130] In an embodiment of the present application, by sorting the number of column indices of each non-zero element in the first matrix in descending order, a descending sort result of the number of multiplications of each row in the second matrix can be obtained. The non-zero elements of each row in the second matrix are read and cached in sequence according to the sorting result. This can be combined with the position rearrangement of the elements of each row of the first matrix to improve the hit rate of the cache unit during the operation, and then the rows that need to participate in the operation multiple times can be calculated as soon as possible, thereby improving the speed and efficiency of the operation.

[0131] For example, in order to improve the on-chip data reuse efficiency of the cache unit, a read / write strategy based on a frequency lock may be used to decide which data in the stored data will be retained first and which data will be replaced first.

[0132] In one possible implementation, the cache unit is further used to: determine the number of multiplications of the row containing any non-zero element in the second matrix as the number of multiplications of the any non-zero element; if any operation unit reads a non-zero element from the cache unit once, the number of multiplications of the read non-zero element is reduced by one; when the cache unit is full and a new non-zero element needs to be read and cached, by comparing the number of multiplications of the cached non-zero elements with the number of multiplications of the new non-zero elements, the non-zero elements with the larger number of multiplications are preferentially cached.

[0133] For example, during the operation, all non-zero elements in a row of the second matrix need to be multiplied with a non-zero element in the first matrix corresponding to the row. Therefore, the number of multiplications of the row can also be understood as the number of multiplications of each non-zero element in the row. After obtaining the number of multiplications of the row, the cache unit can mark the number of multiplications of the row as the number of multiplications of the element for each non-zero element in the row when reading and caching each element of the row. In addition, the cache unit can also reduce the number of multiplications one by one based on the number of times each non-zero element is called by the operation unit to obtain the updated number of multiplications of the non-zero element. The updated number of multiplications can be understood as the remaining number of times the non-zero element will be called later.

[0134] New non-zero elements can be understood as non-zero elements that are not currently cached in the cache unit. When the cache unit is full and a new non-zero element needs to be read and cached, to prioritize caching non-zero elements with a larger number of multiplications, the number of multiplications of the cached non-zero elements can be compared with the number of multiplications of the new non-zero element, prioritizing the caching of non-zero elements with a larger number of multiplications. Because non-zero elements with a relatively large number of multiplications are more likely to be repeatedly called, comparing the number of multiplications can improve data reuse efficiency.

[0135] Figure 8 A schematic diagram of a read-write strategy based on a frequency lock provided in an embodiment of the present application is shown in FIG. Figure 8As shown, the HBM stores the elements of each row of the second matrix MatB. MatB includes 5 rows, namely row 0, row 1, row 2, row 3 and row 4. For the convenience of description, each row can be identified as R0 to R4. Each non-zero element included in any row can be represented by a different serial number, for example Figure 8 In the box, R0-0 represents the first non-zero element in row 0, and R0-1 represents the second non-zero element in row 0; for example, R1-2 represents the third non-zero element in row 1, and so on.

[0136] Assume that the number of multiplications corresponding to rows 0 to 4 is 2, 1, 2, 3, and 2, respectively. According to the read and write strategy of the cache unit for reading and caching each row, the elements required to participate in the operation can be read and cached and called respectively through the following steps S1-S9 according to the operation requirements of the operation unit. It should be understood that when an element is read from the main memory to the cache unit, it is called for operation once by the operation unit. Therefore, when reading and caching non-zero elements, the number of multiplications of the non-zero elements when caching, that is, the number of multiplications when reading, is reduced by one.

[0137] like Figure 8 As shown, step S1 is to read all non-zero elements in row 0. During reading, since the cache unit has no cached data before, the storage space of the cache unit is relatively sufficient, and all non-zero elements in row 0 can be cached. Moreover, since the original number of multiplications of each non-zero element in row 0 is 2, it is called once by the operation unit during reading. Therefore, after subtracting 1, the number of multiplications of each non-zero element in row 0 is reduced to 1, as shown in FIG. Figure 8 The number 1 in the box below R0-0 indicates the number of multiplications after decrement.

[0138] Step S2 is to read all non-zero elements of row 1. Since the remaining storage space of the cache unit can meet the storage space requirements of caching all non-zero elements of row 1, there is no need to delete any non-zero elements of row 0 that have been cached.

[0139] In step S3, the computation unit calls all non-zero elements in row 0 from the cache unit for computation. The multiplication counts for each non-zero element in row 0 decrease from 1 to 0 after the element is read. At this point, the cache unit's storage space is full, and the multiplication counts for each non-zero element have also been reduced to 0. In the next step, these non-zero elements can be deleted sequentially or randomly to make room for elements required for subsequent computations.

[0140] In step S4, the calculation unit needs to call all non-zero elements of the 2 rows to participate in the calculation. At this time, the cache unit will read and cache these elements according to the addresses of the non-zero elements of the 2 rows in the main memory. Figure 8As shown, row 2 includes two non-zero elements. The initial multiplication count of these two non-zero elements is 2. When they are read, they are called once by the operation unit. After being cached in the cache unit, the multiplication count of these two non-zero elements decreases to 1. In addition, in order to cache the two non-zero elements in row 2, the two non-zero elements in row 0 are deleted to cache the two non-zero elements in row 2.

[0141] Similar to step S4, in step S5, multiple elements with a multiplication count of 0 are deleted for each non-zero element in the cache 3 rows. In step S5, the operation unit calls each non-zero operation in the 2 rows for calculation, so the multiplication count of R2-0 and R2-1 is reduced to 0.

[0142] In step S6, all non-zero elements in the four rows need to be read and cached. Since there are many non-zero elements in the four rows, the storage space of the current cache unit cannot meet the requirements for caching all non-zero elements in the four rows. Therefore, it is necessary to compare the multiplication counts of the cached non-zero elements with the multiplication counts of the new non-zero elements to determine which elements to cache in the cache unit.

[0143] like Figure 8 As shown, the number of multiplications of the four non-zero elements R2-0, R2-1, R1-5 and R1-6 is 0, which is less than the number of multiplications of the non-zero elements in the four rows, which is 1. Therefore, the four non-zero elements R2-0, R2-1, R1-5 and R1-6 can be deleted, and the non-zero elements in the four rows are cached first. Since the number of multiplications of the non-zero elements in the four rows, which is 1, is less than the number of multiplications of the non-zero elements in the three rows that have been cached, which is 2, the non-zero elements in the three rows are retained first, and only some of the non-zero elements in the four rows are cached. The non-zero elements in the four rows that cannot be cached can be left uncached and can be read from the main memory again when they are needed for subsequent calculations. Figure 8 In the example, R4-0, R4-1, R4-2, and R4-3 replace R2-0, R2-1, R1-5, and R1-6, and R4-4 that cannot be cached is not cached.

[0144] Steps S7 and S8 are similar. In these two steps, the calculation unit calls all non-zero elements in three rows from the cache unit once for calculation. The number of multiplications of each non-zero element in the three rows decreases from 2 to 0 after the element is called twice. At this point, the number of multiplications of each non-zero element in the three rows is 0, and they can be replaced by new non-zero elements in the next step.

[0145] In step S9, the arithmetic unit needs to call all non-zero elements in rows 4. Since R4-0, R4-1, R4-2, and R4-3 are already cached in the cache unit, they can be directly called. The cache unit can then call R4-4 from main memory again. Since the number of multiplications for R4-4 has also been reduced to 0 after this call, the cache unit can either not cache it or replace it with an already stored element with a multiplication number of 0 and cache R4-4 in the cache unit. At this point, the calculations for rows 0 through 4 of the second matrix are complete.

[0146] As can be seen from the above example, the frequency lock-based read and write strategy can prevent the stored non-zero elements from being directly replaced before they are used up. Instead, by comparing the number of multiplications, non-zero elements with a large number of multiplications are retained first, thereby increasing the residence time and reuse times of non-zero elements on the chip, and thus improving the hit rate of reading elements during calculations.

[0147] In an embodiment of the present application, a read-write strategy based on a frequency lock prioritizes caching non-zero elements with a large number of multiplications, which can reduce the early release of non-zero elements that need to frequently participate in calculations, thereby reducing the reading overhead of the cache unit and the external memory, and improving the efficiency of data reuse.

[0148] For example, during the operation process, the merger in the operation unit is also a factor that affects the operation speed. If the merger is more efficient in merging data, the calculation speed of the operation unit can be accelerated.

[0149] In one possible implementation, any operation unit includes at least two combiners, and any combiner is used to combine the partial sum of any element in the result matrix, where the partial sum is an intermediate result when the product of any element in the matrix is ​​gradually accumulated during the multiplication operation.

[0150] Exemplarily, during the calculation process of the operation unit, the product a·b obtained by multiplying a non-zero element a in the first matrix by the corresponding non-zero element b in the second matrix is ​​a partial sum, the row pointer of the partial sum is the same as the row pointer of the non-zero element a, and the column index of the partial sum is the same as the column index of the non-zero element b. Then the partial sum a·b can be understood as a part of the element located at the row pointer of a and the column index of b in the result matrix, and adding all partial sums located at the row pointer of a and the column index of b in the result matrix can obtain the element value of the element at that position.

[0151] Exemplarily, any operation unit may include two or more mergers, and each merger may be the same merger or a different merger. For example, an operation unit may include mergers with multiple different merging capabilities, wherein the merging capability can be reflected by the number of rows of partial sums merged by the merger. For example, the merging capability of a merger that can simultaneously merge 4 rows of partial sums is greater than the merging capability of a merger that can only simultaneously merge 2 rows of partial sums. A merger with multi-way concurrent sorting can increase the hardware resource overhead in a linear relationship to increase the concurrent merging throughput. The specific implementation of the partial sums and the merger can also refer to the description in other embodiments of the present application.

[0152] In an embodiment of the present application, multiple mergers in the computing unit can accelerate the processing of intermediate results in a parallel or multi-stage merger manner, thereby improving the speed and efficiency of merging partial sums, thereby improving the overall computing efficiency of the computing device.

[0153] In one possible implementation, the operation unit includes two combiners, one of the two combiners is a first-level combiner, and the other combiner is a second-level combiner; the first-level combiner is used to serially merge m groups of partial sums to obtain m first-level merged results, any group of partial sums in the m groups of partial sums includes n rows of partial sums, and the n rows correspond to n rows in the second matrix, where m and n are both positive integers; the second-level combiner is used to merge the m first-level merged results to obtain a second-level merged result, and the second-level merged result is used to obtain the elements of a row in the result matrix.

[0154] Illustratively, the operation unit in this embodiment includes two mergers, and the two mergers can be configured to perform multi-stage merging processing on the partial sums of multiple rows in a multi-stage merging manner.

[0155] Figure 9 A schematic diagram of a multi-stage combiner provided in an embodiment of the present application is shown in FIG. Figure 9 As shown, the L1 Merger is a first-level merger, and the L2 Merger is a second-level merger. The first-level merger can serially process m groups of partial sums, each of which can include n rows of partial sums. The second-level merger is used to perform a secondary merge on the results of the first-level merge.

[0156] For ease of understanding and description, assume that both m and n are 4 and combine Figure 9 Give an example, such as Figure 9 As shown, each small square represents a partial sum. The 16 rows of partial sums are evenly divided into four groups, each containing four rows of partial sums. L1Merger can use time division multiplexing to serially process these four groups of partial sums in four time periods: T0, T1, T2, and T3.

[0157] For example, during time period T0, the first group of parts is merged with the input L1 Merger to output the first level-one merged result. During time period T1, the second group of parts is merged with the input L1 Merger to output the second level-one merged result. Similarly, after time periods T2 and T3, a total of four level-one merged results are obtained. These four level-one merged results still need to be further merged into a row. Therefore, these four level-one merged results can be input into the L2 Merger for further merging to obtain a level-two merged result. This level-two merged result is a row of partial sums, which can be used to obtain the elements of a row in the result matrix.

[0158] It should be understood that the second-level merger can be started at any time after the first-level merger has generated at least one merger result for the last set of partial sums. It can be understood that if there are m first-level merger results participating in the second-level merger, the second-level merger can begin merging after data has been generated in the mth first-level merger result, and is not limited to waiting until the mth first-level merger result is completely generated before starting merging. In this way, the first-level and second-level mergers can merge simultaneously, shortening the merging time and improving merging efficiency.

[0159] In an embodiment of the present application, the computing unit includes two combiners, each of which has different levels. The first-level combiner can use time-division multiplexing to perform a preliminary merge of the partial sums of multiple rows, and then pass the merged results to the second-level combiner for further merging. This allows the two combiners to merge the partial sums of multiple rows in batches with limited hardware resource overhead. Compared to the hardware resource overhead of deploying more combiners, this approach can effectively improve the merging efficiency of the combiners while maintaining a lower hardware resource overhead, thereby improving the computing efficiency of the computing device.

[0160] Illustratively, if the secondary merge result output by the L2 Merger is the result of merging all partial sums of a row in the result matrix, then the secondary merge result corresponds to the element value of each element in the row in the result matrix.

[0161] If the L2 Merger outputs a second-level merge result that doesn't merge all partial sums for a row in the result matrix, it remains an intermediate result and needs to be merged with all other partial sums for that row before the values ​​for the elements in that row in the result matrix are obtained. In this case, consider using an idle Merger to further merge multiple second-level merge results. This allows these second-level merge results to be merged as quickly as possible, quickly obtaining the values ​​for each element in that row in the result matrix.

[0162] In one possible implementation, the computing device includes multiple computing units. When the second matrix includes k rows, k is a positive integer, and the product of m and n is less than k, the scheduling unit is further used to: call a combiner in an idle state among the multiple computing units to merge partial sums of multiple secondary merging results to obtain at least one multi-level merging result, and the multi-level merging result is used to obtain an element of a row in the result matrix.

[0163] For example, the product of m and n can be understood as the total number of rows of partial sums that the first-level combiner in an operation unit currently needs to merge. If the total number of rows is less than the number of rows k of the second matrix, it indicates that the merging capability of the first-level combiner is insufficient to merge multiple rows of partial sums corresponding to a row in the result matrix within one merging round.

[0164] In this case, the scheduling unit can call the idle mergers in multiple operation units, and then use the mergers that are not in working state in these operation units to merge the obtained secondary merging results, thereby forming a merge tree structure of multiple mergers, which can improve the merging efficiency.

[0165] Figure 10 A schematic diagram of a merge tree provided in an embodiment of the present application is shown in FIG. Figure 9 and Figure 10 As shown in the figure, assuming the first-level merger of the first arithmetic unit (MMU1) needs to merge a large number of partial sum rows, the first-level merger can only process 16 rows of partial sums in the first merging round (the four time periods from T0 to T3), obtaining the first second-level merged result. In the second merging round, the first-level merger can process another 16 rows of partial sums in the time periods from T4 to T7, obtaining the second second-level merged result. Similarly, another 16 rows of partial sums are processed in the time periods from T8 to T11, obtaining the third second-level merged result. Another 16 rows of partial sums are processed in the time periods from T12 to T15, obtaining the fourth second-level merged result.

[0166] The scheduling unit invokes idle mergers in multiple computation units, for example, the merger of the second computation unit (MMU2). It then uses MMU2's L1 Merger to merge the four secondary merge results, resulting in a multi-level merge result, which can be understood as a three-level merge result. After accumulating four three-level merge results, MMU2's L2 Merger merges them together to produce a four-level merge result. This allows idle mergers to participate in merging partial sums, effectively improving merge efficiency.

[0167] It should be understood that while MMU2 is merging the secondary merge results, MMU1 can continue processing partial sums from other groups in the period after T15, and MMU2 can also process other secondary merge results in the period after Tn. Therefore, the merge tree composed of MMU1 and MMU2, or the mergers in more computing units, can be understood as a topology of multiple mergers merging in parallel.

[0168] It is understandable that in tasks such as data processing, image processing, or model learning, any matrix involved in the calculation may be a large-scale matrix, potentially including tens of thousands, millions, or even tens of millions of rows of elements. Therefore, the numbers in the above embodiments are provided for illustrative purposes only. In actual applications, whether a first-level or second-level merger is used, the number of rows merged simultaneously is not limited to four, and a single merger round is not limited to processing four partial sums. In addition, the number of operation units or mergers that make up the merger tree is not limited to two, three, or four. A merger tree can be composed of more mergers to meet the computational requirements of large-scale matrices.

[0169] In an embodiment of the present application, by calling a merger in an idle state, the mergers in other computing units can be used to merge the secondary merging results, and a tree-structured merge tree can be formed, which can process the partial sums of more rows in parallel, thereby improving the merging speed and efficiency, and achieving the purpose of improving the computing speed and computing efficiency of the computing device.

[0170] The computing device provided in the embodiment of the present application has a computing unit in which the computing unit reduces the number of iterations required to merge all parts and rows by increasing the number of rows processed simultaneously by the merger, thereby reducing the total amount of off-chip IO and alleviating bandwidth pressure.

[0171] Existing computing devices or accelerators have the defects of redundant parts and storage in the calculation task of matrix multiplication, as well as the defects of hardware power consumption and area constraints. The computing device of the embodiment of the present application can improve the overall merging ability of the computing unit through a multi-level merger or merger tree, and can merge multiple rows of partial sums at the same time, thereby reducing the number of times the partial sums are redundantly stored during merging, and overcoming the defects of redundant parts and storage. The computing device of the embodiment of the present application can effectively reduce the hardware resource overhead of the merger through a time-division multiplexing merger, or a merger tree composed of multiple mergers, overcome the defects of hardware power consumption and area constraints, and can also increase the maximum number of merged rows of the merger, for example, it can process 65536 rows or even more rows of data, and solve the problem of iterative memory access computing overhead caused by the inability to completely merge on-chip.

[0172] In one possible implementation, the cache unit includes p groups of cache areas, any one group of cache areas includes q cache slices, any one cache slice is used to cache non-zero elements in the second matrix, and any one cache slice corresponds to one operation unit, wherein p and q are both positive integers; the scheduling unit is specifically used to: when it is necessary to call a merger in an idle state in multiple operation units, give priority to calling the operation units corresponding to the cache slices in the same group of cache areas.

[0173] For example, a cache unit may be divided into p groups of cache areas, each of which may include q cache slices, where p and q are both positive integers. A cache slice may correspond to an operation unit, and the operation unit may read non-zero elements required for operation through the corresponding cache slice.

[0174] Figure 11 A schematic diagram of multiple cache slices provided in an embodiment of the present application, such as Figure 11 As shown, in the DRB, the boxes 0-3 represent cache slices No. 0 to No. 3 respectively, and these 4 cache slices constitute the first group of cache areas; the boxes 4-7 represent cache slices No. 4 to No. 7 respectively, and these 4 cache slices constitute the second group of cache areas; the boxes 8-11 represent cache slices No. 8 to No. 11 respectively, and these 4 cache slices constitute the third group of cache areas; the boxes 12-15 represent cache slices No. 12 to No. 15 respectively, and these 4 cache slices constitute the fourth group of cache areas. Figure 11 The solid arrows in the middle represent multiple cache slices in the same cache area. Multiple cache slices in the same cache area can quickly call data from each other.

[0175] Since the cache slices in the same group of cache areas are adjacent to each other, the operation units corresponding to these cache slices can call each other's parameters such as element values ​​and column indices of non-zero elements with low latency. Therefore, when it is necessary to call the idle mergers in multiple operation units, the scheduling unit can reduce the time cost of data calls and improve the merging efficiency by giving priority to calling the operation units corresponding to the cache slices in the same group of cache areas.

[0176] In the embodiments of the present application, a cache area refers to multiple cache slices grouped together. When forming a merge tree, if the combiners corresponding to the operation units of each cache slice within the same cache area group are preferentially called, the matrix multiplication operation requirements can be met at a lower hardware resource cost, thereby achieving the purpose of improving operation efficiency.

[0177] In order to improve the overall data reading and writing efficiency of the cache unit, a data path can be established between each cache interval.

[0178] In a possible implementation, between p groups of cache areas, a data path is provided between any two adjacent groups of cache areas, and the data path is used to transmit non-zero elements cached in the cache slices.

[0179] For example, a data path can be a communication channel set up for transferring data between cache slices. If a data path is set up between any two adjacent cache slices in p groups of cache areas, non-zero elements cached in the cache slices can be transferred via the data path. In this way, when the arithmetic units corresponding to these cache slices need to access certain non-zero elements, they can use the data path to access non-zero elements cached in cache slices that do not correspond to them, thereby improving the hit rate of data reads.

[0180] like Figure 11 As shown by the dotted arrows in the figure, a data path can be established between cache slices 0, 4, 8 and 12; a data path can be established between cache slices 1, 5, 9 and 13; a data path can be established between cache slices 2, 6, 10 and 14; and a data path can be established between cache slices 3, 7, 11 and 15. In this way, a ring-shaped distributed cache structure can be formed between the four cache areas of groups 1 to 4, dividing the entire cache and interconnecting them through a ring path, which can replace the cache topology of the crossbar structure, effectively reducing the communication delay during local access and improving the pipeline performance.

[0181] In an embodiment of the present application, a data path is set in the cache interval, and the non-zero elements of the cache can be pushed through the data path, so that all cache areas can form a ring structure. This structure can replace the complex structure of the crossbar, reduce the delay of local access, and improve the data flow performance of the cache unit.

[0182] It can be seen from the above embodiments that the cache unit with a ring-type separation topology can effectively reduce the communication delay between cache slices on a chip and save the crossbar overhead of load data between the cache and multiple computing units.

[0183] For example, since there is always data communication between the cache unit and the main memory during the operation process, even if the number of invalid data repeated reads is reduced through some algorithms or strategies, the bandwidth bottleneck between the cache unit and the main memory will still affect the overall operation efficiency of the operation device. Therefore, in order to further reduce the impact of the bandwidth bottleneck on the overall operation efficiency, the embodiment of the present application can perform corresponding compression according to the type of the second matrix to reduce the amount of data transmitted between the cache unit and the main memory, thereby achieving the purpose of improving the operation speed by reducing the amount and increasing the speed.

[0184] In one possible implementation, the computing device includes a first storage unit and a second storage unit, the first storage unit is used to store the second matrix, and the second storage unit is used to store the result matrix; when the second matrix is ​​a sparse matrix, the first storage unit is used to store a compressed second matrix obtained by sparse matrix compression of the second matrix, and the cache unit reads the non-zero elements of each row in the compressed second matrix in the first storage unit.

[0185] For example, the first storage unit is used to store the elements of the second matrix, and the first storage unit can also be understood as the main memory in the above embodiments, such as HBM. The second storage unit is used to store the result matrix, and the second storage unit can also be understood as the write-back buffer WBB in the above embodiments.

[0186] A diluted matrix can be understood as a matrix in which many elements are zero. Since 0 multiplied by any number is equal to 0, zero element compression can be performed on sparse matrices. This compression does not affect the calculation results and can also reduce the amount of matrix data.

[0187] If the second matrix is ​​a sparse matrix, when storing the second matrix in the first storage unit, the second matrix can be first sparsely compressed to obtain a compressed second matrix, which is then stored in the first storage unit. This not only reduces the storage space occupied by the second matrix in the first storage unit, but also reduces the amount of data read and written between the cache unit and the first storage unit during operations.

[0188] Exemplarily, when sparse matrix compression is performed on the second matrix, compression can be performed using any method that can compress and store a sparse matrix. For example, compression can be performed to store the matrix in a coordinate list (COO) format, or in a compressed sparse row (CSR) format, or in a compressed sparse column (CSC) format, although this embodiment of the present application is not limited thereto.

[0189] In an embodiment of the present application, the first storage unit can store a compressed sparse matrix, which can reduce the space occupancy rate of the second matrix in the first storage unit, save the storage space of the first storage unit, and reduce the amount of data transmitted between the cache unit and the first storage unit, improve data transmission efficiency, and facilitate the operation unit to quickly read the non-zero elements that need to participate in the operation, thereby improving the overall operation speed of the operation device.

[0190] Because existing sparse matrix compression methods have a low compression rate and cannot achieve matrix compression with a high compression rate, the present invention provides an improved CSR compression method for compressing sparse rows (CSRs), namely, a prefix compressed sparse row (PFCSR) compression method, to further compress the second matrix and reduce the amount of data when reading the second matrix.

[0191] For example, the compression method of PFCSR can be used for the first matrix and / or the second matrix, and the first matrix and / or the second matrix can be sparsely represented to obtain the corresponding matrix after compression. Figure 12 The description is made by taking compression of the second matrix as an example.

[0192] In one possible implementation, the compressed second matrix is ​​obtained by performing sparse matrix compression in the following manner: compressing the second matrix by compressing the sparse row CSR to obtain a preliminarily compressed second matrix; for the column index of each non-zero element in any row of the preliminarily compressed second matrix, dividing the column index according to bits of a preset length to obtain at least two levels of sub-indexes, and any two adjacent levels of sub-indexes have a superior-subordinate relationship; for any level of sub-index in the at least two levels of sub-indexes, merging the sub-indexes with the same sub-index of the upper level and the same sub-index of the current level into one sub-index, and marking the correspondence between the merged sub-index and the sub-index of the same column index in the sub-index of the lower level; replacing the column indexes in the preliminarily compressed second matrix with the merged sub-indexes and the marked correspondence to obtain the compressed second matrix.

[0193] Exemplarily, compressed sparse row CSR is a method of storing a sparse matrix through a value array, a column index array, and a row pointer array. Among them, the value array stores the values ​​of all non-zero elements, the column index array stores the column index of each non-zero element, the row pointer array stores the index of the first non-zero element of each row in the value array, and the length of the row pointer array is equal to the number of rows of the matrix plus one. Under normal circumstances, the compression rate of the row pointer array in the compressed sparse row CSR is higher than that of the column index array, and there is still a certain amount of compressible space in the column index array. The PFCSR compression method is to further compress the column indexes in the column index array obtained after the compressed sparse row CSR.

[0194] The bit length of the preset length can be one of multiple preset lengths, for example, it can be a preset length of 4 bits, 8 bits or 16 bits. Figure 12 The PFCSR compression method is described with examples.

[0195] Figure 12A schematic diagram of the PFCSR compression method provided in the embodiment of the present application is shown in FIG. Figure 12 As shown in the figure, assuming that each column index in the column index array is a hexadecimal number and each column index is 32 bits long, for example, a row contains six non-zero elements, and the column indices of these non-zero elements are "00000000", "00000005", "000001EF", "000101CF", "000101FE", and "001039E3". This means that each column index is the 32-bit original column index. Therefore, the six column indices occupy a length of 32 bits × 6 = 192 bits.

[0196] For any column index, splitting it according to the preset length can produce at least two levels of sub-indexes. For example, splitting the 32-bit original column index "001039E3" according to the preset length of 8 bits can produce four levels of sub-indexes: the first-level sub-index "E3", the second-level sub-index "39", the third-level sub-index "10", and the fourth-level sub-index "00". Splitting the column index according to the preset length can also be understood as prefix decomposition. The first-level sub-index can also be understood as an 8-bit cut-down column index; the second-level sub-index can also be understood as an 8-bit L2 prefix; the third-level sub-index can also be understood as an 8-bit L1 prefix; and the fourth-level sub-index can also be understood as an 8-bit row ID. Since the column index is split according to the bits of the column index value, the resulting sub-indexes have a hierarchical relationship. This hierarchical relationship can be understood as the relationship between high and low bits. The relationship between sub-indexes at all levels can help restore the compressed data to its original column index representation during data decompression.

[0197] For any sub-index in at least two levels of sub-indexes, merge all sub-indexes with the same sub-index in both the previous level and the current level into one sub-index. The correspondence between the merged sub-index and the sub-indexes in the next level that share the same column index is marked. This step can be understood as prefix compression.

[0198] like Figure 12 As shown, for example, the two "00"s in the second-level sub-index are merged into one "00", and the two "01"s are merged into one "01"; for another example, the three "00"s in the third-level sub-index are merged into one "00", and the two "01"s are merged into one "01".

[0199] If any level-one sub-index does not have a parent sub-index, the same sub-indexes in that level-one sub-index are merged into one sub-index. For example, if a fourth-level sub-index does not have a parent sub-index, the six "00"s in the fourth-level sub-index can be directly merged into one "00" regardless of whether the parent sub-index is the same.

[0200] The correspondence between the merged sub-index and the sub-indexes of the same column index in the sub-indexes of the lower level can be marked in each sub-index. For example, the correspondence between the upper sub-index and the lower sub-index can be indicated by a base offset. The base offset can be understood as the correspondence between a merged upper sub-index and several lower sub-indexes.

[0201] like Figure 12 In the example, the second-level sub-index "00" is added with the base offset "02", indicating that the second-level sub-index "00" corresponds to two sub-indexes in the first-level sub-index. That is, the second-level sub-index "00" is the parent sub-index of the two sub-indexes "00" and "05" in the first-level sub-index. During decompression, "00" will be added to the leading bits of "00" and "05" to restore the column index representation before compression. For another example, the second-level sub-index "01" is added with the base offset "02", indicating that the second-level sub-index "01" corresponds to two sub-indexes in the first-level sub-index. That is, the second-level sub-index "01" is the parent sub-index of the two sub-indexes "CF" and "FE" in the first-level sub-index. During decompression, "01" will be added to the leading bits of "CF" and "FE" to restore the column index representation before compression. Other details are not repeated here.

[0202] from Figure 12 As can be seen from the example, after the six original column indexes are split and merged after compression, the sub-indexes and base address offsets are "00", "05", "EF", "CF", "FE", "E3", "02", "01", "02", "01", "00", "01", "01", "39", "02", "01", "01", "00", "01", "10", "03" and "00", totaling 22. Each sub-index or base address offset occupies 8 bits in length, so the total length after compression is 8 bits × 22 = 176 bits, which is 16 bits less than the 192 bits before compression.

[0203] For example, for the second matrix, after merging and compressing the sub-indexes at all levels in the second matrix, the corresponding relationships between the merged sub-indexes and labels are used to replace the column indexes in the preliminarily compressed second matrix, thereby obtaining a compressed second matrix.

[0204] Existing diluted representations of sparse matrices, such as BCSR or TileSpGEMM, convert the CSR of a large matrix into a block CSR, thereby reducing the scale of simultaneous calculations. However, the blocked matrix still needs to be cyclically calculated at the block granularity, and there are also repeated IO operations in the merging process, which cannot improve the computational efficiency. Alternatively, the diluted representation of C2SC continues to perform a secondary sparse representation of the row pointer based on the CSC format, thereby compressing the storage space, but does not decompose the column index. At the same time, this method increases the complexity of computation scheduling and decoding and is only applicable to specific hardware structures, which is not flexible and efficient enough.

[0205] In order to further improve the storage performance of sparse matrix multiplication and provide a data format suitable for hardware acceleration, the embodiment of the present application proposes a new sparse matrix representation format PFCSR. On the basis of the basic compressed sparse row CSR, PFCSR further performs prefix decomposition of the column index vector in the original CSR, splits the column index into at least two levels of sub-indexes, and at the same time, for the same sub-index in the same level of sub-index, further performs format compression representation by merging, and finally represents the compressed format of PFCSR. By testing the compression rate of the compressed format of PFCSR on the general sparse matrix multiplication benchmark SuiteSparseMatrix Collection, an average compression of 20% of the original matrix data volume is achieved, indicating that the compressed format of PFCSR can further improve the compression rate of the matrix compared with the compression format of the general compressed sparse row CSR.

[0206] The sparse representation method of the PFCSR provided in the embodiments of the present application divides the column index into multiple levels of sub-indexes based on a preset bit length, merges the same sub-indexes, and marks the corresponding relationship between the sub-indexes that originally belonged to the same column index, thereby achieving compression using the PFCSR method. Because this compression method performs secondary compression on each column index based on the CSR, it can reduce the overall bit width of the column index and the storage space occupied by the column index, thereby achieving a higher compression ratio, saving storage space, reducing the amount of data during data reading and writing, and improving computing efficiency.

[0207] After the compressed second matrix is ​​stored in the main memory of the computing device, a cache unit in the computing device can read the compressed second matrix, thereby reducing the amount of data transmitted between the cache unit and the main memory. After reading the compressed second matrix, the cache unit can decompress the compressed second matrix to represent it as a sparse matrix representation before compression, and then input each element into the computing unit for calculation.

[0208] The present application also provides a calculation method. Figure 13 A schematic diagram of an operation method provided in an embodiment of the present application, wherein the operation method is applied to an operation device as in any of the above embodiments, wherein the operation device includes a scheduling unit, a cache unit, and at least one operation unit, and the method is used for multiplying a first matrix by a second matrix, such as Figure 13 As shown, this includes:

[0209] S1301: Determine the number of times any row in the second matrix participates in the multiplication operation according to the non-zero elements in the first matrix, and indicate the number of times each row in the second matrix participates in the multiplication operation to a cache unit.

[0210] S1302 : Read and cache the non-zero elements of each row in the second matrix according to the number of multiplications of each row.

[0211] S1303: Input each non-zero element in any row of the first matrix into an operation unit.

[0212] S1304: Read the non-zero elements of the corresponding rows of the second matrix participating in the multiplication operation from the cache unit according to the non-zero elements of any row in the first matrix, and perform matrix multiplication and merging operations to obtain the corresponding elements in the result matrix.

[0213] Optionally, based on the non-zero elements in the first matrix, the number of multiplications of any row in the second matrix participating in the multiplication operation is determined, including: counting the number of any column indexes based on the column indexes of the non-zero elements in the first matrix; and based on the index value of any column index, determining the number of any column index as the number of multiplications of the rows in the second matrix whose row pointers are the index values ​​of any column indexes participating in the multiplication operation.

[0214] Optionally, the method further includes: for the non-zero elements of multiple rows in the first matrix, comparing the sizes according to the number of column indexes of each non-zero element, and adjusting the position of the non-zero element with a larger number of column indexes to the front column of the row where the non-zero element is located, so as to give priority to multiplication operations on the non-zero element.

[0215] Optionally, the method further includes: sorting in descending order according to the number of column indices of each non-zero element in the first matrix to obtain a sorting result of each column index, and indicating the sorting result to the cache unit; reading and caching the non-zero elements of each row in the second matrix in sequence according to the sorting result.

[0216] Optionally, the method further includes: determining the number of multiplications of the row containing any non-zero element in the second matrix as the number of multiplications of the any non-zero element; if any operation unit reads a non-zero element from the cache unit each time, the number of multiplications of the read non-zero element is reduced by one; when the cache unit is full and a new non-zero element needs to be read and cached, by comparing the number of multiplications of the cached non-zero elements with the number of multiplications of the new non-zero elements, the non-zero elements with the larger number of multiplications are preferentially cached.

[0217] Optionally, any operation unit includes at least two mergers, and any merger is used to merge the partial sum of any element in the result matrix, where the partial sum is an intermediate result when the product of any element is gradually accumulated during the multiplication operation of the matrix.

[0218] Optionally, the operation unit includes two mergers, one of the two mergers is a first-level merger, and the other merger is a second-level merger; the first-level merger is used to serially merge m groups of partial sums to obtain m first-level merger results, any group of partial sums in the m groups of partial sums includes n rows of partial sums, and the n rows correspond to n rows in the second matrix, where m and n are both positive integers; the second-level merger is used to merge the m first-level merger results to obtain a second-level merger result, and the second-level merger result is used to obtain the elements of a row in the result matrix.

[0219] Optionally, the computing device includes multiple computing units. When the second matrix includes k rows, k is a positive integer, and the product of m and n is less than k, the method further includes: calling a combiner in an idle state among the multiple computing units to perform partial sum merging on multiple secondary merging results to obtain at least one multi-level merging result, and the multi-level merging result is used to obtain an element of a row in the result matrix.

[0220] Optionally, the cache unit includes p groups of cache areas, any group of cache areas includes q cache slices, any cache slice is used to cache non-zero elements in the second matrix, and any cache slice corresponds to an operation unit, wherein p and q are both positive integers; the method also includes: when it is necessary to call a merger that is in an idle state in multiple operation units, the operation unit corresponding to the cache slice in the same group of cache areas is called first.

[0221] Optionally, between the p groups of cache areas, a data path is provided between any two adjacent groups of cache areas, and the data path is used to transmit non-zero elements cached in the cache slices.

[0222] Optionally, the computing device includes a first storage unit and a second storage unit, the first storage unit is used to store the second matrix, and the second storage unit is used to store the result matrix; when the second matrix is ​​a sparse matrix, the first storage unit is used to store the compressed second matrix obtained by sparse matrix compression of the second matrix, and the cache unit reads the non-zero elements of each row in the compressed second matrix in the first storage unit.

[0223] Optionally, the compressed second matrix is ​​obtained by performing sparse matrix compression in the following manner: compressing the second matrix by compressing the sparse row CSR to obtain a preliminarily compressed second matrix; for the column index of each non-zero element in any row of the preliminarily compressed second matrix, dividing the column index according to bits of a preset length to obtain at least two levels of sub-indexes, and any two adjacent levels of sub-indexes have a superior-subordinate relationship; for any level of sub-index in the at least two levels of sub-indexes, merging the sub-indexes with the same sub-index of the upper level and the same sub-index of the current level into one sub-index, and marking the correspondence between the merged sub-index and the sub-index of the same column index in the sub-index of the lower level; replacing the column indexes in the preliminarily compressed second matrix with the merged sub-indexes and the marked correspondence to obtain the compressed second matrix.

[0224] The computing method provided in the embodiment of the present application has similar implementation principles and technical effects as those of the embodiments of the above-mentioned computing devices, and will not be described in detail in this embodiment.

[0225] An embodiment of the present application further provides a board card, which may be a board card in an electronic device. Figure 14 A schematic diagram of a board provided in an embodiment of the present application is shown as follows: Figure 14 As shown, the board includes a chip 1401, which can be a system-on-chip (SoC), or system-on-chip, integrated with one or more computing devices, which can be any of the computing devices of the above-mentioned embodiments of the present application. Chip 1401 can be used to support various deep learning and machine learning algorithms, and meet the matrix operation processing requirements in complex scenarios in fields such as computer vision, speech, natural language processing, and data mining. For example, deep learning technology is widely used in the field of cloud intelligence. One of the characteristics of cloud intelligence applications is the large amount of input data, which places high demands on the storage capacity and computing power of the platform. The board of this embodiment is suitable for cloud intelligence applications, has high matrix multiplication efficiency, and can meet the needs of deep learning tasks.

[0226] Optionally, chip 1401 is connected to an external device 1403 via an external interface device 1402. External device 1403 may be, for example, a server, computer, camera, display, mouse, keyboard, network card, or Wi-Fi interface. Data to be processed can be transmitted from external device 1403 to chip 1401 via external interface device 1402. The calculation results of chip 1401 can be transmitted back to external device 1403 via external interface device 1402. Depending on the application scenario, external interface device 1402 may have different interface formats, such as a PCIe interface.

[0227] The board also includes a memory device 1404 for storing data, which includes one or more storage units 1405. Memory device 1404 connects to a control device 1406 and chip 1401 via a bus for data transmission. The control device 1406 in the board is configured to control the state of chip 1401. To this end, in one application scenario, control device 1406 may include a microcontroller (MCU).

[0228] It should be noted that for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all optional embodiments, and the actions and modules involved are not necessarily required by this application.

[0229] It should be further noted that, although the various steps in the flowchart are shown in sequence as indicated by the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps may be performed in other orders. Moreover, at least a portion of the steps in the flowchart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily performed at the same time, but may be performed at different times. The execution order of these sub-steps or stages is not necessarily to be performed in sequence, but may be performed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0230] It should be understood that the above-described device embodiments are merely illustrative, and the device of the present application may also be implemented in other ways. For example, the division of units / modules in the above-described embodiments is merely a logical functional division, and actual implementations may employ other division methods. For example, multiple units, modules, or components may be combined or integrated into another system, or some features may be omitted or not implemented.

[0231] In addition, unless otherwise specified, the functional units / modules in the various embodiments of the present application may be integrated into a single unit / module, each unit / module may exist physically separately, or two or more units / modules may be integrated together. The aforementioned integrated units / modules may be implemented in the form of hardware or software program modules.

[0232] If the integrated units / modules are implemented in hardware, the hardware can be digital circuits, analog circuits, etc. The physical implementation of the hardware structure includes, but is not limited to, transistors, memristors, etc. Unless otherwise specified, the AI ​​processor can be any suitable hardware processor, such as a CPU, GPU, FPGA, DSP, and ASIC. Unless otherwise specified, the storage unit can be any suitable magnetic storage medium or magneto-optical storage medium, such as resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high-bandwidth memory (HBM), hybrid memory cube (HMC), etc.

[0233] If the integrated unit / module is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a memory and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned memory includes: U disk, read-only memory (ROM), random access memory (RAM), mobile hard disk, magnetic disk, or optical disk, etc., various media that can store program code.

[0234] In the above embodiments, the description of each embodiment has its own emphasis. For parts not described in detail in a particular embodiment, please refer to the relevant description of other embodiments. The technical features of the above embodiments can be combined in any way. To keep the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A computing device, characterized in that: The operation device is applied to the multiplication operation of the first matrix and the second matrix, and the operation device includes a scheduling unit, a cache unit and at least one operation unit; The scheduling unit is configured to determine, based on the non-zero elements in the first matrix, the number of multiplications of any row in the second matrix that participates in the multiplication operation, and indicate the number of multiplications of each row in the second matrix to the cache unit; The cache unit is configured to read and cache the non-zero elements of each row in the second matrix according to the number of multiplications of each row; the cache unit is an on-chip cache; The scheduling unit is further configured to input each non-zero element of any row in the first matrix into one of the operation units; The operation unit is configured to read, from the cache unit, the non-zero elements of corresponding rows of the second matrix involved in the multiplication operation based on the non-zero elements of any row in the first matrix, and perform a matrix multiplication and merging operation to obtain corresponding elements in the result matrix; The cache unit is further configured to delete, from the cached non-zero elements, non-zero elements that have been read by the operation unit for a number of times that reaches the multiplication number.

2. The device according to claim 1, characterized in that The scheduling unit is specifically used for: Counting the number of any column index according to the column indexes of the non-zero elements in the first matrix; According to the index value of any column index, the number of any column indexes is determined as the number of times rows in the second matrix whose row pointers are the index value of any column index participate in the multiplication operation.

3. The device according to claim 2, characterized in that The scheduling unit is specifically used for: For the non-zero elements in multiple rows of the first matrix, a size comparison is performed based on the number of column indexes of each non-zero element, and the position of the non-zero element with a larger number of column indexes is adjusted to the front column of the row where the non-zero element is located, so as to give priority to multiplication operations on the non-zero element.

4. The device according to claim 3, characterized in that The scheduling unit is further configured to: Sorting the column indexes of the non-zero elements in the first matrix in descending order according to the number of the column indexes, obtaining a sorting result of each column index, and indicating the sorting result to the cache unit; The cache unit is specifically used for: The non-zero elements of each row in the second matrix are read and cached in sequence according to the sorting result.

5. The device according to claim 1, characterized in that The cache unit is further configured to: Determine the number of multiplications of the row where any non-zero element in the second matrix is ​​located as the number of multiplications of the any non-zero element; If any of the arithmetic units reads a non-zero element from the cache unit once, the number of multiplications of the non-zero element read is reduced by one; When the cache unit is full and a new non-zero element needs to be read and cached, the non-zero element with a larger multiplication number is cached first by comparing the multiplication times of the cached non-zero elements with the multiplication times of the new non-zero elements.

6. The device according to claim 1, characterized in that Any of the operation units includes at least two mergers, and any of the mergers is used to merge the partial sum of any element in the result matrix, where the partial sum is an intermediate result when the product of any element in the matrix is ​​gradually accumulated during the multiplication operation.

7. The device according to claim 6, characterized in that The operation unit includes two combiners, one of the two combiners is a primary combiner, and the other is a secondary combiner; The first-level combiner is configured to serially combine m groups of partial sums to obtain m first-level combined results, wherein any group of partial sums in the m groups of partial sums includes n rows of partial sums, and the n rows respectively correspond to n rows in the second matrix, where m and n are both positive integers; The secondary merger is used to merge the m primary merger results to obtain a secondary merger result, and the secondary merger result is used to obtain an element of a row in the result matrix.

8. The device according to claim 7, characterized in that The computing device includes a plurality of computing units. When the second matrix includes k rows, k is a positive integer, and the product of m and n is less than k, the scheduling unit is further configured to: The idle combiners in the plurality of operation units are called to merge the partial sums of the plurality of secondary merge results to obtain at least one multi-level merge result, wherein the multi-level merge result is used to obtain an element of a row in the result matrix.

9. The device according to claim 8, characterized in that The cache unit includes p groups of cache areas, each group of the cache areas includes q cache slices, each cache slice is used to cache non-zero elements in the second matrix, and each cache slice corresponds to one operation unit, wherein p and q are both positive integers; the scheduling unit is specifically configured to: When it is necessary to call a merger in an idle state among the plurality of operation units, the operation units corresponding to the cache slices in the same group of cache areas are called first.

10. The device according to claim 9, characterized in that Between the p groups of cache areas, a data path is provided between any two adjacent groups of cache areas, and the data path is used to transmit non-zero elements cached in the cache slices.

11. The device according to any one of claims 1 to 10, characterized in that The computing device includes a first storage unit and a second storage unit, the first storage unit is used to store the second matrix, and the second storage unit is used to store the result matrix; When the second matrix is ​​a sparse matrix, the first storage unit is used to store a compressed second matrix obtained by sparse matrix compression of the second matrix, and the cache unit reads non-zero elements of each row in the compressed second matrix in the first storage unit.

12. The device according to claim 11, characterized in that The compressed second matrix is ​​obtained by compressing the sparse matrix in the following manner: Compressing the second matrix by using a compressed sparse row CSR method to obtain a preliminarily compressed second matrix; For the column index of each non-zero element in any row of the preliminarily compressed second matrix, the column index is split according to a preset length of bits to obtain at least two levels of sub-indexes, where any two adjacent levels of sub-indexes have a hierarchical relationship; For any one level of sub-indexes in the at least two levels of sub-indexes, merge the sub-indexes with the same sub-index in the upper level and the same sub-index in the current level into one sub-index, and mark the correspondence between the merged sub-index and the sub-indexes with the same column index in the sub-indexes in the lower level; The corresponding relationship between each merged sub-index and label is used to replace each column index in the preliminary compressed second matrix to obtain the compressed second matrix.

13. A calculation method, characterized in that: The computing method is applied to the computing device according to any one of claims 1 to 12, the computing device comprising a scheduling unit, a cache unit, and at least one computing unit, and the method is used for multiplying a first matrix by a second matrix, comprising: determining, based on non-zero elements in the first matrix, the number of multiplications of any row in the second matrix that participates in the multiplication operation, and indicating the number of multiplications of each row in the second matrix to the cache unit; Reading and caching non-zero elements of each row in the second matrix according to the number of multiplications of each row; Inputting each non-zero element of any row in the first matrix into one of the operation units; According to each non-zero element in any row of the first matrix, the non-zero elements corresponding to each row of the second matrix participating in the multiplication operation are read in the cache unit, and a matrix multiplication and merging operation is performed to obtain the corresponding elements in the result matrix.

14. A board, characterized in that: The board includes the computing device according to any one of claims 1-12.

Images