Hardware accelerator for triple sparse matrix multiplication, device and method of application thereof

Abstract

The application discloses a hardware accelerator for triple sparse matrix multiplication, a device and an application method thereof. The hardware accelerator comprises a high bandwidth memory (HBM), a crossbar network and an on-chip processing unit connected in sequence. The on-chip processing unit comprises a hierarchical cache module, a global controller and a plurality of computing tiles. Each computing tile comprises an RA computing array, a TP computing array and a local controller. The RA computing array and the TP computing array are respectively used for performing front-end operation T=RxA and back-end operation C=TxP in the triple sparse matrix multiplication. The application aims to solve the problem that a conventional general-purpose processor faces a serious performance and energy efficiency bottleneck due to massive off-chip data transfer caused by irregular memory access, uneven computing load and explosive intermediate part and results when processing the triple sparse matrix multiplication, and improve the computing performance and energy efficiency of the triple sparse matrix multiplication.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of hardware accelerator of high performance computing, and particularly relates to a hardware accelerator for triple sparse matrix multiplication, a device and an application method thereof. BACKGROUND

[0002] In the field of high performance computing, solving sparse linear equations Ax=b obtained by discretizing partial differential equations is the core computing load and performance bottleneck in many scientific and engineering applications. The algebraic multigrid (AMG) algorithm is one of the most efficient and widely used algorithms for solving such sparse linear equations, and is widely used in key engineering fields such as aerospace aerodynamic design, large-scale integrated circuit wiring, power and oil reservoir simulation.

[0003] The AMG algorithm usually includes a setup stage and a solve stage, wherein the setup stage needs to perform coarse-fine point selection, interpolation operator construction and coarse grid matrix generation operations, and the setup stage needs to use triple sparse general matrix-matrix multiplication (TriSpGEMM), such as the matrix operation form of C=RxAxP, wherein the matrix C is the calculation result, the matrix R, the matrix A and the matrix P are three input matrices, the triple sparse matrix multiplication is the core content in the setup stage, usually accounting for 60% to 80% of the total time of the setup stage, and in the solution of transient problems and nonlinear problems, the triple sparse matrix multiplication often needs to be repeatedly executed at each time step.

[0004] The triple sparse matrix multiplication not only appears in the AMG algorithm, but also is an important kernel in graph computing (such as triangle counting, graph contraction, etc.), and the performance optimization thereof has a decisive influence on the overall efficiency of multiple types of applications. However, there are three major challenges in implementing efficient triple sparse matrix multiplication on a general CPU / GPU platform: (1) irregular memory access. The data layout and operation characteristics of sparse matrices result in highly irregular access patterns, making it difficult for traditional cache hierarchies to play a role, and frequently triggering high-latency off-chip memory access. (2) Non-uniform distribution of non-zero elements. The sparsity and skewness of non-zero elements in rows and columns make it difficult to divide parallel tasks, and the load imbalance problem is serious, which reduces the overall parallelism. (3) The intermediate sparse matrix T far exceeds the on-chip cache capacity. In triple sparse matrix multiplication, the intermediate sparse matrix T=RxA generated by T=RxA far exceeds the on-chip capacity, and frequent intermediate result write-back and read-back result in significant increase in memory access delay and energy consumption, which becomes the core bottleneck of the whole process acceleration.

[0005] Current software optimization strategies mainly develop under the general hardware constraints. Common solutions include twice sparse matrix multiplication (SpGEMM) and RxAxP implementation of kernel fusion, whose calculation sequence can be combined in different bracketing ways, such as (RxA) xP and R x (A x P), and cooperate with inner product, outer product, row division, column division and other data flow strategies. However, existing solutions are mostly optimized for single sparse matrix multiplication, and it is difficult to simultaneously consider bandwidth efficiency, on-chip storage occupation and throughput in triple sparse matrix multiplication. Therefore, when processing triple sparse matrix multiplication, the traditional general processor (CPU / GPU) faces serious performance and energy efficiency bottlenecks due to irregular memory access, uneven calculation load and explosive intermediate part and results, which lead to huge off-chip data transfer. Therefore, an accelerator architecture is needed to cooperatively optimize data flow and storage management at the hardware level to improve the performance and energy efficiency of triple sparse matrix multiplication in AMG algorithm and graph computing applications. SUMMARY

[0006] The technical problem solved by the present application: In view of the above problems of the prior art, the present application provides a hardware accelerator for triple sparse matrix multiplication, a device and an application method thereof. The present application aims to solve the problem that the traditional general processor faces serious performance and energy efficiency bottlenecks due to irregular memory access, uneven calculation load and explosive intermediate part and results, which lead to huge off-chip data transfer, when processing triple sparse matrix multiplication, and to improve the calculation performance and energy efficiency of triple sparse matrix multiplication.

[0007] In order to solve the above technical problems, the technical scheme adopted by the present application is:

[0008] A hardware accelerator for triple sparse matrix multiplication, comprising an on-chip processing unit, a crossbar network and a high-bandwidth memory connected in sequence.

[0009] The on-chip processing unit comprises a global controller, a plurality of computing tiles and a hierarchical cache module. The global controller is configured to assign computing tasks to the computing tiles and monitor the execution status of the computing tiles, and dynamically adjust the task distribution strategy to achieve load balancing and maximize parallel efficiency. Each computing tile comprises an RA computing array, a TP computing array and a local controller. The RA computing array comprises a plurality of computing units configured to perform front-end operation T=R×A in triple sparse matrix multiplication C=R×A×P. The TP computing array comprises a plurality of computing units and a hash table configured to perform back-end operation C=T×P in the triple sparse matrix multiplication C=R×A×P. The local controller is configured to control the RA computing array and the TP computing array to form an asynchronous two-stage computing pipeline through T-buffer coupling, and write the result matrix C in the C-cache back to the high-bandwidth memory. The hierarchical cache module comprises an R-tag memory configured to provide the RA computing array with an input matrix R data cache, an A-two-level cache configured to provide the RA computing array with an input matrix A data cache, a T-buffer configured to temporarily store an output matrix T data, a P-two-level cache configured to provide the TP computing array with an input matrix P data cache, and a C-cache configured to store a result matrix C data.

[0010] Optionally, a plurality of data channels of the high-bandwidth memory are divided into a plurality of groups according to their functions to respectively carry the reading flow of the matrices R, A and P and the writing back flow of the result matrix C. The crossbar network comprises a P crossbar, a C crossbar, an R crossbar and an A crossbar. The group for storing the matrix P in the high-bandwidth memory is connected to the P-two-level cache through the P crossbar. The group for storing the matrix C in the high-bandwidth memory is connected to the C-cache through the C crossbar. The group for storing the matrix R in the high-bandwidth memory is connected to the R-tag memory through the R crossbar. The group for storing the matrix A in the high-bandwidth memory is connected to the A-two-level cache through the A crossbar.

[0011] Optionally, in the hierarchical cache module, the R-tag memory, the A-two-level cache, the T-buffer and the C-cache are unique to each computing tile, and the P-two-level cache is shared by all the computing tiles. The TP computing arrays in all the computing tiles are connected to the P-two-level cache shared by all the computing tiles through the same global crossbar.

[0012] Optionally, the RA computing unit comprises a task queue, an A-level one cache, a non-ready request buffer, a computing queue, a multiplier, a partial sum queue, an accumulator and an accumulation window buffer, the task queue is connected with a global controller for receiving front-end operation tasks, when the front-end operation tasks in the task queue are scheduled, elements of matrix A are obtained through a two-level cache formed by the A-level one cache and an A-level two cache, elements of matrix R are obtained through an R-register memory, and requests sent but not responded are put into the non-ready request buffer, after the elements of matrix A and matrix R receive memory responses, the elements of matrix A and matrix R in the non-ready request buffer that are ready are sent into the computing queue, when the elements of matrix A and matrix R in the computing queue are scheduled, multiplication operations of the elements of matrix A and matrix R are first performed through the multiplier, partial sums obtained through the multiplication operations are put into the partial sum queue, and the partial sums in the partial sum queue complete accumulation operations in the accumulation window buffer according to the coordinates of matrix T through the accumulator, and data of matrix T after the accumulation is completed is output to a T-buffer under the control of a local controller for the TP computing array to consume asynchronously.

[0013] Optionally, the TP computing unit comprises a task queue, a P-level cache, a non-ready request buffer, a computing queue, a multiplier, a partial sum queue and a hash table for implementing accumulation, the task queue is connected with a local controller for receiving a back-end operation task containing elements of a cached matrix T in a T-buffer, and when the back-end operation task in the task queue is scheduled, elements of a matrix P are obtained through a two-level cache formed by the P-level cache and a P-two-level cache, and a request that is sent but has not received a response is put into the non-ready request buffer, after the elements of the matrix P receive a memory response, the elements of the matrix T and the elements of the matrix P that are ready in the non-ready request buffer are sent to the computing queue, when the elements of the matrix T and the elements of the matrix P in the computing queue are scheduled, multiplication of the elements of the matrix T and the elements of the matrix P is first performed through the multiplier, a partial sum obtained through the multiplication is put into the partial sum queue, and the partial sum in the partial sum queue is fed as an input stream to be accumulated to the hash table for implementing accumulation; the hash table implementing accumulation comprises: calculating a hash value for a column index of a partial sum result, and mapping the hash value to four adjacent table entries in the hash table to form a 2*2 shared sub-area, the 2*2 shared sub-area completes insertion and hit matching in the sub-area in an actual access order through row buffer matching logic, when the 2*2 shared sub-area is full and still conflicts occur, replacement is triggered by data eviction logic, and an approximate least recently used replacement strategy is executed according to the number of hits / use information, and the evicted data in the 2*2 shared sub-area is written into a high-bandwidth memory according to a row address; when the partial sum queue in the TP computing unit has sent out all the data of a certain row of the matrix T and completed accumulation, a row completion signal and a row address signal are sent out, the data eviction logic cleans up all the table entries of the row in the hash table in batches according to the row address, and writes the table entries into the high-bandwidth memory according to the row address order to release space in time for subsequent accumulation; after the accumulation operation is completed by using the hash table, the partial sum result replaced out is written back into the high-bandwidth memory, at this time, the final result is not stored in the high-bandwidth memory, and a software interface is used to sort and merge the partial sum result stored in the high-bandwidth memory to obtain the final result of the matrix C.

[0014] In addition, the application further provides an electronic device comprising a processor and a hardware accelerator connected with each other, and the hardware accelerator is the hardware accelerator for the triple sparse matrix multiplication.

[0015] Further, the application also provides an application method of the aforementioned hardware accelerator for triple sparse matrix multiplication, comprising the following steps: providing the data of matrix R for the RA computing array through the R-tag memory, providing the elements of matrix A for the two-level cache formed by the A-first-level cache and the A-second-level cache of the RA computing unit, returning the elements of matrix A immediately if the A-first-level cache hits, otherwise accessing the A-second-level cache, returning the elements of matrix A if the A-second-level cache hits, and initiating a request to the high-bandwidth memory to return the elements of matrix A if the A-second-level cache misses, performing the front-end operation T=R×A of the triple sparse matrix multiplication C=R×A×P through each RA computing unit in the RA computing array, and temporarily storing the output elements of matrix T to the T-buffer for the TP computing array to consume asynchronously; providing the data of matrix T for the TP computing array through the T-buffer, providing the elements of matrix P for the two-level cache formed by the P-first-level cache and the P-second-level cache of the TP computing unit, returning the elements of matrix P immediately if the P-first-level cache hits, otherwise accessing the P-second-level cache, returning the elements of matrix P if the P-second-level cache hits, and initiating a request to the high-bandwidth memory to return the elements of matrix P if the P-second-level cache misses, performing the back-end operation C=T×P of the triple sparse matrix multiplication C=R×A×P through each TP computing unit in the TP computing array, and finally writing the result matrix C back to the high-bandwidth memory.

[0016] Optionally, the method further comprises pre-storing the matrix A in a windowed compressed sparse row format, wherein the windowed compressed sparse row format storage comprises dividing the matrix A into a plurality of column windows, storing the matrix A in the column windows in a compressed sparse row format, and storing information of the compressed matrix using a row index array, a row pointer array, a non-zero element number array, a column index array, and a value array, the row index array is used to record the starting row of all column windows and the ending row of the last column window, the row pointer array is used to record the number of non-zero elements of each row in each column window in sequence, the non-zero element number array is used to record the count of non-zero elements in each column window in sequence, the column index array is used to record the column index of non-zero elements in each column window in sequence, and the value array is used to record the value of non-zero elements in each column window in sequence; and the method further comprises rearranging and restoring the windowed compressed sparse row format stored matrix A before reading the windowed compressed sparse row format stored matrix A into the memory.

[0017] Optionally, when each RA computing unit in the RA computing array performs the front-end operation of the triple sparse matrix multiplication C = R x A x P, the element multiplication in T = R x A, the element multiplication in T = R x A includes taking the row of matrix R as the tile-level task unit of the computing tile, and taking the column window of matrix A as the computing unit-level task unit of the RA computing unit, so as to improve the data locality and cache efficiency and maximize the memory bandwidth utilization. In order to ensure the efficient use of computing resources, each computing tile and RA computing unit maintains a task counter, and the scheduler of the tile-level task and the computing unit-level task always allocates a new task to the currently most idle computing tile or RA computing unit to realize hierarchical minimum load balancing. When each TP computing unit in the TP computing array performs the back-end operation of the triple sparse matrix multiplication C = R x A x P, the element multiplication in C = T x P, the element in the matrix T obtained by the front-end operation is mapped to the corresponding computing tile by row, the column index is hashed and accumulated based on the multiplication result in the computing tile, and the task allocation is performed among the TP computing units according to the minimum task number or the least number of unfinished elements to realize dynamic balancing.

[0018] Optionally, when the request is initiated to the high-bandwidth memory to obtain the elements of the matrix A, in order to overcome the access delay bottleneck caused by the high-bandwidth memory, the RA computing unit adopts an out-of-order window to record the data request for accessing the high-bandwidth memory, so that the RA computing unit can continue to issue a new data request while waiting for the requested data to return, as long as the transmission window of the RA computing unit is not full, the next data request can be issued when the previous data request is not ready for calculation; when the request is initiated to the high-bandwidth memory to obtain the elements of the matrix P, the TP computing unit adopts an out-of-order window to record the data request for accessing the high-bandwidth memory, so that the TP computing unit can continue to issue a new data request while waiting for the requested data to return, as long as the transmission window of the TP computing unit is not full, the next data request can be issued when the previous data request is not ready for calculation; part or all of the high-bandwidth memory, R-tag memory, T-buffer, A-second-level cache and P-second-level cache adopt a request merging mechanism to reduce the influence of memory delay, and the request merging mechanism includes merging multiple requests for the same cache line into one downstream access by using a miss state holding table to reduce the number of memory accesses.

[0019] Compared with the prior art, the hardware accelerator of the present application has the following advantages: the hardware accelerator of the present application comprises a high-bandwidth memory, a crossbar network and a processing unit on a chip connected in sequence, the processing unit on a chip comprises a hierarchical cache module, a global controller and a plurality of computing tiles, each computing tile comprises an RA computing array, a TP computing array and a local controller, the RA computing array and the TP computing array are respectively used to perform front-end operation T=RxA and back-end operation C=TxP in triple sparse matrix multiplication, the complex triple sparse matrix multiplication operation is decomposed into two relatively independent and parallel executable sub-tasks through a decoupled two-stage pipelined computing data flow, and a specially optimized data path and a computing unit are designed for each sub-task, thereby solving the problem of huge off-chip data transfer and serious performance and energy efficiency bottleneck caused by irregular memory access, uneven computing load and intermediate data (including intermediate matrices and partial and results of result matrices) when a conventional general-purpose processor processes triple sparse matrix multiplication, and improving the computing performance and energy efficiency of triple sparse matrix multiplication. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 It is a top-level structure diagram of the hardware accelerator in the embodiment of the present application.

[0021] Figure 2 It is a computing tile internal structure diagram of the hardware accelerator in the embodiment of the present application.

[0022] Figure 3 It is an RA computing unit structure diagram in the embodiment of the present application.

[0023] Figure 4 It is a TP computing unit structure diagram in the embodiment of the present application.

[0024] Figure 5 It is a hash table internal structure diagram in the embodiment of the present application.

[0025] Figure 6 It is a windowed compressed sparse row format principle diagram in the embodiment of the present application.

[0026] Figure 7 It is a data flow diagram of front-end operation and back-end operation in the embodiment of the present application, wherein the left side of the arrow is a data flow diagram of front-end operation, and the right side of the arrow is a data flow diagram of back-end operation. DETAILED DESCRIPTION

[0027] In order to enable personnel in the technical field to better understand the technical solutions of the present application, the technical solutions of the present application will be further described in detail below in combination with the drawings in the embodiments of the present application.

[0028] As Figure 1As shown, the hardware accelerator for triple sparse matrix multiplication in the embodiment includes a high bandwidth memory (HBM), a crossbar network, and an on-chip processing unit connected in sequence. The crossbar network is tightly coupled with the high bandwidth memory. The on-chip processing unit includes a hierarchical cache module, a global controller, and a plurality of computing tiles (the number is expandable, and in the embodiment, there are two computing tiles: tile 0 and tile 1). The global controller is configured to issue a computing task to the computing tiles. As shown, Figure 2 Each computing tile includes an RA computing array, a TP computing array, and a local controller. The RA computing array is configured to perform a front-end operation T=RxA in the triple sparse matrix multiplication C=RxAxP. The TP computing array is configured to perform a back-end operation C=TxP in the triple sparse matrix multiplication C=RxAxP. The hierarchical cache module includes an R-tag memory configured to provide a matrix R data cache for the RA computing array, an A-two-level cache configured to provide a matrix A data cache, a T-buffer configured to temporarily store a matrix T data, a P-two-level cache configured to provide a matrix P data cache for the TP computing array, and a C-cache configured to provide a result matrix C data cache. The computing tile is further decoupled into the front-end operation and the back-end operation by the RA computing array and the TP computing array. The two operations are connected by the T-buffer to form a macro two-level computing pipeline. The local controller is configured to control the RA computing array and the TP computing array to form an asynchronous two-level computing pipeline through the T-buffer, and write the result matrix C in the hash table back to the high bandwidth memory HBM. The A-one-level cache and the A-two-level cache in the RA computing unit form a two-level cache for the matrix A data. The P-one-level cache and the P-two-level cache in the TP computing unit form a two-level cache for the matrix P data.

[0029] To improve parallelism, reduce read-write interference and direction switching overhead, and make bandwidth utilization and latency more predictable, in the embodiment, a plurality of data channels of the high bandwidth memory HBM are divided into a plurality of groups according to their responsibilities to respectively carry the matrix R, A, and P reading streams and the result matrix C writing back stream. For example, as an optional implementation, two channels are configured to carry the matrix R reading stream, eight channels are configured to carry the matrix A reading stream, four channels are configured to carry the matrix P reading stream, and two channels are configured to carry the result matrix C writing back stream. As shown, Figure 1As shown, the crossbar network in the embodiment includes a P crossbar, a C crossbar, an R crossbar, and an A crossbar, a group for storing matrix P in the high bandwidth memory HBM is connected to the P-2nd cache through the P crossbar, a group for storing matrix C in the high bandwidth memory HBM is connected to the hash table through the C crossbar, a group for storing matrix R in the high bandwidth memory HBM is connected to the R-Note memory through the R crossbar, and a group for storing matrix A in the high bandwidth memory HBM is connected to the A-2nd cache through the A crossbar.

[0030] As shown in Figure 1 and Figure 2 In the hierarchical cache module of the embodiment, the R-Note memory, the A-2nd cache, the T-buffer, and the hash table are unique to each computing tile, the P-2nd cache is shared by all computing tiles, and the TP computing arrays in all computing tiles are connected to the P-2nd cache shared by all computing tiles through the same global crossbar, so that the cache overhead of matrix P data can be reduced, and the utilization rate of matrix P data in all computing tiles can be improved.

[0031] The front-end operation is responsible for T = R x A. Overall, the data is processed according to the rows of matrix R, and each computing unit is responsible for multiplying a row of R by a column window of matrix A. In the embodiment, the R-Note memory is a dedicated on-chip static memory for prefetching and temporarily storing row data of matrix R, and the A-1st cache and the A-2nd cache are used to cache the total data of the column window of matrix A to provide low-latency data supply for the RA computing unit.

[0032] Specifically, the RA computing array includes a plurality of (for example, 8) RA computing units, as shown in Figure 2 Computing unit 0 to computing unit 7. To cope with irregular access to matrix A, a two-level cache hierarchy is designed, each computing unit is equipped with a private first-level data cache (A-1st cache), and all computing units in a computing tile share a multi-body second-level cache (P-2nd cache), which includes a plurality of second-level cache bodies, such as Figure 2 second-level cache body 0 to second-level cache body 7. As shown in Figure 3As shown, the RA computation unit of the present embodiment includes a task queue, an A-1 cache, a non-ready request buffer, a compute queue, a multiplier, a partial sum queue, an accumulator, and an accumulation window buffer. The task queue is connected to a global controller for receiving front-end operations tasks. When a front-end operation task in the task queue is scheduled, the elements of matrix A are fetched through a two-level cache formed by the A-1 cache and an A-2 cache, the elements of matrix R are fetched through an R-register memory, and the requests that are issued but have not received a response are put into the non-ready request buffer. After the elements of matrix A and matrix R have received a memory response, the complete elements of matrix A and matrix R in the non-ready request buffer are sent to the compute queue. When the elements of matrix A and matrix R in the compute queue are scheduled, the multiplication of the elements of matrix A and matrix R is first performed through the multiplier, the partial sum obtained by the multiplication is put into the partial sum queue, the partial sum in the partial sum queue is accumulated in the accumulation window buffer according to the coordinates of matrix T by using the accumulator, and the data of matrix T after the accumulation is output to a T-buffer under the control of a local controller for the TP computation array to consume asynchronously. The internal structure of each RA computation unit includes three queues (a task queue, a compute queue, and a partial sum queue), two buffers (a non-ready request buffer and an accumulation window buffer), one multiplier, and one accumulator. The accumulator is used to on-chip combine the partial sums generated for the elements of T[i,j] that come from the same row of matrix R and the same column of matrix A when a non-zero element of matrix T is generated. After one or more accumulators in the front end complete the final computation of one or more T non-zero elements, the result is pushed into an on-chip pipeline buffer (T-buffer) for the back end to consume asynchronously.

[0033] The back-end operation consumes the intermediate result (matrix T) and efficiently completes the multiplication with P and the final combination, i.e., performs C = T x P, processes data according to the elements of the intermediate result matrix T, and each computation unit is responsible for performing the multiplication of one element of T with a corresponding row of matrix P. The TP computation queue also includes a plurality of (e.g., 8) computation units.

[0034] As Figure 4As shown, the TP computing unit of the embodiment includes a task queue, a P-level cache, a non-ready request buffer, a computing queue, a multiplier, a partial sum queue and a hash table for implementing accumulation. The task queue is connected to the local controller for receiving a back-end operation task containing elements of the cached matrix T in the T-cache, and when the back-end operation task in the task queue is scheduled, the elements of the matrix P are obtained through the two-level cache formed by the P-level cache and the P-second cache, and the requests sent but not responded are put into the non-ready request buffer, and after the memory response of the elements of the matrix P is received, the ready elements of the matrix T and the matrix P in the non-ready request buffer are sent to the computing queue, and when the elements of the matrix T and the matrix P in the computing queue are scheduled, the multiplication operation of the elements of the matrix T and the matrix P is first performed through the multiplier, the partial sum obtained by the multiplication operation is put into the partial sum queue, and the partial sum in the partial sum queue is fed as an input stream to be accumulated to the hash table for implementing accumulation.

[0035] Unlike traditional hash tables, each key is usually uniquely mapped to a hash slot in traditional hash tables; when a conflict occurs, methods such as open addressing, chain addressing, rehashing or common overflow area are often used to resolve it. However, these methods have their own shortcomings: first, although perfect hashing can significantly reduce conflicts, it requires offline modeling of the data set and high-cost function search, which has poor engineering feasibility; second, although simple expansion can alleviate conflicts, the capacity cannot be increased indefinitely and is prone to space waste; third, open addressing will form a long probe chain when conflicts are concentrated, and the lookup and insertion delay is difficult to control; fourth, chain addressing or common overflow area introduces a large number of irregular chain accesses, which destroys locality, resulting in increased time delay and energy consumption of hardware implementation; fifth, rehashing requires multiple sets of hash functions and multiple determinations, and the control and timing complexity increases. For example, Figure 5As shown, to overcome the above problems, this embodiment proposes a method for sharing adjacent hash value entries: A hash value is calculated for the column index of the partial result (preferably using a bitmask and its offset version for easier hardware implementation), and this hash value is mapped to four adjacent entries in the hash table, thus forming a 2×2 shared sub-region. This 2×2 shared sub-region serves as the priority storage area for the key. Insertion and hit matching are completed within the sub-region in the actual access order using row buffer matching logic. When the 2×2 shared sub-region is full and a conflict still occurs, a data eviction logic triggers replacement, and an approximate least recently used replacement strategy is executed based on the hit count / usage information. The evicted data is written to the high-bandwidth memory (HBM) by row address, thereby avoiding the formation of long probe chains and random chained access. That is, when a conflict overflow occurs, instead of attempting complex data transfer control at the hardware level, the problem is simply passed to memory for software to handle later. Furthermore, when the TP computing unit has sent out all the data of a certain row of matrix T and completed the accumulation, a row completion signal and a row address signal are issued. The data eviction logic then batch-cleans all entries of that row in the hash table and writes them to the high-bandwidth memory (HBM) in row address order, freeing up space for subsequent accumulations. Preferably, the hash table width (number of slots) is set to a power of 2 (e.g., 128) to enable hashing and addressing of adjacent sub-regions through bitmasks, reducing hardware overhead and simplifying timing control. For example, when the table width is 128, the bitmask hash values ​​of keys 0, 128, and 256 are all 0, while the hash value of key 1 is 1. According to the adjacent sharing strategy of this embodiment, the above four keys will naturally fall into the adjacent 2×2 sub-regions corresponding to hash values ​​0 and 1, avoiding the long probe chains and degradation caused by traditional open addressing on the "0 bucket". After the accumulation operation is completed using a hash table, the replaced parts and the result are written back to the high-bandwidth memory (HBM). At this point, the HBM does not store the final result. To make the design cost-effective in terms of both performance and area, this embodiment uses a lightweight software interface to handle the sorting and merging of the results to obtain the final result of matrix C. For efficient access to matrix P, the backend also includes a two-level cache system. The difference is that the backends of different computing chips share a multi-body P-level cache through a global crossbar switch. The hash table of each backend (used for the accumulation of matrix C at this stage) calculates the partial sums. When the capacity of the hash table is insufficient to hold all active partial sums, a least recently used replacement strategy is adopted to safely overflow the "cold" partial sums into the HBM. After the calculation is completed, the software takes over and performs post-processing on the result, avoiding pipeline pauses and overhead caused by the back-and-forth movement of partial sums between the HBMs.

[0036] In addition, this embodiment also provides an electronic device, including a processor and a hardware accelerator interconnected, wherein the hardware accelerator is the hardware accelerator for triple sparse matrix multiplication.

[0037] As an optional implementation, in this embodiment, the hardware accelerator is also optimized by jointly optimizing data flow and storage management at the hardware level, efficiently merging intermediate results in on-chip implementation, and pipelining intermediate matrix results, minimizing expensive off-chip data transfer, reconstructing the bottleneck dominated by random delay into a throughput-dominated pipelined data path, thereby significantly improving the computing performance and energy efficiency of triple sparse matrix multiplication. Specifically, the application method of the foregoing hardware accelerator for triple sparse matrix multiplication also includes the following steps: providing the data of matrix R for the RA calculation array through the R-tag memory, providing the elements of matrix A through the two-level cache formed by the A-first cache and the A-second cache of the RA calculation unit, returning the elements of matrix A immediately if the A-first cache hits, otherwise accessing the A-second cache, returning the elements of matrix A if the A-second cache hits, and initiating a request to the high-bandwidth memory HBM to return the elements of matrix A if the A-second cache misses, performing the front-end operation T=RxA of triple sparse matrix multiplication C=RxAxP in the RA calculation array through each RA calculation unit, and temporarily storing the output elements of matrix T to the T-buffer for the TP calculation array to consume asynchronously; providing the data of matrix T for the TP calculation array through the T-buffer, providing the elements of matrix P through the two-level cache formed by the P-first cache and the P-second cache of the TP calculation unit, returning the elements of matrix P immediately if the P-first cache hits, otherwise accessing the P-second cache, returning the elements of matrix P if the P-second cache hits, and initiating a request to the high-bandwidth memory HBM to return the elements of matrix P if the P-second cache misses, performing the back-end operation C=TxP of triple sparse matrix multiplication C=RxAxP in the TP calculation array through each TP calculation unit, and finally writing the result matrix C back to the high-bandwidth memory HBM.

[0038] The high-bandwidth memory HBM returns one cache line at a time when returning data. In order to better utilize the incoming data, in this embodiment, the matrix A is divided into column windows, and the PE processes one column window at a time. The data in the column window is stored continuously according to the non-zero elements. The memory data is relatively more continuous and friendly, and the utilization rate of the data in the cache line is also higher. Specifically, in this embodiment, the matrix A and the matrix P are stored in a pre-adopted windowed compressed sparse row format, as shown in Figure 6 The windowed compressed sparse row format storage includes dividing the matrix A into a plurality of column windows, for example Figure 6The example of windows 0-3 is included in the column window, stored in compressed sparse row format CSR, and uses a row index array, a row pointer array, a non-zero element number array, a column index array, and a value array to store the information of the compressed matrix, wherein: the row index array is used to record the starting row of all column windows and the ending row of the last column window, for example, the starting row indexes of windows 0-3 are 0, 8, 16, and 24 respectively, and the ending row of window 3 is 32; the row pointer array is used to record the number of non-zero elements of each row in each column window in turn, for example, the column indexes of the non-zero elements of the first row of window 0 are 2, 3, and 5 respectively, and the value of the second element in the row pointer array is 3, indicating that the number of non-zero elements of the first row of window 0 is 3-0=3; the non-zero element number array is used to record the count of non-zero elements in each column window in turn, for example, the values of the gray cells in window 0 are 1-26 respectively, and the value of the second element in the non-zero element number array is 26, indicating that the count of non-zero elements in window 0 is 26-0=26; the column index array is used to record the column index of the non-zero elements of each column in each column window in turn, for example, the indexes of the non-zero elements of the first column of window 0 are all 0, and the values at the 13th and 20th positions in the column index array are 0; the value array is used to record the values of the non-zero elements in each column window in turn, for example, the values of the first row of window 0 are 1, 2, and 3 respectively; and before reading the matrix A stored in the windowed compressed sparse row format into the memory, the matrix A stored in the windowed compressed sparse row format is also rearranged and restored to generate a physically continuous and block format.

[0039] In this embodiment, on-chip front-end and back-end operations are both assigned tasks using a hierarchical dynamic load balancing strategy. When scheduling tasks, the front-end operation is divided by rows. Specifically, when each RA calculation unit in the RA calculation array performs element multiplication in the front-end operation T=RxA in the triple sparse matrix multiplication C=RxAxP, the row of the matrix R is taken as the tile-level task unit of the calculation tile, and the column window of the matrix A is taken as the calculation unit-level task unit of the RA calculation unit. Each calculation tile and RA calculation unit maintains a task counter, and the scheduler of the tile-level task and the calculation unit-level task always assigns new tasks to the currently most idle calculation tile or RA calculation unit to achieve hierarchical minimum load balancing. Figure 7 as shown on the left side of the arrow, wherein the column window size is 4, and each RA calculation unit is responsible for processing a column window.

[0040] The back-end task scheduling also adopts row division. When performing element multiplication in the back-end operation C=T×P in the triple sparse matrix multiplication C=R×A×P by each TP computing unit in the TP computing array, it includes mapping elements in the matrix T obtained by the front-end operation to corresponding computing tiles by row, taking a row of the matrix P as a task unit at the computing unit level of the TP computing unit, performing hash lookup and accumulation on the row of the matrix P in the computing tile, and performing task allocation among the TP computing units according to the minimum number of tasks or the least number of unfinished elements to achieve dynamic balancing. Figure 7 As shown on the right side of the arrow in FIG. 1, each TP computing unit is responsible for in-place multiplication and accumulation with a ready row in a column window.

[0041] Since random access to the matrix A and the matrix P is required in the computing process, corresponding caches are designed for data reuse. Frequent cache misses caused by random access will result in a large number of requests to the high-bandwidth memory HBM, and once data is absent, the computing unit will be blocked. In order to improve the throughput of the processing system, the front-end and back-end operations in the embodiment are both designed with an out-of-order window, that is, the computing unit is allowed to continue to issue a new data request while waiting for the return of the requested data, thereby effectively hiding the high latency of the HBM and alleviating the impact of cache misses. As long as the transmission window is not full, the computing unit can take the next data request from the task queue and transmit it when the previous data request is not ready. The unresponsive memory requests that have been transmitted are stored in the unready request buffer. Specifically, when a request is initiated to the high-bandwidth memory HBM to obtain elements of the return matrix A, the RA computing unit uses an out-of-order transmission window to record data requests for accessing the high-bandwidth memory HBM, thereby allowing the RA computing unit to continue to issue a new data request while waiting for the return of the requested data. As long as the transmission window of the RA computing unit is not full, the next data request can be issued when the previous data request is not ready. When a request is initiated to the high-bandwidth memory HBM to obtain elements of the return matrix P, the TP computing unit uses an out-of-order transmission window to record data requests for accessing the high-bandwidth memory HBM, thereby allowing the TP computing unit to continue to issue a new data request while waiting for the return of the requested data. As long as the transmission window of the TP computing unit is not full, the next data request can be issued when the previous data request is not ready.

[0042] To further reduce the impact of high bandwidth memory HBM high latency, in the embodiment, part or all of the high bandwidth memory HBM, R-Note memory, T-buffer, A-second level cache, P-second level cache adopts a request merging mechanism to reduce the impact of memory latency, which includes adopting a miss status holding table MSHR (Miss Status Holding Register) to merge multiple requests for the same cache line into a downstream access to reduce the number of memory accesses. By adding a request merging mechanism in the DRAM controller of each level of cache and high bandwidth memory HBM, that is, supporting the miss status holding table MSHR, multiple requests for the same cache line can be merged into a downstream access to reduce the number of memory accesses. The memory controller of the high bandwidth memory HBM supports write hit forwarding optimization and burst access optimization. Since the A-second level cache of the calculation chip front-end operation is not shared, the same address request may be initiated to the high bandwidth memory HBM. Therefore, the DRAM controller also implements request merging, thereby effectively improving the data access efficiency of matrices A and P and reducing the number of memory accesses.

[0043] As an optional implementation, in the embodiment, the entire data path from the RA calculation unit / TP calculation unit to the high bandwidth memory HBM is controlled by a credit-based queue or port read-write state judgment, which realizes a perfect back pressure flow control mechanism. This ensures that when the back-end processing speed cannot keep up with the front-end (for example, the T-buffer is full), the upstream module will automatically pause, thereby preventing data loss and system deadlock, and avoiding the huge data transfer overhead caused by writing the intermediate matrix to off-chip and then reading it back. In addition, the data flow of the hardware accelerator for triple sparse matrix multiplication in the embodiment is globally asynchronous. All inter-module communications are matched by a unique request ID for accurate request-response matching, which naturally supports out-of-order memory access and out-of-order task completion.

[0044] The working process of the hardware accelerator for triple sparse matrix multiplication in the embodiment can include the following stages:

[0045] 1) Data warm-up phase: (1) A part of R-matrix data is pre-fetched into the R-Note memory in the start-up phase; (2) the hot A-matrix data and P-matrix data are preheated through the second level cache (A-second level cache and P-second level cache);

[0046] 2) calculation stage, including front-end operation first calculating T = R x A, obtaining the result of intermediate matrix T, and then the back-end starts calculating C = T x P, specifically including: (1) taking matrix R: the RA calculation unit reads the elements of matrix R from the R-note storage, and when calculating, a cache line is discarded after one cache line is used, and a request for the next cache line is issued; (2) taking matrix A: the RA calculation unit reads the elements of matrix A through two-level cache, if A-level cache hits, the data is returned immediately, otherwise access A-two-level cache, if A-two-level cache hits, it is also returned, if A-two-level cache misses, a request is initiated to high bandwidth memory HBM; (3) front-end operation: calculation:

[0047] T[i,j] = R[i,k] x A[k,j];

[0048] Wherein T[i,j] represents the element of the i-th row and j-th column in the matrix T, R[i,k] represents the element of the i-th row and k-th column in the matrix R, and A[k,j] represents the element of the k-th row and j-th column in the matrix A; the multiplication result is sent to the accumulator, and accumulation is performed according to the T matrix coordinates; (4) temporary storage: the complete result obtained in the accumulator is sent to the T-buffer for temporary storage; (5) taking matrix T: the elements of matrix T are taken from the T-buffer, and according to the column index of the elements of matrix T, a row of matrix P is accessed, and a data request for matrix P is initiated; (6) taking matrix P: the TP calculation unit reads P elements through two-level cache, if P-level cache hits, the data is returned immediately, otherwise access P-two-level cache, if P-two-level cache hits, it is also returned, if P-two-level cache misses, a request is initiated to high bandwidth memory HBM; (7) back-end operation: calculation:

[0049] C[i,k] = T[i,j] x P[j,k];

[0050] wherein C[i, k] represents the element in the ith row and kth column of matrix C, T[i, j] represents the element in the ith row and jth column of matrix T, and P[j, k] represents the element in the jth row and kth column of matrix P, and the result is flowed into the partial sum queue; for each obtained multiplication result, a hash lookup is performed according to the column index, and in this embodiment, the hash function is based on right shifting the column index by one bit and then taking the modulus (the modulus size can be adjusted according to the matrix size); if a hit is made, direct accumulation is performed, if a miss is made and there is no space left, a number is sacrificed according to the least recently used replacement principle, and the number is written back to the high bandwidth memory HBM;(9) result writing back: when the partial sum queue in the TP calculation unit has sent out all the data of a certain row of matrix T and completed accumulation, a row completion signal and a row address signal are sent out, and the data pruning logic cleans up all the entries of the row in the hash table in batches according to the row address order, and writes them into the high bandwidth memory HBM.(10) The calculation is completed, the data is written back to the high bandwidth memory HBM from the hash table according to the row address, and the software interface is notified to rearrange and merge in the specified storage range to obtain a complete, correct and ordered result. Through the above architecture and method, a complex, single triple sparse matrix multiplication problem can be decoupled into two sub-problems which are parallel in macro and highly pipelined in micro, and through the dedicated data path, hierarchical cache design and software and hardware collaborative scheduling strategy, the three major bottlenecks of irregular memory access, uneven load and intermediate result overflow are effectively solved, thereby improving the performance and energy efficiency. The hardware accelerator for triple sparse matrix multiplication in this embodiment can perform two-level decoupling macro-pipelining: the RA calculation unit and the TP calculation unit can be executed in parallel, and the intermediate result matrix can be transported back and forth between the memory and the on-chip without using the memory and the on-chip, which significantly improves the execution efficiency; through the dedicated crossbar in the grouped high bandwidth memory HBM and the crossbar network Crossbar, the read and write channels are physically isolated, and the bandwidth utilization and latency are more controllable; through windowed compression of the sparse row format, the cache line utilization and continuous access probability can be improved, and the memory transaction number can be reduced; through hierarchical load balancing, the irregular workload can be balanced across the computing tiles / computing units, and the throughput can be improved; through non-blocking execution + request merging, the long memory delay can be effectively masked, and the pipeline stall can be avoided, which can solve the problem of massive off-chip data transport caused by irregular memory access, uneven calculation load and explosive intermediate partial sum results in the traditional general-purpose processor when processing triple sparse matrix multiplication, and face the serious performance and energy efficiency bottleneck, thereby improving the calculation performance and energy efficiency of triple sparse matrix multiplication.

[0051] The above only describes the preferred embodiments of the present application, and the protection scope of the present application is not limited to the above-mentioned embodiments. Any technical solutions falling within the concept of the present application shall fall within the protection scope of the present application. It should be noted that for ordinary technical personnel in the technical field, some improvements and refinements without departing from the principles of the present application shall also be considered as the protection scope of the present application.

Claims

1. A hardware accelerator oriented towards triple sparse matrix multiplication, characterized in that, The application relates to a high-bandwidth memory, a crossbar network and a processing unit connected in sequence, the processing unit comprising a hierarchical cache module, a global controller and a plurality of computing tiles, the global controller being configured to assign computing tasks to the computing tiles, each computing tile comprising an RA computing array, a TP computing array and a local controller, wherein the RA computing array is configured to perform a front-end operation T=RxA in a triple sparse matrix multiplication C=RxAxP, the TP computing array is configured to perform a back-end operation C=TxP in the triple sparse matrix multiplication C=RxAxP, and the hierarchical cache module comprises an R-tag memory configured to provide the RA computing array with a matrix R data cache, an A-two-level cache configured to provide the RA computing array with a matrix A data cache, a T-buffer configured to provide the RA computing array with a matrix T data cache, a P-two-level cache configured to provide the TP computing array with a matrix P data cache, and a C-cache configured to provide the TP computing array with a result matrix C data cache. The local controller is configured to control the RA computing array and the TP computing array, form an asynchronous two-stage computing pipeline through the T-buffer, and write the result matrix C in the C-cache back to the high-bandwidth memory, the A-one-level cache and the A-two-level cache of the RA computing unit in the RA computing array form a two-stage cache for the matrix A data, the P-one-level cache and the P-two-level cache of the TP computing unit in the TP computing array form a two-stage cache for the matrix P data, a plurality of data channels of the high-bandwidth memory are divided into a plurality of groups according to their functions to respectively carry a matrix R, A and P reading stream and a result matrix C writing back stream, the crossbar network comprises a P crossbar, a C crossbar, an R crossbar and an A crossbar, a group for storing the matrix P in the high-bandwidth memory is connected to the P-two-level cache through the P crossbar, a group for storing the matrix C in the high-bandwidth memory is connected to the C-cache through the C crossbar, a group for storing the matrix R in the high-bandwidth memory is connected to the R-tag memory through the R crossbar, and a group for storing the matrix A in the high-bandwidth memory is connected to the A-two-level cache through the A crossbar, the R-tag memory, the A-two-level cache, the T-buffer and the C-cache in the hierarchical cache module are unique to each computing tile, the P-two-level cache is shared by all the computing tiles, and the TP computing array in all the computing tiles is connected to the P-two-level cache shared by all the computing tiles through a same global crossbar.

2. The hardware accelerator for triply sparse matrix multiplication, according to claim 1, wherein, The RA computing unit comprises a task queue, an A-level-1 cache, a non-ready request buffer, a computing queue, a multiplier, a partial sum queue, an accumulator and an accumulation window buffer, the task queue is connected with a global controller to receive a front-end operation task, when the front-end operation task in the task queue is scheduled, elements of a matrix A are obtained through a two-level cache formed by the A-level-1 cache and an A-level-2 cache, elements of a matrix R are obtained through an R-register memory, and a request sent but not responded is put into the non-ready request buffer, after the elements of the matrix A and the matrix R are responded by the memory, the elements of the complete matrix A and the matrix R in the non-ready request buffer are sent into the computing queue, when the elements of the matrix A and the matrix R in the computing queue are scheduled, multiplication of the elements of the matrix A and the matrix R is first performed through the multiplier, a partial sum obtained by the multiplication is put into the partial sum queue, and the partial sum in the partial sum queue is used to complete accumulation in the accumulation window buffer according to coordinates of a matrix T by using the accumulator, and data of the matrix T after the accumulation is output to a T-buffer under control of a local controller to be consumed by a TP computing array asynchronously.

3. The hardware accelerator for triply sparse matrix multiplication, according to claim 1, wherein, The TP computing unit comprises a task queue, a P-level-1 cache, a non-ready request buffer, a computing queue, a multiplier, a partial sum queue and a hash table for implementing accumulation, the task queue is connected with a local controller to receive a back-end operation task containing elements of a matrix T stored in the T-buffer, when the back-end operation task in the task queue is scheduled, elements of a matrix P are obtained through a two-level cache formed by the P-level-1 cache and a P-level-2 cache, and a request sent but not responded is put into the non-ready request buffer, after the elements of the matrix P are responded by the memory, the elements of the ready matrix T and the matrix P in the non-ready request buffer are sent into the computing queue, when the elements of the matrix T and the matrix P in the computing queue are scheduled, multiplication of the elements of the matrix T and the matrix P is first performed through the multiplier, a partial sum obtained by the multiplication is put into the partial sum queue, and the partial sum in the partial sum queue is fed as an input stream to be accumulated to the hash table for implementing accumulation; The hash table implements accumulation by calculating a hash value for a column index of a partial sum result and mapping the hash value to four adjacent table entries in the hash table to form a 2*2 shared sub-area, the 2*2 shared sub-area completes insertion and hit matching in the sub-area in an actual access order by means of row buffer matching logic, when a conflict still occurs after the 2*2 shared sub-area is full, replacement is triggered by data eviction logic, and an approximate least recently used replacement strategy is performed according to a hit number, and the evicted data in the 2*2 shared sub-area is written into a high-bandwidth memory according to a row address; When the partial sum queue in the TP computing unit has sent out all the data of a certain row of matrix T and completed the accumulation, a row completion signal and a row address signal are sent out, based on which the data pruning logic bulk cleans all the entries of the hash table of the row in the high bandwidth memory and writes them into the high bandwidth memory in row address order to release space for subsequent accumulation and time; after the accumulation operation is completed by the hash table, the partial sum result replaced is written back into the high bandwidth memory, at this time the high bandwidth memory stores not the final result, and a software interface is used to sort and merge the partial sum result stored in the high bandwidth memory to obtain the final result of matrix C.

4. An electronic device comprising a processor and a hardware accelerator connected to each other, characterized in that, The hardware accelerator is the hardware accelerator for triple sparse matrix multiplication according to any one of claims 1-3.

5. A method of using the hardware accelerator for triple sparse matrix multiplication of any one of claims 1 to 3, characterized in that, The method comprises the following steps: providing the data of matrix R for the RA computing array through the R-tag memory, providing the elements of matrix A for the two-level cache formed by the A-first cache and the A-second cache of the RA computing unit, returning the elements of matrix A immediately if the A-first cache hits, otherwise accessing the A-second cache, returning the elements of matrix A if the A-second cache hits, and initiating a request to the high bandwidth memory to return the elements of matrix A if the A-second cache misses, performing the front-end operation T=R×A of the triple sparse matrix multiplication C=R×A×P through each RA computing unit in the RA computing array, and caching the output elements of matrix T to the T-buffer for the TP computing array to consume asynchronously; providing the data of matrix T for the TP computing array through the T-buffer, providing the elements of matrix P for the two-level cache formed by the P-first cache and the P-second cache of the TP computing unit, returning the elements of matrix P immediately if the P-first cache hits, otherwise accessing the P-second cache, returning the elements of matrix P if the P-second cache hits, and initiating a request to the high bandwidth memory to return the elements of matrix P if the P-second cache misses, performing the back-end operation C=T×P of the triple sparse matrix multiplication C=R×A×P through each TP computing unit in the TP computing array, and finally writing the result matrix C back into the high bandwidth memory.

6. The method of claim 5, wherein the hardware accelerator is applied to a triple sparse matrix multiplication. The method also comprises storing the matrix A and the matrix P in a windowed compressed sparse row format, which comprises dividing the matrix A and the matrix P into a plurality of column windows, storing the matrix A and the matrix P in the compressed sparse row format in the column windows, and storing information of the compressed matrix by using a row index array, a row pointer array, a non-zero element number array, a column index array and a value array, the row index array being used to record starting rows of all column windows and an ending row of a last column window, the row pointer array being used to sequentially record numbers of non-zero elements of each row in each column window, the non-zero element number array being used to sequentially record counts of non-zero elements in each column window, the column index array being used to sequentially record column indexes of non-zero elements in each column window, and the value array being used to sequentially record values of non-zero elements in each column window; and the method also comprises rearranging and restoring the windowed compressed sparse row format stored matrix A before reading the windowed compressed sparse row format stored matrix A into the memory.

7. The method of claim 6, wherein the hardware accelerator for triple sparse matrix multiplication is applied to a system for solving a system of linear equations. When each RA computing unit in the RA computing array performs element multiplication in front-end operation T=R×A in the triple sparse matrix multiplication C=R×A×P, the method comprises taking a row of the matrix R as a tile-level task unit of a computing tile, taking a column window of the matrix A as a computing unit-level task unit of the RA computing unit, each computing tile and RA computing unit maintaining a task counter, and a scheduler of the tile-level task and the computing unit-level task always allocating a new task to a currently most idle computing tile or RA computing unit to realize hierarchical minimum load balancing; when each TP computing unit in the TP computing array performs element multiplication in back-end operation C=T×P in the triple sparse matrix multiplication C=R×A×P, the method comprises mapping elements in the matrix T obtained by the front-end operation to corresponding computing tiles according to rows, taking a row of the matrix P as a computing unit-level task unit of the TP computing unit, accessing and accumulating the row of the matrix P in place according to a column index in the computing tile, and performing task allocation among the TP computing units according to minimum task numbers or minimum unfinished elements to realize dynamic balancing.

8. The method of claim 7, wherein the hardware accelerator is applied to a triple sparse matrix multiplication. When the request to the high bandwidth memory for the elements of matrix A is initiated, the RA computation unit uses an out-of-order transmission window to record data requests to the high bandwidth memory, thus allowing the RA computation unit to continue to issue new data requests while waiting for requested data to return, as long as the transmission window of the RA computation unit is not full, the next data request is issued while the previous data request is not yet ready; when the request to the high bandwidth memory for the elements of matrix P is initiated, the TP computation unit uses an out-of-order transmission window to record data requests to the high bandwidth memory, thus allowing the TP computation unit to continue to issue new data requests while waiting for requested data to return, as long as the transmission window of the TP computation unit is not full, the next data request is issued while the previous data request is not yet ready; part or all of the high bandwidth memory, R-tag memory, T-buffer, A-2nd level cache, P-2nd level cache use a request merging mechanism to reduce the impact of memory latency, the request merging mechanism includes using a miss state holding table to merge multiple requests for the same cache line into one downstream access to reduce the number of memory accesses.

Images