A near-memory computing device for direct method solving of sparse linear systems

Abstract

The application provides a near-memory computing device for solving sparse linear equations by direct method, comprising a logic layer and a memory layer, the memory layer comprising a plurality of memory banks hierarchically organized; the logic layer comprising a plurality of processing units, each processing unit coupled with its corresponding memory bank through an interconnection structure, the processing units asynchronously executing independent instruction sequences, processing local data of sparse matrices stored in the corresponding memory banks in a behavior granularity, and performing calculations when dependent external data is ready, the processing units transferring data through an interconnection network to realize indirect data interaction among the memory banks, a hierarchical scratchpad memory deployed in the interconnection network to cache data at each level of the data transmission path across the processing units to support local reuse.

Description

Technical Field

[0001] This invention relates to the field of computer architecture, more specifically to the field of integrated circuits, and more specifically to a near-in-memory computing device for solving sparse linear equations using direct methods. Background Technology

[0002] Solving sparse linear equations is a fundamental workload in scientific computing and machine learning, but its performance is severely limited by memory bandwidth. Processing-near-memory (PNM) computing utilizes logic cells located near the memory array to perform high-precision computations within the memory chip, thereby alleviating the bandwidth bottleneck. Therefore, this technology shows great potential in the field of sparse linear algebra. Current research focuses heavily on PNM architecture design in the area of ​​sparse matrix multiplication, such as those for sparse matrix-dense vector multiplications (SpMV), sparse matrix-dense matrix multiplications (SpMM), and sparse matrix-sparse matrix multiplications (SpMSpM). These sparse matrix multiplication operators are characterized by simple data flow and ease of parallelization. PNM acceleration architectures designed for these operators have achieved good speedups on their respective operators.

[0003] Current near-memory computing methods do not yet support direct solutions to sparse linear equations. Current 3D near-memory computing architectures based on High-Bandwidth Memory (HBM) and other methods typically include a logic layer and a DRAM layer. The logic layer integrates processing elements (PEs) with computational capabilities, transmitting data to the DRAM layer via through-silicon vias (TSVs) or hybrid bonding. In scenarios involving simple data flows and high parallelism in various sparse matrix multiplication operations, each PE can utilize its high internal memory bandwidth to quickly retrieve data from the DRAM layer and perform parallel computations, thereby achieving acceleration.

[0004] However, in scenarios requiring direct solutions (such as LU decomposition), existing architectures suffer from the following two main technical drawbacks:

[0005] First, complex task dependencies lead to severe computational pauses. The data in the direct solution process exhibits strong coupling dependencies. This characteristic renders existing simple parallel solutions inefficient. Many process objects (PEs) become stalled while waiting for the results of other units, failing to fully utilize the hardware's computing power. This highlights the shortcomings of the existing architecture's pipeline design.

[0006] Second, cross-processor (PE) data transfer negates the advantages of near-memory computation, resulting in low efficiency of cross-processor data reuse. Direct method solutions require frequent sharing and transfer of intermediate computation results between PEs. Existing architectures lack optimizations for on-chip data reuse and local communication in direct method solutions. The large amount of cross-processor data movement not only becomes a new performance bottleneck, but the huge communication overhead it generates also completely overshadows the high bandwidth benefits of near-memory computation.

[0007] It should be noted that the background information presented here is only for illustrating relevant information about the present invention to aid in understanding the technical solution of the present invention, and does not imply that the relevant information is necessarily prior art. The relevant information was submitted and disclosed together with the present invention, and should not be considered prior art unless there is evidence that the relevant information was disclosed before the filing date of the present invention. Summary of the Invention

[0008] Therefore, the purpose of this invention is to overcome the shortcomings of the prior art and provide a near-in-memory computing device for solving sparse linear equation systems using direct methods.

[0009] The objective of this invention is achieved through the following technical solution:

[0010] According to a first aspect of the present invention, a near-memory computing device is provided, comprising a logic layer and a storage layer integrated by stacking, wherein the storage layer and the logic layer are connected by an interconnect structure to transmit data, the storage layer comprising a plurality of hierarchically organized storage banks; the logic layer comprising a plurality of processing units, each processing unit having a corresponding storage bank and being coupled thereto by an interconnect structure; each processing unit is configured to asynchronously execute an independent instruction sequence, each instruction sequence corresponding to sparse matrix local data divided at least once and stored in its corresponding storage bank, the processing unit performing asynchronous computation on the local data when the external data on which the local data computation depends is ready; data transmission between processing units is performed through an interconnect network of the logic layer, and indirect data interaction between storage banks is performed through the interconnect network, wherein the storage banks corresponding to the hierarchical organization of the interconnect network are deployed with hierarchical sticky-note memories for caching data at each level of the cross-processing unit data transmission path to support near-memory reuse. This solution can achieve at least the following beneficial technical effects: by restricting data interaction between storage units to the logical layer interconnection network, it simplifies the design complexity of the storage layer without disrupting the mature structure of the storage layer; each processing unit adopts an asynchronous execution mode, which fully considers the locality differences of sparse data and significantly shortens the waiting time caused by global synchronization; the hierarchical note-based storage intercepts repeated remote data requests nearby and uses its cached data to achieve faster access, significantly reducing the average access latency and global bandwidth pressure.

[0011] Optionally, the device is configured to: divide the sparse matrix into multiple storage banks by rows; during the computation process, each processing unit independently advances its local instruction sequence asynchronously, responsible for processing the sparse matrix rows allocated to its corresponding storage bank, wherein: when the current processing unit encounters data dependencies during the computation of the corresponding sparse matrix row, it initiates a remote data request to the remote processing unit corresponding to the remote storage bank to obtain the updated data of the required dependent row; if the updated data of the dependent row is ready, the current processing unit obtains the updated data and continues to advance the computation process of the sparse matrix row; if the updated data of the dependent row is not ready, the current processing unit enters a waiting state for the sparse matrix row until the updated data arrives, and after the updated data is transmitted to the local storage bank via the Internet, it continues to advance the computation process of the sparse matrix row. This scheme can achieve at least the following beneficial technical effects: After dividing by row, the global synchronization barrier can be removed, allowing each processing unit to execute independently according to its local data and instruction sequence, maximizing parallel execution time and improving the overall computational efficiency of the direct method; through the event-driven local dependency waiting mechanism, the computation pause is strictly limited to the local processing unit where the data is truly missing, avoiding resource idleness and inefficient operation caused by overall synchronization; the waiting mechanism ensures the correctness of data dependencies, thereby ensuring the correctness of the computation results.

[0012] Optionally, each processing unit is configured to perform operations based on a three-stage pipeline. Each processing unit can perform multiple locally data-related operations in an overlapping manner through this pipeline. In this three-stage process, the remote acquisition, waiting, and reception of row-dependent update data are merged into a variable-latency transmission stage; index matching operations are configured as a lookup stage; and multiply-accumulate and divide operations are configured as computation stages. This scheme can achieve at least the following beneficial technical effects: by masking variable remote access latency through pipelined execution, the instruction issuance rate, computation, and index matching resource utilization of a single processing unit are improved; the overlapping execution of the transmission, lookup, and computation stages allows multiple operations within a processing unit to proceed in parallel, keeping local components busy as much as possible, thereby improving single-node throughput and overall performance.

[0013] Optionally, the device is configured such that when the current processing unit performs a remote read operation to obtain updated data for its dependent rows, the updated data is written and cached level by level in a note-based memory on each node of the interconnection network during the downlink path from the remote processing unit to the current processing unit via the interconnection network; wherein the downlink path is the path from the highest-level node of the interconnection network to the current processing unit; when any processing unit initiates a read request for the same data, it searches for and returns the data from the nearest-level note-based memory along the uplink path. This scheme can achieve at least the following beneficial technical effects: multiplexing data transmission across processing units, significantly reducing the number of repeated accesses to remote storage; reducing congestion and power consumption of higher-level links in the interconnection network, confining most data transmission within lower-level nodes; and significantly reducing the overall data transmission volume of the system, thereby improving the overall performance and energy efficiency of the architecture when solving sparse linear equation systems using direct methods.

[0014] Optionally, the hierarchical structure of the multiple storage units in the layered organization includes: channels, partitions, storage unit groups, and storage units. Multiple storage units are divided into multiple channels, each channel contains multiple partitions, each partition contains multiple storage unit groups, and each storage unit group contains several storage units. No interconnection structure is established between storage units at the storage unit layer; data interaction between storage units is achieved through processing units in the logical layer and an interconnection network. This scheme can achieve at least the following beneficial technical effects: restricting data transmission within the logical layer simplifies the design complexity of the storage layer without disrupting its mature structure; decoupling vertical access to local storage units from horizontal requests for data from other storage units, allowing both processes to occur simultaneously; and ensuring the correctness of asynchronous operation and laying the foundation for data reuse through only logical layer interconnections.

[0015] Optionally, the interconnection network is an H-tree structure, with the processing unit serving as a leaf node in the H-tree structure. Multiple leaf nodes converge to form a storage group layer node, multiple storage group layer nodes converge to form a partition layer node, multiple partition layer nodes converge to form a channel layer node, and multiple channel layer nodes converge to form a root node. The sticky note-style memory is deployed on each intermediate node of the H-tree structure. This solution achieves at least the following beneficial technical effects: the H-tree structure provides a deterministic low-latency transmission path; the sticky note-style memory constitutes a hierarchical data cache, intercepting repeated remote data requests at lower-level nodes, significantly reducing average access latency and global bandwidth pressure; and it allows multiple processing units to operate efficiently in parallel, avoiding root node communication congestion.

[0016] Optionally, the storage group layer nodes, partition layer nodes, and channel layer nodes are each equipped with corresponding sticky-label storage. This scheme can achieve at least the following beneficial technical effects: providing data reuse opportunities for subsequent read requests at each level of branch node, restricting most cross-processing unit data interactions within the underlying network branches; effectively avoiding root node communication congestion that is very likely to occur in interconnected network topologies, and significantly reducing global data transport overhead.

[0017] Optionally, the processing unit includes: a near-memory computing core, a local SPM, an indexed content-addressable memory, a direct memory access unit, and a transmission unit, wherein: the near-memory computing core includes multiple floating-point computing units for performing floating-point operations; the local SPM is used to temporarily store data from remote processing units; the indexed content-addressable memory is used to store the index of sparse data and perform index search matching; the direct memory access unit is used by the near-memory computing core and the content-addressable memory to access data in their corresponding memory banks; the transmission unit is connected to an interconnection network for data interaction between the processing unit and other processing units. This scheme can achieve at least the following beneficial technical effects: the processing unit integrates index matching, data transmission, storage access, and local caching units, fully ensuring its independence and providing a foundation for its asynchronous computing mode; the index matching unit accelerates the sparse data addressing process, and the local caching unit accelerates the data transmission process, thereby accelerating sparse computing tasks on a single processing unit.

[0018] According to a second aspect of the present invention, a method for directly solving a system of sparse linear equations using a near-memory computing device as described in the first aspect is provided. The method includes: dividing a sparse matrix to be solved into multiple memory banks by rows, with each processing unit responsible for processing the sparse matrix rows in its corresponding memory bank; each processing unit independently advances its local instruction sequence asynchronously; when encountering data dependencies during the calculation process, it initiates a remote data request to a remote processing unit to obtain updated data for the required dependent rows, and performs the following operations according to the readiness status of the updated data: if the updated data is ready, it obtains the updated data and continues the calculation process for the sparse matrix row; if the updated data is not ready, it enters a waiting state for the sparse matrix row until the updated data is transmitted to the local machine via the Internet, and then continues the calculation process for the sparse matrix row; within each processing unit, the remote acquisition, waiting, and reception of dependent row data are merged into a variable-latency data transmission stage, the index matching operation is configured as a lookup stage, and the multiplication-accumulation and division operations are configured as a calculation stage; the data transmission stage, the lookup stage, and the calculation stage constitute a three-stage fine-grained pipeline to execute multiple operations in an overlapping manner. This scheme can achieve at least the following beneficial technical effects: removing global synchronization barriers, allowing each processing unit to execute independently according to its local data and instruction sequence, maximizing parallel execution time; masking variable remote access latency through pipelines, improving the throughput of a single processing unit; ensuring the correctness of data dependencies by relying on waiting mechanisms, ensuring the accuracy of calculation results; and taking into account both complex data dependency processing and irregular communication optimization, significantly improving the overall performance of the direct method for solving sparse linear equation systems.

[0019] Optionally, the method includes: the updated data is written and cached level by level in the sticky notes memory on each level node of the Internet during the downlink path from the remote processing unit to the current processing unit via the Internet, and the replacement strategy of the sticky notes memory on each level node is set as follows: when the storage space is full, the least accessed data row is automatically evicted; when any processing unit initiates a read request for the same data, the data is searched for and returned from the nearest level sticky notes memory along the uplink path. This scheme can achieve at least the following beneficial technical effects: multiplexing data transmission across processing units, significantly reducing the number of repeated accesses to remote storage; reducing congestion and power consumption of higher-level links in the Internet; and ensuring efficient utilization of cache space through a hardware-managed hierarchical sticky notes memory replacement strategy, without introducing complex global cache consistency overhead, thereby further improving the overall performance and energy efficiency of the architecture. Attached Figure Description

[0020] The embodiments of the present invention will be further described below with reference to the accompanying drawings, wherein:

[0021] Figure 1A schematic diagram of the numerical calculation process for the LU decomposition of an existing sparse matrix;

[0022] Figure 2 This is a schematic diagram of the overall structure of a near-memory computing device for direct method solution according to an embodiment of the present invention;

[0023] Figure 3 This is a schematic diagram of an H-tree hierarchical interconnection network topology according to an embodiment of the present invention;

[0024] Figure 4 This is a schematic diagram of the internal modular structure of the processing unit according to an embodiment of the present invention;

[0025] Figure 5 This is a schematic diagram of coarse-grained asynchronous parallel scheduling between processing units according to an embodiment of the present invention;

[0026] Figure 6 This is a schematic diagram of fine-grained three-stage pipeline execution within a processing unit according to an embodiment of the present invention;

[0027] Figure 7 This is a schematic diagram of a data reuse mechanism based on a hierarchical sticky note storage according to an embodiment of the present invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0029] As mentioned in the background section, the inventors, while researching the design of near-in-memory computing architectures for direct method solutions, discovered that the underlying reasons for the computational pauses and low efficiency of cross-cell data reuse in existing technologies are as follows: existing near-in-memory architectures are mainly designed for conventional simple parallel scenarios (such as sparse matrix multiplication), and their pipeline schemes do not consider the parallel design issues when there are strong data dependencies between PEs; at the same time, since the amount of cross-cell data transmission in simple scenarios is minimal and the data flow is regular, existing architectures do not require the deployment of on-chip data reuse mechanisms. However, in scenarios where direct methods are used to solve sparse linear equation systems (such as LU decomposition), these previously ignored strong dependencies and irregular communications become the core bottlenecks hindering the performance improvement of near-in-memory computing architectures.

[0030] Through research on the decomposition of the direct method algorithm and the design of near-memory computing architecture components, the inventors discovered that this deficiency can be addressed by proposing a method combining coarse-grained asynchronous parallelism with fine-grained pipelines and introducing scratchpad memory (SPM) to achieve dynamic data reuse. Specifically, this invention designs a near-memory computing device for direct method solutions, employing a coarse-grained asynchronous parallelism scheme between processing units (PEs) and a fine-grained pipeline scheme within each PE. This allows for the exploitation of the parallel capabilities of near-memory PEs at different levels, while considering complex data dependencies. Furthermore, this invention introduces an H-tree-based interconnection network for cross-PE communication at the logical layer, and further introduces hierarchical scratchpad memory (SPM) at each node of the H-tree to achieve dynamic data caching and reuse. This improves data reuse rates while avoiding complex global cache consistency overhead, reducing cross-PE data transfer volume, and further enhancing the overall architecture performance.

[0031] Figure 1 This is a schematic diagram of the computation process for LU (Lower-Upper) decomposition. LU decomposition is a key step in the direct method for solving sparse linear equation systems. This embodiment uses it as an example to introduce the acceleration process. The core principle of LU decomposition numerical computation is row-by-row dependency update and normalization: processing is performed sequentially according to the natural order of matrix rows. Each row first reads its own non-zero element structure, then uses the non-zero element data of the previous decomposed row pointed to by its column index for multiplication and accumulation update, and finally divides the non-zero elements of the row by the diagonal element, thereby generating the corresponding rows of the lower triangular matrix L and the upper triangular matrix U row by row. This invention can also support the triangular solution steps of the direct method and fully support the direct method solution on the architecture proposed in this invention. Schematic, LU decomposition decomposes a sparse matrix A into the product of the lower triangular matrix L and the upper triangular matrix U. Its computation process includes symbolic decomposition and numerical decomposition, where symbolic decomposition is a preprocessing process, which is completed by the CPU. The numerical decomposition process requires calculating the values ​​of matrices L and U, which is executed by the architecture proposed in this invention. The numerical decomposition process requires updating and calculating each row, and for each row, this update process consists of multiple steps. For example... Figure 1As shown, for the i-th row, this process involves five steps. The first step is to read the i-th row and obtain the non-zero element structure of its L matrix. There are three non-zero elements in the diagram: a, b, and c, with column indices j1, j2, and j3 respectively. These column coordinates correspond to rows j1, j2, and j3, respectively. The i-th row needs to be updated based on the U values ​​of these three rows. The second to fourth steps update the i-th row using these three rows. Taking the second step as an example, the column index of a corresponds to row j1, and the j1-th row of U is used to update the i-th row. First, determine the non-zero elements in the j1-th row of U that have the same column index as the i-th row. Here, g is the same as c, and h is the same as e. Therefore, c is updated to ca*g, and e is updated to ea*h. The third and fourth steps are similar, updating d and f, and e and f respectively. The fifth step divides all non-zero elements in the i-th row by the diagonal element d to obtain the final new i-th row. Then, the calculation continues for the next row until all rows have been calculated.

[0032] It should be noted that although this embodiment describes the row update process in detail using LU decomposition as an example, in other specific embodiments, the near-memory computing architecture and its fine-grained pipeline and coarse-grained asynchronous parallel mechanism proposed in this invention are also fully applicable to other direct methods or preprocessing sparse solution processes such as Cholesky decomposition, QR decomposition, and incomplete LU decomposition (ILU) of symmetric positive definite matrices. In the above embodiment, the sparse matrix computation task is mapped to the processing unit using "rows" as the basic unit. In a modified scheme for practical application, to further improve the locality of computation, the task mapping unit can also be a block composed of multiple consecutive rows, and the pipeline scheduling within the processing unit will be adaptively adjusted to block-based data extraction and matrix multiplication updates.

[0033] Figure 2 This is a schematic diagram of the architecture of an embodiment of the near-memory computing device proposed in this invention. The device includes a storage layer (corresponding to a DRAM layer) and a logic layer. The storage layer contains N (e.g., 128 or 256) memory banks, corresponding to N (e.g., 128 or 256) processing units (PEs) in the logic layer. The storage layer and the logic layer are connected by an interconnect structure to transmit data. The interconnect structure can be a through-silicon via (TSV) structure or a hybrid bonding structure. For example, a vertical interconnect structure can be used to achieve cross-layer data transmission to obtain extremely high local data access bandwidth, meeting the data read and write requirements of the PEs to the upper-layer memory banks. Figure 2The right side illustrates a schematic storage hierarchy of this embodiment, divided into four layers from top to bottom: channels, partitions, storage groups, and storage blocks. These four layers are connected via an H-tree, and each layer integrates a sticky note memory (SPM). Schematic, the SPM capacity at the storage group level is 1MB, at the partition level it is 2MB, and at the channel level it is 4MB. The storage layer stores sparse matrix data, while the logic layer handles near-memory computation and data transfer across PEs. The hierarchical SPM temporarily stores sparse matrix row data flowing through the H-tree interconnect network to provide reuse opportunities. Through this hierarchical structure, the device can provide data reuse opportunities for subsequent read requests at each branch node, restricting most cross-processing unit data interactions to the lower-level network branches. This mechanism effectively avoids root node communication congestion, which is highly prone to occur in H-tree topologies, and significantly reduces global data transport overhead.

[0034] It should be noted that the above embodiments regarding the four microarchitectural layers of the storage layer, the correspondence between the 128 memory banks and processing units, and the specific capacities of each SPM (e.g., 1MB, 2MB, 4MB) are only a preferred embodiment of this architecture. In other modified embodiments, the storage layer division can be simplified to three layers or expanded to five layers based on the actual chip area and routing complexity; the capacity configuration of each SPM can also be scaled proportionally according to the size of the target sparse matrix and the on-chip network bandwidth budget. This invention does not impose any limitations on such adjustments to the layer and capacity parameters based on the basic communication multiplexing concept of this invention.

[0035] Figure 3This diagram illustrates the hierarchical topology of an H-tree interconnect network. The root node (1×) is the highest level of the tree, connecting downwards to two channel layer nodes (2×). Each channel layer node connects downwards to two partition layer nodes (2×). Each partition layer node connects downwards to two memory group layer nodes (2×). Each memory group layer node ultimately connects downwards to sixteen memory layer leaf nodes (16×, with PEs serving as leaf nodes in the H-tree). Thus, the interconnect network, from top to bottom, includes one root node, two channel layer nodes, four partition layer nodes, eight memory group layer nodes, and one hundred and twenty-eight memory layer nodes (corresponding to processing units) serving as leaf nodes. The diagram clearly shows the convergence connections between nodes using diagonal lines. There are no horizontal direct connections between nodes at the same level; data is only converged or distributed along the vertical path between parent and child nodes. Several SPMs are deployed on each hierarchical intermediate node of this structure. The H-tree interconnect network is used to implement cross-processing unit data transmission, and the SPMs are used to temporarily store data transmitted across PEs to provide reuse opportunities. This inverted pyramid-shaped hierarchical design, with fewer top-level nodes and more bottom-level leaf nodes, restricts the vast majority of data transmission to within the lower-level storage group or between adjacent leaf nodes. Only a small amount of global communication across groups, partitions, or even channels needs to pass through upper-level nodes. As a result, data interaction between processing units can be completed locally with the shortest path, significantly reducing the average distance and latency of cross-processing unit data transmission. At the same time, it provides a physical structural basis for deploying tag-like storage at each level of intermediate nodes to intercept and reuse data locally.

[0036] It should be noted that the branch fan-out ratio of each level of the H-tree interconnection network in this embodiment (i.e., a 1:2:2:2:16 ratio for 128 leaf nodes) is only a schematic hardware configuration. In other modified embodiments, the hierarchical depth and number of branches of each layer of nodes can be flexibly adjusted according to the total number of processing units (e.g., adjusted to a 1:4:4:8 ratio). Implementers can use similar multi-level tree-structured hierarchical networks to realize the interconnection topology for local communication and data multiplexing across processing units, and adjust the node ratio of each layer; this invention does not impose any limitations on this.

[0037] Figure 4This is a schematic diagram of the internal structure of the processing unit, illustrating the core components of the basic execution node constituting the near-memory computing device and their data interaction relationships. In this embodiment, each processing unit includes a near-memory computing core (corresponding to the PIM core), an index content addressing memory (corresponding to the index CAM), a local note-taking memory (corresponding to the local SPM), a direct memory access unit (corresponding to DMA), and a transmission unit. Illustratively, the PIM core includes 32 floating-point units (FPUs), the index CAM has a capacity of 256KB, the local SPM has a capacity of 64KB, and the transmission unit storage portion has a capacity of 10KB. The FPU is used to perform addition, subtraction, multiplication, and division operations on floating-point data. The index CAM is used to store the indexes of sparse row non-zero elements and to perform index search and matching of sparse row non-zero elements. The local SPM is used to temporarily store sparse data from a remote location. The DMA provides data read and write functions between the PIM core and the memory. The transmission unit is used for data transmission and reception between the local PE (corresponding to the current processing unit) and the remote PE (corresponding to the remote processing unit, or other processing units).

[0038] In existing technologies, due to the different data distributions in different parts of a sparse matrix and the varying loads on each processing unit, speed differences exist. Using existing global synchronization schemes would cause these processing units to compute simultaneously at the speed of the slowest unit. However, in this invention, each processing unit is configured to asynchronously execute an independent instruction sequence (instruction stream) based on its independent near-memory computing core and its instruction sequence to compute local data in the memory bank. In other words, each processing unit can independently compute data in its corresponding memory bank without needing to synchronize with other processing units. When data from other memory banks is needed, a request is sent to the corresponding processing unit, and the required data is obtained through the on-chip interconnect network. Therefore, the asynchronous design of this invention avoids global synchronization, ensuring that each processing unit can process its local data at the highest speed and can begin computation immediately after the required data is available, making it suitable for sparse direct method solutions. The modular design of "computation-addressing-caching-transmission" adopted in this embodiment enables each processing unit to have independent and complete execution capabilities, providing a hardware foundation for its asynchronous parallel operation. At the same time, the index matching unit accelerates the non-zero addressing of sparse data, and the local caching unit reduces repeated remote access, thereby improving the throughput efficiency of a single node in processing sparse computing tasks.

[0039] Figure 4In the diagram, solid arrows represent data flow, and dashed arrows represent index flow. The schematic data flow is as follows: DMA reads data from the upper-layer storage, sending the data content (solid lines) to the PIM core or local SPM, and simultaneously sending the index information (dashed lines) to the CAM for search matching. The PIM core determines the non-zero element positions based on the index matching results returned by the CAM and performs multiplication-accumulation or division operations. During the operation, the PIM core can directly access remote dependency data temporarily stored in the local SPM, and the calculation results can also be written back to storage via DMA. The transmission unit interacts bidirectionally with the remote processing unit through the interconnection network, receiving updated data from the remote processing unit and storing it in the local SPM, and also sending locally calculated data to the remote unit.

[0040] According to one example of the invention, in combination Figure 4 Schematic representation Figure 1 The LU decomposition calculation process on this device is as follows: In the first step, since the update of the i-th row depends on the data of these three rows, if these three rows of data are located in other storage, the current PE will initiate a remote data request through the H-tree interconnect network (i.e., enter the transmission stage). During this process, DMA is used to obtain local storage data and transmit it to the PIM core and index CAM to determine the non-zero element structure. Then, the transmission unit is used to obtain the data of each row from the remote end. In the second step, after the data of the j1-th row from the remote end reaches the current PE through the H-tree network and SPM multiplexing mechanism, it is transferred to the local SPM for temporary storage via the transmission unit. Then, the index enters the index CAM to perform index lookup and matching to determine the elements in the j1-th row that have the same column index as the i-th row (i.e., c and g, e and h). Then, the PIM core performs the calculation process of updating c to ca*g and e to ea*h. The same process is repeated in the third and fourth steps. In the fifth step, the floating-point calculation unit of the PIM core in the PE is used to calculate the division and update the i-th row. The updated i-th row is written back to the storage layer via DMA and then transmitted to the remote PE via the transfer unit to perform the update calculation on the remote PE.

[0041] It should be noted that the number of components and storage capacity within the PE (e.g., 32 FPUs, 256KB index CAM, 64KB local SPM) in the above embodiments are merely examples of this invention. In other implementations for practical applications, the number of FPUs can be flexibly increased or decreased based on overall chip area constraints, power budget, and the size of the target sparse matrix, and the capacity of each level of storage components (CAM, SPM) can also be adjusted proportionally. Furthermore, in addition to using the standard CAM structure, the hardware performing index matching can also be replaced by an SRAM-based hash lookup logic unit. The above-mentioned hardware scaling and equivalent component replacement are based on specific application scenarios.

[0042] Figure 5This diagram illustrates coarse-grained asynchronous parallel processing between processing units, showing the task progress status and dependency relationships of different processing units at adjacent time points. For example... Figure 5 As shown in the timeline on the left, each processing unit independently and asynchronously performs its local computation task. Specifically, the processing unit corresponding to memory bank 0 is responsible for updating rows that depend on the computation results of rows 1 and 2. It needs to wait for row 1 to complete its computation before continuing its local row update; therefore, it is in a paused state. Meanwhile, the processing unit corresponding to memory bank 1 is executing the update computation for row 1, and the processing unit corresponding to memory bank 2 is executing the update computation for row 2. (The rest of the text is missing.) Figure 5 As shown in the next time node on the right, the processing unit of storage bank 1 has completed the update of row 1 and begins calculating row 3. The processing unit of storage bank 0 obtains the update result of row 1 from the remote end, then uses the updated row 1 to perform the update of its local row, and begins to wait for the calculation result of row 2 and pauses. At the same time, the PE corresponding to storage bank 2 is still performing the update calculation of row 2. After its update is completed, it will transmit the data to the PE corresponding to storage bank 0 and release its paused state. This coarse-grained asynchronous parallel mechanism aims to maximize the parallelism of the overall system, thereby avoiding the inefficient operation caused by overall synchronization. Through the event-driven local dependency waiting mechanism, the device allows processing units without any data dependencies to continuously and fully perform calculations. This mechanism strictly limits the calculation pause to the local processing unit where data is truly missing, thereby avoiding resource idleness and inefficient operation caused by overall synchronization, and significantly improving the actual throughput of the system.

[0043] It should be noted that the waiting mechanism in this embodiment, which triggers subsequent updates upon the arrival of a single line of computation results, is only a preferred implementation. Implementers can adopt variations of the control scheme that implement local asynchronous scheduling between processing units based on data dependency states. In other variations, if task mapping is performed at the data block level, the pause and wake-up mechanism of the processing unit can also be set to a "multi-dependency wait" mode. That is, after the processing unit gathers all the dependent lines required for a data block in its internal buffer pool, it then uniformly releases the pause state and starts the computation pipeline.

[0044] Figure 6 This diagram illustrates a fine-grained pipeline within a processing unit, demonstrating how the update operation of a sparse matrix row is broken down into multiple functional stages and executed in overlapping pipelines to improve instruction issue rate and component utilization. Indicatively, the update process for a single sparse matrix row is divided into three stages:

[0045] Phase 1 (the area indicated by the green arrow, corresponding to the data transmission phase): This phase consists of three sub-phases: fetching, waiting, and receiving. The fetching sub-phase utilizes the direct memory access unit and the transmission unit to obtain local row data and initiates a data request for dependent rows to the remote processing unit. The waiting sub-phase is the process where the local processing unit waits for the remote processing unit to complete the update of the corresponding dependent rows. The receiving sub-phase is the process where the local processing unit receives the updated data from the remote unit via the interconnection network. Due to the irregularity of the non-zero element structure of the sparse matrix and the dynamic changes in the communication state of the interconnection network, the overall duration and internal composition (i.e., the relative proportions of fetching, waiting, and receiving) of this data transmission phase are dynamically fluctuating. For example, if the remote dependent data has already been updated and cached in the interconnection network when a fetching request is initiated, this transmission phase may not include the waiting sub-phase at all.

[0046] Phase 2 (the range indicated by the blue arrow, corresponding to the lookup phase): Using the local index content addressing memory, perform non-zero column index matching between the remote row and the local row to determine the position of the corresponding element that needs to participate in the operation.

[0047] Phase 3 (the area indicated by the red arrow, the calculation phase): Utilizing the floating-point processing unit within the near-memory computing core, multiplication, accumulation, and division operations are performed based on the matched data to complete the local row update.

[0048] Regarding pipeline overlap mechanisms, such as Figure 6 As shown in ①②③, the fine-grained pipelining mechanism allows the three stages of transmission, lookup (search), and computation to overlap. Illustratively, when the previous data transmission stage ends, the next data transmission stage can begin without waiting for the subsequent lookup and computation stages to complete; when the previous data transmission stage ends and the previous lookup stage completes, the next lookup stage can begin; when the previous lookup stage ends and the previous computation stage completes, the next computation stage can begin. Through this stage overlap, multiple processing tasks can be executed in parallel within the processing unit, keeping local components such as the direct memory access unit, indexed content addressing memory, and floating-point processing unit busy as much as possible, thereby improving single-node throughput.

[0049] In addition, such as Figure 6 As indicated by the "bubble" annotation, due to the differences in the non-zero element structure of different sparse matrix rows, the time required for each stage may vary (for example, the lookup stage for some rows may be longer than the computation stage, or the transmission stage may be extended due to waiting at the remote end), resulting in bubbles (cavitation) in the pipeline. These bubbles are a natural manifestation of the inherent structural irregularities of sparse matrices in pipeline execution.

[0050] It should be noted that the "transfer, search, and computation" three-stage pipeline in this embodiment is only a preferred implementation. Implementers can adopt other modified implementation schemes based on the hardware control pipeline of the above-mentioned stage overlap and dynamic scheduling ideas. In other modified implementation schemes, the number of pipeline stages and functional division can be flexibly adjusted according to the timing constraints of the underlying hardware resources. For example, the operation of writing the update result back to memory can be separated from the computation stage to form an independent "write-back stage," constituting a four-stage pipeline. In addition, to eliminate pipeline bubbles caused by sparsity, the control unit can also introduce local out-of-order execution logic to prioritize scheduling rows with ready data to enter subsequent stages.

[0051] Figure 7 This diagram illustrates a data reuse mechanism based on hierarchical sticky note storage. The left side shows the dependencies between sparse matrix rows and the task mapping (partitioning relationship) between different storage banks, while the right side shows the caching and reuse paths of each level of sticky note storage during cross-storage data transfer.

[0052] like Figure 7 As shown on the left, the circled numbers 0 to 10 represent different rows of the sparse matrix, and the directed edges with arrows represent the data dependencies between rows. Row 8 is a common dependency of rows 9 and 10 (row 8 depends on row 9, and row 8 depends on row 10), and row 9 is also a dependency of row 10 (row 9 depends on row 10). According to the task mapping, row 8 is assigned to the processing unit corresponding to storage bank 0, and rows 9 and 10 are assigned to the processing unit corresponding to storage bank 2. Since storage bank 0 and storage bank 2 belong to different storage bank groups, the updated data for row 8 will be transferred from the storage bank group containing storage bank 0 to the storage bank group containing storage bank 2.

[0053] like Figure 7As shown on the right, when the processing unit corresponding to storage bank 2 requests data for row 8 while performing calculation for row 9, the updated data for row 8 originates from storage bank 0 and is transmitted via the Internet. In the transmission path, the data for row 8 sequentially passes through storage bank 0, the storage group node where storage bank 0 resides, the partition node, the storage group node where storage bank 2 resides, and storage bank 2. Therefore, the downlink path consists of the last three nodes: the partition node, the storage group node where storage bank 2 resides, and storage bank 2. During this process, data is written and stored level by level in the sticky notes memory corresponding to each level of the downlink path (i.e., the path from the highest-level node in the transmission path to the local processing unit), including the local SPM of the processing unit corresponding to storage bank 2, the storage group-level sticky notes memory, and the partition-level sticky notes memory. When the processing unit corresponding to storage bank 2 begins calculating row 10, it can retrieve the updated data for row 8 from the local SPM without having to retrieve it again from the original remote location where row 8 was located. If the replacement results in no updated data for row 8 in the processing unit corresponding to storage bank 2, the processing unit can also retrieve the data from the nearest storage bank group-level SPM or partition-level SPM along the uplink path, thereby shortening the transmission distance. Additionally, assuming storage bank 3 also needs the updated data for row 8, it can also retrieve it from the nearest storage bank group-level tag-based memory within the same storage bank group. Each level of SPM uses a hardware-integrated cache replacement strategy for management. When the storage space of a certain level of SPM is full, the hardware automatically selects the least recently accessed data row and evicts it according to the Least Recently Used (LRU) algorithm or a pseudo-LRU algorithm, and stores the newly written data into the released entry. This hierarchical tag-based memory reuse mechanism intercepts duplicate data accesses across processing units within the nearest lower-level network node, significantly reducing the total amount of data communication between storage banks and the average transmission distance, alleviating congestion and power consumption in higher-level links of the interconnection network, thereby improving the overall performance and energy efficiency of the architecture.

[0054] It should be noted that this embodiment uses the reuse of row 8 between rows 9 and 10 as an example, and the Least Recently Used (LRU) replacement strategy adopted by each level of the SPM is only a preferred example. Implementers can adjust the cache allocation and replacement strategy. In other modified implementations, depending on the overhead constraints of on-chip hardware resources, each level of SPM can also adopt a random replacement strategy or a first-in-first-out (FIFO) strategy. In addition, for sparse matrices with specific reuse patterns, the hardware cache management mechanism can also combine compiler hints to lock globally critical data with extremely high reuse rates in a specific level of SPM for a long period of time.

[0055] In summary, the near-in-memory computing device proposed in this invention for direct method solutions can handle complex data dependencies and optimize irregular communication. Based on the characteristics of direct method solutions, this architecture flexibly utilizes a combination of coarse and fine-grained parallel mechanisms to mask computational pauses and leverages hierarchical SPM to achieve dynamic data reuse, thereby significantly improving the overall performance and energy efficiency of the architecture in directly solving sparse linear equation systems. Specifically, for the direct method solution of sparse linear equation systems, when evaluated using several identical sparse matrix datasets from the SuiteSparse dataset, compared to techniques using traditional CPU computing baselines (such as the PARDISO solver), this invention achieves a 135.7-fold performance improvement and 697.5-fold energy saving in LU decomposition using the direct method, and a 5060.5-fold performance improvement and 10474.9-fold energy saving in trigonometric solving. Compared to GPU-based computing baseline solutions (such as the cuDSS solver), this invention achieves a 146.2-fold performance improvement and 993.2-fold energy saving in direct LU decomposition, and a 1987.0-fold performance improvement and 5434.6-fold energy saving in trigonometric solving. These improvements are due to two main factors: firstly, the near-memory computing architecture of this invention effectively overcomes the memory bandwidth bottleneck caused by lower computational intensity; and secondly, the device architecture, through customized design, alleviates inherent computational pauses and reduces data transfer overhead across processing units.

[0056] It should be noted that although the steps are described in a specific order above, it does not mean that the steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently, or even in a different order, as long as the required function can be achieved.

[0057] This invention can be a system, method, electronic device, computing device, computer program product and / or computer-readable medium.

[0058] Computer program products mainly refer to software products that implement various aspects of the present invention through computer programs, or hardware products that carry software that implements various aspects of the present invention.

[0059] Computer-readable storage media can be tangible devices that hold and store instructions for use by an instruction execution device. Computer-readable storage media can include, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof.

[0060] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A near-memory computing device, comprising a logic layer and a storage layer integrated by stacking, wherein the storage layer and the logic layer are connected by an interconnect structure to transmit data, characterized in that, The storage layer comprises multiple storage units organized in a hierarchical manner; The logic layer includes multiple processing units, each of which has a corresponding memory bank and is coupled to it through an interconnection structure. Each of the processing units is configured to asynchronously execute independent instruction sequences. Each instruction sequence corresponds to sparse matrix local data divided at the granularity of at least one line and stored in its respective corresponding memory. When the external data on which the local data calculation depends is ready, the processing unit performs calculations on the local data asynchronously. Data is transmitted between processing units through an interconnection network at the logical layer, and data is indirectly exchanged between storage units through the same interconnection network. In the interconnection network, the storage units corresponding to the hierarchical organization are deployed with hierarchical sticky notes, which are used to cache data at each level of the cross-processing unit data transmission path to support local reuse.

2. The apparatus according to claim 1, characterized in that, The device is configured as follows: Divide the sparse matrix into multiple storage banks by row; During the computation, each processing unit independently advances its local instruction sequence asynchronously, responsible for processing the sparse matrix rows allocated to its corresponding memory bank, where: When the current processing unit encounters data dependencies during the calculation of the corresponding sparse matrix rows, it initiates a remote data request to the remote processing unit corresponding to the remote storage to obtain the updated data for the required dependent rows. If the update data for the dependent row is ready, the current processing unit obtains the update data and continues the calculation process for the sparse matrix row. If the update data for the dependent row is not ready, the current processing unit enters a waiting state for the sparse matrix row until the update data arrives. After the update data is transmitted to the local machine via the Internet, the calculation process for the sparse matrix row continues.

3. The apparatus according to claim 2, characterized in that, Each processing unit is configured to perform operations based on a three-stage pipeline, through which each processing unit can perform multiple locally data-related operations in an overlapping manner. Among the three stages, the remote acquisition, waiting and receiving of update data for dependent rows are merged into a transmission stage with variable delay, the index matching operation is configured as a lookup stage, and the multiplication-accumulation and division operations are configured as a calculation stage.

4. The apparatus according to claim 2, characterized in that, The device is configured to: When the current processing unit performs a remote read operation to obtain updated data for its dependent rows, the updated data is written and cached level by level in the note-like storage on each level node of the Internet during the downlink path from the remote processing unit to the current processing unit via the Internet. The downlink path is the path from the highest-level node of the interconnected network to the current processing unit; when any processing unit initiates a read request for the same data, it searches for and returns the data from the nearest-level sticky memory along the uplink path.

5. The apparatus according to claim 1, characterized in that, The hierarchical structure of the multiple storage units in the hierarchical organization includes: channels, partitions, storage unit groups, and storage units. The multiple storage units are divided into multiple channels, each channel contains multiple partitions, each partition contains multiple storage unit groups, and each storage unit group contains several storage units. There is no interconnection structure between storage units at the storage unit layer; instead, data interaction between storage units is achieved through processing units and interconnection networks at the logic layer.

6. The apparatus according to claim 5, characterized in that, The interconnection network is an H-tree structure, the processing unit is a leaf node of the H-tree structure, multiple leaf nodes are gathered to form a storage group layer node, multiple storage group layer nodes are gathered to form a partition layer node, multiple partition layer nodes are gathered to form a channel layer node, and multiple channel layer nodes are gathered to form a root node. The sticky note storage is deployed on each of the intermediate nodes of the H-tree structure.

7. The apparatus according to claim 6, characterized in that, Each storage group layer node, partition layer node, and channel layer node is equipped with a corresponding sticky note-style storage device.

8. The apparatus according to any one of claims 1 to 7, characterized in that, The processing unit includes: a near-memory computing core, a local SPM, an indexed content-addressable memory, a direct memory access unit, and a transmission unit, wherein: The near-memory computing core includes multiple floating-point computing units for performing floating-point operations; The local SPM is used to temporarily store data from the remote processing unit; Index content-addressable memory is used to store indexes of sparse data and perform index search matching; The direct memory access unit is used by the near-memory computing core and the content-addressable memory to access data in their corresponding memory blocks; The transmission unit is connected to the Internet and is used for data interaction between the processing unit and other processing units.

9. A method for directly solving a system of sparse linear equations using a near-memory computing device as described in any one of claims 1 to 7, the method comprising: The sparse matrix to be solved is divided into multiple storage blocks by rows, and each processing unit is responsible for processing the sparse matrix rows in its corresponding storage block; Each processing unit independently advances its local instruction sequence asynchronously. When encountering data dependencies during computation, it initiates a remote data request to the remote processing unit to obtain the updated data for the required dependent rows, and performs the following operations based on the readiness status of the updated data: If the updated data is ready, obtain the updated data and continue the calculation process for the sparse matrix rows. If the updated data is not ready, the sparse matrix row enters a waiting state until the updated data is transmitted to the local machine via the Internet, and then the calculation process of the sparse matrix row continues. Within each processing unit, the remote acquisition, waiting, and reception of data that depends on rows are merged into a data transmission phase with variable latency. The index matching operation is configured as a lookup phase, and the multiplication-accumulation and division operations are configured as a computation phase. The data transmission phase, the lookup phase, and the computation phase constitute a three-stage fine-grained pipeline to execute multiple operations in an overlapping manner.

10. The method according to claim 9, characterized in that, The method includes: The updated data is written and cached level by level in the note-like storage on each level node of the Internet during the downlink path from the remote processing unit to the current processing unit via the Internet. The replacement strategy of the note-like storage on each level node is set as follows: when the storage space is full, the data row that has not been accessed for the longest time is automatically evicted. When any processing unit initiates a read request for the same data, the data is located and returned from the nearest-level note-based storage along the uplink path.

Images