Conditional storage system and method

Abstract

The invention discloses a conditional storage system and method. The system comprises a wafer-level computing engine, a network-on-chip and a distributed Engram embedded storage subsystem, the wafer-level computing engine comprises at least one core cluster, and each core cluster is provided with at least one computing core and a prefetching scheduler; the distributed Engram embedded storage subsystem comprises a plurality of physical storage blocks, and each physical storage block is mapped with a fragment of the Engram embedded table; the calculation core is used for asynchronously and concurrently calculating an embedded vector address corresponding to a time sequence position of an input sequence on a subsequent Transform layer when the current Transform layer is calculated; the prefetching scheduler is used for prefetching the embedded vector from the physical storage block asynchronously and concurrently according to the embedded vector address through the network-on-chip; according to the method, efficient and low-delay storage and access of super-large-scale Transform parameters can be realized.

Description

Technical Field

[0001] This invention relates to the field of wafer-level chip technology, and more particularly to a conditional memory system and method. Background Technology

[0002] With the explosive growth in the scale of large language models, conditional computation paradigms, represented by hybrid experts, have become the mainstream for expanding model capacity. However, the current Transformer architecture lacks native knowledge lookup primitives, forcing the model to simulate static knowledge retrieval through dynamic computation, which wastes computational resources. For example, recognizing a common multi-word entity requires multiple attention layers and feedforward network layers for progressive feature combination, essentially using expensive runtime computation to reconstruct a static lookup table.

[0003] The Engram module of the Transformer model uses conditional memory as a novel sparsity axis complementary to conditional computation. Through hashed N-gram embeddings, it achieves O(1) constant-time lookup of static knowledge patterns, thus freeing the Transformer model from the low-level static pattern reconstruction task and allowing it to focus more on high-level inference. However, efficiently integrating Engram embedding tables with billions or even tens of billions of parameters into hardware systems presents significant challenges: traditional GPUs have limited high-bandwidth memory capacity; storing parameters in external host memory leads to frequent PCIe data transfers, causing severe latency and bandwidth bottlenecks, limiting system throughput.

[0004] Wafer-level chips (such as Cerebras WSE) offer unprecedented on-chip storage and computing resources, but their architecture differs significantly from traditional GPUs. A key challenge is how to deeply integrate the deterministic and sparse lookup characteristics of args with the hardware features of wafer-level chips to achieve efficient, low-latency storage and access of ultra-large-scale Transformer parameters.

[0005] This section is intended to provide background or context for the embodiments of the invention set forth in the claims. The description herein is not an admission that it is prior art simply because it is included in this section. Summary of the Invention

[0006] This invention provides a conditional storage system for achieving efficient, low-latency storage and access of ultra-large-scale Transformer parameters. The conditional storage system includes: a wafer-level computing engine, an on-chip network, and a distributed engram embedded storage subsystem. The wafer-level computing engine includes at least one core cluster, and each core cluster is equipped with at least one computing core and a prefetch scheduler. The distributed Engram embedding storage subsystem includes multiple physical storage blocks, each of which is mapped to a shard of the Engram embedding table. The Engram embedding table stores the embedding vectors of the Transformer layer of the Transformer model. The computational core is used to: asynchronously and in parallel calculate the address of the embedding vector corresponding to the temporal position of the input sequence in the subsequent Transformer layer when calculating the current Transformer layer based on the input sequence and the embedding vector; The prefetch scheduler is used to: asynchronously and in parallel prefetch the embedded vector from the physical storage block to the corresponding computing core according to the embedded vector address via the on-chip network.

[0007] In one embodiment, each computing core is provided with a computing unit, a deterministic hash addressing unit, and a local cache; The computing unit is used to: perform Transformer model calculations based on the input sequence of the Transformer model and the corresponding embedding vectors in the local cache; The deterministic hash addressing unit is used to: asynchronously and in parallel calculate the embedding vector address corresponding to the subsequent Transformer layer of the input sequence at the current Transformer layer when the computing unit is computing the current Transformer layer; The prefetch scheduler is used to: asynchronously and in parallel prefetch the embedding vector from the physical storage block to the local cache of the corresponding computing core according to the embedding vector address via the on-chip network.

[0008] In one embodiment, the deterministic hash addressing unit is used for: Based on the compressed suffix of the input sequence temporal position, a multi-head hash function is executed in parallel to obtain the embedding vector address corresponding to the input sequence temporal position. The embedding vector address includes physical storage block coordinates and intra-block offset address.

[0009] In one embodiment, the prefetch scheduler stores an address mapping table, which is used to record the mapping relationship between multiple physical storage blocks and multiple fragments of the Enggram embedding table; The prefetch scheduler is used to: query the address mapping table based on the physical storage block coordinates in the embedded vector address to determine the physical storage block and the mapped fragment corresponding to the physical storage block coordinates; locate the single embedded vector corresponding to the fragment in the physical storage block based on the intra-block offset address in the embedded vector address; and prefetch the located embedded vector into the local cache of the corresponding computing core through the on-chip network.

[0010] In one embodiment, the mapping relationship is determined by a load balancing algorithm based on the size of the Enggram embedding table and the number, capacity, and distribution of physical storage blocks.

[0011] In one embodiment, the core cluster also includes a shared cache; The local cache stores embedding vectors whose access frequency is greater than a first access frequency threshold; The shared cache stores embedding vectors whose access frequency is no greater than a first access frequency threshold; The computing unit is used for: If the embedding vector for each Transformer layer is not found in the local cache, query the shared cache of the core cluster to read the embedding vector for each Transformer layer.

[0012] In one embodiment, the computing unit is used for: If the embedding vector corresponding to each Transformer layer is not found in the shared cache, the embedding vector is read from the physical storage block according to the embedding vector address.

[0013] In one embodiment, a first number of shards of the Engram embedding table are mapped in the physical memory of the distributed Engram embedding storage subsystem, and a second number of shards are mapped to an external storage pool. The prefetch scheduler is used for: If the embedding vector corresponding to each Transformer layer is not found in the physical storage block, the embedding vector is prefetched from the external storage pool according to the embedding vector address.

[0014] This invention also provides a conditional storage method for achieving efficient, low-latency storage and access of ultra-large-scale Transformer parameters. The conditional storage method includes: When the computational core computes the current Transformer layer of the Transformer model based on the input sequence and the embedding vector, it asynchronously and in parallel computes the embedding vector address corresponding to the temporal position of the input sequence in the subsequent Transformer layer. The embedding vector of the Transformer layer is stored in the Engram embedding table, which is divided into multiple shards when the Transformer model is loaded. Each shard is mapped to a physical storage block of the distributed Engram embedding storage subsystem. The prefetch scheduler uses the on-chip network to asynchronously and in parallel prefetch the embedded vector from the physical memory block to the corresponding computing core according to the embedded vector address.

[0015] This invention also provides a wafer-level chip, including the above-described conditional memory system.

[0016] This invention also provides a board card including the above-mentioned chip.

[0017] This invention also provides an electronic device including the aforementioned circuit board.

[0018] The conditional storage system and method provided in this invention combine a wafer-level computing engine, an on-chip network, and a distributed engram embedding storage subsystem. This allows all embedding vectors required by the Transformer model to be deployed in a distributed manner across the physical storage blocks of the wafer-level chip, achieving on-chip distributed storage of the engram embedding table. This improves the access bandwidth of the embedding vectors and reduces access latency. By using computing cores and a prefetch scheduler on the core cluster, asynchronous parallel execution of address calculation, data prefetching, and model calculation is achieved. This allows deterministic hash addressing units to asynchronously and parallelly calculate the embedding vector addresses required by subsequent Transformer layers while the current Transformer layer is being calculated, avoiding address calculation from occupying the critical path of model calculation and improving the overall system processing efficiency. Through the collaborative work of the prefetch scheduler and the on-chip network, the corresponding embedding vector can be asynchronously and parallelly prefetched from the physical storage blocks of the distributed engram embedding storage subsystem to the corresponding computing core based on the calculated embedding vector address. This allows the embedding vector transmission process to be executed in parallel with the current Transformer layer calculation process, eliminating latency bottlenecks caused by data waiting and improving the resource utilization of the wafer-level computing engine. A globally unified engram embedding table is used to provide embedding vectors for all Transformer layers in the Transformer model, achieving global sharing of embedding vector resources, reducing hardware resource redundancy, simplifying the system architecture, and ensuring consistency in feature acquisition across layers. By setting up local caching within the computation core, the prefetched embedding vectors are stored locally on the computation core. The computation core can then read the required embedding vectors to perform Transformer model computation, further reducing memory access latency and improving the real-time performance and response speed of model computation. The system architecture is highly compatible with the wafer-level chip hardware structure. The combination of distributed storage, parallel addressing, asynchronous prefetching, and local caching efficiently supports the storage and access of large-scale engram embedding tables, improving the system's throughput and scalability when processing Transformer models. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings: Figure 1 This is a schematic diagram of a conditional storage system in an embodiment of the present invention; Figure 2 This is a schematic diagram of the computing core in an embodiment of the present invention; Figure 3 This is another schematic diagram of a conditional storage system in an embodiment of the present invention; Figure 4 This is a flowchart of the conditional storage method in an embodiment of the present invention; Figure 5 This is a flowchart of the asynchronous parallel prefetching method in an embodiment of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. Here, the illustrative embodiments and descriptions of the present invention are used to explain the present invention, but are not intended to limit the present invention. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this application can be arbitrarily combined with each other. The acquisition, storage, use, and processing of data in the technical solutions of this application all comply with relevant laws and regulations. The user information in the embodiments of the present invention is obtained through legal and compliant means, and the acquisition, storage, use, and processing of user information have been authorized and agreed upon by the customer.

[0021] To facilitate understanding of the technical solution provided in this application, the relevant content of the technical solution in this application will be explained below.

[0022] Existing CPU-based engram implementations suffer from limited PCIe bandwidth and high off-chip memory access latency, making the retrieval of billions of embedding vectors from host DRAM a system bottleneck. Even prefetching techniques cannot completely mask this latency. More critically, the lack of dedicated hardware support for sparse, irregular data access patterns in CPUs results in low cache hit rates when processing deterministic but highly distributed lookup requests for engram modules. A significant amount of time is wasted on memory bus contention and data movement rather than actual computation. Therefore, while CPU-based engram implementations are functionally feasible, their efficiency in large-scale deployments falls far short of the high-throughput, low-latency inference requirements of modern large language models. This hardware mismatch severely restricts the full potential of engram modules.

[0023] To address the aforementioned issues, this application proposes a conditional memory system. This system leverages the deterministic nature of engram modules and predictable access patterns based on input, combined with a massive distributed engram embedded memory subsystem on a wafer-level chip and a high-bandwidth, low-latency on-chip network (NoC), resulting in a hierarchical, prefetch-driven memory architecture. Through hardware and software co-design, this architecture efficiently maps a massive engram embedding table onto the wafer-level chip, achieving near-O(1) latency knowledge retrieval while minimizing communication overhead.

[0024] This invention provides a conditional storage system. For example... Figure 1 This is a schematic diagram of a conditional storage system in an embodiment of the present invention. The conditional storage includes: a wafer-level computing engine, an on-chip network, and a distributed engram embedded storage subsystem. The wafer-level computing engine includes at least one core cluster, and each core cluster is equipped with at least one computing core and a prefetch scheduler. The distributed Engram embedding storage subsystem includes multiple physical storage blocks, each of which is mapped to a shard of the Engram embedding table. The Engram embedding table stores the embedding vectors of the Transformer layer of the Transformer model. The computational core is used to: asynchronously and in parallel calculate the address of the embedding vector corresponding to the temporal position of the input sequence in the subsequent Transformer layer when calculating the current Transformer layer based on the input sequence and the embedding vector; The prefetch scheduler is used to: asynchronously and in parallel prefetch the embedded vector from the physical storage block to the corresponding computing core according to the embedded vector address via the on-chip network.

[0025] Depend on Figure 1It is evident that by combining the wafer-level computing engine, on-chip network, and distributed engram embedding storage subsystem, all embedding vectors required by the Transformer model are deployed in a distributed manner within the physical storage blocks of the wafer-level chip. This achieves on-chip distributed storage of the engram embedding table, improving the access bandwidth of the embedding vectors and reducing access latency. Through the computing cores and a prefetch scheduler set up on the core cluster, asynchronous parallel execution of address calculation, data prefetching, and model calculation is achieved. This allows the deterministic hash addressing unit to asynchronously and parallelly calculate the embedding vector addresses required by subsequent Transformer layers while executing the current Transformer layer's calculations, avoiding address calculations occupying the critical path of model calculations and improving the overall system processing efficiency. Through the collaborative work of the prefetch scheduler and the on-chip network, the corresponding embedding vector can be asynchronously and parallelly prefetched from the physical storage blocks of the distributed engram embedding storage subsystem to the corresponding computing core based on the calculated embedding vector address. This allows the embedding vector transmission process to be executed in parallel with the current Transformer layer's calculation process, eliminating latency bottlenecks caused by data waiting and improving the resource utilization of the wafer-level computing engine. A globally unified engram embedding table is used to provide embedding vectors for all Transformer layers in the Transformer model, achieving global sharing of embedding vector resources, reducing hardware resource redundancy, simplifying system architecture, and ensuring consistency in feature acquisition across layers. By setting up local caching in the computation core, the prefetched embedding vectors are stored locally on the computation core. The computation core can read the required embedding vectors to perform Transformer model computation, further reducing memory access latency and improving the real-time performance and response speed of model computation. The system architecture is highly compatible with the wafer-level chip hardware structure. The combination of distributed storage, parallel addressing, asynchronous prefetching, and local caching can efficiently support the storage and access of large-scale engram embedding tables, improving the system's throughput and scalability when processing Transformer models.

[0026] In this embodiment of the invention, the distributed Engram embedding storage subsystem does not rely on off-chip memory and is designed for high-concurrency, high-bandwidth AI inference / training scenarios, especially for Transformer-type models. The massive Engram embedding table (e.g., 100B parameters) is horizontally divided into multiple segments. These segments are not centrally stored but are distributed and mapped to multiple physical storage blocks. These physical storage blocks can be SRAM blocks, and during mapping, they can be mapped to SRAM blocks adjacent to the computing core or dedicated to them. Each physical storage block is responsible for storing a portion of the embedding vectors. A prefetch scheduler uniformly manages the prefetch timing, routing, and cache scheduling of the embedding vectors.

[0027] The Engram embedding table is an embedding vector library organized by Transformer layers, including positional embeddings, word embeddings, and layer embeddings. To support wafer-level parallel access, the entire embedding table is divided into multiple shards, each mapped to a different physical storage block. This distributed storage structure improves bandwidth, reduces access conflicts, and supports parallel addressing and parallel reading. When a computation unit performs a Transformer computation at a certain layer, it directly reads the prefetched corresponding embedding vector from its local cache, without waiting for external storage access, achieving conditional "ready-to-use" retrieval.

[0028] Figure 2 This is a schematic diagram of a computing core in an embodiment of the present invention. In one embodiment, each computing core is provided with a computing unit, a deterministic hash addressing unit, and a local cache. The computing unit is used to: perform Transformer model calculations based on the input sequence of the Transformer model and the corresponding embedding vectors in the local cache; The deterministic hash addressing unit is used to: asynchronously and in parallel calculate the embedding vector address corresponding to the subsequent Transformer layer of the input sequence at the current Transformer layer when the computing unit is computing the current Transformer layer; The prefetch scheduler is used to: asynchronously and in parallel prefetch the embedding vector from the physical storage block to the local cache of the corresponding computing core according to the embedding vector address via the on-chip network.

[0029] Computation unit: performs Transformer forward / reverse computation; Deterministic hash addressing unit: specifically designed to calculate the addresses of future embedded vectors in advance, in parallel, and deterministically; Local cache: stores embedded vectors that will be used soon in the nearest location, reducing memory access latency.

[0030] In one embodiment, the deterministic hash addressing unit is used for: Based on the compressed suffix of the input sequence temporal position, a multi-head hash function is executed in parallel to obtain the embedding vector address corresponding to the input sequence temporal position. The embedding vector address includes physical storage block coordinates and intra-block offset address.

[0031] In this embodiment of the invention, the deterministic hash addressing unit executes a multi-head hash function in parallel based on the compressed suffix N-gram g_{t,n} of the current processing time position of the received input sequence. `_{n,k}` directly generates the physical storage block coordinates and intra-block offset address of the embedding vector required to generate the current processing temporal position of the input sequence. Since the lookup key of the engram is entirely determined by the input sequence, this addressing process can be completed before the computation of this layer begins. Taking the compressed suffix N-gram `g_{t,n}` of the temporal position reduces the hash input bit width and lowers hardware overhead. Parallel computation using a multi-head hash function allows for the simultaneous generation of addresses for multiple future layers, improving prefetch throughput. The final output address is a two-level structured address: 1) Physical storage block coordinates: Identifies the physical location of the embedded vector, such as the storage block, storage bank, or wafer region; 2) Intra-block offset address: The starting address of the specific vector within the segment.

[0032] The above addressing method guarantees parallelism, no conflict, low latency, and hardware-based implementation.

[0033] In one embodiment, the prefetch scheduler stores an address mapping table, which is used to record the mapping relationship between multiple physical storage blocks and multiple fragments of the Enggram embedding table; The prefetch scheduler is used to: query the address mapping table based on the physical storage block coordinates in the embedded vector address to determine the physical storage block and the mapped fragment corresponding to the physical storage block coordinates; locate the single embedded vector corresponding to the fragment in the physical storage block based on the intra-block offset address in the embedded vector address; and prefetch the located embedded vector into the local cache of the corresponding computing core through the on-chip network.

[0034] In this embodiment of the invention, the on-chip network (NoC) and the prefetch scheduler are key components of the system. Based on the pre-generated deterministic embedding vector addresses, a centralized or distributed prefetch scheduler is activated. It analyzes the embedding vector addresses that will be accessed by multiple computing cores and, through the high-bandwidth, low-latency on-chip network (NoC) at the wafer level, asynchronously and in parallel prefetches the embedding vectors required for subsequent Transformer layers (such as layer L+1, layer L+2, etc.) from distributed SRAM storage blocks into the local memory of the corresponding computing core while the computing unit executes the current Transformer layer (e.g., layer L). This completely overlaps communication (embedded vector transfer) with computation (pre-order layer forward propagation).

[0035] The global mapping table records: which embedded table shard → which physical storage block it is stored in. The mapping table is configurable and updatable, supporting dynamic deployment of Transformer models of different sizes. The prefetch scheduler achieves precise granular prefetching, only fetching the necessary vectors and avoiding bandwidth waste.

[0036] In one embodiment, the mapping relationship is determined by a load balancing algorithm based on the size of the Enggram embedding table and the number, capacity, and distribution of physical storage blocks.

[0037] By using load balancing algorithms (such as hash sharding, round-robin, greedy load balancing, and consistent hashing), the access traffic to each storage block is evenly distributed, hotspot shards are not concentrated, on-chip network link congestion is minimized, and the system can be efficiently scaled under different model sizes.

[0038] like Figure 3 This is another schematic diagram of a conditional storage system in an embodiment of the present invention. In one embodiment, the core cluster is further provided with a shared cache. The local cache stores embedding vectors whose access frequency is greater than a first access frequency threshold; The shared cache stores embedding vectors whose access frequency is no greater than a first access frequency threshold; The computing unit is used for: If the embedding vector for each Transformer layer is not found in the local cache, query the shared cache of the core cluster to read the embedding vector for each Transformer layer.

[0039] In this embodiment of the invention, a hierarchical caching structure is implemented, leveraging the Zipfian distribution characteristics of engram access (a few high-frequency patterns occupy the majority of access). A local cache is set up locally on each computing core, and a shared cache is set up in the shared storage of the core cluster to cache high-frequency compressed suffix N-gram embedding vectors. The prefetch scheduler prioritizes searching in the local cache, and only searches in the shared cache if a cache miss is found. The local cache is private to each computing core and stores high-frequency access embedding vectors; the shared cache is shared by multiple computing cores within the core cluster and stores medium-frequency / low-frequency embedding vectors. By dividing access frequency into thresholds, hot data is brought closer to the cache, while cold data is accessed more efficiently.

[0040] In one embodiment, the computing unit is used for: If the embedding vector corresponding to each Transformer layer is not found in the shared cache, the embedding vector is read from the physical storage block according to the embedding vector address.

[0041] When both the local cache and shared cache are missed, the computing unit directly accesses the physical storage block of the distributed Engram storage subsystem to read data based on the embedding vector address obtained from the deterministic hash. This forms a complete multi-level storage access link, ensuring correct data retrieval under all circumstances. In this embodiment of the invention, remote access across the on-chip network is only triggered when the shared cache is missed; most accesses are satisfied by the local cache or shared cache, greatly reducing average access latency and NoC traffic.

[0042] In one embodiment, a first number of shards of the Engram embedding table are mapped in the physical memory of the distributed Engram embedding storage subsystem, and a second number of shards are mapped to an external storage pool. The prefetch scheduler is used for: If the embedding vector corresponding to each Transformer layer is not found in the physical storage block, the embedding vector is prefetched from the external storage pool according to the embedding vector address.

[0043] The system proposed in this invention supports a two-tier storage architecture: one part of the shards are stored in on-chip / wafer physical storage (low latency, high bandwidth); the other part is stored in an external storage pool (such as HBM, DDR, or storage clusters, large capacity). The prefetch scheduler uniformly schedules access to internal and external storage, transparent to the computing unit, enabling conditional, seamless, and hierarchical data provisioning.

[0044] Large-scale models can have their capacity expanded through external storage pools.

[0045] This invention provides a conditional storage method applied to the aforementioned conditional storage system. This conditional storage method is based on the same inventive concept as the conditional storage system in one embodiment of this invention, and the principle of solving the problem is similar. Therefore, the implementation of the conditional storage method is the same as that of the conditional storage system in one embodiment of this invention, and repeated details will not be described again. The terms "unit" or "module" used below can refer to a combination of software and / or hardware that implements a predetermined function. Although the system described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0046] Figure 4 This is a flowchart of the conditional storage method in an embodiment of the present invention, such as... Figure 4 As shown, the method includes: Step 401: When the computation core calculates the current Transformer layer of the Transformer model based on the input sequence and the embedding vector, it asynchronously and in parallel calculates the embedding vector address corresponding to the temporal position of the input sequence in the subsequent Transformer layer; wherein, the embedding vector of the Transformer layer is stored in the Engram embedding table, which is divided into multiple shards when the Transformer model is loaded, and each shard is mapped to the physical storage block of the distributed Engram embedding storage subsystem; Step 402: The prefetch scheduler uses the on-chip network to asynchronously and in parallel prefetch the embedded vector from the physical storage block to the corresponding computing core according to the embedded vector address.

[0047] Figure 5 This is a flowchart of an asynchronous parallel prefetching method according to an embodiment of the present invention. In one embodiment, the method further includes: Step 501: The computation unit performs Transformer model computation based on the input sequence of the Transformer model and the corresponding embedding vector in the local cache; Step 502: While the computation unit is computing the current Transformer layer, the deterministic hash addressing unit asynchronously and in parallel computes the embedding vector address corresponding to the subsequent Transformer layer of the input sequence at the current Transformer layer. Step 503: The prefetch scheduler uses the on-chip network to asynchronously and in parallel prefetch the embedding vector from the physical storage block to the local cache of the corresponding computing core according to the embedding vector address.

[0048] In one embodiment, the mapping relationship is determined by a load balancing algorithm based on the size of the Enggram embedding table and the number, capacity, and distribution of physical storage blocks.

[0049] When the software executes the Transformer model loading at the wafer-level chip, based on the size of the Engram embedding table and the number, capacity and distribution of physical storage blocks, a load balancing algorithm determines that the Engram embedding table is divided into multiple shards, and the multiple shards are mapped to multiple physical storage blocks of the distributed Engram embedding storage subsystem. The address mapping table is recorded in the prefetch scheduler.

[0050] In one embodiment, the deterministic hash addressing unit executes a multi-head hash function in parallel based on the compressed suffix of the temporal position of the input sequence to obtain the embedding vector address corresponding to the temporal position of the input sequence. The embedding vector address includes physical storage block coordinates and intra-block offset address.

[0051] During the inference process of the Transformer model, for each Transformer layer (e.g., the Lth layer) of the Engram module that is about to execute the Transformer model: when processing the temporal position of the corresponding L-1th layer in the input sequence, the corresponding embedding vector address (physical storage block coordinates + intra-block offset address) of each temporal position of the Lth layer in the input sequence is calculated in parallel using the deterministic hash addressing unit, and an address request is formed.

[0052] In one embodiment, the prefetch scheduler stores an address mapping table, which is used to record the mapping relationship between multiple physical storage blocks and multiple fragments of the Enggram embedding table; The prefetch scheduler queries the address mapping table based on the physical storage block coordinates in the embedded vector address to determine the physical storage block and the mapped fragment corresponding to the physical storage block coordinates. Based on the intra-block offset address in the embedded vector address, it locates the single embedded vector corresponding to the fragment within the physical storage block. Through the on-chip network, the located embedded vector is prefetched into the local cache of the corresponding computing core.

[0053] After receiving an address request, the prefetch scheduler locates the corresponding single embedding vector in the fragment, forms all address requests into a data request packet, and uses the on-chip network to initiate an asynchronous embedding vector prefetch operation while the system is performing L-1 layer computation, transferring the embedding vector from the physical storage block to the local cache of the computation core that issued the address request.

[0054] When the computation of layer L begins, the required embedding vectors are already in the local cache. The computation core directly reads the embedding vectors from the local cache and performs context-aware gating and lightweight convolution fusion with the current hidden state to complete the computation of layer L. The result is then injected into the Transformer model through residual connections.

[0055] In one embodiment, the core cluster also includes a shared cache; The local cache stores embedding vectors whose access frequency is greater than a first access frequency threshold; The shared cache stores embedding vectors whose access frequency is no greater than a first access frequency threshold; The method further includes: If a computing unit does not find the embedding vector corresponding to each Transformer layer in its local cache, it queries the shared cache of its core cluster and reads the embedding vector corresponding to each Transformer layer.

[0056] In this embodiment of the invention, the local cache and shared cache can be updated according to the access frequency, and newly acquired high-frequency embedding vectors can be kept in a faster location.

[0057] In one embodiment, the method further includes: If the computing unit does not find the embedding vector corresponding to each Transformer layer in the shared cache, it reads the embedding vector from the physical storage block according to the embedding vector address.

[0058] In one embodiment, a first number of shards of the Engram embedding table are mapped in the physical memory of the distributed Engram embedding storage subsystem, and a second number of shards are mapped to an external storage pool. If the prefetch scheduler does not find the embedding vector corresponding to each Transformer layer in the physical storage block, it prefetches the embedding vector from the external storage pool according to the embedding vector address.

[0059] The following is a specific embodiment to illustrate the specific application of the system and method proposed in this invention.

[0060] 1. Obtaining hardware parameters for wafer-level chips

[0061] Taking a wafer-level chip with a similar architecture to Cerebras WSE-3 as an example, this wafer-level chip has 900,000 AI cores, 44GB of on-chip SRAM, and offers three different off-chip memory capacity configurations of 1.5TB, 12TB, or 1.2PB.

[0062] 2. Obtain model parameters: Obtain the storage requirements of the Engram module of the Transformer model to be deployed, such as 40GB.

[0063] 3. Fragmented Mapping

[0064] During the Transformer loading phase, the logical engram embedding table is divided into millions of identical 1MB fragments. Based on the size of the engram embedding table, as well as the number, capacity, and distribution of physical storage blocks, these fragments are evenly distributed across approximately 900,000 SRAM blocks using a load balancing algorithm. Each SRAM block stores an average of approximately 4KB of embedding vectors, far below its SRAM capacity limit (typically above 120KB). The mapping relationships are recorded in an address mapping table and stored in the prefetch scheduler.

[0065] 4. Deterministic address resolution and prefetch instruction generation

[0066] During the Transformer model compilation phase, for each Transformer layer (e.g., layer 2 and layer 18) where an Engram module is inserted, the hash function of that Transformer layer is analyzed and solidified in advance. At this point, instead of the traditional "load-compute" instruction, a special two-stage instruction is generated for `_{n,k}`. The first step is the prefetch instruction: This prefetch instruction contains the logic of the hash function. When the computation unit begins executing layer 0 computation, this prefetch instruction is simultaneously issued to the deterministic hash addressing units of all computation cores. Each deterministic hash addressing unit, for the input sequence `Token` it is processing, computes the target embedding vector address of its required embedding vector on-site and generates the first address request. The second step is the computation instruction: This computation instruction is actually executed at layer 2.

[0067] 5. Overlapping Data Prefetching and Routing: When a prefetch instruction is executed within a Layer 1 computation cycle, the prefetch scheduler queries the address mapping table based on the physical memory block coordinates in the embedded vector address to determine the physical memory block and mapped fragment corresponding to the physical memory block coordinates. Based on the intra-block offset address in the embedded vector address, it locates the corresponding single embedded vector within the fragment in the physical memory block. The located embedded vector is prefetched into the local cache of the corresponding computation core via the on-chip network. The on-chip network's router has awareness capabilities. During prefetching, all address requests are generated into data request packets. Then, the on-chip network efficiently routes the data request packets to the correct SRAM memory block using an all-to-all communication mode. Upon receiving the data request packet, the SRAM memory block reads the corresponding embedded vector, encapsulates it into a data response packet, and sends it back to the address request sender via the on-chip network. The entire process is completed autonomously by the network hardware, without CPU core intervention, and is completely parallel in time to the Layer 1 computation.

[0068] 6. Local Computation Fusion: When Layer 1 computation is complete, all (or most) of the embedding vectors required for Layer 2 have been delivered via the on-chip network and stored in the computation core's local memory. At this point, the "computation instructions" for Layer 2 are activated. The computation core directly reads the embedding vectors from local memory and performs low-latency gating and convolution fusion operations with the hidden state of the current layer. Since the embedding vectors are readily available, this computation is as fast as performing a regular matrix multiplication.

[0069] 7. Handling "misses" and near-memory expansion.

[0070] If the engram size exceeds the SRAM block capacity limit (e.g., 200GB cache requirement > 40GB SRAM memory requirement), an "on-chip caching" strategy is employed. Only the most frequently used embedding vectors (approximately 10%–20%) (approximately 20GB–40GB) reside in the SRAM block, covering >80% of access requests. Requests for less frequently used embedding vectors are marked as "missed" by the SRAM block and redirected to an external storage pool accessed via an optical interconnect. For these external accesses, the prefetch scheduler issues prefetch requests earlier (e.g., at level 0) to mask the longer off-chip latency. Through meticulous pipeline design, even the latency of off-chip accesses can be partially hidden.

[0071] In summary, the system and method proposed in this invention combine a wafer-level computing engine, an on-chip network, and a distributed engram embedding storage subsystem. This allows all embedding vectors required by the Transformer model to be deployed in a distributed manner within the physical storage blocks of the wafer-level chip, achieving on-chip distributed storage of the engram embedding table. This improves the access bandwidth of the embedding vectors and reduces access latency. By using computing cores and a prefetch scheduler on the core cluster, asynchronous parallel execution of address calculation, data prefetching, and model calculation is achieved. This enables deterministic hash addressing units to asynchronously and parallelly calculate the embedding vector addresses required by subsequent Transformer layers while the current Transformer layer is being calculated, avoiding address calculation from occupying the critical path of model calculation and improving the overall system processing efficiency. Through the collaborative work of the prefetch scheduler and the on-chip network, the corresponding embedding vector can be asynchronously and parallelly prefetched from the physical storage blocks of the distributed engram embedding storage subsystem to the corresponding computing core based on the calculated embedding vector address. This allows the embedding vector transmission process to be executed in parallel with the current Transformer layer calculation process, eliminating latency bottlenecks caused by data waiting and improving the resource utilization of the wafer-level computing engine. A globally unified engram embedding table is used to provide embedding vectors for all Transformer layers in the Transformer model, achieving global sharing of embedding vector resources, reducing hardware resource redundancy, simplifying the system architecture, and ensuring consistency in feature acquisition across layers. By setting up local caching within the computation core, the prefetched embedding vectors are stored locally on the computation core. The computation core can then read the required embedding vectors to perform Transformer model computation, further reducing memory access latency and improving the real-time performance and response speed of model computation. The system architecture is highly compatible with the wafer-level chip hardware structure. The combination of distributed storage, parallel addressing, asynchronous prefetching, and local caching efficiently supports the storage and access of large-scale engram embedding tables, improving the system's throughput and scalability when processing Transformer models.

[0072] This invention also provides a wafer-level chip, including the above-described conditional memory system. Therefore, the wafer-level chip can achieve the same technical effects as the conditional memory system, which will not be described in detail here.

[0073] This application also provides a board that includes the aforementioned wafer-level chip. Therefore, the board can achieve the same technical effects as the wafer-level chip, which will not be described in detail here.

[0074] This invention also provides an electronic device, which includes the aforementioned circuit board. The electronic device provided by this invention, since it includes the aforementioned circuit board, can achieve the same technical effects as the circuit board described above, and will not be described again here.

[0075] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0076] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0077] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0078] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0079] In the description of this specification, the references to terms such as "an embodiment," "a specific embodiment," "some embodiments," "for example," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0080] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A conditional storage system, characterized in that, include: Wafer-level computing engine, on-chip network, and distributed engram embedded storage subsystem; The wafer-level computing engine includes at least one core cluster, and each core cluster is equipped with at least one computing core and a prefetch scheduler. The distributed Engram embedding storage subsystem includes multiple physical storage blocks, each of which is mapped to a shard of the Engram embedding table. The Engram embedding table stores the embedding vectors of the Transformer layer of the Transformer model. The computational core is used to: asynchronously and in parallel calculate the address of the embedding vector corresponding to the temporal position of the input sequence in the subsequent Transformer layer when calculating the current Transformer layer based on the input sequence and the embedding vector; The prefetch scheduler is used to: asynchronously and in parallel prefetch the embedded vector from the physical storage block to the corresponding computing core according to the embedded vector address via the on-chip network.

2. The system according to claim 1, characterized in that, Each computing core contains a computing unit, a deterministic hash addressing unit, and a local cache; The computing unit is used to: perform Transformer model calculations based on the input sequence of the Transformer model and the corresponding embedding vectors in the local cache; The deterministic hash addressing unit is used to: asynchronously and in parallel calculate the embedding vector address corresponding to the subsequent Transformer layer of the input sequence at the current Transformer layer when the computing unit is computing the current Transformer layer; The prefetch scheduler is used to: asynchronously and in parallel prefetch the embedding vector from the physical storage block to the local cache of the corresponding computing core according to the embedding vector address via the on-chip network.

3. The system according to claim 2, characterized in that, The deterministic hash addressing unit is used for: Based on the compressed suffix of the input sequence temporal position, a multi-head hash function is executed in parallel to obtain the embedding vector address corresponding to the input sequence temporal position. The embedding vector address includes physical storage block coordinates and intra-block offset address.

4. The system according to claim 3, characterized in that, The prefetch scheduler stores an address mapping table, which is used to record the mapping relationship between multiple physical storage blocks and multiple fragments of the Enggram embedding table; The prefetch scheduler is used to: query the address mapping table according to the physical storage block coordinates in the embedding vector address, determine the physical storage block and the mapped fragment corresponding to the physical storage block coordinates, and locate the corresponding single embedding vector in the fragment within the physical storage block according to the intra-block offset address in the embedding vector address. The on-chip network prefetches the located embedding vectors into the local cache of the corresponding computing core.

5. The system according to claim 4, characterized in that, The mapping relationship is determined by a load balancing algorithm based on the size of the Enggram embedding table, as well as the number, capacity, and distribution of physical storage blocks.

6. The system according to claim 2, characterized in that, The core cluster also has a shared cache; The local cache stores embedding vectors whose access frequency is greater than a first access frequency threshold; The shared cache stores embedding vectors whose access frequency is no greater than a first access frequency threshold; The computing unit is used for: If the embedding vector for each Transformer layer is not found in the local cache, query the shared cache of the core cluster to read the embedding vector for each Transformer layer.

7. The system according to claim 6, characterized in that, The computing unit is used for: If the embedding vector corresponding to each Transformer layer is not found in the shared cache, the embedding vector is read from the physical storage block according to the embedding vector address.

8. The system according to claim 1, characterized in that, The first number of shards of the Enggram embedding table are mapped in the physical memory of the distributed Enggram embedding storage subsystem, and the second number of shards are mapped to an external storage pool. The prefetch scheduler is used for: If the embedding vector corresponding to each Transformer layer is not found in the physical storage block, the embedding vector is prefetched from the external storage pool according to the embedding vector address.

9. A conditional storage method, characterized in that, include: When the computational core computes the current Transformer layer of the Transformer model based on the input sequence and the embedding vector, it asynchronously and in parallel computes the embedding vector address corresponding to the temporal position of the input sequence in the subsequent Transformer layer. The embedding vector of the Transformer layer is stored in the Engram embedding table, which is divided into multiple shards when the Transformer model is loaded. Each shard is mapped to a physical storage block of the distributed Engram embedding storage subsystem. The prefetch scheduler uses the on-chip network to asynchronously and in parallel prefetch the embedded vector from the physical memory block to the corresponding computing core according to the embedded vector address.

10. A wafer-level chip, characterized in that, The conditional storage system includes any one of claims 1 to 8.

11. A circuit board, characterized in that, Includes the chip described in claim 10.

12. An electronic device, characterized in that, Includes the board as described in claim 11.

Images