Method, device and storage medium for kv cache loading across streaming multiprocessors

Abstract

The application provides a KV cache loading method and device across stream multiprocessors and a storage medium. The method comprises: in response to an access request of a first SM for a target KV block, querying a directory structure to obtain storage location information of the target KV block; in response to the directory structure indicating that the target KV block has been loaded to an on-chip cache by a second SM, obtaining the target KV block from the on-chip cache of the second SM; and in response to the directory structure indicating that the target KV block has not been loaded, loading the target KV block from an off-chip memory to the on-chip cache of the first SM, and registering location information of the target KV block in the directory structure.

Description

Technical Field

[0001] This application generally relates to the field of GPU architecture and large model inference acceleration technology, and specifically to a key-value (KV) cache loading method, apparatus and storage medium for Streaming Multiprocessor (SM) in the process of Transformer-class large model inference. Background Technology

[0002] In recent years, large-scale language models (LLMs) based on the Transformer architecture have achieved great success in fields such as natural language processing. The inference process of these models is generally divided into a prefill phase and a decoding phase. In the decoding phase, the model generates tokens one by one in an autoregressive manner. Each generation requires the use of the key and value tensors of all previous historical tokens, i.e., a KV cache, to avoid redundant calculations. Summary of the Invention

[0003] The embodiments of this application provide a KV cache loading method, apparatus, and storage medium for cross-stream multiprocessors, aiming to solve the problems of repeated KV cache loading, high off-chip bandwidth pressure, and limited inference performance caused by the lack of explicit coordination mechanisms in the prior art. The technical solution of this application is not only applicable to GPU architecture, but also to other types of processor architectures such as CPU, NPU, and TPU, as well as customized chip designs such as FPGA and ASIC.

[0004] According to one aspect of this application, a key-value (KV) cache loading method for a cross-stream multiprocessor (SM) is provided, comprising: in response to an access request for a target KV block by a first SM, querying a directory structure to obtain storage location information of the target KV block; in response to a directory structure indicating that the target KV block has been loaded into an on-chip cache by a second SM, retrieving the target KV block from the on-chip cache of the second SM; and in response to a directory structure indicating that the target KV block has not been loaded, loading the target KV block from off-chip memory into the on-chip cache of the first SM, and registering the location information of the target KV block in the directory structure.

[0005] According to another aspect of this application, a computing device is provided, comprising: one or more streaming multiprocessors (SMs); a directory structure configured to maintain location information, reference counts, and lifecycle states of key-value blocks; and a memory coupled to the one or more SMs and storing instructions that, when executed by the one or more SMs, cause the device to perform the methods described above. In alternative embodiments, the device may also be a central processing unit (CPU), a neural network processor (NPU), a tensor processor (TPU), a digital signal processor (DSP), or other types of processors.

[0006] According to another aspect of this application, a computer-readable storage medium is provided that stores instructions which, when executed by one or more processors, cause the one or more processors to perform the method described above. Attached Figure Description

[0007] Figure 1 This is a schematic diagram of the architecture of a cross-SM KV cache loading system according to an embodiment of this application.

[0008] Figure 2 This is a flowchart illustrating a cross-SM KV cache loading method according to an embodiment of this application.

[0009] Figure 3 This is a schematic diagram of a computing device according to an embodiment of this application. Detailed Implementation

[0010] The features and exemplary embodiments of various aspects of the present invention will now be described in detail. Numerous specific details are set forth in the following detailed description in order to provide a thorough understanding of the invention. However, it will be apparent to those skilled in the art that the invention may be practiced without requiring some of these specific details. The following description of embodiments is merely intended to provide a better understanding of the invention by illustrating examples of the invention. The invention is by no means limited to any specific configurations and algorithms presented below, but covers any modifications, substitutions, and improvements to elements, components, and algorithms without departing from the spirit of the invention. Well-known structures and techniques are not shown in the drawings and the following description in order to avoid unnecessarily obscuring the invention.

[0011] As the scale and context window of large-scale language models based on the Transformer architecture continue to increase, the amount of data in the key-value cache becomes enormous, requiring storage in off-chip memory (such as high-bandwidth memory, HBM) with large capacity but high bandwidth and access latency. On AI-accelerated chips such as GPUs, the computation in the decoding stage is executed in parallel by multiple memory managers (SMs). Each SM needs to frequently access the same key-value cache data consisting of historical tokens when processing a decoding request.

[0012] In existing technologies, data is typically cached passively using the hardware's own cache hierarchy (such as L2 cache). When a Storage Manager (SM) accesses a key-value (KV) block, if it is not found in the L2 cache, it will be loaded from the Hardware Memory Block (HBM). Since multiple SMs concurrently access the same historical KV data, this passive caching mechanism can lead to the same KV block being repeatedly loaded from the HBM into the L2 cache or even locally by different SMs, resulting in significant waste of off-chip bandwidth, increased average access latency, and increased overall system power consumption. Furthermore, current GPU architectures lack explicit cross-SM collaborative access mechanisms for KV caches, which have highly read-only and highly reusable characteristics.

[0013] To address the significant bandwidth waste and performance bottlenecks caused by existing technologies that typically rely on passive hardware caching mechanisms to handle key-value cache access in large model inference, this application proposes a key-value cache loading method, apparatus, and storage medium for cross-stream multiprocessors (SM).

[0014] Figure 1 A schematic diagram of a cross-SM KV cache loading architecture 100 according to an embodiment of this application is shown. This architecture can be integrated into a GPU or a dedicated AI acceleration chip. In alternative embodiments, the architecture can also be integrated into a CPU, NPU, TPU, FPGA, or ASIC chip. Figure 1 As shown, this architecture includes multiple streaming multiprocessors SM0, SM1, ..., SM2. N The interconnection network 102 between SMs (such as Network-on-Chip, NoC), the directory structure (KV Block Directory) 106, and the shared last-level cache 110 (such as L2 cache) and off-chip memory (such as HBM 112).

[0015] Each SM (Service Provider) is responsible for performing computational tasks, that is, handling one request or a batch of requests during the large model decoding phase. Each SM has its own local cache (in...). Figure 1 The L1 cache (shown as L1) can be directly accessed. In a CPU architecture, each core can have its own L1 cache; in an NPU architecture, each computing unit can have its own local memory.

[0016] The inter-SM interconnect network 102 provides a physical path for data exchange between SMs. To strike a balance between performance and hardware cost, this network can employ a constrained reachability design, where multiple streaming multiprocessor SMs are grouped into a cluster. SMs within a cluster can freely access each other, while cross-cluster access requires a higher-level switching switch, thus avoiding the high hardware overhead of a fully interconnected structure. In alternative embodiments, the inter-SM interconnect network can also employ other topologies such as a fully interconnected design, a ring interconnected design, a mesh interconnected design, or a tree interconnected design.

[0017] In one or more embodiments, multiple streaming multiprocessors (SMs) are divided into at least one cluster. For example, all SMs on the chip can be divided into cluster 0, cluster 1, ..., cluster N, with each cluster containing several SMs. SMs within a cluster can directly communicate with each other; SMs in different clusters are connected via switches to form a hierarchical interconnect topology. This multi-cluster design can support a larger number of SMs while controlling hardware complexity.

[0018] In one or more embodiments, a directory access network 104 is configured to connect each SM to a directory structure 106. Each SM (including SM0 and SM1 of cluster 0, SM2 and SM3 of cluster 1, and SM of cluster N) N All are connected to the directory structure separately through a directory access network. This directory access network is used to transmit metadata operations such as directory query requests, location information returns, new key-value block registrations, and reference count updates. It is separate from the data transmission network to avoid interference between control signaling and data flow. In alternative embodiments, the directory access network can adopt various topologies such as point-to-point connections, bus connections, or tree connections; it can also adopt the same physical network as the inter-SM interconnection network, but with logical separation through virtual channels or priorities.

[0019] In one or more embodiments, a storage network 108 is configured to connect the directory structure, the last-level cache, and the HBM. The directory structure communicates with the last-level cache via the storage network, and the last-level cache is connected to the HBM via the storage network. The storage network is used to load KV block data from the HBM to the last-level cache when needed, and to perform data eviction under the management of the directory structure. In alternative embodiments, the storage network may be implemented using the same physical network as the directory access network; it may also be implemented using a dedicated high-bandwidth bus; or it may be implemented using novel communication technologies such as wireless interconnect or optical interconnect.

[0020] The directory structure is used to implement cross-SM KV cache loading according to this application, and it maintains the metadata of each active KV block. The directory structure can be implemented using a hardware lookup table or content-addressable memory (CAM) for fast lookup of KV block metadata. In one or more embodiments, the directory structure uses KV block identifiers as indexes to store corresponding location information, reference counts, and lifecycle states, supporting O(1) time complexity lookup and update operations. In alternative embodiments, the directory structure can also be implemented using static random access memory (SRAM), dynamic random access memory (DRAM), register files, non-volatile memory (such as MRAM, ReRAM, PCM), or other types of storage devices; it can also be implemented using software-maintained data structures (such as hash tables, red-black trees, B-trees), suitable for scenarios with low latency requirements. In one or more embodiments, the directory structure may include the following items or other items:

[0021] Location information identifies which SM's local cache, last-level cache, or HBM the KV block is currently stored in. In alternative embodiments, location information may also include finer-grained location descriptions, such as specific cache groups within the SM, cache line addresses, etc.; for distributed directory implementations, location information may also include identifiers of directory fragments.

[0022] Reference counting is used to identify the number of SMs currently using the KV block, enabling shared management. In alternative embodiments, reference counting may also include other types of counters, such as prediction counters for predicting future accesses, age counters for aging algorithms, etc.; updates to the reference count can be implemented using atomic operations to avoid consistency issues caused by concurrent accesses.

[0023] Lifecycle states are used to identify whether the KV block is in an "active" state (being used), a "reclaimable" state (no longer used by any SM), or a "loading" state (being read from HBM). In some embodiments, lifecycle states may also include a "preloaded" state (predicted to be used soon), a "locked" state (prohibited from being evicted), a "dirty" state (data has been modified and needs to be written back), etc.

[0024] The directory structure can be implemented in various ways. In one or more embodiments, the directory structure can be a centralized directory located in the chip, where all SMs access the same directory instance through a directory access network. In alternative embodiments, the directory structure can also be a distributed directory maintained by each SM, where each SM holds a portion of the directory and works collaboratively through a consensus protocol; a hybrid directory can also be used, where some directories are centralized and some are distributed; or a hierarchical directory can be used, which is a multi-level structure of local directories plus global directories.

[0025] The size and associativity of the directory structure can be adjusted according to the system scale. For example, for a system with 32 SMs, the directory can be designed as a 32-way group association; for a system with 64 SMs, the directory can be designed as a 64-way group association. The directory replacement strategy can adopt algorithms such as Least Recently Used (LRU), Least Frequently Used (LFU), First-In-First-Out (FIFO), or random replacement.

[0026] Figure 2 A flowchart illustrating a cross-SM KV cache loading method according to an embodiment of this application is shown. This method 200 can be applied to... Figure 1 The system shown includes the following steps:

[0027] Step S202: In response to the first SM's access request for the target KV block, query the directory structure to obtain the storage location information of the target KV block.

[0028] Specifically, in the large model decoding stage, when the first SM (e.g.) Figure 1 When the SM0 in the algorithm needs to access a specific key-value block, it first generates an access request. This request contains a unique identifier for the target key-value block, which can be composed of the position index of the token corresponding to the key-value block in the input sequence, the Transformer layer number, and the attention head number. In alternative embodiments, this identifier may also include other dimensions such as model identifier, batch identifier, and sequence identifier to support more complex application scenarios, such as multi-model parallelism and multi-batch processing.

[0029] SM0 sends the request to the directory structure via the directory access network to obtain the latest location information of the KV block, that is, to determine which SM's local cache, last-level cache, or unloaded KV block is currently located in (i.e., still in HBM). In alternative embodiments, SMs may also send the request in other ways, such as via a shared bus, via wireless communication, via optical interconnect, etc.; the format of the request may be a data packet, message, signal, or other form.

[0030] Upon receiving a query request, the directory structure searches its maintained metadata for a record of the target key-value block, i.e., it determines whether the key-value block has been loaded. If the record exists, it means the key-value block has been loaded into the local cache or last-level cache of a certain storage manager (SM), and the directory structure returns the storage location information of the key-value block to SM0 via the directory access network. If the record does not exist, it means the key-value block has not yet been loaded by any SM and is still located in HBM, and the directory structure returns a not found indication to SM0. In an alternative embodiment, the directory structure may also return other auxiliary information, such as the key-value block's version number, access statistics, priority, etc.

[0031] Step S204: In response to the directory structure indicating that the target KV block has been loaded into the on-chip cache by the second SM, retrieve the target KV block from the on-chip cache of the second SM.

[0032] Specifically, if the query results returned by the directory structure indicate that the target KV block has already been loaded into its on-chip cache by another SM, then the first SM does not need to reload the target KV block from off-chip memory. The data retrieval method between the first SM and the second SM depends on their positional relationship:

[0033] If the second SM is located in the same cluster as the first SM (e.g.) Figure 1 If SM1 and SM0 belong to the same cluster (0), then the first SM can initiate a data retrieval request to the second SM through direct intra-cluster communication. This direct communication can be achieved through a shared local cache, a dedicated intra-cluster bus, or other methods that do not require network routing. After receiving the request, the second SM reads the data of the target KV block from its local cache and sends it to the first SM through direct communication.

[0034] If the second SM is located in a different cluster, the first SM needs to initiate a data retrieval request to the second SM through the inter-SM interconnection network. Data is transmitted through the inter-SM interconnection network and may undergo cross-cluster routing via switching switches, but it is always transmitted at the inter-SM interconnection network level, without passing through the directory access network or storage network. In alternative embodiments, cross-cluster data transmission can employ various routing algorithms such as source routing, distributed routing, or centralized routing; and can employ various switching methods such as circuit switching, packet switching, or virtual circuit switching.

[0035] Once the first SM receives the data from the target KV block, it can use it for subsequent attention mechanism calculations. In an alternative embodiment, the data acquisition request can also be sent via other networks, such as a directory access network (if the network supports data transmission) or a storage network; data transmission can be in streaming mode, packet mode, or burst mode; data can be transmitted in fragments or all at once.

[0036] In one or more embodiments, a lightweight consistency check is performed before retrieving the target KV block data from the second SM, i.e., verifying whether the KV block replica in the second SM is still valid. Considering the read-only nature of the KV cache during the inference phase, this consistency check does not require introducing a complex cache consistency protocol and can be implemented using various lightweight methods. For example, each KV block can be assigned a globally incrementing version number upon initial loading. Subsequent accesses can query the version number via the directory access network for comparison; if the version numbers match, the data is valid. The version number can be global, partitioned, or independent for each KV block, and can be monotonically incrementing or cyclically used.

[0037] Lightweight consistency checks can also be implemented in various other ways. For example, besides version number mechanisms, timestamp mechanisms can be used to record the loading time of key-value blocks and compare timestamps to determine if the data is still valid; alternatively, checksum mechanisms can be used to calculate lightweight checksums for key-value block data and verify checksum matching during retrieval; ECC error correction codes can be used to detect and correct some errors; or a remote verification mechanism using a Trusted Execution Environment (TEE) can be used to verify the trustworthiness of the data source. Those skilled in the art can choose the appropriate lightweight consistency check method based on actual needs.

[0038] Step S206: In response to the directory structure indicating that the target KV block has not been loaded, load the target KV block from off-chip memory into the on-chip cache of the first SM, and register the location information of the target KV block in the directory structure.

[0039] Specifically, if the query results returned by the directory structure indicate that the target KV block has not yet been loaded by any SM (i.e., it is still in HBM), the first SM needs to perform an initial load operation. In one or more embodiments, the first SM initiates a read request to HBM through the memory controller, loading the data of the target KV block from HBM into the last-level cache. After the data arrives at the last-level cache, it can be selectively transferred to the first SM's local cache to further reduce the latency of subsequent accesses and reduce contention for the last-level cache. In alternative embodiments, the first SM can also directly access the data from the last-level cache without transferring it to the local cache. In alternative embodiments, data can be directly loaded from HBM into the SM's local cache, bypassing the last-level cache; data loading can be performed through a direct memory access (DMA) engine, freeing up the SM's computing resources; data loading can employ prefetching techniques to predictively load data in advance based on access patterns.

[0040] After data loading is complete, the first SM (Directory Management Provider) sends registration information to the directory structure via the directory access network to create a new record for the target KV block. This registration information is transmitted via the directory access network and is completely separate from the data loading path. The new record may include: the location information of the KV block (e.g., "located in the local cache of the first SM"), the initial reference count (set to 1, indicating that the first SM is using it), and the lifecycle status (set to "active"). After receiving the registration information, the directory structure updates its maintained metadata so that other SMs can know its location when querying the KV block. In an alternative embodiment, the registration information may also include additional attributes such as the size, format, version number, access permissions, encryption information, and compression information of the KV block; the registration process may employ a transaction mechanism to ensure atomicity; for distributed directories, the registration information may need to be broadcast to multiple directory shards.

[0041] After the first SM completes its computation task on the target KV block, it no longer needs to hold a reference to that block. At this point, the first SM sends a reference release signal to the directory structure via the directory access network. This signal is transmitted via the directory access network and does not consume data transmission network bandwidth. Upon receiving the signal, the directory structure locates the record of the target KV block and decrements its reference count by 1. In an alternative embodiment, the reference release signal may contain reference release information for multiple KV blocks to reduce signaling overhead; reference release may be batch-processed and periodically aggregated; for shared references, it is necessary to handle the situation where multiple SMs release references simultaneously to ensure the accuracy of the reference count.

[0042] When the reference count becomes 0, the directory structure marks the lifetime status of the KV block as "reclaimable," indicating that the block is no longer used by any SM and can be evicted to free up on-chip cache space. In alternative embodiments, KV blocks marked as reclaimable can be evicted immediately or with a delayed evicting period; they can be prioritized for evicting or used as backup evicting candidates; and they can be weighted and evaluated based on factors such as access frequency and access timeliness.

[0043] In actual operation, on-chip cache space is limited. In one or more embodiments, when the system detects cache space shortage, i.e., the free cache capacity is lower than a preset threshold, a cache eviction mechanism is triggered. The eviction policy prioritizes key-value blocks marked as "reclaimable," which are blocks with a reference count of 0 and the least recently accessed. The eviction operation involves removing the corresponding data from the last-level cache via the storage network and marking the corresponding space in the HBM as available, thereby freeing up space for newly loaded key-value blocks. This eviction process is executed by the memory controller or cache controller and does not involve the directory structure. In alternative embodiments, the eviction policy may also consider other factors, such as access frequency, access locality, data importance, priority, etc.; LRU, LFU, FIFO, random replacement, or mixed strategies may be used; eviction may be performed synchronously or asynchronously; if the evicted data needs to be retained (e.g., dirty data), it needs to be written back to the HBM first.

[0044] While the embodiments described above illustrate the separation of control signaling and data transmission through three independent networks (directory access network, inter-SM interconnection network, and storage network), those skilled in the art should understand that this is merely a preferred implementation of the invention. In actual chip design, the functions of the three networks can be merged into a unified physical network based on cost, power consumption, and performance requirements.

[0045] For example, in an alternative embodiment, a single on-chip network (NoC) can carry all types of communication, including control signaling between the SM and the directory structure, data transmission between SMs, and storage access between the last-level cache and the HBM. Under this unified network architecture, effective isolation between control signaling and data transmission can still be achieved by assigning different priorities, virtual channels, or Quality of Service (QoS) levels to different types of communication, avoiding mutual interference. For example, the highest priority can be assigned to control signaling for directory access to ensure low latency for directory queries; medium priority can be assigned to data transmission between SMs; and lower priority can be assigned to background data loading for the storage network.

[0046] In another alternative embodiment, a combination of two networks instead of three networks can be used. For example, the directory access network and storage network can be merged into a single control / storage network, while the inter-SM interconnect network remains independent; or the inter-SM interconnect network and storage network can be merged into a single data network, while the directory access network remains independent. These variations all fall within the scope of this application.

[0047] In summary, the cross-SM KV cache loading mechanism of this application can include the following three steps:

[0048] Location phase: SM determines the location of the target KV block by querying the directory structure, that is, whether the data is in the cache of other SMs or needs to be loaded from HBM.

[0049] Acquisition process: If the data is already in another SM, it is acquired directly through the interconnection network, which enables cross-SM data sharing to avoid duplicate loading; if the data has not yet been loaded, it is read from HBM and registered in the directory.

[0050] Status update process: After SM finishes using the data, it updates the reference count in the directory. When the count reaches zero, it is marked as recyclable, which means that the system is notified that the data is no longer needed and can be overwritten by subsequent data.

[0051] Through the above mechanism, when multiple SMs need to access the same KV block, only the first SM to access it needs to load it from outside the chip. Subsequent SMs can obtain data from the SM that has already loaded the block through directory bootstrapping and inter-SM interconnection, thus achieving efficient reuse.

[0052] According to embodiments of this application, compared to existing technologies that rely on passive caching, this application can bring some or all of the following beneficial effects:

[0053] Significantly reduces off-chip bandwidth pressure: By explicitly sharing and reusing across SMs, the number of repeated reads of off-chip memories such as HBM is greatly reduced, thereby freeing up valuable off-chip bandwidth for more necessary memory access operations.

[0054] Reduce average access latency: Data is retrieved from the on-chip cache of the adjacent SM, which has a much lower latency than loading from the off-chip HBM, thereby speeding up the processing speed of the SM and improving the throughput of the decoding stage.

[0055] Improved energy efficiency: Reduced access to high-energy-consuming off-chip memory, thus reducing the overall energy consumption of the system.

[0056] Efficient resource management: Through directory structure and reference counting, fine-grained management of on-chip KV cache resources is achieved, thereby improving performance while ensuring data consistency and the correctness of lifecycle control.

[0057] Hardware-friendly implementation: The proposed SM interconnection and directory structure can reduce hardware complexity through a restricted reachable network design, while the lightweight consistency mechanism avoids the huge overhead of traditional cache consistency protocols, making it very suitable for the specific application scenario of KV caching.

[0058] The technical solution of this application is not only applicable to KV cache loading of GPU architecture, but also to other processor architectures such as CPU, NPU, TPU, as well as multiple technical fields such as computer vision, recommendation system, database, and scientific computing, and has good versatility and portability.

[0059] The technical solution of this application provides a variety of implementation variations, including different network topologies, different directory structures, and different consistency checking mechanisms. Those skilled in the art can choose the most suitable implementation method according to specific application scenarios and performance requirements.

[0060] The technical solution of this application supports extended features such as multi-level cache hierarchy, multi-cluster structure, and distributed directory, which can be expanded as the processor scale increases to meet the needs of future large-scale processors.

[0061] Figure 3 This is a schematic diagram of a computing device according to an embodiment of this application. This device can be implemented as a computing device 300 and can be used to execute the methods described above. Figure 3 As shown, computing device 300 may include bus 302 or other communication mechanisms for transmitting information, and one or more hardware processors 304 coupled to bus 302 for processing information. The one or more hardware processors 304 may include, for example, one or more general-purpose microprocessors, or may include dedicated processors such as GPUs, NPUs, TPUs, DSPs, FPGAs, and ASICs.

[0062] like Figure 3 As shown, in some embodiments, computing device 300 may further include main memory 306 coupled to bus 302. Main memory 306 is used to store information and instructions executed by one or more processors 304, such as random access memory (RAM), cache, and / or other dynamic storage devices. Main memory 306 may also be used to store temporary variables or other intermediate information during the execution of instructions executed by one or more processors 304. When these instructions are stored in storage media accessible to one or more processors 304, they can cause computing device 300 to become a dedicated machine customized to perform the operations specified in the instructions. Storage device 308 may include non-volatile and / or volatile storage media. Non-volatile storage media may include, for example, optical discs or magnetic disks. Volatile storage media may include dynamic memory. Common forms of storage media may include, for example, floppy disks, hard disks, solid-state drives, magnetic tape, or any other magnetic data storage media, CD-ROMs, any other optical data storage media, any physical media with a perforated pattern, RAM, DRAM, PROM, and EPROM, FLASH-EPROM, NVRAM, any other memory chip or cartridge, or their networking versions.

[0063] like Figure 3 As shown, in some embodiments, computing device 300 may further include one or more communication interfaces or network interfaces 310 coupled to bus 302. Network interface 310 may provide bidirectional data communication coupling to one or more network links connected to one or more networks. As another example, network interface 310 may be a local area network (LAN) card to provide data communication connectivity to a LAN-compatible (or WAN component communicating with a WAN) network. Wireless links may also be implemented. The computing device 300 may be a graphics processing unit (GPU) or artificial intelligence (AI) accelerator chip, or an electronic device including the KV cache loading device of the cross-stream multiprocessor SM described in the embodiments of this application, such as a server, workstation, personal computer, mobile device, embedded device, Internet of Things device, etc.

[0064] The execution of certain operations can be distributed across processors rather than residing within a single machine, but rather deployed across multiple machines. In some example embodiments, the processor or processor-implemented engine may reside in a single geographic location (e.g., in a home environment, office environment, or server farm). In other example embodiments, the processor or processor-implemented engine may be distributed across multiple geographic locations.

[0065] Each of the processes, methods, and algorithms described in the preceding sections may be embodied in code modules executed by one or more computer systems or computer processors including computer hardware, and may be fully or partially automated by these code modules. The processes and algorithms may be implemented, partially or fully, in dedicated circuit systems.

[0066] When the functions disclosed herein are implemented as software functional units and sold or used as stand-alone products, they may be stored in a processor-executable, non-volatile, computer-readable storage medium. Specific technical solutions (all or part) disclosed herein, or aspects contributing to the prior art, may be embodied in the form of a software product. The software product may be stored in a storage medium and includes several instructions that cause a computing device (which may be a personal computer, server, network device, etc.) to perform all or some steps of the methods of the embodiments of this application. The storage medium may include a flash drive, portable hard disk drive, ROM, RAM, magnetic disk, optical disk, other media operable to store program code, or any combination thereof.

[0067] Specific embodiments further provide a system including a processor and a non-transitory computer-readable storage medium storing instructions executable by the processor to cause the system to perform operations corresponding to steps in any method of the embodiments disclosed above. Specific embodiments further provide a non-transitory computer-readable storage medium storing instructions executable by one or more processors to cause the one or more processors to perform operations corresponding to steps in any method of the embodiments disclosed above.

[0068] The embodiments disclosed herein can be implemented via a cloud platform, server, or server cluster (collectively referred to below as the "Service System") that interacts with a client. The client can be a terminal device or a client registered by a user at the platform, wherein the terminal device can be a mobile terminal, a personal computer (PC), or any device that can have the platform application installed.

[0069] The various features and processes described above can be used independently of each other or combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of this disclosure. Additionally, certain method or process blocks may be omitted in some embodiments. The methods and processes described herein are not limited to any particular order, and their associated blocks or states may be executed in other suitable orders. For example, described blocks or states may be executed in an order other than that specifically disclosed, or multiple blocks or states may be combined into a single block or state. Example blocks or states may be executed sequentially, in parallel, or in some other manner. Blocks or states may be added to or removed from the disclosed example embodiments. The exemplary systems and components described herein may be configured differently than described. For example, components may be added to, removed from, or rearranged compared to the disclosed example embodiments.

[0070] The various operations of the exemplary methods described herein can be performed at least in part by an algorithm. The algorithm may be included in program code or instructions stored in memory (e.g., the aforementioned non-transitory computer-readable storage medium). This algorithm may include a machine learning algorithm. In some embodiments, the machine learning algorithm may not explicitly refer to the computer as performing the function but may learn from training data to generate a predictive model of the function.

[0071] The various operations of the exemplary methods described herein can be performed, at least in part, by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, these processors can constitute an engine of processor implementation that operates to perform one or more of the operations or functions described herein.

[0072] Similarly, the methods described herein may be implemented at least in part by a processor, wherein one or more specific processors are instances of hardware. For example, at least some operations of the methods may be performed by one or more processors or an engine implemented by a processor. Furthermore, one or more processors may also be operable to support the execution of relevant operations in a “cloud computing” environment or as the execution of relevant operations in a “Software as a Service” (SaaS) context. For example, at least some operations may be performed by a group of computers (as an example of a machine containing processors), wherein these operations are accessible via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., application programming interfaces (APIs)).

[0073] The execution of certain operations can be distributed across processors rather than residing within a single machine, and can be deployed across multiple machines. In some example embodiments, the processor or processor-implemented engine may reside in a single geographic location (e.g., in a home environment, office environment, or server farm). In other example embodiments, the processor or processor-implemented engine may be distributed across multiple geographic locations.

[0074] Throughout this specification, multiple instances may be implemented as components, operations, or structures described as a single instance. Although individual operations of one or more methods are illustrated and described as separate operations, one or more of these individual operations may be performed simultaneously, and not necessarily in the order illustrated. Structures and functions presented as separate components in the example configuration may be implemented as composite structures or components. Similarly, structures and functions presented as single components may be implemented as single components. These and other variations, modifications, additions, and improvements fall within the scope of this document.

[0075] As used herein, "or" is inclusive rather than exclusive unless explicitly indicated by the context. Therefore, in this document, "A, B, or C" means "A, B, A and B, A and C, B and C, or A, B, and C" unless explicitly indicated by the context. Furthermore, "and" is combined and separate unless explicitly indicated by the context. Therefore, in this document, "A and B" means "A and B, combined or separate" unless explicitly indicated by the context. Additionally, multiple instances of resources, operations, or structures described herein may be provided as a single instance. Furthermore, the boundaries between various resources, operations, engines, and data storage devices are somewhat arbitrary and specific operations are illustrated within the context of a particular illustrative configuration. Other functional assignments are foreseeable and fall within the scope of various embodiments of this disclosure. Generally, structures and functions presented as individual resources in example configurations may be implemented as combined structures or resources. Similarly, structures and functions presented as single resources may be implemented as single resources. These and other changes, modifications, additions, and improvements fall within the scope of the embodiments of this disclosure as expressed in the appended claims. Therefore, this specification and drawings should be considered illustrative rather than restrictive.

[0076] The terms “comprising” or “including” are used to indicate the presence of a subsequently claimed feature, but do not preclude the addition of other features. Unless otherwise specifically stated or otherwise understood in the context in which they are used, conditional language such as “may,” “can,” “may,” and “can” is generally intended to convey that certain embodiments include certain features, components, and / or steps that are not included in other embodiments. Therefore, this conditional language is generally not intended to imply that one or more embodiments require features, components, and / or steps in any way, or that one or more embodiments must include logic for determining whether such features, components, and / or steps are included in or performed in any particular embodiment, with or without user input or prompts.

[0077] Although the general outline of the subject matter has been described with reference to specific exemplary embodiments, various modifications and changes may be made to these embodiments without departing from the broad scope of embodiments of this disclosure. Where more than one embodiment is disclosed, these embodiments of the subject matter may be referred to individually or collectively herein as the term "invention," this is for convenience only and is not intended to automatically limit the scope of this application to any single disclosure or concept.

[0078] The embodiments illustrated herein are described in detail to enable those skilled in the art to practice the disclosed teachings. Other embodiments may be used and derived therefrom, such that structural and logical substitutions and changes may be made without departing from the scope of this disclosure. Therefore, “implementation” is not intended to be limiting, and the scope of the various embodiments is defined only by the appended claims and their full scope.

Claims

1. A method for loading a key-value cache in a multi-processor system, characterized in that, The method includes: In response to an access request from the first streaming multiprocessor for a target key-value block, the directory structure is queried through the directory access network to obtain the storage location information of the target key-value block. The directory structure maintains metadata for multiple key-value blocks, and the metadata for each key-value block includes location information, reference count, and lifecycle status. In response to the directory structure indicating that the target KV block has been loaded into the on-chip cache by the second streaming multiprocessor, a lightweight consistency check is performed before retrieving the target KV block from the on-chip cache of the second streaming multiprocessor, and then the target KV block is retrieved from the on-chip cache of the second streaming multiprocessor via the inter-SM interconnect network; In response to the directory structure indicating that the target KV block has not been loaded, the target KV block is loaded from off-chip memory into the on-chip cache of the first streaming multiprocessor, and the location information of the target KV block is registered in the directory structure; In response to the first streaming multiprocessor releasing a reference to the target KV block, the reference count of the target KV block in the directory structure is updated; and In response to the reference count of the target key-value block in the directory structure becoming 0, the target key-value block is marked as reclaimable. In this system, multiple streaming multiprocessors are divided into at least one cluster. Streaming multiprocessors within the same cluster can directly communicate with each other, while streaming multiprocessors in different clusters are connected via a switch to enable communication.

2. The method according to claim 1, characterized in that, The location information is used to identify which streaming multiprocessor's local cache, last-level cache, or high-bandwidth memory the KV block is currently stored in; The reference count is used to identify the number of streaming multiprocessors currently using the KV block; and The lifecycle status is used to identify whether the KV block is in an active state, a reclaimable state, or a loading state.

3. The method according to claim 1, characterized in that, The on-chip cache includes local caches for streaming multiprocessors or shared last-level caches.

4. The method according to claim 1, characterized in that, The target KV block stores the key tensors and value tensors of historical tokens generated by the Transformer model during the inference phase.

5. The method according to claim 4, characterized in that, KV blocks marked as reclaimable can be evicted to free up on-chip cache space.

6. A computing device, characterized in that, The device includes: Multiple streaming multiprocessors; The directory structure is configured to maintain the location information, reference count, and lifecycle state of key-value blocks; and A memory, coupled to the plurality of streaming multiprocessors and storing instructions that, when executed by the plurality of streaming multiprocessors, cause the apparatus to perform the method according to any one of claims 1 to 5.

7. The apparatus according to claim 6, characterized in that, The device is a graphics processor or an artificial intelligence acceleration chip.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed by one or more processors, cause the one or more processors to perform the method according to any one of claims 1 to 5.

Images