Hbm hierarchical cache management method and system for ai inference parameters

Abstract

This application provides a method and system for HBM hierarchical caching management of AI inference parameters, including a host and a solid-state drive (SSD). The host includes a hardware accelerator and a high-bandwidth memory (HBM), while the SSD includes a solid-state drive controller, a high-speed cache memory (SCM), and flash memory chips. By establishing a dedicated hardware command stream queue between the host and the SSD, the SSD controller can accurately perceive the real-time computing status of the current network layer in conjunction with the computing topology issued by the host, and read the inference parameters required for the next network layer in advance accordingly. The read inference parameters are subjected to sparse filtering to retain the effective parameters required for subsequent inference calculations, and then written into the high-bandwidth memory (HBM) cache in advance through a direct bus connection channel, so that the hardware accelerator can directly obtain the corresponding inference parameters from the HBM for calculation when switching to the next network layer.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to an HBM hierarchical caching management method and system for AI inference parameters. Background Technology

[0002] With the development of artificial intelligence (AI) technology, the scale of inference parameters in AI models, such as large language models, multimodal models, and other deep neural network models, is constantly increasing, and the amount of data that needs to be accessed during model inference is continuously increasing. In order to meet the requirements of high-performance inference and access, hardware accelerators are usually used in conjunction with high-speed storage resources to complete model computation, and HBM (High Bandwidth Memory) is used as a high-speed memory near the hardware accelerator to improve parameter access speed, model inference speed, and computational efficiency.

[0003] However, since HBM has a relatively limited capacity, it is usually unable to accommodate all the inference parameters of a large-scale AI model. Therefore, during the model inference process, the corresponding inference parameters need to be continuously loaded from a larger storage medium into HBM according to the execution process of the network layer, and then read by the hardware accelerator to complete the calculation of the corresponding network layer.

[0004] As model size increases, the frequency of parameter migration between different storage layers rises, making it difficult to maintain coordination between parameter supply speed and model computation progress. When the required inference parameters have not yet been loaded, the hardware accelerator must wait for the data to arrive before continuing subsequent calculations, resulting in idle computing resources, increased storage access overhead, and reduced overall model inference throughput and computational efficiency. Therefore, improving the coordination between parameter supply and model computation during AI model inference under HBM capacity constraints, and reducing the performance overhead caused by parameter migration to improve model inference efficiency, has become a pressing technical problem to be solved in this field. Summary of the Invention

[0005] This application provides a hierarchical HBM cache management method and system for AI inference parameters to solve the problem that the limited capacity of HBM leads to insufficient coordination in the computation process and large parameter migration overhead, resulting in a decrease in the inference efficiency of AI models.

[0006] In a first aspect, embodiments of this application provide an HBM hierarchical cache management method for AI inference parameters. This method is applied to an HBM hierarchical cache management system composed of a host and a solid-state drive. The host includes a hardware accelerator and a high-bandwidth memory (HBM). The solid-state drive includes a solid-state drive controller, a high-speed cache memory (SCM), and flash memory chips. The flash memory chips cache the AI ​​model, and the high-bandwidth memory (HBM) is used to cache the inference parameters of the currently running network layer in the AI ​​model for the hardware accelerator to access. The execution entity of this method is a solid-state drive controller, including: A dedicated hardware command stream queue is established with the host to monitor the real-time execution status of the AI ​​model on the host, thereby enabling dual-end timing and space collaborative control between the host and the solid-state drive. Obtain the computational topology of the AI ​​model issued by the host. The computational topology is used to characterize the operator connection relationships and execution sequence of each network layer in the AI ​​model. Based on the computing topology, the execution status of the AI ​​model on the host side is monitored through a dedicated hardware command stream queue to obtain the real-time computing status of the currently running network layer. Based on the real-time computing status of the current network layer, the inference parameters of the next network layer are read and sparsified to obtain the sparsed target inference parameters. The sparsed target inference parameters are written to the high-speed bandwidth memory HBM as inference parameters for the next network layer via a direct bus connection, so that the hardware accelerator can directly call them when switching to the next network layer for inference.

[0007] Secondly, this application provides an HBM hierarchical caching management system for AI inference parameters. The system includes a host and a solid-state drive. The host includes a hardware accelerator and a high-bandwidth memory (HBM). The solid-state drive includes a solid-state drive controller, a high-speed cache memory (SCM), and flash memory chips. The flash memory chips cache the AI ​​model. The high-bandwidth memory (HBM) is used to cache the inference parameters of the currently running network layer in the AI ​​model for the hardware accelerator to access. The solid-state drive controller is used for: A dedicated hardware command stream queue is established with the host to monitor the real-time execution status of the AI ​​model on the host, thereby enabling dual-end timing and space collaborative control between the host and the solid-state drive. Obtain the computational topology of the AI ​​model issued by the host. The computational topology is used to characterize the operator connection relationships and execution sequence of each network layer in the AI ​​model. Based on the computing topology, the execution status of the AI ​​model on the host side is monitored through a dedicated hardware command stream queue to obtain the real-time computing status of the currently running network layer. Based on the real-time computing status of the current network layer, the inference parameters of the next network layer are read and sparsified to obtain the sparsed target inference parameters. The sparsed target inference parameters are written to the high-speed bandwidth memory HBM as inference parameters for the next network layer via a direct bus connection, so that the hardware accelerator can directly call them when switching to the next network layer for inference.

[0008] In one embodiment, the cache memory SCM is divided into a first physical buffer and a second physical buffer that are isolated from each other to form a double-buffered pipeline structure; the first physical buffer is used to cache the inference parameters of the current network layer, and the second physical buffer is used to write the target inference parameters of the next network layer. When the host accesses the inference parameters corresponding to the first physical buffer to perform the calculation of the current network layer, the solid-state drive controller controls multiple physical channels of the flash memory chip in parallel, reads the pre-fetched target inference parameters of the next network layer into the second physical buffer, and issues a hardware fence instruction to instantly swap the global physical address pointers of the first physical buffer and the second physical buffer when the host completes the network calculation of the current layer.

[0009] Thirdly, embodiments of this application provide a computer program product, which includes a computer program that implements the above-described method when executed by a processor.

[0010] The aforementioned HBM hierarchical caching management method and system for AI inference parameters provides a solution that establishes a dedicated hardware command stream queue between the host and the solid-state drive (SSD). This allows the SSD controller to obtain the execution status of the AI ​​model in real time and accurately perceive the real-time computing status of the current network layer by combining the computing topology issued by the host. Based on the real-time computing status of the current network layer, it can pre-read the inference parameters required for the next network layer. Simultaneously, the read inference parameters undergo sparse filtering, retaining only the valid parameters required for subsequent inference calculations. These parameters are then pre-written into the high-speed bandwidth memory HBM cache via a direct bus connection. This allows the hardware accelerator to directly obtain the corresponding inference parameters from the HBM for calculation when switching to the next network layer. This reduces data migration and redundant data transmission of invalid parameters between different storage layers, improves the utilization of the limited HBM cache space, reduces parameter loading waiting time and storage access latency, and ensures continuous coordination between the inference parameter supply process and the network layer computing process. This improves the throughput and overall computing efficiency of the AI ​​model inference, solving the technical problems of insufficient coordination between inference parameter supply and network layer computing processes and high parameter migration overhead leading to decreased AI model inference efficiency under limited HBM capacity conditions in existing technologies. Attached Figure Description

[0011] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a schematic diagram of the structure of an HBM hierarchical caching management system for AI inference parameters in one embodiment of this application; Figure 2 This is a flowchart illustrating an HBM hierarchical cache management method for AI inference parameters in one embodiment of this application; Figure 3 yes Figure 2 A schematic diagram of the implementation process of step S30; Figure 4 yes Figure 2 A schematic diagram of the implementation process of step S40; Figure 5 This is a schematic diagram of the dynamic management process of swapping out dynamic key-value cache parameters in one embodiment of this application. Detailed Implementation

[0013] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0014] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of a described feature, integral, step, operation, element, and / or component, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof. It should also be understood that, as used in this specification and the appended claims, the term "and / or" refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0015] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0016] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0017] To illustrate the technical solution of this application, specific embodiments are described below.

[0018] The HBM hierarchical caching management method for AI inference parameters provided in this application embodiment can be applied to, for example... Figure 1The HBM tiered cache management system (hereinafter referred to as the system) for the AI ​​inference parameters shown includes a host side and a solid-state drive (SSD). The host side includes a hardware accelerator and high-bandwidth memory (HBM), while the SSD includes an SSD controller, a cache memory (SCM), and flash memory chips. The host side, often simply referred to as the host, is the computing platform responsible for performing core computing tasks, running the operating system, and main applications; it is the core brain of the entire computing system, responsible for controlling, managing, and scheduling all connected peripheral devices, such as the SSD. The SSD controller communicates with the host side via a high-speed bus.

[0019] In practical applications, when AI inference is required, the SSD controller establishes a dedicated hardware command stream queue with the host to monitor the real-time execution status of the AI ​​model on the host, enabling dual-end timing and spatial collaborative control between the host and the SSD. The SSD controller acquires the computational topology of the AI ​​model issued by the host, which characterizes the operator connections and execution sequence of each network layer in the AI ​​model. Then, during inference, based on the computational topology, the SSD controller monitors the host's execution status of the AI ​​model through the dedicated hardware command stream queue to obtain the real-time computational status of the currently running network layer. Based on this status, it reads and sparsifies the inference parameters for the next network layer, obtaining the sparsified target inference parameters. Finally, the SSD controller writes the sparsified target inference parameters as the inference parameters for the next network layer to a cache in high-speed bandwidth memory (HBM) via a direct bus connection, allowing the hardware accelerator to directly access them when switching to the next network layer for inference.

[0020] In existing AI model inference processes, as the model parameter scale continuously increases, the limited capacity of the HBM (Hardware Body for Modeling) becomes insufficient to continuously accommodate the inference parameters of each network layer. This leads to frequent migration of inference parameters between different storage media, and the parameter supply cannot effectively match the network layer computation process, resulting in problems such as hardware accelerator waiting for data, increased storage access overhead, and decreased inference throughput. In this embodiment, a dedicated hardware command stream queue is established between the host and the solid-state drive (SSD) to obtain the AI ​​model execution status in real time. This allows the SSD to organize the reading of parameters for the next network layer in advance based on the real-time computation status of the current network layer, achieving synchronous coordination between the parameter supply process and the computation process, and reducing parameter waiting time. By performing sparsity filtering on the inference parameters of the next network layer, only the parameters that actually participate in subsequent inference computations are transmitted, reducing the data transfer of invalid parameters and lowering the data transfer overhead across storage layers.

[0021] Meanwhile, because the data entering the HBM cache is filtered, redundant parameters occupying HBM capacity are reduced, allowing the limited HBM space to cache more effective inference parameters and improving HBM cache resource utilization. Since the inference parameters for the next network layer can be pre-written into the HBM, the corresponding parameters can be directly called when the hardware accelerator switches network layers, reducing idle periods caused by waiting for parameter loading during computation and improving computational resource utilization.

[0022] Furthermore, by coordinating execution status monitoring, parameter pre-organization, parameter filtering, and cache pre-writing, storage access latency is reduced and parameter supply efficiency is improved, thereby enhancing the overall inference throughput and inference performance of the AI ​​model.

[0023] In one embodiment, such as Figure 2 As shown, an HBM hierarchical cache management method for AI inference parameters is provided. This method is applied to... Figure 1 The system comprises a host-side HBM hierarchical caching management system and a solid-state drive (SSD). The host-side system includes a hardware accelerator and high-bandwidth memory (HBM), while the SSD includes an SSD controller, a high-speed cache memory (SCM), and flash memory chips. The flash memory chips cache the AI ​​model, and the high-bandwidth memory (HBM) caches the inference parameters of the currently running network layer in the AI ​​model for the hardware accelerator to access. This method is applied in… Figure 1 Taking the solid-state drive controller in the system as an example, the following steps are included: S10: Establish a dedicated hardware command stream queue with the host to monitor the real-time execution status of the AI ​​model on the host, so as to realize dual-end timing and space collaborative control between the host and the solid-state drive.

[0024] A dedicated hardware command stream queue is established between the SSD controller and the host computer. This queue is a dedicated communication channel implemented in hardware between the SSD controller and the host hardware accelerator, used to transmit commands and status information. Unlike traditional software-level message queues, it features low latency and high determinism. The SSD controller uses this dedicated hardware command stream queue to monitor the execution status of the AI ​​model on the host computer in real time, including information such as which network layer the host computer is currently calculating and its calculation progress.

[0025] Through this monitoring mechanism, the SSD controller can sense the operating rhythm of the host, thereby achieving coordinated control between the host and the SSD in two dimensions: timing (i.e., coordination in execution time) and space (i.e., coordination in data storage location). For example, in a typical Transformer large language model inference scenario, the AI ​​model contains multiple Transformer layers. When the host hardware accelerator is calculating the Nth layer, the SSD controller can detect this state through a dedicated hardware command stream queue and simultaneously prepare the inference parameters for the N+1th layer, realizing pipelined parallel operation between the two storage devices.

[0026] S20: Obtain the computational topology of the AI ​​model issued by the host. The computational topology is used to characterize the operator connection relationships and execution sequence of each network layer in the AI ​​model.

[0027] The solid-state drive (SSD) controller receives a descriptive data structure from the host, which is the computational topology. This topology is a graph-like structure describing the operator connections between network layers in an AI model and the execution order of those layers. It can be understood as a static computational graph of the AI ​​model, where each node represents a network layer or operator, and edges represent data dependencies and execution order. By acquiring this computational topology, the SSD controller can gain a comprehensive understanding of the entire AI model, including the number of network layers, the dependencies between layers, which layer must execute after which layer, and the data flow between layers. For example, for a GPT-type model containing an embedding layer, 12 Transformer layers, a normalization layer, and a linear output layer, the computational topology explicitly indicates the sequential execution of these layers and the specific operator types within each layer (such as multi-head self-attention, feedforward neural networks, etc.). This computational topology is generated by the host during the model loading phase based on the model configuration file or model description file such as ONNX, and is then sent to the SSD controller via a dedicated hardware command stream queue or a separate configuration channel.

[0028] S30: Based on the computing topology, it monitors the execution status of the AI ​​model on the host side through a dedicated hardware command stream queue to obtain the real-time computing status of the currently running network layer.

[0029] The solid-state drive controller, combining the computational topology obtained in step S20, utilizes the dedicated hardware command stream queue established in step S10 to continuously monitor the execution of the AI ​​model by the host-side hardware accelerator, thereby determining the currently running network layer and its real-time computation status. The real-time computation status of the currently running network layer refers to the computational progress corresponding to the network layer currently being executed by the host-side hardware accelerator, which can be characterized by information such as the unique identifier of the current network layer, the percentage of computation completed, and the activation state vector.

[0030] In one specific implementation, the solid-state drive (SSD) controller integrates a hardware processing engine and a bus interface. The bus interface supports high-speed interconnect protocol standards (such as CXL or CCIX) to map the high-speed cache memory (SCM) and high-bandwidth memory (HBM) within a globally unified virtual address space. The hardware accelerator issues a synchronization token with a unique identifier for the current network layer to a dedicated hardware command stream queue when the current network layer computation begins. This synchronization token is a special data packet generated by hardware and contains information about the current computation state; it includes not only the unique identifier of the current network layer but also the real-time computation state of the currently running network layer. The SSD controller polls the dedicated hardware command stream queue via hardwired polling (i.e., continuous checking using dedicated hardware circuitry, rather than software-level polling) to trigger the capture of the synchronization token, thereby monitoring the real-time computation state of the currently running network layer (i.e., the network layer currently running on the host side, referred to as the current network layer).

[0031] For example, in a large language model inference scenario with 24 Transformer layers, when the host-side hardware accelerator starts computing the 5th Transformer layer, it generates a synchronization token carrying information such as Layer_ID=5 and computation status=start execution. The solid-state drive controller's hardware polling circuitry captures this token in a dedicated hardware command stream queue, thus confirming that the currently running network layer is the 5th layer.

[0032] S40: Based on the real-time computing status of the current network layer, read and sparsify the inference parameters of the next network layer to obtain the sparsed target inference parameters.

[0033] Based on the real-time computing status of the currently running network layer obtained in step S30, the solid-state drive controller determines which network layer's computing stage it has entered. Then, based on the connection relationships between network layers in the computing topology, it determines the next network layer and performs reading and sparsity filtering processing on the inference parameters of that next network layer. Inference parameters refer to the weight data and intermediate cache data required by the AI ​​model during network layer inference computation. Sparsity filtering is a data compression method that uses hardware circuitry to filter out unimportant or inactive elements from the parameter data, retaining only the important parameter data, thereby reducing the data volume.

[0034] Specifically, the solid-state drive (SSD) controller initiates a prefetch operation on the target inference parameters in the next network layer based on the physical storage address of the next network layer (i.e., the specific storage location of the parameters in the flash memory chip). The prefetched target inference parameters are then read from the flash memory chip into the SCM (Supply Cache Memory) via the SSD's internal data read channel. Subsequently, the SSD controller invokes its integrated hardware processing engine to perform hardware-level sparsity filtering on the target inference parameters in the SCM. This involves using dedicated hardware circuitry rather than software algorithms to quickly identify and filter out unimportant parameter elements, thereby reconstructing the sparsely filtered target inference parameters.

[0035] For example, in a feedforward neural network layer with 4096 neurons, the weight parameters are a 4096×4096 matrix. After sparsity filtering, if only about 30% of the neurons are activated, the number of parameters retained is only about 30% of the original, greatly reducing the amount of data that needs to be transmitted.

[0036] S50: The sparsed target inference parameters are written to the high-speed bandwidth memory HBM as inference parameters for the next network layer via the bus direct connection channel, so that the hardware accelerator can directly call them when switching to the next network layer for inference.

[0037] The solid-state drive (SSD) controller writes the target inference parameters, filtered through sparsification in step S40, to the host-side high-bandwidth memory (HBM) for caching via a direct bus connection. The direct bus connection refers to a direct data path established between the SSD controller and the host-side HBM via a high-speed interconnect bus (such as the CXL.mem protocol), eliminating the need for intermediaries like the host CPU or system memory. Data can be directly transferred from the SSD's SCM to the HBM. These sparsified target inference parameters written to the HBM constitute all the valid parameter data required for the next network layer computation. When the host-side hardware accelerator completes the current network layer computation and switches to the next network layer for inference computation, it can directly retrieve these prepared parameters from the HBM without waiting to read them from the SSD's flash memory.

[0038] In the specific implementation, the bus interface is used to write the sparsely filtered target inference parameters to the high-speed bandwidth memory (HBM) cache via the direct bus connection channel, so that the hardware accelerator can directly access them when switching to the next network layer for inference. For example, when the host is calculating layer N, the solid-state drive controller has already written the sparsification parameters for layer N+1 into the designated address region of the HBM. When the calculation of layer N is completed and the hardware accelerator switches to layer N+1, the inference parameters for that layer already exist in the HBM, and the hardware accelerator can directly read and use them, achieving zero-wait parameter loading.

[0039] In this embodiment, a dedicated hardware command stream queue enables real-time hardware-level communication between the host and the solid-state drive (SSD), allowing the SSD controller to accurately perceive the inference execution progress on the host. By acquiring the computational topology of the AI ​​model, the SSD controller gains the ability to predict the global computational process of the model. Based on this, the SSD controller uses the dedicated hardware command stream queue to monitor and capture synchronization tokens, obtains the computational status of the currently running network layer in real time, and accordingly performs proactive prefetching and hardware-level sparsity filtering on the inference parameters for the next network layer. Finally, the sparsified parameters are directly written to the HBM cache via a direct bus connection. The entire solution constructs an intelligent, pipelined data path from flash memory chips → SCM → HBM, completely overlapping the parameter preparation work for the next layer with the computation process of the current layer, eliminating the idle time of the hardware accelerator waiting for parameter loading during layer switching in traditional solutions. Simultaneously, sparsity filtering significantly reduces the amount of data that needs to be transmitted and cached, reducing the pressure on bus bandwidth and HBM capacity, and achieving efficient and low-latency inference parameter supply in large model inference scenarios.

[0040] In one embodiment, such as Figure 3 As shown, step S30, which involves monitoring the execution status of the AI ​​model on the host side through a dedicated hardware command stream queue based on the computational topology to obtain the real-time computational status of the currently running network layer, specifically includes the following steps: S31: Based on the computational topology, predict the access order of inference parameters in different network layers of the AI ​​model to obtain a deterministic memory access sequence of inference parameters in the AI ​​model. The deterministic memory access sequence includes the access order and physical storage address of inference parameters in different network layers.

[0041] Based on the computational topology obtained in step S20, the solid-state drive controller predicts and analyzes the order in which the inference parameters of each network layer in the AI ​​model are accessed by the hardware accelerator during the inference execution process, thereby generating a deterministic memory access sequence.

[0042] A deterministic memory access sequence is a parameter access sequence that is predetermined based on the computational topology of an AI model and arranged in the order of inference execution. This sequence explicitly records the order in which the inference parameters of each network layer will be accessed during the model inference process, as well as the physical storage address of each network layer parameter in the flash memory chip.

[0043] Unlike traditional cache-based random access models, the inference process of AI models has a strict execution order of the computation graph, meaning the execution order of network layers is deterministic, and therefore the access order of inference parameters is also deterministic. Solid-state drive (SSD) controllers leverage this characteristic to extract the dependencies between network layers from the computation topology, determine the execution order of each layer according to the topology, and then combine this with the storage layout information of each layer's parameters in the flash memory to generate a complete memory access sequence from the first layer to the last.

[0044] For example, for a sequentially executed Transformer model, the deterministic memory access sequence might be: [Layer 1 parameter address → Layer 2 parameter address → Layer 3 parameter address → ... → Layer N parameter address], where each parameter address contains the physical block address and page address information of the weight data of that layer in the flash memory chip.

[0045] S32: When performing computational tasks based on the AI ​​model on the host side, based on the deterministic memory access sequence, the network layer of the currently running AI model on the host side is monitored through a dedicated hardware command stream queue to obtain the real-time computational status of the currently running network layer.

[0046] When the host-side hardware accelerator starts performing inference computation tasks based on the AI ​​model, the solid-state drive controller continuously monitors which network layer of the AI ​​model the host-side hardware accelerator is currently executing, based on the deterministic memory access sequence generated in step S31 and the dedicated hardware command stream queue established in step S10, thereby obtaining the real-time computation status of the currently running network layer.

[0047] Specifically, the solid-state drive (SSD) controller captures synchronization tokens issued by the hardware accelerator via a dedicated hardware command stream queue. These tokens carry a unique identifier for the current network layer. The SSD controller compares the captured current network layer identifier with the order in the deterministic memory access sequence to accurately determine the current execution position within the entire inference process.

[0048] For example, if the deterministic memory access sequence is [layer 1, layer 2, ..., layer 24], when the synchronization token shows that the current network layer is layer 5, the solid-state drive controller knows that it has reached the 5th position in the sequence, and can predict that the next parameter to be prefetched is layer 6.

[0049] It is important to note that the solid-state drive controller first generates a complete deterministic memory access sequence by calculating the topology, and then uses this sequence to parse and locate the position of the current network layer in the entire inference process. It has a stronger global predictive ability and can predict the access order and parameter position of all subsequent network layers.

[0050] This solution extracts and generates deterministic memory access sequences from the computational topology, transforming the parameter access pattern during AI model inference from unpredictable random access to predictable sequential access, thus fully leveraging the deterministic execution characteristics of AI model inference. Combined with real-time monitoring of a dedicated hardware command flow queue, the solid-state drive controller can not only know the execution status of the current network layer but also predict the access order and parameter storage location of all subsequent network layers based on the deterministic memory access sequence. This allows for advance planning of the prefetch scheduling strategy throughout the entire inference lifecycle, improving the accuracy and foresight of prefetch operations.

[0051] In one embodiment, such as Figure 4 As shown, in step S40, based on the real-time computing state of the currently running network layer, the inference parameters of the next network layer are read and sparsified to obtain the sparsely filtered target inference parameters. This specifically includes the following steps: S41: Based on the real-time computing state and computing topology of the current network layer, predict the activation probability weight of each computing unit in the next network layer, and determine the physical memory address of the next network layer according to the deterministic memory access sequence.

[0052] Based on the real-time computing state of the current network layer obtained in step S30 or S32, and combined with the connection relationship between the current network layer and the next network layer in the computing topology obtained in step S20, the solid-state drive controller predicts the probability of each computing unit (i.e., basic computing units such as neurons or attention heads) in the next network layer being activated and accessed, thus obtaining the activation probability weight of each computing unit. The activation probability weight is a value between 0 and 1, used to characterize the likelihood that a certain computing unit in the next network layer will be actually called by the hardware accelerator in the upcoming inference computation; a higher weight indicates that the computing unit is more likely to be activated and participate in the computation. Simultaneously, based on the deterministic memory access sequence generated in step S31, the solid-state drive controller determines the physical storage address of the next network layer inference parameters in the flash memory chip.

[0053] In one specific embodiment, the real-time computing state of the current network layer is represented by an activation state vector, which is a binary vector consisting of 0s and 1s, where each element corresponds to a computing unit in the current network layer, with a value of 1 indicating activation and 0 indicating inactivation. The solid-state drive controller analyzes the activation state vector of the current network layer and combines it with the connection weights between layers in the computing topology to predict the activation probability of each computing unit in the next network layer.

[0054] The difference between this step and step S40 is that step S40 directly reads and sparsifies the parameters of the next layer based on the real-time computation state of the currently running network layer, while this solution S41 further introduces a prediction mechanism for activation probability weights, making the subsequent sparsity filtering more accurate and evidence-based. For example, in the Transformer model, which attention heads are activated in the current Transformer layer's attention computation directly affects which computational units in the next layer are more likely to be invoked. If the output activation value of the third attention head in the current layer is high, then according to the connection weights in the computational topology, the activation probability weight of the computational units connected to the third attention head in the next layer will be correspondingly higher.

[0055] S42: While performing network computation for the current layer on the host side, a prefetch operation for the target inference parameters in the next network layer is initiated based on the physical storage address of the next network layer, and the prefetched target inference parameters are read from the flash memory chip into the cache memory SCM.

[0056] While the host-side hardware accelerator performs inference computation for the current network layer, the solid-state drive (SSD) controller utilizes this time window to initiate a prefetch operation for the target inference parameters of the next network layer, based on the physical storage address of the next network layer determined in step S41. The target inference parameters refer to the inference parameters of the next network layer that need to be prefetched from the flash memory chip into the SCM for processing. The SSD controller reads the prefetched target inference parameters from the flash memory chip through its internal bus interface and DMA engine, and transfers them to the SCM for temporary storage.

[0057] In a preferred embodiment, the cache memory (SCM) is divided into a first physical buffer and a second physical buffer, which are isolated from each other, to form a dual-buffered pipeline structure (i.e., a mechanism in which the two buffers work alternately). The first physical buffer is used to cache the inference parameters of the current network layer, which can be accessed by the host-side hardware accelerator via a direct bus connection. The second physical buffer is used to write the target inference parameters of the next network layer prefetched from the flash memory. These two buffers are completely isolated in physical address space and do not overlap, ensuring that no data conflicts occur during parallel access.

[0058] Specifically, dual-buffered pipeline interleaving control can be initiated. That is, when the host accesses the parameters corresponding to the first physical buffer already opened in the cache memory SCM to perform the current network layer calculation, multiple physical channels of the flash memory chip are controlled in parallel (for example, 4 or 8 NAND flash memory channels are used for parallel reading at the same time) to read the pre-fetched target inference parameters of the next network layer into the isolated second physical buffer in the cache memory SCM.

[0059] In other words, when the host accesses the inference parameters corresponding to the first physical buffer to perform calculations for the current network layer, the SSD controller controls multiple physical channels of the flash memory chips in parallel to read the pre-fetched target inference parameters for the next network layer into the second physical buffer. Upon completion of the current layer network calculation on the host side, a hardware fence instruction is issued to instantly swap the global physical address pointers of the first and second physical buffers. In other words, when the host completes the calculation for the current network layer, the SSD controller immediately issues a hardware fence instruction. This instruction is a hardware-executed synchronization barrier instruction used to instantly swap the global physical address pointers of the first and second physical buffers, thereby achieving immediate switching between the roles of the two buffers.

[0060] S43: Perform sparsity filtering on the target inference parameters read into the cache memory (SCM) to filter out sparse elements in the target inference parameters whose activation probability weights are lower than a preset threshold, and reconstruct the sparsely filtered target inference parameters.

[0061] The solid-state drive controller utilizes its integrated hardware processing engine to perform sparsity filtering on the target inference parameters already read into the second physical buffer of the SCM (Solid State Cache). The core logic of this process is as follows: Each parameter element in the target inference parameters is examined sequentially for its corresponding activation probability weight (predicted in step S41). Parameter elements with activation probability weights below a preset threshold are identified as sparse elements—that is, unimportant parameter data that is unlikely to be used in the next network layer computation—and are filtered out. Parameter elements with activation probability weights reaching or exceeding the preset threshold are considered important and should be retained as inference parameters. After filtering, the solid-state drive controller reorganizes and arranges these retained parameter data to form the sparsely filtered target inference parameters.

[0062] For example, if the weight matrix of the next network layer is 4096×4096, after activation probability weight prediction, 70% of the computational units have activation probability weights lower than the preset threshold of 0.1. Then, the weight rows / columns corresponding to these computational units are filtered out, and only the remaining 30% of the important parameters are retained. Finally, the amount of parameter data after sparse filtering is only about 30% of the original data.

[0063] It is worth noting that this scheme introduces the prediction of activation probability weights to guide sparsity filtering, and describes in detail the prefetching operation under the double-buffered pipeline structure and the specific filtering logic, thereby achieving more refined and evidence-based sparsity processing and higher filtering accuracy.

[0064] This scheme, building upon prefetching and sparsity filtering, introduces an activation probability weight prediction mechanism based on the current network layer's activation state. This transforms sparsity filtering from indiscriminate uniform filtering into selective filtering based on precise predictions of the actual call probabilities of each computational unit in the next network layer, maximizing parameter data compression while maintaining inference accuracy. Simultaneously, through the SCM's double-buffered pipeline structure and flash memory multi-channel parallel read technology, the time overlap between current layer computation and next layer parameter prefetching is achieved, hiding flash read latency and sparsity processing latency entirely within the computation time. Hardware fence instructions enable instantaneous zero-copy switching of the buffer role, eliminating the switching latency caused by data copying or address remapping in traditional schemes.

[0065] In one embodiment, the real-time computation state of the current network layer is represented by an activation state vector. Step S41, which involves predicting the activation probability weights of each computational unit in the next network layer based on the current real-time computation state and computational topology of the network layer, specifically includes the following steps: S411: Obtain the prediction weight matrix pre-configured for the AI ​​model. The prediction weight matrix is ​​used to establish the mapping relationship between the activation state of the current network layer and the probability of the next network layer being accessed.

[0066] The solid-state drive controller acquires a pre-configured prediction weight matrix for the current AI model. This prediction weight matrix is ​​a two-dimensional numerical matrix used to establish a mathematical mapping relationship between the activation state of each computational unit in the current network layer and the probability of each computational unit being accessed in the next network layer. This prediction weight matrix is ​​typically pre-calculated during the model training phase or the offline analysis phase before model deployment by analyzing the activation propagation characteristics between the model's layers, and is stored as metadata along with the AI ​​model in the flash memory chip.

[0067] For example, for a neural network with a hidden layer dimension of 1024, if the current network layer has 1024 computational units and the next network layer also has 1024 computational units, then the size of the prediction weight matrix is ​​1024×1024, where the element in the i-th row and j-th column represents the influence weight of the activation state of the i-th computational unit in the current layer on the probability of the j-th computational unit in the next layer being accessed. This matrix can be understood as a simplified, linear approximate model of inter-layer activation propagation.

[0068] S412: Based on the activation state vector of the current network layer, use the prediction weight matrix to perform linear mapping calculation to obtain the access score of each computing unit in the next network layer.

[0069] The solid-state drive controller performs matrix multiplication on the predicted weight matrix obtained in step S411 and the activation state vector of the current network layer, i.e., performs linear mapping calculation. The activation state vector is a binary vector with a length equal to the number of computational units in the current network layer, where an element value of 1 indicates that the corresponding computational unit is activated in the current inference, and 0 indicates that it is not activated.

[0070] The mathematical expression for linear mapping calculation is: Next layer access score vector = Current layer activation state vector × Prediction weight matrix. The result is a numerical vector with a length equal to the number of computational units in the next network layer, where each element is the access score (a real value) of the corresponding computational unit, representing the probability that the computational unit will be accessed in the next network layer. The higher the score, the greater the probability of being accessed.

[0071] For example, the current layer activation state vector is [1, 0, 1, 0, ...] (indicating that the 1st and 3rd computation units are activated). The 1st and 3rd rows of the prediction weight matrix describe the influence of these two activation units on the units of the next layer, respectively. After linear mapping calculation, the access scores of each unit of the next layer will be obtained.

[0072] S413: Normalize the access scores of each computational unit in the next network layer to obtain the activation probability weights of each computational unit in the next network layer.

[0073] The solid-state drive controller performs normalization processing on the access scores of each computing unit in the next network layer calculated in step S412, mapping each score value to a probability value between 0 and 1, thereby obtaining the activation probability weight of each computing unit. Here, normalization processing refers to the process of adjusting a set of values ​​to have a uniform scale (such as a sum of 1 or a range of [0,1]) through mathematical transformation.

[0074] In this embodiment, the normalization method used can be Softmax normalization, which involves taking the exponent of each element in the score vector and dividing it by the sum of the exponents of all elements, so that the sum of all probability weights is 1. Alternatively, a simplified normalization method can be used, such as dividing each score by the maximum value in the score vector, making the highest score 1, and scaling the remaining scores proportionally to the [0,1] interval. The normalized activation probability weights directly reflect the probability of each computational unit being activated and invoked in the next layer of inference, and subsequent sparsity filtering will use these weights as the basis for judgment.

[0075] For example, the next layer has 4 computation units with original access scores of [2.0, 0.5, 1.5, 0.0]. After Softmax normalization, the activation probability weights are [0.5, 0.12, 0.35, 0.03], indicating that the first computation unit has a 50% probability of being activated, while the fourth computation unit has only a 3% probability of being activated.

[0076] This solution establishes an extremely low-overhead inter-layer activation propagation approximation model through a pre-configured lightweight prediction weight matrix and simple linear mapping calculations. This allows the SSD controller to quickly predict the activation probabilities of each computational unit in the next network layer within a few clock cycles, without requiring a full model computation capability. Normalization further transforms the raw scores into activation probability weights with clear probabilistic meaning, enabling a unified setting of sparsity thresholds across different layers and models. The entire prediction process is implemented entirely in hardware, with fixed and predictable latency, avoiding additional timing bottlenecks to the existing prefetch pipeline and providing accurate, real-time, and low-overhead decision-making basis for subsequent sparsity filtering.

[0077] In one embodiment, step S43 involves performing sparsity filtering on the target inference parameters read into the cache memory (SCM) to filter out sparse elements in the target inference parameters whose corresponding activation probability weights are lower than a preset threshold, and reconstructing the sparsely filtered target inference parameters. This specifically includes the following steps: S431: By establishing a hard-wired data selection channel, filter out the parameter data of the computational units whose activation probability weights are lower than a preset threshold in the target inference parameters, and obtain the inference parameters of the computational units that have not been filtered out.

[0078] Solid-state drive controllers filter target inference parameters in the SCM (Solid State Cache) through hard-wired data selection channels in their internal hardware processing engine. A hard-wired data selection channel is a data path implemented using dedicated hardware logic circuitry (rather than a programmable processor or software algorithm) that determines which data passes through and which is blocked based on input selection signals. Its characteristics include high processing speed, fixed latency, and no software overhead.

[0079] The solid-state drive controller compares the activation probability weight of each computing unit calculated in steps S41 to S413 with a preset threshold (e.g., 0.1). For computing units with activation probability weights lower than the threshold, their corresponding parameter data is determined to be sparse elements, and the hard-wired data selection channel blocks and filters out these data. For computing units with activation probability weights greater than or equal to the threshold, their corresponding parameter data is determined to be important data, and the hard-wired data selection channel allows these data to pass through, thereby obtaining the inference parameters of the unfiltered computing units.

[0080] For example, if the next network layer has 1024 computational units and the preset threshold is 0.05, it is predicted that the activation probability weights of 300 computational units are lower than 0.05 and the weights of 700 computational units are higher than 0.05. The hard-wired data selection channel will filter out the parameter data corresponding to these 300 low-weight units (such as the corresponding rows in the weight matrix) and only retain the parameter data of the 700 high-weight units, thereby compressing the amount of data to about 68% of the original.

[0081] S432: Merge the inference parameters and their location index information of the unfiltered computing units, reconstruct them into dense matrix data, and use the dense matrix data as the target inference parameters after sparse filtering.

[0082] After filtering in step S431, only some parameters of the computational units in the original sparse parameter matrix are retained. However, the positions of these retained parameters in the original matrix are discrete and non-contiguous. If they are directly transmitted and stored in this sparse format, they will be difficult for the hardware accelerator to utilize efficiently. Therefore, the hardware processing engine of the solid-state drive controller merges and encapsulates the inference parameters of the computational units (i.e., the parameter values ​​themselves) obtained in step S431 with the position index information of these computational units in the original matrix (i.e., the row number, column number, or serial number of the computational unit in the original matrix). Then, these parameter data are reorganized and arranged according to a continuous memory layout to form a dense matrix data—that is, a data structure in which data is stored continuously in memory, without gaps, and is easy to access continuously.

[0083] This dense matrix data is the final sparse filtered target inference parameters. It contains the values ​​of all the important inference parameters that need to be retained and their positional mapping relationships in the original parameter space, enabling the hardware accelerator to correctly restore the parameters to their original calculation positions based on the position index information during subsequent calculations.

[0084] For example, the original weight matrix is ​​8×8 in size. After filtering, only the weight data of rows 1, 3, and 5 are retained. The hardware processing engine stores the weight data of these three rows consecutively in the order of row 1, row 3, and row 5. At the same time, a position index array [1, 3, 5] is constructed to record the position of these rows in the original matrix. The reconstructed dense matrix is ​​3×8 in size. With the small amount of additional overhead of the position index array, the total amount of data is much smaller than the original 8×8 matrix.

[0085] This solution achieves real-time hardware-level filtering of target inference parameters through a hard-wired data selection channel, avoiding the instruction overhead and latency uncertainties associated with software processing. Data filtering is completed with a deterministic hardware latency, ensuring the real-time nature and determinism of the filtering operation and enabling seamless integration with the data reading pipeline. Subsequently, by reconstructing the sparsely filtered parameters into a dense matrix format with positional indexes, the data volume is significantly reduced while ensuring the hardware accelerator can correctly parse and use these parameters. This achieves a balance between data compression and availability, resulting in a significant reduction in the data volume of subsequent bus transmissions and HBM storage, effectively lowering bus bandwidth usage and HBM capacity requirements.

[0086] In one embodiment, in step S30 or step S32, the real-time computing state of the current network layer is obtained in the following way: S301: In terms of spatial synchronization, the cache memory SCM and the host-side HBM are mapped in a globally unified virtual address space determined based on the bus protocol.

[0087] To achieve spatial collaboration between the host and the solid-state drive (SSD), a bus interface that supports high-speed interconnect protocol standards (such as CXL or CCIX) is used to map the SSD's internal high-speed cache memory (SCM) and the host's high-speed bandwidth memory (HBM) into a globally unified virtual address space.

[0088] In this context, a globally unified virtual address space refers to a bus protocol that supports shared memory semantics (such as CXL.mem), enabling the host-side CPU, hardware accelerator, and solid-state drive controller to access physical data distributed in HBM and SCM through the same virtual address, without the need for explicit data copying or address translation.

[0089] Specifically, the bus interface hardware circuitry is responsible for registering the SCM's physical address range into the host's system address mapping table. This allows the host's access requests for specific virtual address ranges to be transparently routed by the bus hardware to the SCM's physical storage units. Similarly, the SSD controller can also access the address regions specified in the HBM through this unified address space. After space synchronization is complete, the host's hardware accelerator and the SSD controller see the same address space view, enabling direct data exchange without host CPU intervention.

[0090] S301: In terms of timing synchronization, the dedicated hardware command stream queue is polled via a hard connection to capture the synchronization token issued by the hardware accelerator when the current network layer computation begins, as a starting signal to trigger the prefetching operation of the target inference parameters of the next network layer. The synchronization token carries the unique identifier of the current network layer and the real-time computation status of the current network layer.

[0091] To achieve timing coordination between the host and the solid-state drive (SSD), the SSD controller continuously checks the status of a dedicated hardware command stream queue via hard-wired polling. Hard-wired polling refers to using dedicated hardware circuitry (rather than CPU polling or interrupt mechanisms) to continuously check the command queue for new data arrivals at fixed clock cycles. This method has extremely low polling latency (typically only a few clock cycles) and the latency is deterministic, avoiding jitter caused by software scheduling.

[0092] When the host-side hardware accelerator begins computation of the current network layer, it writes a synchronization token to a dedicated hardware command stream queue. This synchronization token is a special data packet generated by the hardware accelerator, carrying a unique identifier for the current network layer (e.g., Layer_5) and the real-time computation status of the current network layer (e.g., computation started, computation in progress, or computation completed). Upon detecting this synchronization token in the command stream queue, the hard-wired polling circuitry of the solid-state drive controller immediately captures and parses the information within, using it as the starting signal to trigger the prefetching operation of the target inference parameters for the next network layer.

[0093] This solution constructs a complete master-slave collaboration mechanism through spatial and temporal synchronization. Spatially, a globally unified virtual address space mapping enables zero-copy data sharing between the host and the SSD. The hardware accelerator can directly access SCM data, and the SSD controller can directly write data to HBM, eliminating the overhead of multiple data copies during transit in traditional solutions. Temporally, a hard-wired polling mechanism precisely captures synchronization tokens with nanosecond-level latency, allowing the SSD controller to receive this information the instant the hardware accelerator begins computation at the current layer and immediately initiate prefetching operations for the next layer, maximizing the time overlap window between prefetching and computation operations. This dual collaboration of spatial and temporal synchronization provides a solid underlying foundation for the pipelined operation of the entire HBM tiered cache management system.

[0094] In one embodiment, the high-speed cache memory (SCM) includes a first physical buffer and a second physical buffer that are isolated from each other. The first physical buffer is used to cache the inference parameters of the current network layer, and the second physical buffer is used to write the target inference parameters of the next network layer. After step S50, i.e., after writing the sparsified target inference parameters as inference parameters of the next network layer into the high-speed bandwidth memory (HBM) for caching via a direct bus connection, the method further includes the following steps: S60: When the host completes the calculation of the current network layer and the received synchronization token is updated to the unique identifier of the next network layer, it sends a hardware fence instruction to the bus direct connection channel. Through the mutex lock mechanism, it instantly swaps the global physical address pointers of the first physical buffer and the second physical buffer, realizing zero-copy switching of the master and slave data spaces.

[0095] Based on the dual-buffer structure described in step S42 (the first physical buffer caches the parameters of the current layer, and the second physical buffer writes the parameters of the next layer), after the host-side hardware accelerator completes all inference calculations for the current network layer, the synchronization token received by the SSD controller through the dedicated hardware command stream queue is updated to the unique identifier of the next network layer (indicating that the hardware accelerator is about to switch to the next network layer for computation). At this time, the SSD controller issues a hardware barrier instruction to the bus direct connection channel. This instruction is a synchronization barrier instruction executed by hardware circuitry to ensure that all previous data write operations (i.e., data writes to the second physical buffer) have been completed and are globally visible.

[0096] Subsequently, using a mutex mechanism (a hardware locking mechanism that ensures atomic access to shared resources and prevents multiple execution units from simultaneously modifying address pointers, thus preventing data corruption), the global physical address pointers of the first and second physical buffers are instantaneously swapped within a single bus clock cycle. That is, after the swap, the pointer that previously pointed to the first physical buffer now points to the second physical buffer, and vice versa. This address pointer swapping is zero-copy—the data itself is not moved within the SCM; only the data access mapping is changed.

[0097] This solution utilizes hardware fence instructions and mutex mechanisms to atomically swap the address pointers of the dual buffers in the SCM the instant the current layer computation is completed on the host side, achieving zero-copy buffer role switching. The switching process involves no data movement; the address mapping relationship is changed only within a single bus clock cycle. This avoids the data movement overhead required in traditional solutions, which necessitates copying next-layer parameters from the SCM to the HBM or performing page migrations. This achieves a seamless transition from the current layer to the next, reducing switching latency to the nanosecond level and effectively ensuring continuous computational capabilities during hardware accelerator layer switching.

[0098] In one embodiment, the inference parameters of the AI ​​model include static weight parameters and dynamic key-value cache parameters; the method further includes performing a dynamic management step of swapping out the dynamic key-value cache parameters of each network layer. Wherein, for example... Figure 5 As shown, the dynamic management process for swapping out specifically includes the following steps: S71: During the current network layer computation on the host side, the attention score of each token in the current inference sequence generated by the current network layer computation is extracted periodically using the attention mechanism algorithm.

[0099] During the inference computation of the current network layer (especially the self-attention computation part in the Transformer layer) performed by the hardware accelerator on the host side, the solid-state drive controller monitors the attention score matrix generated by the hardware accelerator in HBM and uses the attention mechanism algorithm to periodically extract the attention score of each token in the current inference sequence.

[0100] In this context, a Token refers to a basic semantic unit obtained after the input text has been segmented; it can be a word, a subword, or a character. The attention score is the attention weight value calculated for each Token relative to all other Tokens during the self-attention mechanism of the Transformer model. It reflects the strength of the semantic association between Tokens; a higher score indicates that the Token is more important in the current context.

[0101] Specifically, in multi-head self-attention computation, each attention head generates an attention weight matrix, where the element in the i-th row and j-th column represents the attention weight of the i-th token in the sequence to the j-th token. The solid-state drive controller periodically (e.g., every inference step or every few inference steps) samples and extracts the attention score corresponding to each token in the attention weight matrix from the HBM through hardware monitoring circuitry. Typically, the sum or average of the attention scores of the token to all other tokens is taken as the overall attention score of the token. These scores reflect the importance of each token in the current inference context and will be used to guide the eviction decision of the dynamic key-value cache parameters.

[0102] For example, when reasoning about a sequence of 2048 tokens, the attention mechanism generates a 2048×2048 attention weight matrix. The solid-state drive controller extracts the average of the attention weights of each token to all other tokens, resulting in 2048 attention scores. Some key tokens (such as subjects, predicates, etc.) tend to have higher scores, while stop words or auxiliary words have lower scores.

[0103] S72: Performs streaming hierarchical quantization on the dynamic key-value cache parameters of the corresponding network layer, dynamically converting high-precision floating-point format data into low-bit integer format data.

[0104] The solid-state drive controller performs streaming hierarchical quantization on the dynamic key-value cache parameters of the current network layer or other specified network layers. The dynamic key-value cache parameters refer to the key and value vectors generated for each token during the self-attention computation of the Transformer model. These vectors are dynamically increased during inference as each new token is generated, and are usually stored in a high-precision floating-point format (such as FP16 or FP32).

[0105] Streaming hierarchical quantization refers to a compression operation that converts high-precision floating-point data into low-bit integer data in real time during data flow (i.e., along the path from HBM to SCM). Each network layer can independently set quantization parameters. Specifically, when the hardware processing engine of the solid-state drive controller receives the dynamic key-value cache parameters that need to be swapped out, it performs the quantization operation online in real time along the data flow (read from HBM → quantization processing → write to SCM).

[0106] The quantization process includes: determining the numerical range (maximum and minimum values) of the current layer key-value cache data, calculating the scaling factor and zero offset based on the target number of bits (such as INT8 or INT4), and then mapping each high-precision floating-point value to the corresponding low-bit integer value.

[0107] S73: Perform iterative calculation of heat decay for the corresponding dynamic key-value cache page based on the attention score. When the heat score of the specified dynamic key-value cache page decays to the preset lifetime boundary, disconnect the page table mapping of the specified dynamic key-value cache page in the high-speed bandwidth memory HBM, and refresh the specified dynamic key-value cache page from the high-speed bandwidth memory HBM to the high-speed cache memory SCM.

[0108] Based on the attention score extracted in step S71, the solid-state drive controller performs iterative calculations on the popularity decay of the dynamic key-value cache page corresponding to each token in HBM. A dynamic key-value cache page refers to a memory page (typically 4KB or 2MB in size) in HBM used to store the dynamic key-value cache parameters of a group of consecutive tokens. Popularity decay iterative calculation is a process that simulates the decay of the access frequency of a cache page over time. In each iteration (i.e., each inference step), the page popularity is updated based on the attention score of the current step, and the popularity value decreases over time according to a certain decay rate.

[0109] Specifically, each dynamic key-value cache page is assigned an initial popularity score, which is based on the attention score of the corresponding token in the newly generated inference step. In each subsequent inference step, the SSD controller performs popularity decay on the page: new popularity = old popularity × decay factor + attention score contribution of the current step.

[0110] When the popularity score of a dynamic key-value cache page continues to decay to below a preset lifetime boundary (i.e., a preset popularity threshold, below which it means that the data on the page is no longer important and can be removed from HBM), the solid-state drive controller disconnects the page table mapping of the cache page in HBM through the hardware address translation circuit (i.e., removes the mapping relationship between virtual address and HBM physical address), and then refreshes (writes back) the data of the page from HBM to the cache memory SCM for persistent storage, thereby freeing up HBM space for other more important cache pages.

[0111] For example, in a long sequence reasoning scenario, the key-value cache of the initial token has a high heat value, but as reasoning progresses (such as when the 1000th token is generated), the attention scores of the first few tokens approach zero, and the heat value of their corresponding cached pages continues to decay. When the heat value of a certain page decays to below the preset lifetime boundary of 0.01, the solid-state drive controller disconnects the HBM page table mapping of that page and swaps its data out to the SCM for storage.

[0112] This solution addresses the HBM capacity bottleneck caused by the linear growth of dynamic key-value cache with sequence length in large language model inference. It proposes a complete cache management mechanism integrating heat awareness, online compression, and intelligent swapping. By extracting attention scores in real-time as a quantitative indicator of token importance and combining this with iterative calculation of heat decay, it achieves fine-grained lifecycle management of the dynamic key-value cache in HBM—allowing high-attention tokens to reside in HBM for extended periods while promptly swapping low-attention tokens to the SCM, thus supporting longer inference sequences within the limited HBM capacity.

[0113] In one embodiment, the method further includes performing low-level physical channel isolation control on the inference data of the AI ​​model; the solid-state drive controller internally includes a flash memory conversion layer, which is used to perform low-level physical channel isolation control on the inference data of the AI ​​model. Specifically, the process of the flash memory conversion layer performing low-level physical channel isolation control on the inference data of the AI ​​model includes the following steps: S81: During the system initialization phase, a portion of the independent physical channels of the flash memory chips are divided into dedicated physical channels for AI inference, and the dedicated physical channels for AI inference are given the highest execution priority so that I / O requests from the regular operating system are prohibited from entering within the dedicated physical channels for AI inference.

[0114] During the system initialization phase (i.e., before system power-on or AI inference tasks begin), the solid-state drive controller uses its internal Flash Translation Layer (FTL) to dedicate a portion of the independent physical channels within the flash memory chips to dedicated physical channels for AI inference. The Flash Translation Layer (FTL) is a core abstraction layer in the solid-state drive controller firmware, responsible for mapping logical addresses issued by the host to physical addresses of the flash memory chips and managing low-level operations such as garbage collection and wear leveling.

[0115] AI inference dedicated physical channels refer to independent channels separated from the multiple physical channels of flash memory chips, specifically used for accessing AI inference data. These channels are isolated from regular storage data physical channels at the hardware level. These dedicated channels are given the highest execution priority; that is, in the bus arbitration and command scheduling within the SSD controller, command requests involving the AI ​​inference dedicated physical channels are always processed with priority over requests from other channels. Simultaneously, the flash translation layer is configured with routing rules to prevent all I / O requests generated by the regular operating system (such as file system read / write requests) from being routed to these AI inference dedicated physical channels, ensuring that AI inference data access is not interfered with by regular storage operations.

[0116] S82: During the lifecycle of inference computation based on the AI ​​model on the host side, suspend the regular garbage collection operation of the flash memory block to which the dedicated physical channel for AI inference belongs.

[0117] Throughout the entire lifecycle of the inference computation task performed on the host based on the AI ​​model (i.e., from the start of inference to the end of inference), the flash translation layer of the SSD controller suspends (suspends) the regular garbage collection operation on all flash blocks managed by the dedicated physical channel for AI inference. Garbage collection is a background maintenance operation in NAND flash storage used to move valid data from one flash block to another and then erase the original flash block to free up storage space. This operation consumes a large amount of read, write, and erase resources, and its execution time and resource consumption are uncertain.

[0118] Because AI inference is extremely sensitive to latency, garbage collection operations can introduce unpredictable latency jitter. Therefore, during the inference lifetime, FTL marks all flash memory blocks in the dedicated physical channel as paused, meaning that no valid data migration or erasure operations are triggered or performed on these flash memory blocks. Normal garbage collection of these flash memory blocks is resumed after the inference lifetime ends.

[0119] S83: When a write-back request from the HBM (High-Speed ​​Cache Memory) to swap out dynamic inference parameters to the solid-state drive is intercepted, the swapped-out parameters are treated as continuous sequential large blocks of data and directly allocated to a pre-erased designated flash superblock via the dedicated physical channel for AI inference to perform a non-overwrite append operation, thereby eliminating read / write timing conflicts at the physical media level.

[0120] When step S73 or other swap-out mechanisms trigger the write-back (swap-out) of dynamic inference parameters from HBM to the SSD, the SSD controller intercepts the write-back request. The controller organizes the parameter data to be swapped out into contiguous sequential large blocks of data (i.e., data blocks arranged in a continuous memory address order, rather than randomly scattered small data blocks), and then directly allocates and writes this data into the designated flash superblocks that have been pre-erased in the flash memory chips through the previously allocated dedicated physical channels for AI inference.

[0121] A flash superblock is a logical block collection composed of flash memory blocks on multiple physical channels. It improves data throughput by accessing multiple physical channels in parallel. Non-overwrite append operations refer to a writing method where newly written data does not overwrite existing data. Instead, it is written to the next available blank page in the flash memory block. Each write operation appends to existing data, rather than modifying or overwriting it. This writing method eliminates read / write timing conflicts caused by erasing before writing in flash memory (because flash memory cannot overwrite in place and requires erasing the entire block before writing).

[0122] This solution constructs a completely isolated and dedicated optimized storage channel path for AI inference data access at the underlying physical resource management level of solid-state drives (SSDs). Through hardware isolation and highest priority guarantees of the physical channel, AI inference data access obtains dedicated resources unaffected by conventional storage I / O, ensuring deterministic latency and stable quality of service for prefetch operations. By suspending garbage collection operations on the dedicated AI inference physical channel, the uncertain latency jitter that garbage collection might introduce is completely eliminated, ensuring stable performance for data access during inference. Through non-overwrite append write operations and pre-erased superblock allocation, the latency overhead of erasing operations required before flash memory writing is eliminated, enabling deterministic, low-latency write performance for AI inference parameter swapping. These three elements work synergistically to completely eliminate read / write timing conflicts and resource contention at the physical media level, providing stable, efficient, and predictable underlying storage support for the entire HBM tiered caching management system.

[0123] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0124] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0125] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0126] This application provides a computer program product that, when run on an electronic device, enables the electronic device to perform the steps described in the various method embodiments above.

[0127] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographing device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0128] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0129] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0130] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for HBM hierarchical caching management of AI inference parameters, characterized in that, The method is applied to an HBM hierarchical cache management system consisting of a host and a solid-state drive. The host includes a hardware accelerator and a high-speed bandwidth memory (HBM). The solid-state drive includes a solid-state drive controller, a high-speed cache memory (SCM), and flash memory chips. The AI ​​model is cached in the flash memory chips. The high-speed bandwidth memory (HBM) is used to cache the inference parameters of the currently running network layer in the AI ​​model for use by the hardware accelerator. The method is executed by the solid-state drive controller, including: A dedicated hardware command stream queue is established with the host to monitor the real-time execution status of the AI ​​model on the host, thereby achieving dual-end timing and spatial collaborative control between the host and the solid-state drive. Obtain the computational topology of the AI ​​model sent by the host, wherein the computational topology is used to characterize the operator connection relationship and execution sequence of each network layer in the AI ​​model; Based on the aforementioned computing topology, the execution status of the AI ​​model on the host side is monitored through the dedicated hardware command stream queue to obtain the real-time computing status of the currently running network layer. Based on the real-time computing state of the current network layer, the inference parameters of the next network layer are read and sparsified to obtain the sparsed target inference parameters. The sparsed target inference parameters are written to the high-speed bandwidth memory (HBM) cache via a direct bus connection channel as inference parameters for the next network layer, so that the hardware accelerator can directly call them when switching to the next network layer for inference.

2. The method according to claim 1, characterized in that, Based on the computational topology, the process of monitoring the execution status of the AI ​​model on the host side through the dedicated hardware command stream queue to obtain the real-time computational status of the currently running network layer includes: Based on the computational topology, the access order of inference parameters in different network layers of the AI ​​model is predicted to obtain a deterministic memory access sequence of inference parameters in the AI ​​model. The deterministic memory access sequence includes the access order and physical storage address of inference parameters in different network layers. When the host executes a computation task based on the AI ​​model, the network layer of the currently running AI model in the host is monitored through the dedicated hardware command stream queue based on the deterministic memory access sequence to obtain the real-time computation status of the currently running network layer.

3. The method according to claim 2, characterized in that, Based on the real-time computing state of the currently running network layer, the inference parameters of the next network layer are read and sparsely filtered to obtain the sparsely filtered target inference parameters, including: Based on the real-time computing state and computing topology of the current network layer, the activation probability weight of each computing unit in the next network layer is predicted, and the physical storage address of the next network layer is determined according to the deterministic memory access sequence. While performing network computation for the current layer on the host side, a prefetching operation for the target inference parameters in the next network layer is initiated based on the physical storage address of the next network layer, and the prefetched target inference parameters are read from the flash memory chip into the cache memory (SCM). The target inference parameters read into the high-speed cache memory (SCM) are subjected to sparsity filtering to remove sparse elements in the target inference parameters whose activation probability weights are lower than a preset threshold, and the sparsely filtered target inference parameters are reconstructed.

4. The method according to claim 3, characterized in that, The step of performing sparse filtering on the target inference parameters read into the high-speed cache memory (SCM) to filter out sparse elements in the target inference parameters whose corresponding activation probability weights are lower than a preset threshold, and reconstructing the sparsely filtered target inference parameters, includes: By establishing a hard-wired data selection channel, the parameter data of the calculation units whose activation probability weight is lower than a preset threshold in the target inference parameters are filtered out, and the inference parameters of the unfiltered calculation units are obtained. The inference parameters and their location index information of the unfiltered computing units are merged and reconstructed to form dense matrix data, which is then used as the target inference parameters after sparse filtering.

5. The method according to claim 1, characterized in that, The real-time computing state of the currently running network layer is obtained in the following way: In terms of spatial synchronization, the high-speed cache memory (SCM) and the host-side HBM are mapped within a globally unified virtual address space determined based on the bus protocol. In terms of timing synchronization, the dedicated hardware command stream queue is polled via hardwired to capture the synchronization token issued by the hardware accelerator when the current network layer computation begins, which serves as the starting signal for triggering the prefetching operation of the target inference parameters of the next network layer. The synchronization token carries the unique identifier of the current network layer and the real-time computation status of the current network layer.

6. The method according to any one of claims 1-5, characterized in that, The method further includes performing low-level physical channel isolation control on the inference data of the AI ​​model, including: During the system initialization phase, a portion of the independent physical channels of the flash memory chips are divided into dedicated physical channels for AI inference through a flash memory conversion layer, and the dedicated physical channels for AI inference are given the highest execution priority so as to prohibit I / O requests from the regular operating system from entering within the dedicated physical channels for AI inference. During the lifetime of inference calculations based on the AI ​​model on the host side, the regular garbage collection operation of the flash memory block to which the dedicated physical channel for AI inference belongs is suspended. When a write-back request from the High Speed ​​Cache (HBM) to the Solid State Drive to swap out dynamic inference parameters is intercepted, the swapped-out parameters are treated as continuous sequential large blocks of data and directly allocated to a pre-erased designated flash superblock via the AI ​​inference dedicated physical channel for non-overwrite append write operations, thereby eliminating read / write timing conflicts at the physical media level.

7. An HBM hierarchical caching management system for AI inference parameters, characterized in that, The system includes a host and a solid-state drive (SSD). The host includes a hardware accelerator and a high-bandwidth memory (HBM). The SSD includes a solid-state drive controller, a high-speed cache memory (SCM), and flash memory chips. The AI ​​model is cached in the flash memory chips. The high-bandwidth memory (HBM) is used to cache the inference parameters of the currently running network layer in the AI ​​model for the hardware accelerator to access. The solid-state drive controller is used for: A dedicated hardware command stream queue is established with the host to monitor the real-time execution status of the AI ​​model on the host, thereby achieving dual-end timing and spatial collaborative control between the host and the solid-state drive. Obtain the computational topology of the AI ​​model sent by the host, wherein the computational topology is used to characterize the operator connection relationship and execution sequence of each network layer in the AI ​​model; Based on the aforementioned computing topology, the execution status of the AI ​​model on the host side is monitored through the dedicated hardware command stream queue to obtain the real-time computing status of the currently running network layer. Based on the real-time computing state of the current network layer, the inference parameters of the next network layer are read and sparsified to obtain the sparsed target inference parameters. The sparsed target inference parameters are written to the high-speed bandwidth memory (HBM) cache via a direct bus connection channel as inference parameters for the next network layer, so that the hardware accelerator can directly call them when switching to the next network layer for inference.

8. The system according to claim 7, characterized in that, The solid-state drive controller integrates a hardware processing engine and bus interface. The solid-state drive controller initiates a prefetch operation on the target inference parameters in the next network layer based on the physical storage address of the next network layer, and reads the prefetched target inference parameters from the flash memory chip into the cache memory (SCM). Then, the hardware processing engine performs hardware-level sparsity filtering on the target inference parameters in the cache memory (SCM) to reconstruct the sparsely filtered target inference parameters. The bus interface is used to write the sparsely filtered target inference parameters into the high-speed bandwidth memory (HBM) cache via a direct bus connection channel, so that the hardware accelerator can directly call it when switching to the next network layer for inference.

9. The system according to claim 8, characterized in that, The bus interface supports high-speed interconnect protocol standards to map the high-speed cache memory (SCM) and the high-speed bandwidth memory (HBM) in a globally unified virtual address space. The hardware accelerator is used to issue a synchronization token with a unique identifier of the current network layer to the dedicated hardware command stream queue when the current network layer computation begins. The synchronization token also carries the real-time computation status of the currently running network layer. The solid-state drive controller triggers the capture of the synchronization token by polling the dedicated hardware command stream queue via a hardwired connection.

10. The system according to any one of claims 7-9, characterized in that, The solid-state drive controller internally includes a flash memory conversion layer, which is used for: During the system initialization phase, a portion of the independent physical channels of the flash memory chips are divided into dedicated physical channels for AI inference, and the dedicated physical channels for AI inference are given the highest execution priority so that I / O requests from the regular operating system are prohibited from entering the dedicated physical channels for AI inference. During the lifetime of inference calculations based on the AI ​​model on the host side, the regular garbage collection operation of the flash memory block to which the dedicated physical channel for AI inference belongs is suspended. When a write-back request from the High Speed ​​Cache (HBM) to the Solid State Drive to swap out dynamic inference parameters is intercepted, the swapped-out parameters are treated as continuous sequential large blocks of data and directly allocated to a pre-erased designated flash superblock via the AI ​​inference dedicated physical channel for non-overwrite append write operations, thereby eliminating read / write timing conflicts at the physical media level.

Images