Data management method, device and equipment and readable storage medium

Abstract

The invention provides a data management method, device and equipment and a readable storage medium, and the method comprises the steps: obtaining key value cache data generated by a first type of computing power unit in a pre-filling stage, transmitting the key value cache data to a memory pool, and caching the key value cache data in the memory pool; and in response to a data acquisition demand of the second type of computing power unit in a decoding stage, transmitting the specified key value cache data cached in the memory pool to the second type of computing power unit according to the data acquisition demand. According to the technical scheme of the invention, the memory pool is arranged and the aggregated transmission channel is utilized, so that the transmission efficiency of the key value cache data between the computing power devices is effectively improved, the data transmission delay is reduced, and the network transmission protocol overhead is reduced, thereby improving the data interaction efficiency of the computing power devices in a pre-filling stage and a decoding stage, and improving the user experience. The utilization of computing power resources is optimized, and the throughput and performance of large model reasoning are improved.

Description

Technical Field

[0001] This specification relates to the field of communication technology, and in particular to a data management method, apparatus, device, and readable storage medium. Background Art

[0002] The inference process for large models presents challenges in terms of high demands on computing resources and memory capacity. These large models are typically based on the Transformer architecture, where the attention mechanism is one of the most computationally expensive components. Traditional inference methods require recalculating the attention of all previous tokens every time a new token is generated, resulting in a large amount of redundant computation. KV Cache technology caches previous Key and Value (KV) values, allowing new token predictions to be read directly from the cache, thus avoiding duplicate computation and effectively optimizing this process.

[0003] The inference process of a large language model (LLM) is generally divided into two stages: Prefill and Decode. The Prefill stage is a compute-intensive process, primarily responsible for processing user input and calculating the corresponding KV cache (key-value cache). This stage requires a large amount of computing power to quickly generate initial context information. In contrast, the Decode stage is a memory-intensive process, primarily generating tokens one at a time, calculating only one token at a time, and therefore requiring high memory bandwidth and access speed. However, when processing large-scale models using traditional GPU and memory architectures, there is often an imbalance between computing power and memory capacity. Although high-computing-power GPUs can handle complex calculations, their limited memory capacity cannot meet the data and parameter requirements for inference on large models.

[0004] In order to address the above challenges, a PD (Prefill-Decode) separation architecture is proposed. By deploying the Prefill stage and the Decode stage on different machine resources, computing power and memory resources can be more effectively utilized to improve overall system performance. Although this method improves resource utilization, when the KV Cache (i.e., KV cache, key-value cache) is migrated from the Prefill stage to the Decode stage, data transmission is inevitably involved. This process will introduce additional delays and computing overhead. In particular, in the existing technical solutions, the method of transmitting KvCache through the network is difficult to meet the low-latency requirements of the inference business because the delay of network transmission is larger than the delay of memory semantic access. Summary of the Invention

[0005] In view of this, this specification provides a data management method, device, electronic device, and readable storage medium to improve the above-mentioned problem of insufficient data transmission performance.

[0006] The specific technical solutions are as follows:

[0007] This specification provides a data management method, which is applied to a computing power device, wherein the computing power device includes a first type of computing power unit applied to several pre-filling stages and several second type of computing power units applied to the decoding stage. The computing power device also includes a memory pool. The method includes: obtaining key-value cache data generated by the first type of computing power unit in the pre-filling stage, transferring the key-value cache data to the memory pool and caching it in the memory pool; in response to the data acquisition requirements of the second type of computing power unit in the decoding stage, transferring the specified key-value cache data cached in the memory pool to the second type of computing power unit according to the data acquisition requirements, so that the second type of computing power unit performs calculations based on the specified key-value cache data; the memory pool includes several aggregated memory units, the memory pool and the first type of computing power unit transmit data through an aggregated first data transmission channel, and the memory pool and the second type of computing power unit transmit data through an aggregated second data transmission channel.

[0008] As a technical solution, the key-value cache data generated by the first type of computing power unit in the pre-filling stage is obtained, the key-value cache data is transferred to the memory pool and cached in the memory pool, including: counting the active parameters of the key-value cache data generated by the first type of computing power unit, caching the key-value cache data with lower active parameters to the memory pool according to a preset ratio, and deleting the key-value cache data with lower active parameters from the cache unit of the first type of computing power unit.

[0009] As a technical solution, the response to the data acquisition requirements of the second type of computing power unit in the decoding stage, the specified key-value cache data cached in the memory pool is transferred to the second type of computing power unit according to the data acquisition requirements, including: responding to the data acquisition requirements of the second type of computing power unit in the decoding stage, determining the specified key-value cache data according to the data acquisition requirements; if the specified key-value cache data is cached in the memory pool, transferring the specified key-value cache data cached in the memory pool to the second type of computing power unit; if the specified key-value cache data is cached in the cache unit of the first type of computing power unit, caching the specified key-value cache data cached in the cache unit of the first type of computing power unit to the memory pool, and then transferring the specified key-value cache data cached in the memory pool to the second type of computing power unit.

[0010] As a technical solution, the activity parameter is positively correlated with the access frequency of the associated key-value cache data.

[0011] As a technical solution, the computing requirements of the currently executed computing task are obtained, the hardware attributes of each first-class computing power unit and second-class computing power unit are obtained, a configuration target is generated based on the computing requirements and hardware attributes, and a specific first-class computing power unit is configured as a second-class computing power unit and / or a specific second-class computing power unit is configured as a first-class computing power unit based on the configuration target.

[0012] This specification also provides a data management device, which is applied to a computing power device, wherein the computing power device includes a first type of computing power unit applied to several pre-filling stages and several second type of computing power units applied to the decoding stage. The computing power device also includes a memory pool, and the device includes: a first module, which is used to obtain key-value cache data generated by the first type of computing power unit in the pre-filling stage, transfer the key-value cache data to the memory pool and cache it in the memory pool; a second module, which is used to respond to the data acquisition requirements of the second type of computing power unit in the decoding stage, and transfer the specified key-value cache data cached in the memory pool to the second type of computing power unit according to the data acquisition requirements, so that the second type of computing power unit performs calculations based on the specified key-value cache data; the memory pool includes several aggregated memory units, and the memory pool and the first type of computing power unit transmit data through an aggregated first data transmission channel, and the memory pool and the second type of computing power unit transmit data through an aggregated second data transmission channel.

[0013] As a technical solution, the key-value cache data generated by the first type of computing power unit in the pre-filling stage is obtained, the key-value cache data is transferred to the memory pool and cached in the memory pool, including: counting the active parameters of the key-value cache data generated by the first type of computing power unit, caching the key-value cache data with lower active parameters to the memory pool according to a preset ratio, and deleting the key-value cache data with lower active parameters from the cache unit of the first type of computing power unit.

[0014] As a technical solution, the response to the data acquisition requirements of the second type of computing power unit in the decoding stage, the specified key-value cache data cached in the memory pool is transferred to the second type of computing power unit according to the data acquisition requirements, including: responding to the data acquisition requirements of the second type of computing power unit in the decoding stage, determining the specified key-value cache data according to the data acquisition requirements; if the specified key-value cache data is cached in the memory pool, transferring the specified key-value cache data cached in the memory pool to the second type of computing power unit; if the specified key-value cache data is cached in the cache unit of the first type of computing power unit, caching the specified key-value cache data cached in the cache unit of the first type of computing power unit to the memory pool, and then transferring the specified key-value cache data cached in the memory pool to the second type of computing power unit.

[0015] As a technical solution, the activity parameter is positively correlated with the access frequency of the associated key-value cache data.

[0016] As a technical solution, it also includes a third module, which is used to obtain the computing requirements of the currently executed computing task, obtain the hardware attributes of each first-class computing power unit and second-class computing power unit, generate configuration targets based on the computing requirements and hardware attributes, and configure a specific first-class computing power unit as a second-class computing power unit according to the configuration target, and / or configure a specific second-class computing power unit as a first-class computing power unit.

[0017] This specification also provides an electronic device, including a processor and a readable storage medium, wherein the readable storage medium stores machine-executable instructions that can be executed by the processor, and the processor executes the machine-executable instructions to implement the aforementioned data management method.

[0018] This specification also provides a readable storage medium, which stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the aforementioned data management method.

[0019] The above technical solutions provided in this specification bring at least the following beneficial effects:

[0020] By setting up a memory pool and utilizing aggregated transmission channels, the transmission efficiency of key-value cache data between computing devices is effectively improved, the data transmission delay is reduced, and the network transmission protocol overhead is reduced, thereby improving the data interaction efficiency of computing devices in the pre-filling and decoding stages, optimizing the utilization of computing resources, and improving the throughput and performance of large model inference. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the implementation methods of this specification or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the implementation methods of this specification or the description of the prior art. Obviously, the drawings described below are only some implementation methods recorded in this specification. For ordinary technicians in this field, other drawings can also be obtained based on these drawings of the implementation methods of this specification.

[0022] Figure 1 is a flow chart of a data management method in one embodiment of this specification;

[0023] Figure 2 is a structural diagram of a data management device in one embodiment of this specification;

[0024] Figure 3 is a structural diagram of a data management device in one embodiment of this specification;

[0025] Figure 4 is a hardware structure diagram of an electronic device in one embodiment of this specification;

[0026] Figure 5 It is a schematic diagram of the architecture in one implementation of this specification.

[0027] Reference numerals: first module 21 , second module 22 , third module 23 . DETAILED DESCRIPTION

[0028] The terms used in the embodiments of this specification are only for the purpose of describing specific embodiments and are not intended to limit this specification. The singular forms "a," "the," and "the" used in this specification and claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to any or all possible combinations of one or more of the associated listed items.

[0029] It should be understood that although the terms first, second, third, etc. may be used in the embodiments of this specification to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of this specification, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" may also be interpreted as "when...", "when...", or "in response to determining."

[0030] The inference architecture can use a single GPU device to simultaneously undertake the computational tasks of the prefill and decode stages. This coupled design has gradually exposed significant performance bottlenecks in practice. The prefill stage requires forward calculation of the complete prompt sequence input by the user, generates the key-value vectors (KV) of each Transformer layer through a multi-head attention mechanism, and temporarily stores these intermediate results in the form of a key-value cache (KV Cache) in the video memory. The computational characteristics of this stage are characterized by intensive matrix operations, which have extremely high demands on GPU computing resources. The decoding stage needs to generate output tokens one by one in an autoregressive manner based on the KV Cache. Although this process has low computational intensity, it requires high-frequency access to the KV Cache data in the video memory, which creates continuous pressure on the video memory bandwidth.

[0031] In one technical solution, a single GPU architecture cannot effectively coordinate the differentiated hardware resource requirements of these two stages. Specifically, the pre-filling stage requires a high-computing-power GPU due to its computationally intensive nature, but the memory bandwidth of such devices often becomes a bottleneck that restricts the generation speed during the decoding stage. Conversely, if a GPU device with high-bandwidth memory is used, its computing power resources will remain idle for a long time during the decoding stage, resulting in low hardware utilization. This resource mismatch is particularly prominent in long text generation scenarios. For example, when processing contexts exceeding 4K tokens, the memory usage of the KV Cache may reach tens of GB. The above architecture not only faces the risk of insufficient memory capacity, but also significantly increases the token generation delay due to memory bandwidth limitations.

[0032] If a multi-GPU distributed inference solution is adopted, the computing load of the pre-population phase can be shared through parallel computing, or the KV Cache storage space can be expanded using video memory aggregation technology. However, this solution still has fundamental flaws: First, statically allocated computing resources cannot adapt to the dynamically changing load characteristics of the pre-population and decoding phases, and are prone to resource idleness or contention conflicts when batch requests fluctuate. Second, the data transmission delay and bandwidth limitations across devices in the traditional PCIe bus architecture make it difficult for the remote access efficiency of the KV Cache to meet the real-time requirements of the decoding phase. Third, the existing system lacks a coordinated scheduling mechanism for heterogeneous computing units (such as high-computing GPUs and high-bandwidth video memory devices), resulting in insufficient hardware resource pooling. Especially in scenarios supporting ultra-long context generation, the linear growth characteristics of the KV Cache further exacerbate the fragmentation of video memory resources. Existing video memory management strategies cannot achieve dynamic memory expansion and efficient recycling across devices, seriously restricting the scalability of the system.

[0033] In view of this, this specification provides a data management method, device, electronic device, and readable storage medium to improve at least one of the above technical problems.

[0034] The specific technical solution is described below.

[0035] In one embodiment, this specification provides a data management method applied to a computing power device, wherein the computing power device includes a first type of computing power unit applied to several pre-filling stages and several second type of computing power units applied to the decoding stage. The computing power device also includes a memory pool. The method includes: obtaining key-value cache data generated by the first type of computing power unit in the pre-filling stage, transferring the key-value cache data to the memory pool and caching it in the memory pool; in response to the data acquisition requirements of the second type of computing power unit in the decoding stage, transferring the specified key-value cache data cached in the memory pool to the second type of computing power unit according to the data acquisition requirements, so that the second type of computing power unit performs calculations based on the specified key-value cache data; the memory pool includes several aggregated memory units, the memory pool and the first type of computing power unit transmit data through an aggregated first data transmission channel, and the memory pool and the second type of computing power unit transmit data through an aggregated second data transmission channel.

[0036] Specifically, if Figure 1 , including the following steps:

[0037] This is primarily used in computing devices, which contain two types of computing units: the first type is responsible for pre-population computing tasks, while the second type focuses on decoding tasks. To optimize the efficiency of data exchange between these two types of computing units, a memory pool is introduced to store and manage the key-value cache data generated by the first type of computing units. The memory pool can use the CXL protocol, specifically CXL 3.0.

[0038] Step S11: obtain the key-value cache data generated by the first type of computing power unit in the pre-filling stage, transfer the key-value cache data to the memory pool and cache it in the memory pool.

[0039] After the first type of computing unit completes the pre-filling phase, it transfers the generated key-value cache data to the memory pool and caches it there. This design ensures that the context information required for the subsequent decoding phase can be efficiently stored and quickly accessed.

[0040] Step S12: In response to the data acquisition requirements of the second type computing power unit in the decoding phase, the specified key-value cache data cached in the memory pool is transmitted to the second type computing power unit according to the data acquisition requirements.

[0041] During the decoding phase, the second-class computing units need to gradually generate output tokens based on user input. At this point, they issue data retrieval requests to the memory pool for specific key-value cache data. Upon receiving these requests, the memory pool retrieves the corresponding key-value cache data from its cache based on the second-class computing unit's needs and transmits it to the second-class computing unit via a dedicated data transmission channel.

[0042] This process not only speeds up data access but also significantly reduces the latency and overhead associated with network transmission. For example, in a large-scale language model application, a user might enter a text message as a question or command. The first type of computing unit processes this input, generating the necessary contextual information and storing it in a memory pool. The second type of computing unit then uses this contextual information to generate a response, word by word, ensuring a smooth and efficient process.

[0043] The design of the memory pool is one of the core elements of this solution. It includes several aggregated memory units that together form an efficient storage resource pool. The memory pool is connected to the first type of computing power unit via an aggregated first data transmission channel, allowing the key-value cache data generated by the first type of computing power unit to be quickly transmitted to the memory pool. Similarly, the memory pool is connected to the second type of computing power unit via an aggregated second data transmission channel, ensuring that the second type of computing power unit can quickly obtain the required key-value cache data when needed. This design greatly improves the efficiency of data transmission, reduces waiting time, and thus improves the response speed of the entire system. The first data transmission channel and the second data transmission channel connection described in this paragraph can be an aggregation of PCIe interfaces.

[0044] Imagine a large model processing a series of complex natural language understanding tasks. The first type of computing unit preprocesses the input for each task and stores the results in a memory pool. The second type of computing unit then processes these tasks sequentially, each time extracting only the contextual information relevant to the task from the memory pool before beginning work. This process arrangement allows the system to maintain high efficiency even in the face of a large number of concurrent requests.

[0045] This implementation also supports a KVCache hierarchical unloading strategy, which means that the memory pool can dynamically adjust its storage method based on the access frequency of different KV cache data. For those KV cache data that are frequently accessed, the system will choose to store them in high-bandwidth memory (such as HBM), while for data with lower access frequency, it can choose to migrate them to conventional memory. Doing so not only maximizes the use of limited high-bandwidth memory resources, but also ensures that system performance is not affected. In addition, due to the use of CXL technology, multiple low-computing power GPUs can share resources in the memory pool through aggregated CXL links, which not only solves the problem of limited bandwidth of a single PCIe interface, but also achieves linear growth in bandwidth, making large-scale parallel reasoning possible. For example, in an actual application scenario, if inference requests from multiple clients are to be processed simultaneously, the system can ensure that all requests are responded to in a timely manner by reasonably allocating the workload of the first and second types of computing power units, and utilizing the efficient data management and transmission mechanism provided by the memory pool.

[0046] This implementation allows for asynchronous reading and writing of KV caches, and can asynchronously write or read KV cache data to or from the memory pool without affecting other operations. This is crucial to improving the overall efficiency of the system, especially when processing high-concurrency requests. Through the dedicated CXL asynchronous read and write interface, the large model framework layer can asynchronously read and write KV cache data in the memory pool at a CXL bandwidth close to the peak. In this way, both the first and second category computing power units can use the resources in the memory pool more flexibly according to their needs without having to worry about performance bottlenecks caused by synchronous operations. For example, when performing complex dialogue tasks, the system may need to continuously update and query KV cache data. The use of an asynchronous read and write mechanism can significantly reduce the time required for these operations, making the dialogue process smoother.

[0047] In one embodiment, the data management method is applied to a special computing device. The internal architecture of the computing device is sophisticatedly designed and includes two types of computing units with different functions and a key memory pool component. Among them, the first type of computing units is specially optimized for the Prefill stage of large model inference. They have powerful computing capabilities and can quickly generate initial context information when the model processes user input and calculate the corresponding KV cache data. The second type of computing units focuses on tasks in the Decode stage. This stage has extremely high requirements for memory bandwidth and access speed. It is mainly responsible for sequentially generating tokens and calculating only one token at a time to achieve efficient decoding. The two types of computing units each perform their duties and jointly promote the efficient execution of large model inference tasks.

[0048] In this architecture, the memory pool is not only the storage center for KV cache data, but also a key bridge for data transmission and sharing. The memory pool is composed of multiple aggregated memory units. This design enables the memory pool to provide storage capacity and bandwidth far exceeding that of a traditional single memory unit, thereby meeting the needs of massive data storage and fast access during large-model inference. The memory pool is connected to the first type of computing power unit through an aggregated first data transmission channel, ensuring that the key-value cache data generated in the Prefill phase can be quickly and stably transmitted to the memory pool and cached. At the same time, the memory pool is also connected to the second type of computing power unit through an aggregated second data transmission channel, so that in the Decode phase, when the second type of computing power unit needs to obtain KV cache data, it can quickly read the required data from the memory pool, greatly improving data transmission efficiency.

[0049] In actual applications, when a large model inference task is initiated, the first type of computing power unit intervenes first and is responsible for handling tasks in the Prefill phase. They receive user input, perform complex computational operations, and generate initial context information and corresponding KV cache data. These KV cache data are crucial for the subsequent decoding process. They contain the key and value values that the model has calculated during the inference process, and can avoid repeatedly calculating the attention of all previous tokens when generating new tokens, thereby significantly reducing the computational cost. The generated key-value cache data is then transmitted to the memory pool through the aggregated first data transmission channel and cached in the memory pool. The memory pool uses its efficient storage management mechanism to ensure that this data can be stored quickly and organized in an easily accessible manner so that it can be efficiently retrieved in the subsequent decoding phase.

[0050] When the model enters the Decode phase, the second type of computing units comes into play. These units are specifically optimized for memory-intensive tasks. They can efficiently read KV cache data from the memory pool and perform decoding calculations based on this data to generate new tokens. Whenever a second type of computing unit needs to obtain KV cache data, it sends a data retrieval request to the memory pool. Upon receiving this request, the memory pool responds quickly, extracting the specified key-value cache data from the cache and transmitting it to the corresponding second type of computing unit via the aggregated second data transmission channel. This efficient data transmission mechanism ensures a smooth decoding process, enabling the model to quickly generate subsequent tokens, thereby improving the efficiency of the entire inference process.

[0051] In order to further optimize data transmission efficiency and storage resource utilization, this data management method also adopts a variety of advanced technical strategies. For example, KV cache hierarchical offloading technology. In large-model multi-round dialogue interaction scenarios, there are significant differences in the access frequencies of different contexts. Some contexts may be frequently accessed in multiple rounds of dialogue, while others may be less frequently used. By dynamically counting the access frequency of the KV cache, this method can intelligently unload context data with lower access frequencies to the ordinary storage area in the memory pool, while retaining the KV cache with higher access frequencies in the high-bandwidth memory (HBM). This not only saves valuable HBM space, but also avoids recalculation caused by excessive cached data, thereby effectively saving GPU computing resources.

[0052] In a PD-separated architecture, multiple GPU server hosts can simultaneously connect to the CXL memory pool by utilizing separate CXL memory pool devices. This design aggregates the bandwidth of the CXL link, breaking the bandwidth bottleneck of a single PCIe link. The collaborative operation of multiple CXL memory modules significantly improves overall bandwidth, enabling even low-power GPUs to access the KV cache data in the memory pool with high bandwidth. Compared to network transmission, CXL technology provides memory semantic access with lower latency and less network transmission protocol overhead, which is a significant advantage for latency-sensitive large-model inference. For example, each low-power GPU server node can achieve a theoretical bidirectional bandwidth of 64GB / s via an x8 CXL link. By aggregating the bandwidth of multiple GPUs and CXL memory devices, the system achieves linear growth in bandwidth capacity, providing virtually unlimited heterogeneous memory bandwidth, thus meeting the stringent data transmission bandwidth requirements of large-scale models.

[0053] Multi-GPU high-concurrency generation technology further improves the overall performance of the system. In traditional inference architectures, the imbalance in computing power between the Prefill and Decode stages can easily lead to low GPU computing power utilization. By introducing the CXL memory pool, multiple GPUs can achieve parallel processing on multiple decoding tasks. Combined with dynamic task scheduling strategies, the system can efficiently handle high-concurrency inference requests. Multiple low-computing-power GPUs can simultaneously obtain KV cache data from the memory pool and execute decoding tasks in parallel during the Decode stage. This concurrent access mechanism not only improves generation throughput, but also ensures that the computing power of high-computing-power GPUs is fully utilized, allowing the entire system to maintain good performance under high inference loads.

[0054] In one embodiment, the method of obtaining the key-value cache data generated by the first type of computing power unit in the pre-filling stage, transferring the key-value cache data to the memory pool and caching it in the memory pool includes: counting the activity parameters of the key-value cache data generated by the first type of computing power unit, caching the key-value cache data with lower activity parameters to the memory pool according to a preset ratio, and deleting the key-value cache data with lower activity parameters from the cache unit of the first type of computing power unit.

[0055] In one embodiment, the response to the data acquisition requirements of the second type of computing power unit in the decoding stage and the transfer of the specified key-value cache data cached in the memory pool to the second type of computing power unit according to the data acquisition requirements include: responding to the data acquisition requirements of the second type of computing power unit in the decoding stage and determining the specified key-value cache data according to the data acquisition requirements; if the specified key-value cache data is cached in the memory pool, the specified key-value cache data cached in the memory pool is transferred to the second type of computing power unit; if the specified key-value cache data is cached in the cache unit of the first type of computing power unit, the specified key-value cache data cached in the cache unit of the first type of computing power unit is cached in the memory pool, and then the specified key-value cache data cached in the memory pool is transferred to the second type of computing power unit.

[0056] In one embodiment, the activity parameter is positively correlated with the access frequency of the associated key-value cache data.

[0057] In one embodiment, the computing requirements of the currently executed computing task are obtained, the hardware attributes of each first-class computing power unit and second-class computing power unit are obtained, a configuration target is generated based on the computing requirements and the hardware attributes, and a specific first-class computing power unit is configured as a second-class computing power unit and / or a specific second-class computing power unit is configured as a first-class computing power unit based on the configuration target.

[0058] The problem of flexible computing power allocation under the PD separation architecture is taken into consideration. By configuring the global scheduler, the proportion of GPU resources used in the Prefill and Decode stages can be dynamically adjusted according to the actual computing power of each GPU. This method not only helps to balance the computing needs between different stages, but also effectively avoids resource waste. For example, in some cases, if it is found that the amount of computing in a certain stage suddenly increases, the system can allocate more computing resources to this stage by adjusting the proportion of PD separation to ensure that the overall performance is not affected. In short, by combining the above-mentioned multiple technical means, this data management method can greatly improve the performance of large model inference systems, enabling them to better cope with growing computing needs.

[0059] In one embodiment, Figure 5 GPU X is responsible for the Prefill phase, performing the highly concurrent, compute-intensive tasks of the Prefill phase and updating the KVCACHE data to the CXL memory pool. Due to its powerful computing power, GPU X can quickly generate context information, thus providing context information to the CXL memory pool for the subsequent Decoder phase. GPUs 1-n are primarily responsible for the Decoder phase, focusing on reading KVCACHE data from the CXL memory pool and generating output. Because the decoding process is relatively memory-intensive, using a low-computing GPU can effectively reduce costs while meeting performance requirements.

[0060] KV Cache stores cached data in the CXL memory pool, including historical session KV Cache and actively accessed KV Cache. By aggregating CXL bandwidth, multiple low-power GPUs can concurrently access KV Cache for efficient decoding. The CXL memory pool provides shared memory resources, allowing multiple GPUs to access and update KV Cache. CXL technology supports high-bandwidth, low-latency data transmission, making data access more efficient.

[0061] KV Cache offloading optimizes memory usage by designing a multi-level KV Cache structure that dynamically manages the loading and unloading of cached data. This primarily involves counting KV Cache access frequency and offloading less frequently accessed contexts to memory during inference, while keeping more frequently accessed KV Cache data in HBM high-bandwidth memory. This is a crucial technology for large-model technology frameworks and multi-round conversational interactions, avoiding cache recalculation and conserving GPU computing resources.

[0062] With PD separation technology, a separate CXL memory pool device can be used. The CXL memory pool can support multiple GPU server hosts connected to the CXL memory pool. This aggregates the bandwidth of the CXL link, as shown in the figure, rather than being bottlenecked by a single PCIe link. Aggregating multiple memory modules under the CXL memory base increases overall bandwidth. Furthermore, CXL technology provides bandwidth comparable to PCIe, while offering lower latency and reducing data copying and network protocol overhead compared to network transmission.

[0063] CXL 3.0 further expands resource pools and bandwidth capabilities, enabling the system to support more devices and achieve more efficient data sharing and consistency, making it ideal for latency-sensitive inference businesses.

[0064] For example, the theoretical bidirectional bandwidth of x8 CXL per low-computing-power GPU server node is 64 GB / s. By aggregating CXL bandwidth, the bandwidth of low-computing-power nodes and high-computing-power nodes can be matched. Furthermore, the bandwidth capacity can scale linearly with the number of CXL memory devices and GPUs, resulting in virtually unlimited CXL heterogeneous memory bandwidth.

[0065] Since the Prefill stage and the decoder stage of large model inference are integrated, it is easy to cause resource waste of GPU computing power utilization under the premise of unbalanced PD computing power. Through the CXL memory pool, high-concurrency inference request processing is achieved by executing decoding tasks in parallel on multiple GPUs, combined with a dynamic task scheduling strategy. GPU1-N concurrently obtains kvcache. In the Decoder stage, multiple GPUs (including GPU1 and other low-computing-power GPUs) obtain the required data from KVCACHE in parallel. This concurrent access strategy can significantly improve the generation throughput, ensure the maximum utilization of high-computing-power GPU computing power, and improve the overall system's ability to maintain good performance under high inference loads.

[0066] Through the global scheduler, it is necessary to achieve the optimal PD separation ratio based on the GPU computing power to reasonably allocate GPU resources according to the computing requirements of different stages (Prefil and Decode), mainly based on the following considerations: GPU computing power, for example, each prefill request consumes 25TFLOP computing power. CXL bandwidth: Evaluate the bandwidth capability of the CXL memory to ensure that the large amount of Kvcache data generated by Prefill in high-concurrency inference requests and the GPU processing the Decoder stage can still effectively support data access through CXL. XLink bandwidth: Consider the bandwidth limitation of XLink, a high-speed bandwidth interface for direct communication between GPUs and GPUs, to avoid bottlenecks in the data transmission process. HBM capacity: Ensure that the HBM memory can accommodate the data of active KVCACHE and the data of the model itself.

[0067] In one embodiment, Figure 2 This specification also provides a data management device, which is applied to a computing power device, wherein the computing power device includes a first type of computing power unit applied to several pre-filling stages and several second type of computing power units applied to the decoding stage. The computing power device also includes a memory pool, and the device includes: a first module, which is used to obtain the key-value cache data generated by the first type of computing power unit in the pre-filling stage, transfer the key-value cache data to the memory pool and cache it in the memory pool; a second module, which is used to respond to the data acquisition requirements of the second type of computing power unit in the decoding stage, and transfer the specified key-value cache data cached in the memory pool to the second type of computing power unit according to the data acquisition requirements, so that the second type of computing power unit performs calculations based on the specified key-value cache data; the memory pool includes several aggregated memory units, and the memory pool and the first type of computing power unit transmit data through an aggregated first data transmission channel, and the memory pool and the second type of computing power unit transmit data through an aggregated second data transmission channel.

[0068] In one embodiment, the method of obtaining the key-value cache data generated by the first type of computing power unit in the pre-filling stage, transferring the key-value cache data to the memory pool and caching it in the memory pool includes: counting the activity parameters of the key-value cache data generated by the first type of computing power unit, caching the key-value cache data with lower activity parameters to the memory pool according to a preset ratio, and deleting the key-value cache data with lower activity parameters from the cache unit of the first type of computing power unit.

[0069] In one embodiment, the response to the data acquisition requirements of the second type of computing power unit in the decoding stage and the transfer of the specified key-value cache data cached in the memory pool to the second type of computing power unit according to the data acquisition requirements include: responding to the data acquisition requirements of the second type of computing power unit in the decoding stage and determining the specified key-value cache data according to the data acquisition requirements; if the specified key-value cache data is cached in the memory pool, the specified key-value cache data cached in the memory pool is transferred to the second type of computing power unit; if the specified key-value cache data is cached in the cache unit of the first type of computing power unit, the specified key-value cache data cached in the cache unit of the first type of computing power unit is cached in the memory pool, and then the specified key-value cache data cached in the memory pool is transferred to the second type of computing power unit.

[0070] In one embodiment, the activity parameter is positively correlated with the access frequency of the associated key-value cache data.

[0071] In one embodiment, Figure 3 , also includes a third module, which is used to obtain the computing requirements of the currently executed computing task, obtain the hardware attributes of each first-class computing power unit and second-class computing power unit, generate a configuration target according to the computing requirements and hardware attributes, and configure a specific first-class computing power unit as a second-class computing power unit according to the configuration target, and / or configure a specific second-class computing power unit as a first-class computing power unit.

[0072] In one embodiment, by constructing a heterogeneous computing resource pool and a distributed memory architecture, full-process decoupling and collaborative optimization of the pre-filling stage and the decoding stage are achieved. The system hardware architecture consists of three core components: a first-class computing unit that deploys a high-computing GPU cluster, a second-class computing unit equipped with a high-bandwidth video memory module, and a distributed memory pool built based on the CXL protocol. When the pre-filling phase starts, the scheduler dynamically allocates computing tasks to the first-class computing unit based on the length of the current pending prompt queue. Taking the processing of 32K input text as an example, after a single first-class computing unit loads the Transformer model parameters, it performs full attention calculations on the input sequence: first, the text is segmented into a token sequence and converted into a 768-dimensional embedding vector, which is then processed in parallel by 12 layers of Transformer layers. During the multi-head attention calculation process of each layer, the unit generates a key matrix and a value matrix with dimensions of [32, 12, 32768, 64]. These matrices are structured and stored according to the layer number and attention head sequence number to form the initial key-value cache data block.

[0073] After computation is complete, the data transfer engine initiates the first aggregation channel to offload the KV cache. This channel utilizes a combination of PCIe 5.0 x16 physical links and the CXL 3.0 protocol stack, achieving a theoretical bidirectional bandwidth of 128 GB / s on a single channel. During the transfer process, the memory pool controller uses an address mapping table to identify the target storage area for the KV cache. For example, it stores the key matrix of the third attention head in layer 5 directly in the third storage partition of memory pool node group B. The memory pool utilizes a distributed hash table management mechanism, with each memory unit node maintaining local address space mapping information and ensuring cross-node data access transparency through a consistency protocol. Upon detecting a DMA write request from a first-class computing unit, the memory pool controller automatically splits the data block into 256KB packets and distributes them to multiple memory unit nodes for parallel storage using a load balancing algorithm. This process utilizes pipeline optimization techniques, allowing a single first-class computing unit to complete a full 12-layer KV cache transfer within 23ms.

[0074] When entering the decoding phase, the second-category computing unit cluster initiates data requests to the memory pool through a publish-subscribe mechanism. For example, in the scenario of generating a conversational response, when the system needs to generate the 152nd output token, the scheduler selects a second-category computing unit with a load factor below 40% to perform this computation. This unit first sends a data retrieval request packet to the memory pool via the second aggregation channel. The request packet contains metadata such as the target Transformer layer number, attention head index, and historical token position range. Upon receiving the request, the memory pool query engine uses a Bloom filter to quickly locate the memory node group where the target data resides. For example, the Value matrix for the fifth head of the seventh layer might be located in the fifth storage partition of node group C. The second aggregation channel utilizes the CXL.cache protocol, optimized for high-frequency, small data packets. This protocol uses address aliasing technology to represent physically dispersed memory units as a continuous logical space, reducing the response time for a single request to as little as 0.8μs.

[0075] A three-level cache acceleration structure is implemented within the memory pool to improve data supply efficiency. The first-level cache, comprised of 16GB of HBM2e memory within each memory unit node, stores the most frequently accessed KV data blocks. The second-level cache utilizes 3DXPoint persistent memory, with a total capacity expanded to 2TB, for storing KV caches within the complete context window. The third-level cache utilizes an FPGA accelerator interconnected via CXL to implement near-data preprocessing, performing preprocessing operations such as half-precision floating-point format conversion on the KV matrix being transmitted. When processing long text generation tasks, the memory pool's dynamic garbage collection mechanism automatically eliminates KV data blocks that are outside the current context range based on a token position sliding window strategy. For example, when generating the 2048th token, the system retains only the KV cache for positions 1536 to 2047, and asynchronously erases and organizes expired data areas through a background thread.

[0076] At the resource scheduling level, the system introduces a dynamic allocation algorithm based on reinforcement learning to achieve precise matching of computing power and memory resources. The pre-filled task scheduler monitors the CUDA core utilization and memory bandwidth occupancy of each first-class computing power unit in real time. When it detects that the computing load of a unit exceeds 85%, it automatically routes the newly arrived Prompt request to the idle unit. For the decoding stage, the scheduler establishes a performance portrait model of the second-class computing power unit, records the actual throughput data of each unit when processing different numbers of layers and attention heads, and allocates computing tasks through a weighted polling algorithm. In sudden high-concurrency scenarios, the system can temporarily switch some first-class computing power units to decoding mode, and by dynamically reconfiguring their PCIe link connection mode, enable their second aggregation channel to access the memory pool, forming an elastically expandable decoding resource pool.

[0077] The physical layer implementation of the data transmission channel adopts an innovative optoelectronic hybrid architecture. The first aggregation channel deploys a silicon photonic engine module at the board-level interconnection stage, replacing the traditional copper interconnection with multi-wavelength optical signal transmission, achieving an energy efficiency ratio of 0.5pJ / bit within a transmission distance of 30 cm. The second aggregation channel adopts time-sensitive networking (TSN) technology to divide time-sensitive traffic channels at the physical layer and reserve deterministic bandwidth for KV cache reading. At the protocol stack level, the two types of channels share the transport layer protocol optimized based on RoCEv2, but adopt different QoS strategies: the first channel enables large-block data transmission mode and sets an 8MB sliding window size; the second channel enables low-latency mode, adopts a 1KB micro-packet encapsulation format and turns on zero-copy optimization.

[0078] The reliability assurance mechanism of the memory pool includes a multi-level error protection design. During the data storage phase, an erasure code redundancy strategy across three memory unit nodes is adopted, encoding each KV data block into a (4+2) shard structure, allowing data to be recovered even if two nodes fail at the same time. During the transmission process, the channel controller implements an end-to-end CRC check and retransmission mechanism, and adds Hamming code protection to key metadata fields. When the bit error rate of the persistent memory unit is detected to exceed 10^-17, the system automatically triggers the bad block replacement process, migrates the affected data to the spare memory unit, and replaces the faulty unit online through a hot-swappable module.

[0079] In a certain scenario, when processing 500 words of consultation content entered by the user (equivalent to approximately 850 tokens), the first-class computing power unit cluster completes the pre-fill calculation within 230ms, generates KV cache data, and completes the transmission to the memory pool through the first aggregation channel. Subsequently, 32 second-class computing power units are connected to the memory pool in parallel, and each unit is responsible for processing conversation generation requests from different users. When serving 200 concurrent sessions at the same time during peak hours, the distributed access architecture of the memory pool makes the average data acquisition delay of each decoding unit stable and low, and the throughput is significantly improved compared to the traditional single GPU architecture. The system uses dynamic voltage and frequency scaling technology to adjust the core voltage of the second-class computing power unit according to the real-time load, reducing overall power consumption while ensuring service quality.

[0080] A dedicated memory pool management middleware is used to achieve unified address space management across heterogeneous memory media. A zero-copy transfer protocol based on RDMA is designed to reduce data replication overhead between user and kernel states. A cache replacement algorithm tailored to KV cache characteristics is constructed, using an improved LRU strategy to balance temporal and spatial locality. At the hardware level, a custom computing accelerator card is designed for the first type of computing power unit, integrating Tensor Core matrix operation units and high-bandwidth HBM graphics memory. A storage-optimized processing module is designed for the second type of computing power unit, equipped with a PCIe 5.0x32 interface and an intelligent prefetch engine. The system software stack includes a real-time monitoring dashboard that dynamically displays key indicators such as the load heat map of each memory unit node, channel bandwidth utilization, and computing power unit task queue depth, providing operations and maintenance personnel with accurate resource allocation decision support.

[0081] During deployment, the memory pool topology is initialized, identifying all available memory unit nodes through an automatic discovery protocol. Next, the physical connection and protocol parameters for the two types of aggregated channels are configured, establishing a bandwidth reservation strategy. Encryption authentication keys are distributed to the computing unit cluster to ensure the security of data transmission. Finally, pre-trained Transformer model parameters are loaded, and a mapping table is established between the number of model layers and the memory pool storage area. During the operation and maintenance phase, the system provides online capacity expansion capabilities, supporting the dynamic addition of new computing units or memory expansion devices without downtime, and automatically optimizing resource allocation strategies through a self-balancing algorithm.

[0082] In one embodiment, this specification provides an electronic device including a processor and a readable storage medium, wherein the readable storage medium stores machine-executable instructions that can be executed by the processor, and the processor executes the machine-executable instructions to implement the aforementioned data management method. From a hardware perspective, a schematic diagram of the hardware architecture can be seen in FIG4 .

[0083] In one embodiment, this specification provides a readable storage medium, wherein the readable storage medium stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the aforementioned data management method.

[0084] Here, the readable storage medium can be any electronic, magnetic, optical or other physical storage device that can contain or store information, such as executable instructions, data, etc. For example, the readable storage medium can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drive (such as hard disk drive), solid state drive, any type of storage disk (such as optical disk, DVD, etc.), or similar storage media, or a combination thereof.

[0085] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email transceiver, game console, tablet computer, wearable device, or any combination of these devices.

[0086] For the convenience of description, the above devices are described as being divided into various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0087] Those skilled in the art will appreciate that embodiments of this specification may be provided as methods, systems, or computer program products. Thus, this specification may take the form of a fully hardware implementation, a fully software implementation, or an implementation combining software and hardware. Furthermore, embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0088] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0089] Furthermore, these computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0090] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable device to implement the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0091] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Thus, this specification may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0092] The foregoing is merely an embodiment of the present invention and is not intended to limit the present invention. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be included within the scope of the claims of the present invention.

Claims

1. A data management method, characterized in that: Applied to a computing device, the computing device includes a first type of computing unit applied to a plurality of pre-filling stages and a plurality of second type of computing units applied to a decoding stage, the computing device also including a memory pool, the method including: Obtain the key-value cache data generated by the first type of computing power unit in the pre-filling phase, transfer the key-value cache data to the memory pool, and cache it in the memory pool; In response to a data acquisition requirement of the second type of computing power unit in the decoding phase, transmitting the specified key-value cache data cached in the memory pool to the second type of computing power unit according to the data acquisition requirement, so that the second type of computing power unit performs calculations according to the specified key-value cache data; The memory pool includes several aggregated memory units. The memory pool and the first type of computing power unit transmit data through an aggregated first data transmission channel, and the memory pool and the second type of computing power unit transmit data through an aggregated second data transmission channel.

2. The method according to claim 1, characterized in that The step of obtaining the key-value cache data generated by the first type computing power unit in the pre-filling phase, transferring the key-value cache data to the memory pool, and caching the key-value cache data in the memory pool includes: Count the activity parameters of the key-value cache data generated by the first type of computing power unit, cache the key-value cache data with lower activity parameters to the memory pool according to a preset ratio, and delete the key-value cache data with lower activity parameters from the cache unit of the first type of computing power unit.

3. The method according to claim 2, characterized in that The method of responding to the data acquisition requirements of the second type computing power unit in the decoding phase and transmitting the specified key-value cache data cached in the memory pool to the second type computing power unit according to the data acquisition requirements includes: In response to the data acquisition requirements of the second type of computing power unit in the decoding phase, determining the designated key-value cache data according to the data acquisition requirements; If the specified key-value cache data is cached in the memory pool, the specified key-value cache data cached in the memory pool is transferred to the second type of computing power unit; If the specified key-value cache data is cached in the cache unit of the first type of computing power unit, the specified key-value cache data cached in the cache unit of the first type of computing power unit is cached in the memory pool, and then the specified key-value cache data cached in the memory pool is transferred to the second type of computing power unit.

4. The method according to claim 2, characterized in that The activity parameter is positively correlated with the access frequency of the associated key-value cache data.

5. The method according to claim 1, wherein Obtain the computing requirements of the currently executed computing task, obtain the hardware attributes of each first-class computing power unit and second-class computing power unit, generate a configuration target based on the computing requirements and hardware attributes, and configure a specific first-class computing power unit as a second-class computing power unit and / or configure a specific second-class computing power unit as a first-class computing power unit based on the configuration target.

6. A data management device, characterized in that: Applied to a computing device, the computing device includes a first type of computing unit applied to a plurality of pre-filling stages and a plurality of second type of computing units applied to a decoding stage, the computing device also includes a memory pool, and the apparatus includes: The first module is used to obtain the key-value cache data generated by the first type of computing power unit in the pre-filling stage, transfer the key-value cache data to the memory pool, and cache it in the memory pool; The second module is configured to respond to the data acquisition requirements of the second type of computing power unit in the decoding phase and transmit the specified key-value cache data cached in the memory pool to the second type of computing power unit according to the data acquisition requirements, so that the second type of computing power unit performs calculations according to the specified key-value cache data; The memory pool includes several aggregated memory units. The memory pool and the first type of computing power unit transmit data through an aggregated first data transmission channel, and the memory pool and the second type of computing power unit transmit data through an aggregated second data transmission channel.

7. The device according to claim 6, characterized in that The step of obtaining the key-value cache data generated by the first type computing power unit in the pre-filling phase, transferring the key-value cache data to the memory pool, and caching the key-value cache data in the memory pool includes: Count the activity parameters of the key-value cache data generated by the first type of computing power unit, cache the key-value cache data with lower activity parameters to the memory pool according to a preset ratio, and delete the key-value cache data with lower activity parameters from the cache unit of the first type of computing power unit.

8. The device according to claim 7, characterized in that The method of responding to the data acquisition requirements of the second type computing power unit in the decoding phase and transmitting the specified key-value cache data cached in the memory pool to the second type computing power unit according to the data acquisition requirements includes: In response to the data acquisition requirements of the second type of computing power unit in the decoding phase, determining the designated key-value cache data according to the data acquisition requirements; If the specified key-value cache data is cached in the memory pool, the specified key-value cache data cached in the memory pool is transferred to the second type of computing power unit; If the specified key-value cache data is cached in the cache unit of the first type of computing power unit, the specified key-value cache data cached in the cache unit of the first type of computing power unit is cached in the memory pool, and then the specified key-value cache data cached in the memory pool is transferred to the second type of computing power unit.

9. The device according to claim 7, characterized in that The activity parameter is positively correlated with the access frequency of the associated key-value cache data.

10. The device according to claim 6, characterized in that Also includes: The third module is used to obtain the computing requirements of the currently executed computing task, obtain the hardware attributes of each first-class computing power unit and second-class computing power unit, generate configuration targets based on the computing requirements and hardware attributes, and configure a specific first-class computing power unit as a second-class computing power unit and / or configure a specific second-class computing power unit as a first-class computing power unit based on the configuration targets.

11. An electronic device, characterized in that: include: A processor and a readable storage medium, wherein the readable storage medium stores machine-executable instructions that can be executed by the processor, and the processor executes the machine-executable instructions to implement the method according to any one of claims 1 to 5.

12. A readable storage medium, characterized in that: The readable storage medium stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the method according to any one of claims 1 to 5.

Images