Model reasoning method, electronic device and storage medium

Abstract

The present application discloses a model inference method, electronic device, and storage medium, which relate to the field of computer technology. The method includes: obtaining data to be processed; performing pre-filling processing on the data to be processed, saving a key-value cache generated by the pre-filling processing to a first buffer, and transmitting the key-value cache to a second buffer; the first buffer belongs to a connector corresponding to a pre-filling node, and the second buffer belongs to a connector corresponding to a decoding node; loading the key-value cache from the second buffer and performing decoding processing; during the decoding process, monitoring the memory utilization of a graphics processor in the decoding node, and adjusting the storage location of the key-value cache according to the memory utilization; wherein the decoding node is provided with a computational fast link memory, and the above-mentioned storage location includes the computational fast link memory and the memory of the graphics processor; and outputting the decoding processing result. The solution provided by the present application can effectively guarantee decoding efficiency, thereby ensuring the effective operation of the model inference process.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a model reasoning method, electronic device, and storage medium. Background Art

[0002] With the rapid development of artificial intelligence (AI) technology, the size of large models (i.e., the number of parameters) has grown at an alarming rate. This growth is reflected not only in the absolute value of the model parameters, but also in the increase in model complexity.

[0003] Because the memory capacity of the Graphics Processing Unit (GPU) limits the size of trainable models, even with more advanced algorithms and more data, if the GPU's memory capacity is insufficient, larger and more complex models cannot be trained. Especially during model inference, insufficient GPU memory capacity can lead to decoding failures in the decoding stage, thus affecting decoding efficiency. Summary of the Invention

[0004] The present application provides a model inference method, an electronic device, and a storage medium to at least solve the technical problem in related technologies of low decoding efficiency due to the limitation of GPU memory capacity.

[0005] This application provides a model reasoning method, which includes:

[0006] Get the data to be processed;

[0007] Performing pre-fill processing on the data to be processed, saving the key-value cache generated by the pre-fill processing to a first buffer, and transmitting the key-value cache to a second buffer; the first buffer belongs to the connector corresponding to the pre-fill node, and the second buffer belongs to the connector corresponding to the decoding node;

[0008] The key-value cache is loaded from the second buffer and decoding processing is performed based on the loaded key-value cache; during the decoding process, memory utilization of a graphics processor in the decoding node is monitored and a storage location of the key-value cache is adjusted according to the memory utilization; the decoding node is provided with a computation fast link memory, and the storage location includes the computation fast link memory and the memory of the graphics processor;

[0009] Output the decoding result.

[0010] This application also provides a model reasoning device, comprising:

[0011] An acquisition module is used to obtain data to be processed;

[0012] A pre-filling module is used to pre-fill the data to be processed, save the key-value cache generated by the pre-filling process to a first buffer, and transfer the key-value cache to a second buffer; the first buffer belongs to the connector corresponding to the pre-filling node, and the second buffer belongs to the connector corresponding to the decoding node;

[0013] A decoding module, configured to load the key-value cache from the second buffer and perform decoding processing based on the loaded key-value cache;

[0014] The scheduling module is used to monitor the memory utilization of the graphics processor in the decoding node during the decoding process and adjust the storage location of the key-value cache based on the memory utilization; the decoding node is provided with a computing fast link memory, and the above storage location includes the above computing fast link memory and the memory of the graphics processor.

[0015] The output module is used to output the decoding processing results.

[0016] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-mentioned model inference methods when executing the computer program.

[0017] The present application also provides a computer-readable storage medium, which stores a computer program, wherein when the computer program is executed by a processor, the steps of any of the above-mentioned model reasoning methods are implemented.

[0018] The present application also provides a computer program product, comprising a computer program, which implements the steps of any of the above-mentioned model reasoning methods when executed by a processor.

[0019] The model inference method, electronic device, and storage medium provided in this application configure a computationally fast link memory at the decoding node. By monitoring the GPU memory utilization of the decoding node during the decoding process, the storage location of the key-value cache can be automatically adjusted, thereby avoiding the reduction in decoding efficiency due to insufficient GPU memory and ensuring the effective operation of the model inference process. In addition, by rationally utilizing the computationally fast link memory, the overall decoding performance can be improved without increasing the GPU memory cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0021] Figure 1A schematic diagram of a model reasoning architecture provided in an embodiment of the present application;

[0022] Figure 2 This is a schematic diagram of another model reasoning architecture provided in an embodiment of the present application;

[0023] Figure 3 A flow chart of a model reasoning method provided in an embodiment of the present application;

[0024] Figure 4 A flowchart of another model reasoning method provided in an embodiment of the present application;

[0025] Figure 5 This is a schematic diagram of the structure of a model reasoning device provided in an embodiment of the present application;

[0026] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described below are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0028] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0029] The following is an explanation of some of the data involved in the examples of this application:

[0030] Large Language Model (LLM): LLM is an artificial intelligence model based on deep learning technology with powerful natural language processing capabilities that can understand and generate human language.

[0031] The main functions of LLM include:

[0032] Text generation: Generate coherent and meaningful text based on a given prompt or context, including writing articles, conversations, stories, code, etc.

[0033] Text understanding: Understanding the meaning, sentiment, and intent of text, and performing tasks such as answering, classification, and summarization based on this information.

[0034] Knowledge Question Answering: Through training, large language models can master a large amount of knowledge and be able to answer users' questions. This knowledge covers various fields such as science, history, and culture.

[0035] Multilingual processing: Many large language models support multiple languages ​​and can handle cross-lingual text generation and understanding tasks.

[0036] Application scenarios of LLM include:

[0037] Intelligent customer service: It can be used to build an intelligent customer service system to automatically answer users' questions and provide 24-hour uninterrupted service.

[0038] Content creation: Models can assist writers, journalists, editors, etc. in creating content, providing inspiration, suggestions, or directly generating text.

[0039] Educational tutoring: In the field of education, large language models can be used to build intelligent tutoring systems to provide students with personalized learning advice and feedback.

[0040] Machine translation: Combined with multilingual processing capabilities, large language models can achieve high-quality machine translation and break down language barriers.

[0041] Transformer model: A deep learning algorithm model that uses the attention mechanism to improve model training speed and serves as the basic architecture of LLM.

[0042] Token: A token is the fundamental unit of text that LLM uses to understand and generate text. It represents the smallest processable fragment of language. Depending on the specific tokenization scheme used, a token can represent a word, part of a word, or just a character.

[0043] Key-Value Cache (KV Cache): An optimization technique widely used in Transformer model inference (especially autoregressive generation tasks). By caching intermediate calculation results, it can significantly reduce repeated calculations and improve decoding speed.

[0044] KV Cache can store the keys and values ​​from previous steps to avoid repeated calculations. When generating a new token, you only need to calculate the attention score of the current token's query and the key in the cache, and then perform a weighted sum based on the cached value.

[0045] With the rapid development of artificial intelligence (AI) technology, the size (i.e., the number of parameters) of large models has grown rapidly. This growth is reflected not only in the absolute number of model parameters, but also in the increase in model complexity. In contrast, due to space and power constraints of GPUs, the growth rate of GPU memory capacity is relatively slow.

[0046] Due to the limitations of GPU memory capacity, the scale of models that can be trained is currently severely restricted. Therefore, even with more advanced algorithms and more data, if the GPU memory capacity is insufficient, larger and more complex models cannot be trained.

[0047] Due to the rapid growth in the size of large models, the availability of large model inference is severely limited and the cost is high. Related technologies use LLM PD separation (Prefill-Decode Separation) to separate the prefill and decode phases of the inference process for calculation, thereby improving the overall performance of large models.

[0048] Among them, PD separation is a technical strategy for optimizing inference of large language models, which divides the inference process into two independent stages: prefill and decoding.

[0049] Prefill phase: This phase processes the input sequence (such as user prompts or questions) all at once and calculates the KV cache for all input tokens. This phase typically uses parallel computing (such as matrix multiplication) for higher efficiency.

[0050] Decode phase: Generates output tokens one by one, calculating the attention score for the query of the current token and the key-value pair cached in the Prefill phase. This phase relies on autoregressive calculations (which must be generated sequentially), but avoids repeated calculations through the KV Cache.

[0051] For example, refer to Figure 1 , Figure 1 A schematic diagram of a large model reasoning architecture provided in an embodiment of the present application.

[0052] The Application Programming Interface Server (API-server) is responsible for receiving and processing external requests and interacting with different parts of the LLM model. It can pass input data to the LLM model for processing and ultimately return the output generated by the model.

[0053] The pre-filling stage is used to process the input sequence and generate the initial key-value cache. The decoding stage is used to use the key-value cache generated in the pre-filling stage to perform multiple iterations of autoregression and gradually generate the output sequence.

[0054] Specifically, during the pre-population phase, the LLM model receives an input sequence from the API server. Using a self-attention mechanism, the LLM model calculates the correlation between each token in the input sequence and generates corresponding keys and values. These key-value pairs are stored in a key-value cache for use in the subsequent decoding phase.

[0055] During the decoding phase, the LLM model retrieves contextual information about the input sequence from the key-value cache generated during the pre-population phase. The LLM model then begins generating output tokens one by one. Each generated token is appended to the end of the input sequence and the key-value cache is updated. This process is autoregressive, meaning that each newly generated token influences the generation of subsequent tokens.

[0056] Reference Figure 2 , Figure 2 A schematic diagram of another large model reasoning architecture provided in an embodiment of the present application.

[0057] in, Figure 2 The large model inference process shown uses LLM PD separation technology, separating the pre-filling and decoding phases of the inference process. The pre-filling phase focuses on performing pre-fill calculations and generating a key-value cache. After the calculations are completed, the generated key-value cache is used by the subsequent decoding phase. The decoding phase reads the key-value cache and performs multiple autoregressive iterative calculations to generate the output sequence.

[0058] PD separation can include the following advantages:

[0059] Resource utilization optimization: Since the pre-filling stage is computationally intensive, while the decoding stage is computationally sparse, separating these two stages can better utilize the GPU's computing resources.

[0060] Improved throughput: The separated pre-filling and decoding stages can process different requests simultaneously. This means that while the pre-filling stage is processing new requests, the decoding stage can continue to process decoding tasks for previous requests, thereby improving overall processing capacity.

[0061] Reduced latency: Since pre-population and decoding are performed in different stages, waiting time can be reduced, especially when multiple requests arrive concurrently.

[0062] In the above-mentioned LLM PD separation technology, PD direct connection technology is usually used for data transmission between the pre-filling node and the decoding node, that is, the pre-filling node directly sends the KV Cache to the decoding node. Its advantage is low latency. Since the KV Cache to be transmitted at one time is usually large, and decoding (autoregressive multiple iterations, storage-intensive) requires the KV Cache of historical tokens for iteration, while the pre-filling node does not need the KV Cache of historical tokens; therefore, the GPU memory requirement of the decoding node is much larger than that of the pre-filling node. However, due to the limited GPU capacity, the KV Cache data of the decoding node will accumulate more and more, and finally the GPU memory of the decoding node will overflow, and decoding will be impossible, affecting the decoding efficiency.

[0063] In response to the above technical problems, a method is provided in an embodiment of the present application. In this method, Compute Express Link (CXL) memory is configured on the decoding node, and each GPU is allocated a CXL memory on the same PCIe switch. A KV Cache scheduler corresponding to the decoding node is set. During the decoding process, the GPU memory utilization of the decoding node is monitored, and the storage location of the KV Cache is adjusted according to the GPU memory utilization, thereby ensuring decoding efficiency and further ensuring the effective operation of the model inference process.

[0064] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0065] Reference Figure 3 , Figure 3 This is a flow chart of a model inference method provided in an embodiment of the present application. In some embodiments, the method includes:

[0066] S301: Obtain data to be processed.

[0067] Optionally, the data to be processed may include text data.

[0068] S302 : Pre-fill the data to be processed, save the key-value cache generated by the pre-filling process to a first buffer, and transmit the key-value cache to a second buffer.

[0069] The first buffer zone belongs to a connector corresponding to a pre-filling node, and the second buffer zone belongs to a connector corresponding to a decoding node.

[0070] In some implementations, each LLM process can create a connector. Furthermore, two types of connectors are created and assigned to pre-population nodes and decoding nodes, respectively. These two types of connectors can be responsible for key-value cache transfer operations and are co-located with the corresponding pre-population nodes and decoding nodes.

[0071] In some implementations, the first buffer belongs to the connector corresponding to the pre-fill node and can be used to temporarily store the key-value cache generated during the pre-fill phase, ensuring stable data storage within the pre-fill node and preventing data loss or corruption. The second buffer belongs to the connector corresponding to the decoding node and can receive the key-value cache transmitted from the first buffer, providing a directly accessible data source for the decoding phase.

[0072] In some implementations, the router may be responsible for scheduling key-value cache transfer operations, determining which tokens need to be loaded from the connector into the key-value cache, or saving the key-value cache to the connector.

[0073] In the pre-filling phase, the LLM model computes the input sequence and generates corresponding key-value pairs for each token in the input sequence through operations such as the self-attention mechanism. These key-value pairs are aggregated to form a key-value cache.

[0074] The generated key-value cache is first saved to a first buffer, which belongs to the memory of the central processing unit (CPU).

[0075] In some implementations, the key-value cache can be transferred from the first buffer to the second buffer (which belongs to the CPU memory). This transfer mechanism ensures that the key-value cache generated in the pre-filling phase can be smoothly transferred to the decoding phase, preparing for subsequent decoding processing.

[0076] In the above implementation, by storing the key-value cache in the first buffer and the second buffer respectively, data isolation is achieved between the pre-filling stage and the decoding stage. This isolation prevents data interference between different stages and ensures that each stage uses accurate data for calculation.

[0077] In addition, the transmission of the key-value cache allows the decoding stage to directly obtain data from the second buffer without recalculating the key-value pairs of the input sequence. This greatly reduces the amount of computation and improves the efficiency of the entire inference process, especially when processing long sequences or a large number of requests.

[0078] S303: Load the key-value cache from the second buffer and perform decoding based on the loaded key-value cache. During the decoding process, monitor the memory utilization of the graphics processing unit (GPU) within the decoding node and adjust the storage location of the key-value cache based on the memory utilization. The decoding node is provided with a computational fast link memory, and the storage location includes the computational fast link memory and the GPU memory.

[0079] During the decoding phase, the model uses the key-value cache generated during the pre-population phase to gradually generate the output sequence. The key-value cache contains contextual information about the input sequence. By loading this cache from the second buffer, the decoding node can calculate and generate the next most likely token based on the existing input information in an autoregressive manner.

[0080] Optionally, the decoding process includes at least one computing task, such as:

[0081] Matrix multiplication: Matrix multiplication is a core operation in attention computation. For example, computing the interaction between query, key, and value requires multiple matrix multiplications. The efficiency of these operations directly impacts the speed of the decoding phase.

[0082] Convolution (optional): In some model architectures, convolution may be introduced to further extract features or process contextual information. Convolution can enhance the model's ability to capture local patterns.

[0083] Because the decoding stage involves a large number of computationally intensive tasks such as matrix multiplication, GPUs are used to perform these calculations. GPUs have powerful parallel computing capabilities, which can significantly accelerate the decoding process and improve the model's inference speed.

[0084] Among them, CXL memory is a new memory solution based on CXL technology, which aims to solve the data transmission bottleneck between CPU, memory, accelerators and storage devices in modern computing systems, and improve system performance and resource utilization.

[0085] CXL supports memory expansion and pooling, allowing external storage devices to be mapped as host memory, enabling "memory-level" storage access. By dynamically allocating memory resources, CXL avoids the fragmentation problem caused by traditional static partitioning and improves resource utilization.

[0086] Because CXL memory has high input / output (I / O), low latency, and direct transmission through GDR (GPU Direct RDMA), it can provide lower latency than CPU memory and higher bandwidth than GDR-supported non-volatile memory express (NVME), making it suitable as the KVCache buffer pool of decoding nodes.

[0087] GDR combines GPU-accelerated computing with Remote Direct Memory Access (RDMA) technology to enable direct data transmission and communication between GPUs and RDMA network devices. For example, the GPU on computer 1 can directly access the GPU on computer 2, bypassing the system memory and CPU.

[0088] Understandably, during the decoding phase, the GPU undertakes a large number of computational tasks, such as matrix multiplication and convolution operations, which require a certain amount of GPU memory. The key-value cache, a crucial data structure in the decoding process, has a direct impact on GPU memory usage due to its storage location. By monitoring GPU memory utilization in real time, the key-value cache's storage location can be dynamically adjusted to optimize memory usage, avoid GPU memory shortages or waste, and thus improve decoding efficiency and system stability.

[0089] Optionally, you can use GPU-specific acquisition tools or operating system-related interfaces to collect GPU memory usage, including indicators such as used memory, free memory, and memory occupancy. During the decoding process, continuously monitor GPU memory utilization to ensure that the latest GPU memory usage status is obtained in a timely manner.

[0090] In some implementations, CXL memory can be pre-configured in the decoding node, and each GPU is allocated a piece of CXL memory on the same PCIe switch, ensuring that each GPU has a dedicated PCIE link connected to its dedicated CXL memory, thereby expanding the memory space available to the GPU.

[0091] Among them, each GPU is connected to the CXL memory through an exclusive PCIE link, which can avoid the bandwidth bottleneck and performance interference caused by multiple GPUs competing for shared resources, and ensure that each GPU can access the memory stably and efficiently during the decoding process.

[0092] In some implementations, a key-value cache scheduler may be further provided at the decoding node as the key-value cache scheduler of the decoding node, and configured to monitor the memory utilization of the GPU within the decoding node.

[0093] In some implementations, a GPU memory utilization threshold can be pre-set, and the real-time monitored GPU memory utilization can be compared with the set threshold. If the GPU memory utilization exceeds the threshold, it indicates that the GPU memory resources are tight; if the GPU memory utilization is below the threshold, it indicates that the GPU memory has a lot of free resources.

[0094] When the GPU memory utilization exceeds the above threshold, the key-value cache is partially or completely transferred from the GPU memory to the CXL memory of the decoding node. This can free up GPU memory space, ensure that the decoding calculation can continue, and avoid calculation errors or performance degradation caused by insufficient GPU memory.

[0095] When GPU memory utilization falls below the threshold, some or all of the key-value cache can be moved from CXL memory back to GPU memory. Because GPU memory is typically accessed faster, moving the key-value cache back to GPU memory can speed up decoding computations.

[0096] During the decoding process, the key-value cache scheduler continuously monitors the GPU memory utilization within the decoding node. Based on changes in memory utilization, it dynamically adjusts the key-value cache storage location. For example, if memory utilization is high, it can migrate some key-value cache from GPU memory to CXL memory to free up GPU memory resources and ensure smooth decoding.

[0097] For example, in a text generation task, GPU memory utilization is low at the beginning of the decoding phase, and the key-value cache is stored entirely in GPU memory to take advantage of the GPU's high-speed computing capabilities. As the decoding process progresses, the generated sequences become longer and longer, the key-value cache continues to grow, and GPU memory utilization gradually increases. When memory utilization exceeds a set threshold, part of the key-value cache is transferred to CXL memory. During subsequent decoding, if GPU memory utilization drops below the threshold again, part of the key-value cache can be moved back to GPU memory to optimize decoding speed.

[0098] S304: Output the decoding result.

[0099] For example, for a text generation task, a text sequence may be output, such as “It will rain tomorrow”.

[0100] The method provided in this embodiment configures CXL memory on the decoding node. By monitoring the GPU memory utilization of the decoding node, it can automatically adjust the storage location of the key-value cache in three heterogeneous memory types: the second buffer (CPU memory), CXL memory, and GPU memory. This ensures decoding efficiency and, in turn, the effective operation of the model inference process. Furthermore, by properly utilizing this CXL memory, overall decoding performance can be improved without increasing GPU memory costs.

[0101] Reference Figure 4 , Figure 4 This is a flowchart of another model reasoning method provided in an embodiment of the present application.

[0102] In some embodiments, the pre-filling node and the decoding node are each provided with a connector, which can be used for communication transmission.

[0103] The router can be used to schedule key-value cache transfer operations, determine which tokens need to be loaded from the connector to the key-value cache, or save the key-value cache to the connector.

[0104] In some embodiments, the above method further comprises:

[0105] A prefilling operation request is sent to the prefilling node; and a decoding node is awakened, and a connector corresponding to the prefilling node is communicatively connected to the decoding node.

[0106] Exemplarily, the router may send a pre-filling operation request to the pre-filling node, and simultaneously send an indication message to a decoding node corresponding to a connector of the pre-filling node to wake up the decoding node.

[0107] Through the above implementation, the communication connection between the connector corresponding to the pre-filled node and the decoding node can be achieved, preparing for subsequent model reasoning.

[0108] In some embodiments, the steps of performing LLM PD separation may include:

[0109] S1: The pre-fill node runs the pre-fill operation and saves the key-value cache generated by the pre-fill process to the first buffer (CPU memory).

[0110] S2: The pre-fill node sends the key-value cache to the second buffer (CPU memory).

[0111] Among them, steps S1 and S2 can be executed in parallel.

[0112] S3: The connector corresponding to the decoding node instructs the connector corresponding to the pre-filling node that the key-value cache has been transferred.

[0113] S4: The connector corresponding to the pre-filled node instructs the router that the key-value cache has been transferred.

[0114] Among them, the connector corresponding to the decoding node indicates that the key-value cache of the connector corresponding to the pre-filling node has been transferred, and the connector corresponding to the pre-filling node further instructs the router. This state synchronization mechanism ensures that all relevant nodes understand the status of data transmission and reduces errors caused by data inconsistency.

[0115] S5: The router sends first information to the decoding node, where the first information is used to instruct the decoding node to perform decoding processing.

[0116] S6: The decoding node loads the key-value cache from the second buffer and runs the first decoding process.

[0117] The decoding node loads the key-value cache only when it receives the first message, implementing event-driven decoding. This approach avoids unnecessary preloading or continuous polling, reducing waste of system resources while ensuring that the decoding node can immediately obtain the required data when needed, thereby improving overall response speed.

[0118] S7: The key-value cache scheduler monitors the memory utilization of the GPU in the decoding node at this time. If the memory utilization is less than a preset threshold, S8 is executed; if the memory utilization is greater than or equal to the preset threshold, S9 is executed.

[0119] S8: Calculate the remaining memory space of the GPU and the space size of the key-value cache layer; based on the remaining memory space of the GPU and the space size of the key-value cache layer, select at least one key-value cache layer to be stored in the GPU memory; and store the remaining key-value cache layers in the CXL memory.

[0120] The space size of the at least one key-value cache layer selected above is smaller than or equal to the remaining memory space of the GPU.

[0121] When the memory utilization is lower than the preset threshold, by storing part of the key-value cache layer in the CXL memory, the GPU memory load can be balanced to avoid reduced decoding efficiency due to insufficient GPU memory.

[0122] In some implementations, the oldest key-value cache of a specified size in the GPU memory may be determined and transferred to the CXL memory.

[0123] By transferring the oldest (i.e., the longest unused) key-value cache to CXL memory, the GPU memory can immediately release the corresponding space for new tasks or critical data.

[0124] Optionally, n first key-value cache layers may be selected from the CXL memory and stored in the GPU memory; m second key-value cache layers may be selected from the second buffer and stored in the GPU memory; n and m are both positive integers.

[0125] In some implementations, a first communication bandwidth P1 between the CPU and the GPU and a second communication bandwidth P2 between the CXL memory and the GPU may be determined; and values ​​of n and m may be determined based on the first communication bandwidth P1 and the second communication bandwidth P2.

[0126] Illustratively, n=P2 / (P1+P2); m=P1 / (P1+P2).

[0127] Through the above implementation, part of the key-value cache layer can be accurately selected and stored in the GPU memory, thereby preventing invalid data from occupying the GPU memory.

[0128] In some implementations, the first n key-value cache layers may be selected from the CXL memory and stored in the GPU memory according to the order of the key-value cache layers during the forward propagation of the model to be trained.

[0129] It's understandable that during the model's forward propagation, early key-value cache layers often have a significant impact on subsequent computations. By prioritizing loading these cache layers into GPU memory, we ensure that the model can quickly access critical data during forward propagation, reducing wait time.

[0130] In some implementations, the last m second key-value cache layers can be selected from the second buffer and stored in the GPU memory according to the front-to-back order of the key-value cache layers during the forward propagation of the model to be trained.

[0131] By selecting the key-value cache layer according to the forward propagation order, we can avoid loading unnecessary or subsequently unused data into GPU memory.

[0132] Exemplarily, the above S8 may include:

[0133] Step a: The key-value cache scheduler calculates the remaining GPU memory space and the space size of each key-value cache layer (the model structure can be layered). Based on the remaining GPU memory space and the space size of the key-value cache layers, it selects several key-value cache layers and transfers the key-value caches in the second buffer to the GPU memory in the order of each key-value cache layer during the LLM forward propagation. After receiving the key-value caches, the decoding node immediately performs decoding calculations.

[0134] Step b: transfer all remaining key-value cache layers to the corresponding CXL memory, and keep the original remaining key-value cache layers in the original second buffer (CPU memory).

[0135] In step c, the key-value cache scheduler searches for the oldest key-value cache of batch size in the GPU memory and evicts it to the CXL memory. In the free GPU memory, the first n key-value cache layers (the first key-value cache layer) are selected from the CXL memory and uploaded to the GPU, following the order of each layer in the LLM forward propagation process. At the same time, the last m key-value cache layers (the second key-value cache layer) are selected from the second buffer (CPU memory) and uploaded to the GPU.

[0136] Step d: Determine whether the decoding process is completed. If not, merge the first key-value cache layer and the second key-value cache layer and wait for decoding process. If it is completed, perform decoding process of the first key-value cache layer first, and then perform decoding process of the second key-value cache layer.

[0137] Step e: Repeat steps S1 to S7.

[0138] S9: Store all key-value caches in CXL memory.

[0139] Among them, when the GPU memory utilization exceeds the preset threshold, by migrating all key-value caches to CXL memory, decoding task crashes or performance degradation caused by insufficient GPU memory can be avoided.

[0140] In some implementations, the remaining key-value caches except for the most recently specified size in the GPU memory may be transferred to the CXL memory.

[0141] In some embodiments, n first key-value cache layers can be selected from the computing quick link memory and stored in the GPU memory; m second key-value cache layers can be selected from the second buffer and stored in the GPU memory; n and m are both positive integers.

[0142] In some implementations, a first communication bandwidth P1 between the CPU and the GPU and a second communication bandwidth P2 between the CXL memory and the GPU may be determined; and values ​​of n and m may be determined based on the first communication bandwidth P1 and the second communication bandwidth P2.

[0143] Illustratively, n=P2 / (P1+P2); m=P1 / (P1+P2).

[0144] In some implementations, the first n key-value cache layers may be selected from the CXL memory and stored in the GPU memory according to the order of the key-value cache layers during the forward propagation of the model to be trained.

[0145] In some implementations, the last m second key-value cache layers can be selected from the second buffer and stored in the GPU memory according to the front-to-back order of the key-value cache layers during the forward propagation of the model to be trained.

[0146] Exemplarily, the above S9 may include:

[0147] Step a: Transfer all key-value caches in the second buffer (CPU memory) to the corresponding CXL memory, and keep the key-value cache in the original second buffer (CPU memory).

[0148] Step b: Evict all key-value caches except the most recent bachsize to CXL memory. In the free GPU memory, select n front-end key-value cache layers (first key-value cache layer) from CXL memory and upload them to the GPU, according to the order of each layer in the LLM forward propagation process. At the same time, select m back-end key-value cache layers (second key-value cache layer) from the second buffer (CPU memory) and upload them to the GPU.

[0149] Step c: Determine whether the decoding process is completed. If not, merge the first key-value cache layer and the second key-value cache layer and wait for decoding process. If completed, perform decoding process on the first key-value cache layer first, and then perform decoding process on the second key-value cache layer.

[0150] Step d: Repeat steps S1 to S7.

[0151] The method provided in the embodiment of the present application sets a key-value cache scheduler in the LLM PD separation architecture and uses the CXL memory cache pool to automatically adjust the storage location of the key-value cache in the second buffer (CPU memory), CXL memory, and GPU memory, three heterogeneous memories, to ensure that the GPU can effectively complete the inference process.

[0152] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0153] The embodiment of the present application also provides a model reasoning device. Figure 5 , Figure 5 This is a structural diagram of a model reasoning device provided in an embodiment of the present application.

[0154] In some embodiments, the model inference device 50 includes:

[0155] The acquisition module 501 is used to acquire data to be processed.

[0156] The pre-filling module 502 is used to pre-fill the data to be processed, save the key value cache generated by the pre-filling process to the first buffer, and transfer the key value cache to the second buffer; the first buffer belongs to the connector corresponding to the pre-filling node, and the second buffer belongs to the connector corresponding to the decoding node.

[0157] The decoding module 503 is configured to load the key-value cache from the second buffer during the decoding phase, and perform decoding processing based on the loaded key-value cache.

[0158] The scheduling module 504 is used to monitor the memory utilization of the graphics processor in the decoding node during the decoding process, and adjust the storage location of the key-value cache according to the memory utilization; the decoding node is provided with a computing fast link memory, and the above storage location includes the above computing fast link memory.

[0159] The output module 505 is used to output the decoding result.

[0160] In some implementations, the scheduling module 504 is configured to:

[0161] When the memory utilization is less than a preset threshold, the remaining memory space of the graphics processor and the space size of the key-value cache layer are calculated;

[0162] Selecting to store at least one key-value cache layer in the memory of the graphics processor based on the remaining memory space of the graphics processor and the space size of the key-value cache layer;

[0163] Store the remaining key-value cache layers in the compute fast link memory.

[0164] In some implementations, the scheduling module 504 is configured to:

[0165] Determine the oldest key-value cache of a specified size in the GPU's memory and transfer it to the compute fast link memory.

[0166] In some implementations, the scheduling module 504 is configured to:

[0167] When the memory utilization is greater than or equal to the preset threshold, all key-value caches are stored in the computational fast link memory.

[0168] In some implementations, the scheduling module 504 is configured to:

[0169] Transfer the remaining key-value caches in the GPU memory except for the most recently specified key-value cache to the computing fast link memory.

[0170] In some implementations, the scheduling module 504 is configured to:

[0171] Select n first key-value cache layers from the computational fast link memory and store them in the GPU memory;

[0172] Select m second key-value cache layers from the second buffer and store them in the memory of the graphics processor; n and m are both positive integers.

[0173] In some implementations, the scheduling module 504 is configured to:

[0174] determining a first communication bandwidth between the central processing unit and the graphics processing unit, and calculating a second communication bandwidth between the quick link memory and the graphics processing unit;

[0175] Based on the first communication bandwidth and the second communication bandwidth, the values ​​of n and m are determined.

[0176] In some implementations, the scheduling module 504 is configured to:

[0177] According to the order of the key-value cache layers in the forward propagation process of the model to be trained, the first n first key-value cache layers are selected from the computing fast link memory and stored in the memory of the graphics processor.

[0178] In some implementations, the scheduling module 504 is configured to:

[0179] According to the front-to-back order of the key-value cache layers in the forward propagation process of the model to be trained, the last m second key-value cache layers are selected from the second buffer and stored in the memory of the graphics processor.

[0180] In some embodiments, the apparatus further includes a router configured to:

[0181] Sending a prefill operation request to the prefill node;

[0182] Wake up the decoding node and establish a communication connection between the connector corresponding to the pre-filled node and the decoding node.

[0183] In some implementations, the decoding module 503 is configured to:

[0184] When the first information is received, the key-value cache is loaded from the second buffer; the first information is used to instruct the decoding module to perform decoding processing.

[0185] For the description of the features in the embodiment corresponding to the above-mentioned model reasoning device, please refer to the relevant description of the embodiment corresponding to the above-mentioned model reasoning method, and no further details will be given here.

[0186] The model inference device provided in the embodiments of the present application configures a computational fast link memory at the decoding node. By monitoring the GPU memory utilization of the decoding node, it can automatically adjust the storage location of the key-value cache in three heterogeneous memories: the second buffer (CPU memory), CXL memory, and GPU memory. This can ensure decoding efficiency and, in turn, the effective operation of the model inference process. In addition, by rationally utilizing the computational fast link memory, overall decoding performance can be improved without increasing GPU memory costs.

[0187] Reference Figure 6 , Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 6 As shown, the electronic device 60 provided in this embodiment includes: at least one processor 601 and a memory 602. Optionally, the electronic device 60 further includes a communication component 603. The processor 601, the memory 602 and the communication component 603 are connected via a bus.

[0188] During the specific implementation process, at least one processor 601 executes the computer execution instructions stored in the memory 602, so that at least one processor 601 executes the above-mentioned model reasoning method embodiment.

[0189] The specific implementation process of the processor 601 can be found in the above-mentioned model reasoning method embodiment. Its implementation principle and technical effects are similar and will not be repeated here in this embodiment.

[0190] In the above embodiments, it should be understood that the processor may be a CPU, or other general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), etc. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the application may be directly implemented by a hardware processor or implemented by a combination of hardware and software modules in the processor.

[0191] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage.

[0192] A bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. Buses can be categorized as address buses, data buses, and control buses. For ease of illustration, the buses in the drawings of this application are not limited to just one bus or just one type of bus.

[0193] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above-mentioned model reasoning method embodiments when run.

[0194] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0195] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any one of the above-mentioned model reasoning method embodiments are implemented.

[0196] An embodiment of the present application also provides another computer program product, including a non-volatile computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps in any of the above-mentioned model reasoning method embodiments.

[0197] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0198] The above is a detailed introduction to a model reasoning method provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core ideas of the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A model reasoning method, characterized in that: The method comprises: Get the data to be processed; Pre-filling the data to be processed, saving a key-value cache generated by the pre-filling process to a first buffer, and transmitting the key-value cache to a second buffer; the first buffer belongs to a connector corresponding to a pre-filling node, and the second buffer belongs to a connector corresponding to a decoding node; The key-value cache is loaded from the second buffer and decoding processing is performed based on the loaded key-value cache; during the decoding process, memory utilization of a graphics processor in the decoding node is monitored and a storage location of the key-value cache is adjusted according to the memory utilization; the decoding node is provided with a computation fast link memory, and the storage location includes the computation fast link memory and a memory of the graphics processor; Output the decoding result.

2. The method according to claim 1, characterized in that The adjusting the storage location of the key-value cache according to the memory utilization includes: When the memory utilization is less than a preset threshold, calculating the remaining memory space of the graphics processor and the space size of the key-value cache layer; Selecting to store at least one key-value cache layer in the memory of the graphics processor according to the remaining memory space of the graphics processor and the space size of the key-value cache layer; The remaining key-value cache layers are stored in the computational fast link memory.

3. The method according to claim 2, characterized in that The adjusting the storage location of the key-value cache according to the memory utilization rate further includes: The oldest key-value cache of a specified size in the memory of the graphics processor is determined and transferred to the computing fast link memory.

4. The method according to claim 1, wherein The adjusting the storage location of the key-value cache according to the memory utilization includes: When the memory utilization is greater than or equal to a preset threshold, all key-value caches are stored in the computing fast link memory.

5. The method according to claim 4, characterized in that The adjusting the storage location of the key-value cache according to the memory utilization rate further includes: Transferring the remaining key-value caches except for the most recently specified key-value cache in the memory of the graphics processor to the computing fast link memory.

6. The method according to claim 3 or 5, characterized in that The adjusting the storage location of the key-value cache according to the memory utilization rate further includes: Selecting n first key-value cache layers from the computing fast link memory and storing them in the memory of the graphics processor; m second key-value cache layers are selected from the second buffer and stored in the memory of the graphics processor; n and m are both positive integers.

7. The method according to claim 6, characterized in that The method further comprises: determining a first communication bandwidth between a central processing unit and the graphics processor, and a second communication bandwidth between the computing quick link memory and the graphics processor; Based on the first communication bandwidth and the second communication bandwidth, values ​​of n and m are determined.

8. The method according to claim 7, characterized in that The selecting n first key-value cache layers from the computing quick link memory and storing them in the memory of the graphics processor includes: According to the front-to-back order of the key-value cache layers in the forward propagation process of the model to be trained, the first n first key-value cache layers are selected from the computing fast link memory and stored in the memory of the graphics processor.

9. The method according to claim 7, characterized in that The selecting m second key-value cache layers from the second buffer and storing them in the memory of the graphics processor includes: According to the front-to-back order of the key-value cache layers in the forward propagation process of the model to be trained, the last m second key-value cache layers are selected from the second buffer and stored in the memory of the graphics processor.

10. The method according to claim 1, characterized in that The method further comprises: Sending a prefill operation request to the prefill node; The decoding node is awakened, and a connector corresponding to the pre-filling node is communicatively connected to the decoding node.

11. The method according to claim 1, wherein The loading the key-value cache from the second buffer includes: When the decoding node receives the first information, the key-value cache is loaded from the second buffer; the first information is used to instruct the decoding node to perform decoding processing.

12. The method according to claim 11, characterized in that The method further comprises: The connector corresponding to the decoding node indicates to the connector corresponding to the pre-filling node that the key-value cache has been transmitted; The connector corresponding to the pre-filled node indicates to the router that the key-value cache has been transferred; The router sends the first information to the decoding node.

13. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the model reasoning method according to any one of claims 1 to 12 when executing the computer program.

14. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the model inference method according to any one of claims 1 to 12.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the model reasoning method according to any one of claims 1 to 12 are implemented.

Images