Model reasoning method, computer program product and chip

Abstract

The embodiment of the invention provides a model reasoning method, a computer program product and a chip. The chip running the high-request-quantity model can send the QKV of the new lexical elements in the unloading text sequence to the chip running the low-request-quantity model after calculating and obtaining the QKV of the new lexical elements in the unloading text sequence, and the chip running the low-request-quantity model can unload the new lexical elements on the basis of the key value cache of the historical lexical elements in the stored unloading text sequence and the QKV of the new lexical elements. And calculating to obtain an attention result corresponding to the unloaded text sequence, and then returning the attention result obtained by calculation to the chip running the high request quantity model so as to execute subsequent calculation. Therefore, the resource utilization rate of a chip running a low-request-quantity model and the overall efficiency of a system can be improved, and as only the QKV tensor of the unloaded text sequence and the attention calculation result need to be transmitted, the extra overhead brought by communication is low, and the overall resource utilization rate of a multi-model service platform can be greatly improved.

Description

Technical Field

[0001] The present application relates to the field of control technology, and in particular to a model reasoning method, a computer program product, and a chip. Background Art

[0002] Multi-Model Service Platform (MaaS) is a service platform that integrates multiple artificial intelligence models, aiming to provide flexible, efficient and diversified AI model services for enterprises and individuals. These platforms usually support multiple model types and provide a unified service interface to meet the needs of different business scenarios. Taking into account the different popularity of various models in the multi-model service platform, for example, a small number of models are popular models, that is, the number of service requests for such models is usually large. But many models are unpopular models, that is, the number of service requests for such models is relatively small. The current service solutions of the multi-model service platform either cannot maximize the utilization of the hardware resources of the chip in the multi-model service platform, or will cause a large overhead of transmission resources. Therefore, it is necessary to provide a solution that can maximize the utilization of the hardware resources of the chip and reduce the consumption of transmission resources. Summary of the Invention

[0003] In view of this, the present application provides a model reasoning method, a computer program product, and a chip.

[0004] According to a first aspect of the present application, a model inference method is provided. The method is applicable to a multi-model service platform, the multi-model service platform including a first chip for running a high-request volume model and a second chip for running a low-request volume model. The number of service requests received by the high-request volume model per unit time is greater than the number of service requests received by the low-request volume model. The high-request volume model is a model based on a Transformer architecture. The method is executed by the first chip, and the method includes:

[0005] In the process of processing multiple input text sequences using the high-request volume model, the QKV of the new word of the unloaded text sequence in the multiple text sequences currently calculated is sent to the second chip, so that the second chip calculates the attention result of the unloaded text sequence based on the key-value cache of the historical word-grams in the unloaded text sequence and the QKV of the new word-gram; wherein the unloaded text sequence is at least a part of the text sequence in the multiple text sequences, and the key-value cache of the historical word-grams in the unloaded text sequence is stored in the second chip;

[0006] Receive the attention result of the uninstalled text sequence returned by the second chip, and calculate the output of the high request volume model based on the attention result of the uninstalled text sequence

[0007] According to a second aspect of the present application, a model inference method is provided. The method is applicable to a multi-model service platform, the multi-model service platform including a first chip for running a high-request volume model and a second chip for running a low-request volume model. The number of service requests received by the high-request volume model per unit time is greater than the number of service requests received by the low-request volume model. The high-request volume model is a model based on a Transformer architecture. The method is executed by the second chip, and the method includes:

[0008] receiving a QKV of a new word in an unloaded text sequence sent by the first chip, wherein the unloaded text sequence is at least a portion of a plurality of text sequences currently processed by the high-request volume model, and a key-value cache of historical words in the unloaded text sequence is stored in the second chip;

[0009] Calculating an attention result of the unloaded text sequence based on a key-value cache of historical word units in the unloaded text sequence and a QKV of the new word unit;

[0010] The attention result of the uninstalled text sequence is sent to the first chip, so that the first chip calculates the output of the high-request model based on the attention result of the uninstalled text sequence.

[0011] According to a third aspect of the present application, a computer program product is provided, which includes a computer program, and when the computer program is executed, the method mentioned in the first aspect is implemented.

[0012] According to a fourth aspect of the present application, an electronic device is provided, comprising a processor, a memory, and a computer program stored in the memory for execution by the processor, wherein the processor implements the method mentioned in the first aspect when executing the computer program.

[0013] According to a fifth aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed, the method mentioned in the first aspect is implemented.

[0014] By applying the solution provided by this application, for multiple text sequences being served by a high-request model, the attention calculation tasks of some of the text sequences can be offloaded to a chip running a low-request model. Considering that the memory of a chip running a high-request model is often limited, part of the KV cache of the historical text sequence of the high-request model usually needs to be offloaded to an external storage unit for storage. Therefore, the KV cache of some text sequences can be offloaded to the chip running the low-request model for storage. At the same time, the attention calculation for this part of the unloaded text sequence can also be performed by the chip model running the low-request model. In this way, the chip running the low-request model can be used to alleviate the storage pressure of the KV cache of the historical text sequence of the high-request model, and at the same time, it can assist it in completing part of the attention calculation tasks. For example, after calculating the QKV of a new word in an unloaded text sequence, the chip running the high-request model can send it to the chip running the low-request model. The chip running the low-request model can then calculate the attention result corresponding to the unloaded text sequence based on the stored key-value cache of historical words in the unloaded text sequence and the QKV of the new word, and then return the calculated attention result to the chip running the high-request model for subsequent calculations. By offloading the attention calculation of part of the text sequence of the high-request model and its corresponding KV cache to the chip running the low-request model, the resource utilization of the chip running the low-request model and the overall efficiency of the system are significantly improved. In addition, since only the QKV tensor and attention calculation results of the unloaded text sequence need to be transmitted, the additional overhead caused by communication is low, which can greatly improve the overall resource utilization of the multi-model service platform.

[0015] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0017] Figure 1 It is a schematic diagram of an application scenario of an embodiment of the present application.

[0018] Figure 2 This is a flowchart of a model reasoning method according to an embodiment of the present application.

[0019] Figure 3 This is a schematic diagram of a first chip and a second chip transmitting data through a shared memory according to an embodiment of the present application.

[0020] Figure 4 This is a schematic diagram of tasks in a queue of tasks to be executed in a second chip according to an embodiment of the present application.

[0021] Figure 5 Schematic diagram of a first chip sending an attention calculation task to a second chip in one embodiment of the present application.

[0022] Figure 6 It is a schematic diagram of the logic structure of a chip according to an embodiment of the present application. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0024] The multi-model service platform integrates various AI models and provides a unified service interface and intelligent scheduling mechanism to meet the diverse needs of enterprises and individuals in different business scenarios. Considering the varying popularity of various models in the multi-model service platform, for example, some models are highly popular and receive a large number of service requests (hereinafter referred to as high-request models), while some models are less popular and receive a smaller number of service requests (hereinafter referred to as low-request models).

[0025] For example, taking the large language model service platform as an example, with the rapid growth of the types of large language models, LLM (LargeLanguage Model) service platforms (such as huggingface, Together.ai) now typically host dozens to hundreds of LLMs, presenting a multi-LLM service model. For example, Model as a Service (MaaS) platforms like Hugging Face, Together.ai, and OpenRouter provide services for a variety of different types of models to meet users with different needs and use cases. Even single model providers, such as OpenAI and Gemini, host a range of models with different capabilities, model sizes, and release versions, enabling users to choose the most suitable model for their specific tasks. For example, OpenAI alone provides more than 20 variants of its chat series LLM.

[0026] Multi-LLM service workloads exhibit skewed access patterns. For example, based on a week-long analysis of multi-LLM service statistics from the popular MaaS platform OpenRouter, the platform contained 209 models and processed over 200 billion tokens. Analysis results show that the top 5% of models consumed 74.8% of tokens, exhibiting a skewed distribution. In other words, within a multi-LLM service, a small number of popular models (such as Llama-3 or OpenAI's latest release) receive the majority of traffic, while many models are unpopular and sparsely used, receiving low but stable request rates most of the time.

[0027] There are two main existing multi-model service methods. The first service method uses a dedicated instance for each model. For example, a service provider can deploy popular models (i.e., high-request models) on a dedicated GPU, while dynamically allocating GPUs to unpopular models (i.e., low-request models) (for example, using server-insensitive computing). However, when processing requests for unpopular models, the associated GPUs have the problem of low memory utilization. For example, the applicant conducted an experiment in which a popular model (15QPS, Llama-3-8B) and an unpopular model (1QPS, Llama-3-8B) each ran on an A100-40GB GPU and measured their GPU memory utilization. Although the popular model has fully utilized the GPU memory, the GPU memory utilization of the unpopular model is still only 43%. It can be seen that using dedicated instances for each model will lead to underutilization of the GPU memory resources of the unpopular model.

[0028] Another approach is to multiplex chips (e.g., GPUs) for popular and unpopular models. AlpaServe uses model parallelism to statistically multiplex GPUs for multiple models. MuxServe further improves on this by integrating NVIDIA MPS capabilities, enabling flexible spatiotemporal sharing. MuxServe primarily utilizes tensor parallelism as its primary model parallelism strategy. While this model-parallelism-based serving approach improves GPU utilization, it also incurs overhead from additional communication and smaller (and therefore less efficient) operators. The applicant conducted an experiment to evaluate the token generation throughput of the Llama-3-8B model on two A100-40GB GPUs connected via NVLink, using two independent instances and a tensor parallelism (TP) degree of 2. The results show that even on platforms with high interconnect bandwidth, simply using tensor parallelism can introduce overhead of up to 26%.

[0029] It can be seen that although simply using model parallelism to reuse chip (e.g., GPU) resources can improve the utilization of chip (e.g., GPU) video memory to a certain extent, model parallelism brings excessively high communication and operator overhead, which reduces the system throughput.

[0030] Therefore, it is necessary to provide a solution that can maximize the use of chip hardware resources while reducing the consumption of transmission resources.

[0031] Models using the Transformer architecture, such as large language models, typically include N repeated network layers (i.e., Transformer layers). Each Transformer layer contains two main modules: an attention module and a feed-forward network module (FFN). The attention module is used to establish information interaction relationships between tokens, and the feed-forward network module is used to perform nonlinear transformations and enhancements on the feature information of tokens. Therefore, the operators in models based on the Transformer architecture can also be divided into two categories: parameterized operators (parameterized projection) and non-parameterized (non-parameterized attention) operators.

[0032] Parameterized operators include fully connected feedforward networks (FFNs) and specialized linear transformations in multi-head attention mechanisms, such as the calculation of Q (Query), K (Key), V (Value), and output projection (i.e., projection operators). As parameterized operators, they rely on a fixed set of learned parameters to process input tokens or their intermediate representations.

[0033] Non-parametric operators are primarily attention operators, which dynamically compute relationships between input tokens to capture contextual dependencies. Unlike parametric operators, attention mechanisms are non-parametric. Therefore, attention computations do not rely on model parameters, but rather on KV tensors of previous tokens. These KV tensors are typically stored in a KV cache to avoid repeated computations during subsequent token generation.

[0034] For example, suppose a large language model based on the Transformer architecture is used for text generation. The task is to generate the sentence "I like to eat apples." The model outputs "I," "like," "eat," and "apple" word by word. During the generation process, each time the model generates a token, it needs to calculate the attention score between the query vector (Q) of the current token and the key vectors (K) and value vectors (V) of all historical tokens. Without a key-value cache, the model needs to recalculate K and V for all historical tokens each time a new token is generated, which results in a huge computational overhead. To reduce computational overhead, a key-value cache of historical tokens can be used.

[0035] The following takes the key-value cache of a certain layer of the model as an example:

[0036] When generating "I", you need to calculate Q_I, K_I, and V_I.

[0037] When generating "like", if there is no cache, you need to recalculate K_me, V_me, and then calculate Q_like and K_me, K_like, V_me, V_like.

[0038] The core idea of ​​key-value caching is to cache the K and V of historical tokens to avoid repeated calculations. The specific process is as follows:

[0039] When generating "I", calculate Q_I, K_I, V_I, and store K_I and V_I in the key-value cache.

[0040] When generating a "like", only Q_like is calculated, K_me and V_me are obtained from the cache, the attention score of Q_like and K_me is calculated, K_like and V_like are calculated, and the cache is updated.

[0041] The generation process of subsequent word units is similar. By continuously caching the K and V values ​​of historical word units, the computing efficiency can be improved.

[0042] As the length of the text sequence increases, the K and V values ​​that need to be cached will also increase. However, the storage space of the chip used to execute the model inference method (such as GPU) is limited, and the key-value cache cannot be stored entirely in the chip's memory. Therefore, the key-value cache can be unloaded, that is, the key-value cache can be saved in a storage unit outside the model inference chip. For example, currently, the key-value cache is usually unloaded to the SSD.

[0043] Taking into account that the utilization rate of hardware resources (such as memory, etc.) in the chip running the low-request model is often relatively low, in order to improve the utilization rate of hardware resources of the chip running the low-request model, the embodiment of the present application conceives that, for multiple text sequences being served by the high-request model, the attention calculation tasks of some of the text sequences can be offloaded to the chip running the low-request model. Taking into account that the memory of the chip running the high-request model is often limited, part of the KV cache of the historical text sequence of the high-request model usually needs to be offloaded to an external storage unit for storage. Therefore, the KV cache of part of the text sequence can be offloaded to the chip running the low-request model for storage. At the same time, the attention calculation for this part of the unloaded text sequence can also be performed by the chip model running the low-request model. In this way, the chip running the low-request model can be used to alleviate the storage pressure of the KV cache of the historical text sequence of the high-request model, and at the same time, it can assist it in completing part of the attention calculation tasks. For example, after the chip running the high-request model calculates the Q (query vector), K (key vector), and V (value vector) of a new word in the unloaded text sequence, it can send it to the chip running the low-request model. The chip running the low-request model can then calculate the attention result corresponding to the unloaded text sequence based on the stored key-value cache of historical words in the unloaded text sequence and the QKV of the new word, and then return the calculated attention result to the chip running the high-request model for subsequent calculations. By offloading the attention calculation of part of the text sequence of the high-request model and its corresponding KV cache to the chip running the low-request model, the resource utilization of the chip running the low-request model and the overall efficiency of the system are significantly improved. In addition, since only the QKV tensor and attention calculation results of the unloaded text sequence need to be transmitted, the additional overhead caused by communication is low, which can greatly improve the overall resource utilization of the multi-model service platform.

[0044] In the embodiment of the present application, a type of model that receives a large number of service requests in the multi-model service platform is called a high-request model, and a type of model that receives a small number of service requests in the multi-model service platform is called a low-request model, that is, the number of service requests received by the multi-model service platform per unit time for the high-request model is greater than the number of service requests received for the low-request model. Among them, the high-request model can be a model based on the Transformer architecture, for example, it can be various large language models, and the low-request model can be a language model, an image generation model or other types of AI models, which are not limited in the embodiment of the present application.

[0045] In the embodiments of this application, the chip used to run the high-request model is referred to as the first chip, and the chip used to run the low-request model is referred to as the second chip. The first chip and the second chip can be various chips used to perform model inference computing tasks, such as CPUs, GPUs, or other chips with the same functions, and are not limited in this embodiment.

[0046] For example, Figure 1 As shown, a schematic diagram of an application scenario shown in an exemplary embodiment of the present application, wherein the multi-model service platform includes a first chip (for example, GPU1) for running a high-request model, and a second chip (for example, GPU2) for running a low-request model, and the first chip and the second chip are connected via a processing unit (for example, CPU). The first chip can interact with the second chip to complete the inference task of the high-request model with the help of the hardware resources of the second chip to improve the utilization rate of the hardware resources of the second chip, wherein the first chip can interact with the second chip as shown in the flowchart. Figure 2 As shown, the following steps are included:

[0047] S201: During processing of multiple input text sequences using a high-request-volume model, a first chip sends a QKV of a new word in an unloaded text sequence from the multiple text sequences currently calculated to a second chip, wherein the unloaded text sequence is at least a portion of the multiple text sequences, and a key-value cache of historical word-grams in the unloaded text sequence is stored in the second chip.

[0048] When a high-request model processes multiple text sequences as input, the attention calculation task for at least some of these text sequences can be offloaded to a second chip, which then assists in completing the corresponding calculations. These multiple text sequences can correspond to a single inference task or multiple inference tasks, depending on actual needs.

[0049] For example, taking the high-request model as a large language model, the large language model can process service requests from multiple users at the same time, such as answering questions from multiple users. Therefore, the user questions of multiple users can be packaged into one task and input into the large language model. Among them, the multiple text sequences can be the multiple user questions, that is, each user question is a text sequence. For the multiple text sequences, the attention calculation task of a part of the text sequence can be offloaded to the second chip and completed by the second chip.

[0050] For the unloaded text sequence in the multiple text sequences, after calculating the QKV of the new word in the unloaded text sequence, the first chip may send the QKV of the new word to the second chip.

[0051] S202: The second chip calculates an attention result of the unloaded text sequence based on the key-value cache of historical word units in the unloaded text sequence and the QKV of the new word unit;

[0052] In step S202, after receiving the QKV of the new word sent by the first chip, the second chip can calculate the attention result of the unloaded text sequence based on the key value cache of the historical words in the stored unloaded text sequence and the QKV of the new word. The attention result is used to characterize the association relationship between the new word and the unloaded text sequence.

[0053] S203: The second chip sends the attention result of the unloaded text sequence to the first chip;

[0054] In step S203, after calculating the attention result of the unloaded text sequence, the second chip can send the attention result to the first chip.

[0055] S204. The first chip receives the attention result of the uninstalled text sequence returned by the second chip, and calculates the output of the high-request model based on the attention result of the uninstalled text sequence.

[0056] In step S204, the first chip may receive the attention result of the uninstalled text sequence returned by the second chip, and calculate the output of the high-request model based on the attention result of the uninstalled text sequence, such as performing linear transformation, projection processing, etc. on the attention result.

[0057] In some embodiments, the first chip is further configured to calculate, for a non-offloaded text sequence among the multiple text sequences, an attention result for the non-offloaded text sequence based on a currently calculated QKV of a new word in the non-offloaded text sequence and a key-value cache of historical words in the non-offloaded text sequence, wherein the key-value cache of historical words in the non-offloaded text sequence is stored in the first chip.

[0058] When calculating the output of the high-request model based on the attention results of the offloaded text sequence, the first chip can concatenate the attention results of the offloaded text sequence with the attention results of the non-offloaded text sequence, and calculate the output of the high-request model based on the concatenated results. For example, a projection calculation is performed on the concatenated results to obtain the final output.

[0059] For example, consider a multi-model serving platform for natural language processing tasks, such as answering user questions. The platform has two models: a high-volume model based on the Transformer architecture, running on GPU 1, and a low-volume model, running on GPU 2.

[0060] The platform simultaneously receives user questions from multiple users for the high-request model. Each question is an independent text sequence, and these questions are part of the high-request task.

[0061] Assume that the platform receives the following five questions entered by users at the same time:

[0062] User 1: "What is artificial intelligence?"

[0063] User 2: "What are the characteristics of the Transformer architecture?"

[0064] User 3: "How to use natural language processing technology?"

[0065] User 4: "What is the difference between deep learning and machine learning?"

[0066] User 5: "What is the attention mechanism in deep learning?"

[0067] The above user questions can be packaged into one task. These multiple user questions are multiple text sequences that the high-request model needs to process.

[0068] When running a high-request model, GPU1 can offload the attention calculation tasks of some text sequences (i.e., some user questions) to GPU2 based on the current load of GPU2. Suppose GPU1 offloads the questions of users 3 and 4 to GPU2, while the questions of users 1, 2, and 5 are still processed by GPU 1.

[0069] GPU 1 will process questions from User 1, User 2, and User 5:

[0070] For example, user 1’s question: “What is artificial intelligence?”

[0071] GPU 1 can calculate the QKV of the new word of the question. Using the QKV of the new word and the key-value cache of historical words (artificial intelligence, is, what) (stored on GPU 1), the attention result of the text sequence is calculated.

[0072] The same process can be applied to the problems of user 2 and user 5.

[0073] After GPU 1 calculates the QKV of the new word in user 3's question, it can send the QKV of the new word to GPU 2:

[0074] User 3's question: "How to use natural language processing technology?"

[0075] GPU2 has stored the key-value cache of the historical tokens (how, use, natural language processing technology) of the question, which is stored in the previous calculation process. GPU2 receives the QKV of the new tokens, and calculates the attention result of the question by using the QKV of the new tokens and the key-value cache of the historical tokens.

[0076] The same processing can be performed for the question of user 4.

[0077] GPU2 can return the calculated attention results of the questions of user 3 and user 4 to GPU1, and GPU1 splices the attention results of user 3 and user 4 with the attention results of user 1, user 2 and user 5, and then calculates the final output of the high request model based on the spliced results to generate the answer to each question.

[0078] In some embodiments, if the first chip needs to perform the attention calculation task by using the hardware resources of the second chip, the first chip can allocate an attention calculation task to the second chip through a processing unit (such as a CPU), that is, the task allocation and scheduling are performed through the processing unit. However, this way is low in efficiency, and since the second chip itself also needs to perform the inference task of the low request model, the attention calculation task of the first chip can be delayed, resulting in a long response time of the high request model to the user and affecting the user experience. In order to avoid the above problems, in some embodiments, as shown in Figure 3 the first chip can delegate the execution and control logic of the attention calculation task to the second chip, for example, a piece of memory can be allocated in the first chip or the second chip as shared memory of the two. In the shared memory, a special storage space can be allocated to store the QKV of the new token and the first state information, wherein the first state information is used to indicate whether the first chip currently issues a new attention calculation task.

[0079] When the first chip sends the QKV of the new token of the offloaded text sequence currently calculated to the second chip, the first chip can write the QKV of the new token in the shared memory of the first chip and the second chip, and update the first state information stored in the shared memory to inform the second chip that there is a new attention calculation task to be processed through the first state information. For example, the first state information can be represented by a counter, and the first chip can increase the counter by 1 after writing the QKV of the new token in the shared memory, to indicate that the second chip currently has a new attention calculation task.

[0080] After the second chip determines that a new attention calculation task exists based on the first state information, it can obtain the QKV of the new word from the shared memory and calculate the attention result of the unloaded text sequence based on the QKV of the new word and the key-value cache of historical words in the unloaded text sequence stored in its own storage. The second chip can also store the KV (i.e., key-value cache) of the new word as the KV cache of historical words for the next new word.

[0081] Similarly, in some embodiments, when the second chip returns the calculated attention result of the unloaded text sequence to the first chip, it can also be implemented using shared memory. For example, a special storage space can be allocated in the shared memory to store the attention result of the unloaded text sequence and the second state information, and the second state information is used to indicate whether the attention calculation task sent to the second chip is completed. After the second chip calculates the attention result of the unloaded text, it can update the second state information in the shared memory to notify the first chip whether the attention calculation task it has sent is completed. For example, the second state information can be represented by a counter, and after the second chip writes the calculated attention result of the unloaded text sequence into the shared memory, it can add 1 to the counter. The first chip can read the second state information from the shared memory, and when it determines that the attention calculation task sent to the second chip has been completed based on the second state information, it reads the attention result of the unloaded text sequence from the shared memory.

[0082] In some embodiments, the first chip and the second chip are both GPUs, and the shared memory is implemented based on the CUDA IPC mechanism. The CUDA IPC (Inter-Process Communication) mechanism is a technology that allows different CUDA processes to share GPU memory. It improves the efficiency of multi-process parallel computing by providing an efficient way to transfer data between processes, avoiding unnecessary data copying. For example, the sending process can use cudaMalloc to allocate a piece of GPU video memory and obtain the IPC handle (Handle) of the video memory through cudaIpcGetMemHandle. The IPC handle is a unique identifier that can be passed between processes. The IPC handle is passed from the sending process to the receiving process (Consumer), which is usually completed through a standard inter-process communication mechanism (such as shared memory, files, etc.). The receiving process uses the cudaIpcOpenMemHandle function to map the shared memory into its own address space through the IPC handle. In this way, the receiving process can directly access the video memory allocated by the sending process. Among them, the process in the first chip can be used as the sending process and the process in the second chip can be used as the receiving process, or the process in the second chip can be used as the sending process and the process in the first chip can be used as the receiving process. The specific settings can be flexibly based on actual needs.

[0083] In some embodiments, considering that the memory resources in the first chip are relatively limited, a memory area in the memory of the second chip can be set as a shared memory. That is, the shared memory area is located in the memory area of ​​the second chip, and the first chip accesses the shared memory based on CUDA IPC.

[0084] Compared with issuing attention calculation tasks through the processing unit (for example, CPU), setting up shared memory directly on the GPU and entrusting the processing and control logic of the attention calculation tasks to CPU2 can avoid multiple copies of data between the processing unit and the chip, significantly improving data transmission efficiency, that is, improving the efficiency of issuing task instructions and reducing latency.

[0085] In some embodiments, considering that the second chip also needs to perform reasoning tasks related to the low-request model, in order to avoid the attention calculation task issued by the first chip having to wait for the reasoning task of the previously issued low-request model to be completed, it takes too long to wait. Figure 4As shown, the queue of tasks to be executed of the second chip includes first tasks related to operators in the low request volume model, and each first task is interspersed with a second task, and the second task is used to implement the following operations: query the first status information from the shared memory, and when the first status information indicates that there is a new attention calculation task, read the QKV of the new word from the shared memory, calculate the attention result of the unloaded text sequence based on the QKV of the new word and the key-value cache of the historical words in the unloaded text sequence, write it to the shared memory, and update the second status information.

[0086] By inserting the second task into the queue of tasks to be executed on the second chip, efficient response and processing of attention calculation tasks is achieved. The second task is responsible for polling the status information in the shared memory. Once a new attention calculation task is detected, it immediately reads the QKV tensor of the new word from the shared memory, and uses the locally stored key-value cache to calculate the attention result of the unloaded text sequence. The result is then written back to the shared memory and the status information is updated. This mechanism effectively avoids the long delay caused by the attention calculation task issued by the first chip waiting for the completion of the previous task, significantly improving the timeliness of task processing and the overall response speed of the system. At the same time, by rationally arranging the task queue, the computing resources of the second chip are fully utilized, resource allocation is optimized, and the parallel processing capability of the system is enhanced, thereby improving the efficiency and performance of the multi-model service platform in processing high-request model and low-request model tasks, providing users with a smoother and more efficient service experience.

[0087] In some embodiments, considering that the second task (i.e., the attention calculation task) needs to wait for the completion of the previous first task before it can be executed, in order to avoid the execution time of the first task being too long, which causes the second task to wait for a long time, resulting in a large delay. The operators on the low request volume model can be split and processed to obtain a target operator set (i.e., a set of operators related to the first task). Among them, the execution time corresponding to each operator in the target operator set is less than the preset execution time, wherein each operator in the target operator set is the original operator in the low request volume model, or the split operator obtained by splitting the original operator in the low request volume model. The embodiment of the present application effectively reduces the execution time of a single operator by splitting the operators in the low request volume model to generate a target operator set, avoiding the large delay of the second task (attention calculation task) caused by the long execution time of the first task. This splitting strategy ensures that the execution time of each operator in the target operator set is less than the preset time, thereby significantly reducing the waiting time between tasks and improving the response speed and overall throughput of the system. In addition, by rationally splitting the original operators, this solution not only optimizes resource allocation, but also enhances the system's parallel processing capabilities, enabling the multi-model service platform to more efficiently perform offloading tasks for high-request models while ensuring that inference tasks for low-request models are not affected, thereby improving the overall system performance and user experience.

[0088] Considering that different operators in the model have different execution times, if the operators are not split enough, the waiting time of the attention calculation task may be too long. If the operators are split too small, it may lead to excessive fragmentation and introduce performance overhead. Therefore, in some embodiments, in order to obtain the best splitting effect, when determining the target operator set, the original operator set constructed by all the original operators in the low-request model can be obtained, and the original operator set can be used as the current operator set for the first round of iteration. The following operator splitting operation is performed until the maximum execution time of the operators in the current operator set is less than the preset time: the target operator with the longest execution time can be obtained from the current operator set; if the execution time of the target operator is greater than the preset time, the target operator is split into two split operators, and the two split operators are used to replace the target operator in the current operator set, wherein the execution time of each split operator is half of the execution time of the target operator.

[0089] For example, assume that the set of all primitive operators in the low-request model is Q. The primitive operators contained in Q and their corresponding execution times are as follows:

[0090] OP1: 10ms

[0091] OP2: 8ms

[0092] OP3: 6ms

[0093] OP4: 4ms

[0094] Assume that the preset time length WTthreshold=5ms.

[0095] First iteration:

[0096] OPmax=OP1(10ms)

[0097] Tmax=10ms

[0098] Tmax>5ms

[0099] Split OP1 into OP1a (5ms) and OP1b (5ms),

[0100] Remove OP1 from the original operator set Q and add OP1a and OP1b.

[0101] At this point, the operators in the current operator set and their corresponding execution times are as follows: OP1a (5ms), OP1b (5ms), OP2 (8ms), OP3 (6ms), OP4 (4ms)

[0102] Second iteration:

[0103] OPmax=OP2(8ms)

[0104] Tmax=8ms

[0105] Tmax>5ms

[0106] Split OP2 into OP2a (4ms) and OP2b (4ms),

[0107] Remove OP2, add OP2a and OP2b,

[0108] At this time, the operators in the current operator set and their corresponding execution times are as follows: OP1a (5ms), OP1b (5ms), OP2a (4ms), OP2b (4ms), OP3 (6ms), OP4 (4ms).

[0109] Third iteration:

[0110] OPmax=OP3(6ms)

[0111] Tmax=6ms

[0112] Tmax>5

[0113] Split OP3 into OP3a (3ms) and OP3b (3ms),

[0114] Remove OP3 and add OP3a and OP3b.

[0115] At this time, the operators in the current operator set and the corresponding execution time are as follows: Queue Q: OP1a (5ms), OP1b (5ms), OP2a (4ms), OP2b (4ms), OP3a (3ms), OP3b (3ms), OP4 (4ms).

[0116] Fourth iteration:

[0117] OPmax=OP1a or OP1b

[0118] Tmax=5ms

[0119] Tmax≤5ms

[0120] End the iteration.

[0121] The final target operator set includes the split operators:

[0122] OP1a (5ms)

[0123] OP1b(5ms)

[0124] OP2a (4ms)

[0125] OP2b (4ms)

[0126] OP3a (3ms)

[0127] OP3b (3ms)

[0128] OP4 (4ms)

[0129] Through the priority-based operator splitting strategy, operators with long execution time can be split into multiple smaller subtasks, thereby reducing head-of-line blocking, improving the efficiency of parallel computing, and alleviating head-of-line blocking without generating a large number of small kernels (i.e., small operators).

[0130] The following is an introduction to the implementation of the attention calculation task and operator splitting calculation through shared memory in the embodiment of the present application with a specific example. Taking the first chip and the second chip as GPUs as an example (assuming that the first chip is GPU1 and the second chip is GPU2), in order to avoid the unloaded attention operator being blocked by the pre-issued kernel in GPU2 (a GPU kernel can be regarded as a task to be executed), a preliminary method is to issue the kernel in a blocking manner: that is, only when a kernel is completed, the CPU of GPU2 will issue the next kernel (if there is an unloaded attention operator, it will be issued first). However, this method will significantly affect the computing throughput of GPU2. This embodiment proposes a GPU-driven dynamic control flow technology to avoid the blocking problem caused by the pre-issued kernel, while minimizing the impact on GPU performance. In this embodiment, the control logic for issuing the unloaded attention operator is delegated to GPU2, and synchronized using the cross-GPU shared memory function (cudaIpcMemory). After adopting the GPU-driven dynamic control flow, the process of unloading the attention operator is as follows: Figure 5 As shown. When GPU1 needs to offload an attention operator, it uses one-sided writes to write the QKV tensor of the latest word to shared memory (②) and atomically increases a task counter (③). GPU2 polls the task counter (④). Once the counter is detected to increase, indicating that there is a new task, GPU2 executes the attention operator, then writes the result tensor to shared memory (⑤), and finally increases a completion counter (⑥). Once GPU1 finds that the completion counter has changed through polling (⑦), it obtains the result tensor from shared memory and continues subsequent calculations. All of the above operations of GPU2, including polling, counter update, data transfer and execution of attention calculation tasks, are encapsulated in a GPU kernel, which is called a polling kernel in this embodiment. Like the existing LLM service system, the CPU pre-issues many kernels (these kernels are tasks related to the low request volume model) to saturate the GPU2 hardware queue, but each kernel is followed by a polling kernel. In this way, the offloaded attention calculation task will only be blocked by the kernel being executed, and will not be blocked by other pre-issued kernels.

[0131] In some embodiments, to simplify the overall process, updates to attention metadata (e.g., sequence page tables) are still performed on the CPU, and these updates are infrequent (i.e., only once per iteration) and therefore do not cause a performance bottleneck.

[0132] In addition, to avoid head-of-line blocking caused by long-running kernels, the present embodiment proposes an operator splitting technique, the core idea of which is to split the kernel of a large operator into smaller kernels, so that the offloaded attention computation tasks can be executed on GPU2 after waiting for a bounded time. However, it is not easy to generate an optimal splitting scheme, considering that: 1) the operators in the LLM service have different execution times; 2) excessive fragmentation caused by operator splitting can introduce performance overhead. To this end, the present embodiment uses a priority-based algorithm to generate a splitting scheme. The present embodiment maintains a priority queue containing all operators of the low-request model on GPU2, and iteratively splits the operator with the longest execution time into two halves. If the split operator still exceeds the threshold, it is reinserted into the queue. This process continues until the waiting time is reduced below a predefined threshold. In some embodiments, the threshold can be set to 5% of the average iteration time of GPU1, where the average iteration time is the time it takes for the high-request model to generate a new token.

[0133] Through this operator splitting technique, the present embodiment can alleviate head-of-line blocking without generating a large number of small kernels.

[0134] In the operator splitting process, for parameterized operators (such as operators for projection calculations), such parameterized operators can be split in the weight dimension, and for non-parameterized operators (such as rms_norm), such operators can be split in the sequence dimension.

[0135] In some embodiments, the number of offloaded text sequences and the proportion of the plurality of text sequences are dynamically adjusted based on the current call volume of the low-request model, where the proportion is inversely related to the current call volume of the low-request model. By dynamically adjusting the number of text sequences offloaded to the low-request model and the proportion of the total number of input text sequences that the high-request model needs to process, flexibility and efficiency of resource allocation can be achieved. Specifically, when the current call volume of the low-request model is low, the number of offloaded text sequences and their proportion are increased, making full use of the idle resources of the GPU where the low-request model is located, thereby improving overall resource utilization. Conversely, when the call volume of the low-request model increases, the number of offloaded text sequences and their proportion are reduced, avoiding excessive impact on the normal inference tasks of the low-request model and ensuring its service quality. This dynamic adjustment mechanism not only optimizes resource allocation, but also enhances the adaptability and stability of the system, enabling the multi-model service platform to more efficiently handle different load situations and improving overall inference efficiency and user experience.

[0136] In some embodiments, the second chip stores a key-value cache of historical word elements of the unloaded text sequence and a key-value cache of word elements in the input text sequence of the low-request volume model, and the storage space of the key-value cache of historical word elements of the unloaded text sequence and the key-value cache of word elements in the input text sequence of the low-request volume model is allocated based on the minimum memory allocation granularity, which is determined based on the number of heads and dimensions of the key-value cache of the high-request volume model and the number of heads and dimensions of the KV cache of the low-request volume model.

[0137] In the Transformer architecture, the attention mechanism is implemented using multi-head attention. Each head independently calculates the attention weights between the query, key, and value vectors. The number of KV cache heads refers to the number of heads used to store the key and value vectors in the multi-head attention mechanism. For example, if a Transformer model uses 8 heads (i.e., num_heads = 8), the key and value vectors of each word are divided into 8 different sub-vectors, one for each head. These sub-vectors are stored separately in the KV cache, and each head has its own KV cache block. The dimension of the KV cache refers to the size of each key and value vector. In the Transformer architecture, the embedding vector of each word is divided into multiple heads, with each head processing a portion of the vector. Therefore, the dimension of the key and value vectors of each head is the dimension of the entire embedding vector divided by the number of heads. In order to support models with different numbers of headers and dimensions, and to use a unified KV block allocator to allocate memory for the key-value caches of different models, the minimum allocation granularity can be determined based on the number of headers and dimensions of the key-value caches of different models, and memory can be allocated based on this minimum allocation granularity. The minimum allocation granularity can be the common divisor of the product of the number of headers and dimensions of the key-value caches of each model, for example, the greatest common divisor can be taken. For example, a unified KV block allocator can be built using the slab allocator, with a minimum allocation granularity of 128 bytes, to support high-request-volume models and low-request-volume models with different numbers of KV cache headers and dimensions.

[0138] This solution significantly improves memory utilization and overall system performance by uniformly managing the key-value cache of high-request models and low-request models based on the minimum memory allocation granularity. Specifically, the minimum memory allocation granularity is dynamically determined based on the number and dimensions of the KV cache heads of the high-request models and the low-request models, and can flexibly adapt to the caching needs of different models. This design not only reduces memory fragmentation and ensures the efficiency and flexibility of memory allocation, but also optimizes the utilization of cache lines through memory alignment, further improving memory access efficiency. In addition, the unified memory management mechanism simplifies system design and reduces management overhead, enabling the multi-model service platform to handle high-request and low-request tasks more efficiently, while supporting the diverse needs of different models, thereby achieving a better balance between resource utilization and performance.

[0139] Among them, the solutions of the above embodiments can be freely combined to obtain new solutions when there is no conflict. Due to space constraints, they will not be listed one by one here.

[0140] In addition, an embodiment of the present application further provides a computer program product, which includes a computer program, and when the computer program is executed, implements the method mentioned in any of the above embodiments.

[0141] Furthermore, the present invention also provides a chip. Figure 6 As shown, the chip includes a processor 61, a memory 62, and computer instructions stored in the memory 62 for execution by the processor 61. When the processor 61 executes the computer instructions, the method described in any of the above embodiments is implemented. The chip can be a CPU, a GPU, etc., and this embodiment of the application does not limit this.

[0142] Accordingly, an embodiment of the present application further provides a computer storage medium, in which a program is stored. When the program is executed by a processor, the method in any of the above embodiments is implemented.

[0143] The embodiments of the present application may take the form of a computer program product implemented on one or more storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing program code. Computer-usable storage media include permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs or other data. Examples of computer storage media include but are not limited to: phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disc, read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, disk or other magnetic storage device or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

[0144] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0145] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements includes not only those elements, but also other elements that are not explicitly listed, or also includes elements that are inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a..." does not exclude the presence of other identical elements in the process, method, article or device that includes the element.

[0146] The above is a detailed introduction to the methods and devices provided in the embodiments of the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the methods and core ideas of the present application. At the same time, for those skilled in the art, based on the ideas of the present application, there may be changes in the specific implementation methods and application scope. In summary, the content of the present application should not be understood as a limitation on the present application.

Claims

1. A model reasoning method, characterized in that: The method is applicable to a multi-model service platform, the multi-model service platform including a first chip for running a high-request volume model and a second chip for running a low-request volume model. The number of service requests received by the high-request volume model per unit time is greater than the number of service requests received by the low-request volume model. The high-request volume model is a model based on the Transformer architecture. The method is executed by the first chip, and the method includes: In the process of processing multiple input text sequences using the high-request volume model, the QKV of the new word of the unloaded text sequence in the multiple text sequences currently calculated is sent to the second chip, so that the second chip calculates the attention result of the unloaded text sequence based on the key-value cache of the historical word-grams in the unloaded text sequence and the QKV of the new word-gram; wherein the unloaded text sequence is at least a part of the text sequence in the multiple text sequences, and the key-value cache of the historical word-grams in the unloaded text sequence is stored in the second chip; Receive the attention result of the uninstalled text sequence returned by the second chip, and calculate the output of the high-request model based on the attention result of the uninstalled text sequence.

2. The method according to claim 1, characterized in that The method further comprises: Calculating an attention result of the non-unloaded text sequence based on the currently calculated QKV of the new word in the non-unloaded text sequence among the multiple text sequences and the key-value cache of the historical word in the non-unloaded text sequence; wherein the key-value cache of the historical word in the non-unloaded text sequence is stored in the first chip; The calculating the output of the high-request model based on the attention result of the uninstalled text sequence includes: The attention results of the uninstalled text sequence and the attention results of the non-uninstalled text sequence are spliced ​​together, and the output of the high-request model is calculated based on the splicing results.

3. The method according to claim 1, characterized in that The step of sending the currently calculated QKV of the new word of the unloaded text sequence in the multiple text sequences to the second chip includes: The QKV of the new word is written into the shared memory of the first chip and the second chip, and the first status information stored in the shared memory is updated, so that after the second chip determines that there is a new attention calculation task based on the first status information, it obtains the QKV of the new word from the shared memory, and obtains the attention result of the unloaded text sequence based on the QKV of the new word and the key-value cache calculation of the historical words in the unloaded text sequence; wherein the first status information is used to indicate whether the first chip issues a new attention calculation task.

4. The method according to claim 1, wherein The receiving the attention result of the uninstalled text sequence returned by the second chip includes: Reading second status information from the shared memory of the first chip and the second chip, where the second status information is used to indicate whether the attention calculation task sent to the second chip is completed; When it is determined based on the second status information that the attention calculation task issued to the second chip has been completed, the attention result of the unloaded text sequence is read from the shared memory, wherein the attention result of the unloaded text sequence is calculated by the second chip based on the QKV of the new word and the key-value cache of the historical word of the unloaded text sequence, and the second status information is updated by the second chip after completing the attention calculation task.

5. The method according to claim 3 or 4, characterized in that The first chip and the second chip are both GPUs, and the shared memory is implemented based on a CUDA IPC mechanism.

6. The method according to claim 1, characterized in that The ratio of the number of the uninstalled text sequences to the total number of the multiple text sequences is dynamically adjusted based on the current call volume of the low request volume model, wherein the ratio is negatively correlated with the current call volume.

7. A model reasoning method, characterized in that: The method is applicable to a multi-model service platform, the multi-model service platform including a first chip for running a high-request volume model and a second chip for running a low-request volume model. The number of service requests received by the high-request volume model per unit time is greater than the number of service requests received by the low-request volume model. The high-request volume model is a model based on the Transformer architecture. The method is executed by the second chip, and the method includes: receiving a QKV of a new word in an unloaded text sequence sent by the first chip, wherein the unloaded text sequence is at least a portion of a plurality of text sequences currently processed by the high-request volume model, and a key-value cache of historical words in the unloaded text sequence is stored in the second chip; Calculating an attention result of the unloaded text sequence based on a key-value cache of historical word units in the unloaded text sequence and a QKV of the new word unit; The attention result of the uninstalled text sequence is sent to the first chip, so that the first chip calculates the output of the high-request model based on the attention result of the uninstalled text sequence.

8. The method according to claim 7, characterized in that The receiving the QKV of the new word of the unloaded text sequence sent by the first chip includes: Reading first status information from a shared memory of the first chip and the second chip, wherein the first status information is used to indicate whether the first chip issues a new attention calculation task; When it is determined based on the first state information that there is a new attention calculation task, the QKV of the new word is read from the shared memory, wherein the QKV of the new word is calculated by the first chip and written into the shared memory, and the first state information is updated by the first chip after writing the QKV of the new word.

9. The method according to claim 7 or 8, characterized in that The sending the attention result of the unloaded text sequence to the first chip includes: The attention result of the unloaded text sequence is written into the shared memory of the first chip and the second chip, and the second status information stored in the shared memory is updated, so that the first chip reads the attention result of the unloaded text sequence from the shared memory when it is determined based on the second status information that the attention calculation task sent to the second chip has been completed; wherein the second status information is used to indicate whether the attention calculation task sent to the second chip is completed.

10. The method according to claim 9, characterized in that The queue of tasks to be executed of the second chip includes first tasks related to the operators in the low-request model, and each first task is interspersed with a second task, and the second task is used to implement the following operations: query the first status information from the shared memory, and when the first status information indicates that there is a new attention calculation task, read the QKV of the new word from the shared memory, calculate the attention result of the unloaded text sequence based on the QKV of the new word and the key-value cache of the historical words in the unloaded text sequence, write it to the shared memory, and update the second status information.

11. The method according to claim 10, characterized in that The set of operators related to the first task is the target operator set, and the execution time corresponding to each operator in the target operator set is less than the preset execution time, wherein each operator in the target operator set is the original operator in the low request volume model, or a split operator obtained by splitting the original operator in the low request volume model.

12. The method according to claim 11, characterized in that The target operator set is obtained based on the following method: Get the primitive operator set constructed by all primitive operators in the low-request-volume model: The original operator set is used as the current operator set for the first iteration, and the following operator splitting operation is performed until the maximum execution time of the operators in the current operator set is less than the preset time: Get the target operator with the longest execution time from the current operator set; If the execution time of the target operator is greater than the preset time, the target operator is split into two split operators, and the two split operators are used to replace the target operator in the current operator set, wherein the execution time of each split operator is half of the execution time of the target operator.

13. The method according to claim 1, wherein The second chip stores the key value cache of historical word elements of the unloaded text sequence and the key value cache of word elements in the input text sequence of the low request volume model. The storage space of the key value cache of historical word elements of the unloaded text sequence and the key value cache of word elements in the input text sequence of the low request volume model is allocated based on the minimum memory allocation granularity, and the minimum memory allocation granularity is determined based on the number of heads and dimensions of the key value cache of the high request volume model and the number of heads and dimensions of the key value cache of the low request volume model.

14. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed, the method according to any one of claims 1 to 13 is implemented.

15. A chip, characterized in that: The chip includes a processor, a memory, and computer instructions stored in the memory for execution by the processor. When the processor executes the computer instructions, the method according to any one of claims 1-6 or 7-13 is implemented.

Images