Resource scheduling method and device of large model, storage medium and electronic equipment

Abstract

The invention provides a large model resource scheduling method, a large model resource scheduling device, a computer storage medium and electronic equipment, and relates to the technical field of computers. The method comprises the following steps: receiving a decoding task request; wherein the decoding task request at least comprises a first key value cache and a first output token, and the first key value cache and the first output token are generated by performing pre-filling processing on a second cue word formed by splicing a first cue word input by a user with a prediction cue word in a pre-filling resource pool; according to the first output token, predicting the number of decoding iterations corresponding to the execution of the decoding task request; determining a target capacity value of the reasoning card based on a first capacity value occupied by the first key value cache, a second capacity value occupied by a second key value cache generated under the decoding iteration times and a third capacity value occupied by the reasoning card in the decoding resource pool; and determining a target inference card according to the target capacity value of the inference card. The method can improve the resource utilization rate.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a resource scheduling method for large models, a resource scheduling device for large models, a computer storage medium, and an electronic device. Background Technology

[0002] In the process of performing inference tasks using a large model, the Prefill and Decode phases are the two core phases. The Prefill phase is the process by which the large model generates the first token after receiving the prompt words input by the user. The Decode phase is the process of generating the next token using the previously generated token and KV cache data, until the iteration ends and the inference task is completed.

[0003] In relevant technical solutions, the high parallel processing advantage of current mainstream inference cards is often used to batch process a large number of inference task requests. However, considering the limitations of the inference card's memory capacity, the number of batch processing requests is restricted. Existing batch processing methods set a memory limit for each request.

[0004] However, the number of KVCaches generated during the decoding phase for each request is positively correlated with the length of the required output content. Since the output length is uncertain, the aforementioned uniform upper limit method suffers from insufficient video memory resources leading to output failures for longer output content, while wasting video memory resources for shorter output content. This results in low resource utilization. Summary of the Invention

[0005] This disclosure provides a resource scheduling method, a resource scheduling device for large models, a computer storage medium, and an electronic device, thereby improving resource utilization.

[0006] In a first aspect, one embodiment of this disclosure provides a resource scheduling method for a large model. The method includes: receiving a decoding task request from a pre-filled resource pool of the large model; wherein the decoding task request includes at least a first key-value cache and a first output token, the first key-value cache and the first output token being generated by pre-filling the resource pool for a second prompt word formed by concatenating a first prompt word input by the user with a predicted prompt word; predicting the number of decoding iterations corresponding to the decoding task request based on the first output token; determining a target capacity value for the inference card based on a first capacity value occupied by the first key-value cache, a second capacity value occupied by the second key-value cache generated under the number of decoding iterations, and a third capacity value already occupied by the inference card in the decoding resource pool; and determining a target inference card based on the target capacity value of the inference card, so as to execute the decoding task corresponding to the decoding task request based on the target inference card.

[0007] Secondly, one embodiment of this disclosure provides a resource scheduling apparatus for a large model. The apparatus includes: a task receiving module configured to receive a decoding task request from a pre-filled resource pool of the large model; wherein the decoding task request includes at least a first key-value cache and a first output token, the first key-value cache and the first output token being generated by pre-filling the pre-filled resource pool for a second prompt word formed by concatenating a first prompt word input by the user with a predicted prompt word; an iteration count prediction module configured to predict the number of decoding iterations corresponding to the decoding task request based on the first output token; a target capacity value determination module configured to determine a target capacity value of an inference card based on a first capacity value occupied by the first key-value cache, a second capacity value occupied by the second key-value cache generated under the decoding iteration count, and a third capacity value already occupied by the inference card in the decoding resource pool; and an inference card determination module configured to determine a target inference card based on the target capacity value of the inference card, so as to execute the decoding task corresponding to the decoding task request based on the target inference card.

[0008] Thirdly, one embodiment of this disclosure provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the resource scheduling method of the large model described above.

[0009] Fourthly, one embodiment of this disclosure provides an electronic device, including: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the resource scheduling method of the large model described above by executing the executable instructions.

[0010] Fifthly, one embodiment of this disclosure provides a computer program product, including a computer program that is executed by a processor to implement the resource scheduling method of the large model described above.

[0011] The technical solution disclosed herein has the following beneficial effects: The resource scheduling method for the aforementioned large model receives decoding task requests from the pre-filled resource pool of the large model. Each decoding task request includes at least a first key-value cache and a first output token. The first key-value cache and the first output token are generated by pre-filling the resource pool with a second prompt word formed by concatenating the first prompt word input by the user with a predicted prompt word. Based on the first output token, the number of decoding iterations corresponding to the decoding task request is predicted. Based on the first capacity value occupied by the first key-value cache, the second capacity value occupied by the second key-value cache generated under the decoding iteration count, and the third capacity value already occupied by the inference card in the decoding resource pool, the target capacity value of the inference card is determined. Based on the target capacity value of the inference card, a target inference card is determined, and the decoding task corresponding to the decoding task request is executed based on the target inference card.

[0012] Firstly, this method pre-fills the resource pool with a second prompt word formed by concatenating the first prompt word with the predicted prompt word. This yields a first output token that can predict the number of decoding iterations and, consequently, the capacity change caused by the KVCache changes during subsequent decoding stages. This not only enables fine-grained scheduling of subsequent decoding tasks and allocates the request to a suitable inference card for processing, avoiding the technical problem of low resource utilization due to insufficient or surplus inference card resources, but also significantly reduces prediction overhead. Secondly, this method calculates the target capacity value of the inference card. By leveraging the characteristic that the number of KVCache entries reaches its peak at the end of each request, the target inference card is selected. This reduces the time and space complexity of the algorithm while improving the accuracy of target inference card selection, thereby enhancing the reliability and security of the system.

[0013] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0014] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0015] Figure 1A This diagram illustrates one example of a request-level batch processing method in this exemplary embodiment. Figure 1B This schematic diagram illustrates one of the iterative batch processing methods of this exemplary embodiment; Figure 2 This schematically illustrates an application architecture diagram of a large-scale resource scheduling system according to one of the exemplary embodiments of this invention. Figure 3 The flowchart schematically illustrates a resource scheduling method for a large model in this exemplary embodiment; Figure 4A The flowchart illustrates a method for generating a first key-value cache and a first output token from a pre-populated resource pool according to one of the exemplary embodiments of the present invention. Figure 4B This schematically illustrates a pre-filled resource pool generating a first key-value cache and a first output token, and processing them based on a decoded resource pool, according to one of the exemplary embodiments of this invention. Figure 5The flowchart illustrating one of the methods for determining the target capacity value of an inference card according to this exemplary embodiment is shown in the diagram. Figure 6 This schematically illustrates the capacity change of one of the inference cards in this exemplary embodiment when it receives a new request; Figure 7 This schematic diagram illustrates a capacity change procedure when an inference card receives a new request, according to one of the exemplary embodiments of the present invention. Figure 8 The flowchart illustrates a resource scheduling method for one overall large model of this exemplary embodiment. Figure 9 This schematic diagram illustrates the structure of a resource scheduling device for a large model in this exemplary embodiment. Figure 10 The schematic diagram illustrates the structure of an electronic device in this exemplary embodiment. Detailed Implementation

[0016] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this disclosure more comprehensive and complete, and to fully convey the concept of exemplary embodiments to those skilled in the art. The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a full understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more specific details omitted, or other methods, components, apparatus, steps, etc., can be employed. In other instances, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0017] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0018] The flowchart shown in the attached diagram is merely an illustrative example and does not necessarily include all steps. For example, some steps may be broken down, while others may be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0019] To help those skilled in the art better understand the technical solutions of this disclosure, the relevant content involved in the technical solutions of this disclosure will be introduced below.

[0020] 1) Large Model Inference: After training a large model, such as a Large Language Model (LLM), pre-trained large language models (e.g., Generative Pre-trained Transformer (GPT), Bidirectional Encoder Representations from Transformers (BERT), ChatGenerative Language Model (ChatGLM), etc.) are used to provide users with real-time or batch text generation, text understanding, question answering, and other services. The inference process is the process by which the large language model generates the answer content. In each inference iteration, the large language model produces an output token until a termination token is generated, at which point the inference iteration ends.

[0021] 2) PD Separation Technology: This is a key technology for optimizing LLM model inference. The large model inference process is divided into two independent stages: Prefill and Decode, which are deployed on different computing resources to improve resource utilization and inference efficiency. Since both Prefill and Decode share the same GPU resources, this can easily lead to idle computing power or bandwidth. PD separation technology achieves precise resource matching through decoupling.

[0022] 3) Prefill: All input prompts requested by the user are fed into the LLM model together. The LLM model generates key-value cache data (also known as KVCache data) for each prompt and generates and outputs the first token.

[0023] 4) Decoding: This is an iterative process. Each iteration outputs a token, which is then used as the input token for the next iteration (the first token comes from prefilling). In each iteration, a KVCache is generated for the input token. Each iteration requires inputting the KVCache generated by prefilling, along with the KVCaches generated by all previous iterations, into the LLM model.

[0024] 5) KVCache (Key-Value Cache): This is a core mechanism used to optimize computational efficiency during LLM model inference. Essentially, it avoids repeatedly calculating the attention weights of historical tokens by caching intermediate results (key and value vectors, i.e., KVCache data) in the Transformer self-attention mechanism. Furthermore, KVCache data is generated during the large model inference process. For each input token, corresponding KVCache data is generated, and the KVCache data is related to the order of the input tokens, thus containing information about all previous tokens.

[0025] 6) Inference Request Scheduling: The resource pool of the large model handles a large number of inference requests. The requests are scheduled to different inference cards in stages using batch scheduling. Since the KVCache data of the inference requests will occupy a large amount of GPU (Graphics Processing Unit) memory, memory resource constraints need to be considered during scheduling. Therefore, the scheduling optimization goal is to minimize resource usage while meeting the user service quality requirements.

[0026] 7) AI Inference Card: This is a hardware acceleration device specifically designed for artificial intelligence inference tasks. It achieves low-latency, high-throughput model deployment by optimizing computing architecture and energy efficiency. It can contain dedicated AI acceleration units, such as NPU (Neural Processing Unit) or TPU (Tensor Processing Unit), for matrix operations; or it can contain general-purpose computing units, such as CPU (Central Processing Unit) or GPU cores, for logic control or hybrid computing.

[0027] In applications using large models for inference tasks, the inference process mainly consists of two stages: Prefill and Decode. The Prefill stage involves the LLM model generating the first token upon receiving user prompts, a process involving numerous matrix operations, making it computationally intensive. The Decode stage utilizes previously generated tokens and KV cache data to generate the next token, continuing until the iteration ends. The main bottleneck in this stage is GPU memory capacity. Large model inference tasks are characterized by high computational demands, large GPU memory usage, and, in particular, a significant increase in the generated KV cache data as the input sequence length increases. Therefore, the resource requirements of inference tasks are difficult to predict, and resource scheduling faces the challenge of balancing memory and computational resources.

[0028] In relevant technical solutions, considering that current mainstream inference cards (such as GPUs and NPUs) all have the advantage of high parallel processing, batch processing technology can be used to improve the utilization efficiency of inference cards. Batch processing technology can package multiple user requests and input them into the inference card together. The inference card can output the results of all requests simultaneously, and then separate the response results of each request, thereby greatly increasing the throughput.

[0029] However, for the computationally intensive pre-filling process described above, a large number of requests can lead to a decrease in computation speed, thereby increasing the perceived latency for users and affecting their user experience. Furthermore, for the memory-intensive decoding stage, the number of batch requests is also limited by the available memory capacity.

[0030] Furthermore, in related technical solutions, considering the iterative nature of the decoding process, the batch processing has evolved from request-level batching to iterative-level batching (also known as continuous batch processing). The following will combine... Figure 1A , Figure 1B The two batch processing methods described above are illustrated by example.

[0031] Figure 1A This schematic diagram illustrates a request-level batch processing method in this exemplary embodiment; see reference. Figure 1A As shown, request-level batching requires all requests to complete before proceeding to the next batch. However, this approach is not advantageous for requests that terminate early, as they must wait for other requests to finish. Figure 1B This schematic diagram illustrates an iterative batch processing method in this exemplary embodiment; see reference. Figure 1B As shown, if a request ends after each iteration, a new request can be added. This approach is compared to... Figure 1A The method shown not only improves request processing efficiency but also greatly increases GPU utilization.

[0032] Existing technical solutions can already perform decoding request scheduling at the iteration level. However, when considering GPU memory constraints, an upper limit is set for the GPU memory used by each request, thereby determining the upper limit of requests that can be scheduled on each GPU (e.g., Figure 1A , Figure 1B The system is set to process 4 requests simultaneously. The number of KVCaches generated during the decoding phase is positively correlated with the length of the output sequence. Since the length of the output sequence is uncertain, this uniform upper limit method is prone to failure due to insufficient video memory resources for output series with long sequence lengths, while it leads to a waste of video memory resources for output series with short sequence lengths.

[0033] Therefore, there is an urgent need for a fine-grained resource scheduling method for large models to avoid the technical problems of output failure due to insufficient video memory resources or resource waste due to excessive video memory resources, so as to improve resource utilization.

[0034] This exemplary embodiment addresses the aforementioned problems by proposing a resource scheduling method for large models. This method leverages the capabilities of the large model itself, enabling it to focus on predicting the final output length of the first prompt word input by the user. It then predicts the KVCache transformation amount generated during the decoding iteration of the inference service request, thereby obtaining the change in video memory capacity caused by the KVCache transformation. This change is then superimposed with the currently occupied capacity value of each inference card to dynamically configure the number of batch requests matching its video memory capacity. This avoids the technical problems of output failure due to insufficient video memory resources or resource waste due to excessive video memory resources in the aforementioned related technical solutions, achieving the technical effect of improving resource utilization during inference task processing.

[0035] To address the aforementioned problems, this disclosure proposes a resource scheduling method and apparatus for large-scale models, which can be applied to... Figure 2 In the system architecture of the exemplary application environment shown.

[0036] like Figure 2 As shown, system architecture 200 may include terminal device 201 and server 202.

[0037] The terminal device 201 can be any device involved in implementing intelligent reasoning, such as a mobile phone, tablet computer (PAD), laptop computer, desktop computer, smart TV, smart in-vehicle device, smart wearable device, or aircraft. The terminal device can install target applications that provide services such as intelligent question answering, text generation, and text understanding. It should be noted that the target application involved in this embodiment can be a software client, a webpage, a mini-program, etc., and the server 202 is the server corresponding to the software, webpage, mini-program, etc., without limiting the specific type of client.

[0038] Server 202 can be a backend server for the target application, used to provide corresponding backend services. It can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, i.e., Content Delivery Network (CDN), as well as big data and artificial intelligence platforms, but it is not limited to these.

[0039] It should be noted that the resource scheduling method for large models in this embodiment can be executed by either the terminal device 201 or the server 202 alone, or by both the server 202 and the terminal device 201. Both the server 202 and the terminal device 201 may include one or more processors, memory, and interactive I / O interfaces. Furthermore, the server 202 may be configured with a database to store data generated during model inference. The memory of both the server 202 and the terminal device 201 may also store the program instructions required for execution in the resource scheduling method for large models provided in this embodiment. These program instructions, when executed by the processor, can be used to implement the resource scheduling process for large models provided in this embodiment.

[0040] When the resource scheduling method for the large model provided in this embodiment is executed by either server 202 or terminal device 201 alone, the above application scenario may also include only a single device, either server 202 or terminal device 201. Alternatively, server 202 and terminal device 201 may be considered as the same device. Of course, in practical applications, when the resource scheduling method for the large model provided in this embodiment is executed by both server 202 and terminal device 201, server 202 and terminal device 201 may also be the same device. That is, server 202 and terminal device 201 may be different functional modules of the same device, or virtual devices virtualized by the same physical device.

[0041] Furthermore, in the embodiments of this disclosure, the terminal device 201 and the server 202 can communicate directly or indirectly through one or more networks 203. The network 203 can be a wired network or a wireless network; for example, the wireless network can be a mobile cellular network or a wireless fidelity network. The network used is Fidelity (Wi-Fi), but it can also be other possible networks; the embodiments disclosed herein are not limited to these. It should be noted that... Figure 2 The examples shown are merely illustrative. In reality, the number of terminal devices and servers is unlimited. Depending on the implementation needs, there can be any number of terminal devices, networks, and servers. No specific limitations are made in the embodiments disclosed herein.

[0042] For example, in one exemplary embodiment of this disclosure, taking a smart question-and-answer application based on a large language model as an example, a user can run the smart question-and-answer application through a terminal device 201 and trigger an inference task request. Upon receiving the inference task request, the server 202 can pre-fill the inference task request using the pre-filled resource pool of the large language model to obtain a first key-value cache and a first output token. Then, it can package the first key-value cache and the first output token to generate a decoding task request and send it to the decoding resource pool. The decoding resource pool receives decoding task requests. Each decoding task request includes at least a first key-value cache and a first output token. The first key-value cache and the first output token are generated by pre-filling the resource pool with a second prompt word formed by concatenating the first prompt word input by the user with a predicted prompt word. Based on the first output token, the decoding iteration number corresponding to the decoding task request can be predicted. Then, the decoding resource pool determines the target capacity value of the inference card based on the first capacity value occupied by the first key-value cache, the second capacity value occupied by the second key-value cache generated under the decoding iteration number, and the third capacity value already occupied by the inference card in the decoding resource pool. Based on the target capacity value of the inference card, the target inference card is determined, and the decoding task corresponding to the decoding task request is executed based on the target inference card.

[0043] However, those skilled in the art will readily understand that the above application scenarios are merely illustrative and are not intended to limit the scope of this exemplary embodiment.

[0044] The following example uses the aforementioned terminal device as the execution subject and illustrates how the resource scheduling method of this large model is applied to the aforementioned terminal device 201. Figure 3 A flowchart illustrating a resource scheduling method for a large model in this exemplary embodiment is shown below. Figure 3 The resource scheduling method for large models provided in this disclosure includes the following steps S301-S303: Step S301: Receive a decoding task request from the pre-filled resource pool of the large model; wherein, the decoding task request includes at least a first key-value cache and a first output token, the first key-value cache and the first output token are generated by the pre-filled resource pool for pre-filling the second prompt word formed by concatenating the first prompt word with the predicted prompt word.

[0045] Step S302: Based on the first output token, predict the number of decoding iterations corresponding to the decoding task request.

[0046] Step S303: Based on the first capacity value occupied by the first key-value cache, the second capacity value occupied by the second key-value cache generated under the decoding iteration number, and the third capacity value already occupied by the inference card in the decoding resource pool, determine the target capacity value of the inference card.

[0047] Step S304: Determine the target inference card based on the target capacity value of the inference card, and execute the decoding task corresponding to the decoding task request based on the target inference card.

[0048] exist Figure 3 In the technical solution provided in the embodiments, firstly, the method pre-fills a resource pool with a second prompt word formed by concatenating the first prompt word with a predicted prompt word. This yields a first output token that can predict the number of decoding iterations and, in subsequent decoding stages, predict the capacity change caused by the KVCache changes resulting from the request. This not only enables fine-grained scheduling of subsequent decoding tasks and allocates the request to a suitable inference card for processing, avoiding the technical problem of low resource utilization caused by insufficient or surplus inference card resources, but also significantly reduces prediction overhead. Secondly, the method calculates the target capacity value of the inference card. By utilizing the characteristic that the number of KVCaches reaches its peak at the end of each request, the target inference card is selected. This reduces the time and space complexity of the algorithm while improving the accuracy of target inference card selection, thereby enhancing the reliability and security of the system operation.

[0049] The following will describe, in conjunction with specific embodiments, Figure 3 The implementation methods of each step in the illustrated embodiment are described in detail below: In step S301, a first key-value cache and a first output token are received from the pre-filled resource pool of the large model; wherein, the decoding task request includes at least the first key-value cache and the first output token, which are generated by the pre-filled resource pool through pre-filling processing of the second prompt word formed by concatenating the input first prompt word with the predicted prompt word.

[0050] The pre-filled resource pool typically contains multiple candidate pre-filled nodes, from which a target pre-filled node is selected to receive inference task requests and pre-fill them, generating a first key-value cache and a first output token.

[0051] For example, after processing the inference task request, the pre-populated resource pool of the large model generates a first key-value cache and a first output token, and then sends the first key-value cache and the first output token to the decoding resource pool. The decoding resource pool can then receive the first key-value cache and the first output token from the pre-populated resource pool of the large model.

[0052] Because the first key-value cache and the first output token are generated by pre-filling the resource pool with the second prompt word formed by concatenating the first prompt word with the predicted prompt word, a change in the input content occurs. Therefore, when the large model performs inference tasks, it also needs to pay attention to the content of the predicted prompt word and generate the corresponding first output token under the guidance of the predicted prompt word. In other words, the predicted prompt word has higher priority, and its role is to guide the large model to generate the first output token based on the content of the predicted prompt word.

[0053] For example, suppose the user's first prompt is "How's the weather today?", and the example of the prediction prompt to be concatenated is "Predict the output length of this question, directly output a number". Since the pre-filling stage segments the input "How's the weather today, predict the output length of this question, directly output a number" into a token sequence, the large model uses a self-attention mechanism to compute the Key (K) and Value (V) vectors of all input tokens in parallel, generating a global KV cache. Then, based on the complete KV cache, the model decoder outputs the probability distribution of the first token. Because the second prompt explicitly requires "directly output a number", "directly output a number" becomes a high-weight context after word segmentation. Therefore, the large model suppresses non-numeric tokens in the probability distribution calculation, prioritizing numeric tokens (such as "15") as the output, rather than weather content.

[0054] The decoding resource pool typically contains multiple candidate decoding nodes. Considering the high parallel processing advantage of current mainstream inference cards, which can batch process a large number of inference task requests, the decoding node in this embodiment is set as an inference card. The target inference card is selected from the inference cards to receive the first key-value cache and the first output token, and then decodes them to generate the final inference result.

[0055] The following will describe in detail the process of generating the first key-value cache and the first output token from the pre-filled resource pool of a large model, with specific embodiments.

[0056] Figure 4A This schematically illustrates a flowchart of a method for generating a first key-value cache and a first output token from a pre-populated resource pool according to one of the exemplary embodiments of this invention. Please refer to [link / reference]. Figure 4A In an optional embodiment of this disclosure, the method for pre-filling the resource pool to generate the first key-value cache and the first output token includes the following steps S401-S403: Step S401: Receive a reasoning task request; wherein the reasoning task request contains at least the first input prompt word.

[0057] Step S402: Concatenate the predicted prompt word after the first prompt word to obtain the concatenated second prompt word.

[0058] Step S403: Input the second prompt word into the pre-filled resource pool for pre-filling processing, generate the first key-value cache and the first output token, and package the first key-value cache and the first output token to generate a decoding task request, so as to send the decoding task request to the decoding resource pool of the large model.

[0059] For example, after a user inputs the first prompt word, a corresponding inference task request can be triggered and generated. Upon receiving the inference task request, the inference system can extract the first prompt word input by the user from the inference task request. Then, a predicted prompt word is concatenated after the first prompt word to obtain the concatenated second prompt word.

[0060] It should be explained that the predicted prompt is related to the user's needs. In this embodiment, the output length needs to be predicted. Therefore, the predicted prompt could be "Predict the output length of this question and output a number directly." For example, assuming the user's first prompt is "How's the weather today?", the predicted prompt to be concatenated is "Predict the output length of this question and output a number directly." By concatenating the first prompt and the predicted prompt, the resulting second prompt would be: "How's the weather today? Predict the output length of this question and output a number directly."

[0061] It should be understood that since the large model uses a forward attention mechanism when pre-filling nodes, a predicted prompt word needs to be concatenated after the first prompt word entered by the user so that the large model can focus on the length prediction of the output corresponding to the first prompt word entered by the user.

[0062] Finally, the second prompt word is input into the pre-filled resource pool, and the pre-filled nodes in the pre-filled resource pool can process it to generate the first key-value cache (i.e., the target KVCache) and the first token corresponding to the inference task request. The first key-value cache and the first output token are then input into the decoding resource pool of the large model. This realizes step S301 where the decoding resource pool receives the first key-value cache and the first output token from the pre-filled resource pool of the large model.

[0063] When performing step S403 above, which involves inputting the first key-value cache and the first output token into the decoding resource pool of the large model, invalid KVCaches can also be removed. The following will describe this in conjunction with specific embodiments: In one optional embodiment of this disclosure, in response to the first key-value cache including a first sub-key-value cache generated for the first prompt word and a second sub-key-value cache generated for the predicted prompt word, the second sub-key-value cache is removed to obtain an updated first key-value cache; the updated first key-value cache and the first output token are input into the decoding resource pool.

[0064] For example, the target KVCache generated by the pre-filled resource pool contains not only the first subkey value cache generated for the first prompt word, but also the second subkey value cache generated for the predicted prompt word. However, the second subkey value cache corresponding to the predicted prompt word is invalid data for the inference task. If it is also synchronously transmitted to the decoding resource pool along with the first subkey value cache, it will not only occupy transmission resources, but also occupy the video memory resources of each decoding node in the decoding resource pool.

[0065] In response to this issue, the embodiments of this disclosure take into account the aforementioned problems. Furthermore, considering that KVCache is only related to the prefix token, the second sub-key value cache for predicting the prompt word concatenated after the user-input first prompt word does not affect the first sub-key value cache generated for the user-input first prompt word. That is, the first and second sub-key value caches are independent and do not affect each other. Therefore, after generating the target KVCache from the pre-filled resource pool, the second sub-key value cache can be directly removed to obtain the updated first key value cache. Then, the updated first key value cache and the first output token are input into the decoding resource pool for decoding processing.

[0066] This embodiment allows for the direct use of the large model's inherent capabilities without the need for additional modules. Specifically, it leverages the large model's forward attention mechanism on the user-input first prompt word to concatenate and predict the predicted prompt word, ensuring that the model focuses on and predicts the final output length of the first prompt word. Simultaneously, it utilizes the characteristic of the large model's transform architecture where the KVCache is only related to the prefix token. It directly uses the KVCache generated from the user-input first prompt word (i.e., the first sub-key value cache) and removes the KVCache of the predicted prompt word (i.e., the second sub-key value cache). This cleverly utilizes the pre-filling process based on the user-input first prompt word to accurately predict the output length, thereby improving overall performance and reducing resource consumption.

[0067] In step S302, the number of decoding iterations corresponding to the decoding task request is predicted based on the first output token.

[0068] For example, in the decoding phase, since each inference task request generates only one output token in each decoding iteration, the number of decoding iterations is the same as the number of output tokens obtained in the pre-filling phase.

[0069] For example, suppose the second prompt word is: "How is the weather today? Predict the output length of this question and output the number directly." Based on the first output token generated in the pre-filling stage being 15, the number of decoding iterations to be performed in the decoding stage is predicted to be 15.

[0070] Through the above embodiments, a large model can be used directly to determine the length of the output content, thereby predicting the unknown number of decoding iterations. Furthermore, it cleverly utilizes the pre-filling process of the original first prompt word, eliminating the need for other prediction modules and greatly reducing the prediction overhead.

[0071] It's important to explain that the first output token generated during the pre-filling stage is not the first token of the first prompt word entered by the user. Continuing with the example above, the second prompt word is: "How's the weather today? Predict the output length of this question and output a number directly." The first output token generated during the pre-filling stage is 15, which is not the first token for the first prompt word "How's the weather today?" entered by the user. Therefore, during the first decoding in the decoding stage, no token is input; only the KVCache (i.e., the first key-value cache or the first subkey-value cache) is used. This generates the first token corresponding to the first prompt word entered by the user, allowing subsequent decoding iterations to proceed normally. The specific process described above can be found in [reference needed]. Figure 4B As shown.

[0072] In step S303, the target capacity value of the inference card is determined based on the first capacity value occupied by the first key-value cache, the second capacity value occupied by the second key-value cache generated under the decoding iteration number, and the third capacity value already occupied by the inference card in the decoding resource pool.

[0073] The first capacity value occupied by the first key-value cache generated in the pre-filling stage is a known value. The number of decoding iterations is a known value. In each decoding iteration, a KVCache of a token is generated and superimposed. The size of the KVCache of each token is also a known value. The corresponding second capacity value is a known value. The third capacity value occupied by the inference card in the decoding resource pool is also a known value. Therefore, the total capacity change generated by this inference task request during the execution of this task can be determined.

[0074] In one embodiment, when calculating the second capacity value, the second key-value cache generated under each decoding iteration number is determined, and the capacity value occupied by the second key-value cache generated under each decoding iteration number is the same; the second capacity value is obtained based on the product of the decoding iteration number and the capacity value.

[0075] For example, a KVCache for a token is generated and superimposed in each decoding iteration, and the KVCache size for each token is the same. Therefore, the second capacity value can be obtained based on the product of the number of decoding iterations and the capacity value.

[0076] Assume that the first key-value cache generated during the pre-filling stage occupies a first capacity of p; the number of decoding iterations is t, and a new key-value cache for each token is generated and added during each decoding iteration, with each token's key-value cache having a size of m. Then, the second capacity of the second key-value cache generated under the number of decoding iterations is: m. t. Accordingly, for this inference task, the total capacity change generated during the execution of this task is requested. It can be determined using the following formula (1): Formula (1) Through the above embodiments, the overall resource capacity change caused by the KVCache of the entire request can be quickly obtained by predicting the number of iterations, so as to serve as the basis for subsequent selection of target inference cards.

[0077] In this embodiment of the disclosure, the following is adopted: Figure 1B The iterative batch processing scheduling method shown is used for batch processing. Therefore, when a decoding task request arrives at the decoding resource pool, it needs to be pushed together with the decoding task request being processed on the inference card to the next decoding iteration.

[0078] Figure 5 This schematically illustrates a flowchart of a method for determining the target capacity value of an inference card according to one of the exemplary embodiments of this invention. Please refer to [link to relevant documentation]. Figure 5 In an optional embodiment of this disclosure, when performing step S303 above, the following steps S501-S503 are included: Step S501: Determine the iteration interval for the decoding task request to be superimposed on the inference card; wherein, the iteration interval includes the number of iterations at which the decoding task request is superimposed on the inference card begins, and the number of iterations at which the decoding task request exits the inference card ends.

[0079] Step S502: Based on the third capacity value already occupied by the decoding task request being processed in the inference card within the iteration interval, the second capacity value of the decoding task request at each iteration number within the iteration interval, and the first capacity value occupied by the first key-value cache, determine the initial capacity peak value in the iteration interval.

[0080] Step S503: Determine the target capacity value of the inference card from the initial capacity peaks that is closest to the number of termination iterations.

[0081] For example, for each inference card in the decoding resource pool, the following is performed: Figure 5 The prediction process shown is used to determine the target capacity value of the inference card, and then select the target inference card to receive the decoding task request and process the inference task.

[0082] To reduce algorithm complexity, in the embodiments of this disclosure, considering that the KVCache for all requests is monotonically increasing before the iteration stops, the peak point must occur at the termination iteration count for each request. Furthermore, the predicted value of the new request is superimposed on the change in the KVCache capacity of existing requests on the inference card (e.g., refer to...). Figure 6 As shown in the figure, in this regard, it is only necessary to calculate the capacity peak closest to the termination iteration number and compare it with the available capacity of the inference card to determine the target inference card.

[0083] The following will combine Figure 6 , Figure 7 As shown above Figure 5 The steps shown are explained below.

[0084] Taking any inference card in the decoding resource pool as an example, the calculation method for other inference cards is the same.

[0085] Reference Figure 6 As shown, there are currently n requests on the inference card, which are predicted to complete inference in X = {x1, x2, … , xn} iterations respectively. When a new decoding task request arrives, the initial iteration count of the inference card is xa, the new request is predicted to exit in xb (i.e., the termination iteration count is xb), and the first capacity value of the KVCache generated by the new request pre-filling is p.

[0086] The iteration interval is defined by the initial iteration number xa to the final iteration number xb (i.e., Figure 6 The KVCache capacity generated by the existing n requests and the KVCache capacity predicted by the new decoding task requests are superimposed at each iteration number within the interval shown in the prediction iteration number, forming a result as shown in the figure. Figure 6 The effect shown in the middle right figure is that multiple initial peak capacities can be obtained within the iteration interval, such as the initial peak capacities corresponding to iteration numbers x1 and x2. Finally, the peak capacity at x2, which is close to the termination iteration number xb, is taken as the final target capacity value of the inference card.

[0087] In addition, to further improve the accuracy of peak capacity calculation and reduce the amount of computation, in an optional embodiment of this disclosure, completed requests within the iteration interval are deleted.

[0088] Among them, the "completed request" is the decoding task request for which the inference iteration task has been completed.

[0089] In step S304, the target inference card is determined based on the target capacity value of the inference card, so as to execute the decoding task corresponding to the decoding task request based on the target inference card.

[0090] After obtaining the target capacity value of the inference card based on the above embodiments, a suitable target inference card can be selected to execute the decoding task corresponding to the decoding task request based on the target inference card.

[0091] In one optional embodiment of this disclosure, the inference card includes an activated first inference card and an inactive second inference card. The activated first inference card indicates that there are currently requests being processed internally, while the inactive second inference card indicates that there are no requests being processed internally.

[0092] Accordingly, when the first inference card includes a first sub-inference card and a second sub-inference card, it is detected whether the target capacity value of the first sub-inference card is less than the first available capacity value of the first sub-inference card; in response to the target capacity value of the first sub-inference card being less than the first available capacity value, the first sub-inference card is identified as the target inference card; in response to the target capacity value of the first sub-inference card being not less than the first available capacity value, it is detected whether the target capacity value of the second sub-inference card is less than the second available capacity value of the second sub-inference card; in response to the target capacity value of the second sub-inference card being less than the second available capacity value, the second sub-inference card is identified as the target inference card.

[0093] For example, during resource scheduling, to further improve resource utilization, new requests are preferentially placed on the first inference card that is already enabled. If it is determined that the predicted peak value of the KVCache after placement will not exceed the available memory capacity M of the inference card, the request is directly pushed to that inference card. Otherwise, the same judgment is made for the next inference card until an inference card that meets the condition that the target capacity value is less than the corresponding available capacity value of the inference card is found. That is, in this method, when a new request is received, it is not necessary to predict and compare the target capacity values ​​of all inference cards, but rather to predict and compare them one by one until a target inference card that meets the above conditions is found.

[0094] Conversely, in an optional embodiment of this disclosure, in response to the target capacity value of the first sub-inference card being not less than the first available capacity value and the target capacity value of the second sub-inference card being not less than the second available capacity value, the second inference card is determined as the target inference card.

[0095] For example, if none of the first inference cards in the decoding resource pool meet the above conditions, then an unactivated second inference card is selected as the target inference card to receive and process the data, thereby obtaining the final inference result.

[0096] Based on the above embodiments, it can be combined with Figure 7 The exemplary program shown is illustrated below: Suppose that the data is used to record the capacity of the KVCache on the inference card at the xth decoding iteration.

[0097] First, delete the records of completed requests, and find the nearest peak point xr before the new request ended (corresponding to...). Figure 7 (Lines 3-11 of the code shown); then find the number of requests that exist after xr (corresponding to) Figure 7 Line 13 of the program code shown); and, backup of the current inference card's prediction value (corresponding to...). Figure 7 (Line 15 of the code shown); then calculate the KVCache capacity at the end of the new request prediction and insert the record (corresponding to...) Figure 7 Lines 16-17 of the program code shown); calculate the change in KVCache capacity on the inference card after adding a new request (corresponding to...). Figure 7 Lines 18-22 of the program code shown); finally, it checks whether the peak capacity of the KVCache after adding the new request exceeds the available video memory capacity (corresponding to...). Figure 7 Lines 23-28 of the program code shown.

[0098] Furthermore, the following will combine Figure 8 The resource scheduling method for large models according to exemplary embodiments of this disclosure provides a detailed description of the entire resource scheduling process of the large model.

[0099] Reference Figure 8 As shown, firstly, the large model receives the first prompt word input by the user. Then, it concatenates the first prompt word with a predicted prompt word used to output the predicted length, resulting in the second prompt word. This second prompt word is then input into the pre-filling stage. In the pre-filling stage, the second prompt word is pre-filled using pre-filling nodes to obtain the predicted length value (i.e., the first output token), and the KVCache corresponding to the predicted prompt word is removed.

[0100] Then, determine whether the first inference card that has been enabled exists in the decoding resource pool.

[0101] If it exists, an already activated first inference card can be selected, and it can be determined whether the target capacity value predicted for the first inference card is less than the available capacity value of the first inference card. If so, the first inference card is selected to place the request, so that it can be decoded to obtain the inference result.

[0102] Conversely, if not, continue to determine whether there is an enabled first inference card in the decoding resource pool, until a first inference card that meets the requirements is selected from the decoding resource pool to place the request and perform decoding processing to obtain the inference result; if none of the first inference cards in the decoding resource pool meet the condition that the predicted target capacity value is less than the available capacity value of the first inference card, then an unenabled second inference card is selected for decoding processing.

[0103] To implement the resource scheduling method for the large model described above, one embodiment of this disclosure provides a resource scheduling apparatus for the large model. Figure 9 The schematic diagram illustrates the architecture of a resource scheduling device for a large model.

[0104] The resource scheduling device 900 of the large model includes a task receiving module 901, an iteration number prediction module 902, a target capacity value determination module 903, and an inference card determination module 904.

[0105] The task receiving module 901 is configured to receive decoding task requests from the pre-filled resource pool of the large model. The decoding task request includes at least a first key-value cache and a first output token. The first key-value cache and the first output token are generated by pre-filling the resource pool with a second prompt word formed by concatenating the first prompt word input by the user with a predicted prompt word. The iteration count prediction module 902 is configured to predict the number of decoding iterations corresponding to the decoding task request based on the first output token. The target capacity value determination module 903 is configured to determine the target capacity value of the inference card based on the first capacity value occupied by the first key-value cache, the second capacity value occupied by the second key-value cache generated under the decoding iteration count, and the third capacity value already occupied by the inference card in the decoding resource pool. The inference card determination module 904 is configured to determine the target inference card based on the target capacity value of the inference card, so as to execute the decoding task corresponding to the decoding task request based on the target inference card.

[0106] In an optional embodiment of this disclosure, the apparatus further includes a request receiving module, a splicing module, and a pre-filling processing module. The request receiving module is configured to receive an inference task request, wherein the inference task request includes at least an input first prompt word. The splicing module is configured to splice a predicted prompt word after the first prompt word to obtain a spliced ​​second prompt word. The pre-filling processing module is configured to input the second prompt word into a pre-filling resource pool for pre-filling processing, generate a first key-value cache and a first output token, package the first key-value cache and the first output token to generate a decoding task request, and send the decoding task request to the decoding resource pool of the large model.

[0107] In an optional embodiment of this disclosure, the pre-filling processing module is configured to, in response to the first key-value cache containing a first sub-key-value cache generated for the first prompt word and a second sub-key-value cache generated for the predicted prompt word, remove the second sub-key-value cache to obtain an updated first key-value cache; package the updated first key-value cache and the first output token to generate a decoding task request, and send the decoding task request to the decoding resource pool of the large model.

[0108] In an optional embodiment of this disclosure, the target capacity value determination module 903 is configured to determine the iteration interval for overlaying decoding task requests onto the inference card; wherein, the iteration interval includes the starting iteration number for overlaying decoding task requests onto the inference card, and the ending iteration number for exiting the inference card; based on the third capacity value already occupied by the decoding task requests being processed in the inference card within the iteration interval, the second capacity value of the decoding task requests at each iteration number within the iteration interval, and the first capacity value occupied by the first key-value cache, the initial capacity peak value in the iteration interval is determined; and the capacity peak value among the initial capacity peak values ​​that is closest to the ending iteration number is determined as the target capacity value of the inference card.

[0109] In an optional embodiment of this disclosure, the apparatus further includes a deletion module configured to delete completed requests within the iteration interval.

[0110] In an optional embodiment of this disclosure, the plurality of inference cards includes an activated first inference card and an inactive second inference card, and the first inference card includes a first sub-inference card and a second sub-inference card. The inference card determination module 904 is configured to perform the following actions: detecting whether the target capacity value of the first sub-inference card is less than the first available capacity value of the first sub-inference card; in response to the target capacity value of the first sub-inference card being less than the first available capacity value, determining the first sub-inference card as the target inference card; in response to the target capacity value of the first sub-inference card being not less than the first available capacity value, detecting whether the target capacity value of the second sub-inference card is less than the second available capacity value of the second sub-inference card; in response to the target capacity value of the second sub-inference card being less than the second available capacity value, determining the second sub-inference card as the target inference card.

[0111] In an optional embodiment of this disclosure, the inference card determination module 904 is configured to determine the second inference card as the target inference card in response to the first sub-inference card's target capacity value being not less than the first available capacity value and the second sub-inference card's target capacity value being not less than the second available capacity value.

[0112] The resource scheduling device 900 for large models provided in this embodiment can execute the technical solution of the resource scheduling method for large models in any of the above embodiments. Its implementation principle and beneficial effects are similar to those of the resource scheduling method for large models. Please refer to the implementation principle and beneficial effects of the resource scheduling method for large models. It will not be repeated here.

[0113] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the methods described above is stored. In some possible embodiments, various aspects of the present invention may also be implemented as a program product comprising program code that, when the program product is run on a terminal device, causes the terminal device to perform the steps of the various exemplary embodiments of the present invention described in the "Exemplary Methods" section above.

[0114] According to embodiments of the present invention, a program product for implementing the above-described method may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0115] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0116] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0117] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, radio frequency (RF), or any suitable combination thereof.

[0118] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0119] In an exemplary embodiment of this disclosure, an electronic device capable of implementing the above-described method is also provided.

[0120] Those skilled in the art will understand that various aspects of the present invention can be implemented as systems, methods, or program products. Therefore, various aspects of the present invention can be specifically implemented in the following forms: entirely in hardware, entirely in software (including firmware, microcode, etc.), or in a combination of hardware and software, collectively referred to herein as “circuit,” “module,” or “system.”

[0121] The following reference Figure 10 To describe an electronic device 1000 according to this embodiment of the present invention. Figure 10 The electronic device 1000 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0122] like Figure 10As shown, the electronic device 1000 is manifested in the form of a general-purpose computing device. The components of the electronic device 1000 may include, but are not limited to: at least one processing unit 1010, at least one storage unit 1020, a bus 1030 connecting different system components (including storage unit 1020 and processing unit 1010), and a display unit 1040.

[0123] The storage unit stores program code, which can be executed by the processing unit 1010 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of the present invention. For example, the processing unit 1010 can perform, as follows: Figure 3 Steps S301 to S304 are shown in the figure.

[0124] Storage unit 1020 may include readable media in the form of volatile storage units, such as random access memory (RAM) 10201 and / or cache memory 10202, and may further include read-only memory (ROM) 10203.

[0125] Storage unit 1020 may also include a program / utility 10204 having a set (at least one) program module 10205, such program module 10205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0126] Bus 1030 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the multiple bus structures.

[0127] Electronic device 1000 can also communicate with one or more external devices 2000 (e.g., keyboard, pointing device, Bluetooth device, etc.), one or more devices that enable a user to interact with electronic device 1000, and / or any device that enables electronic device 1000 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 1050. Furthermore, electronic device 1000 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 1060. As shown, network adapter 1060 communicates with other modules of electronic device 1000 via bus 1030. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 1000, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, Redundant Arrays of Independent Disks (RAID) systems, tape drives, and data backup storage systems.

[0128] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0129] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of the present invention, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0130] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0131] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0132] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is defined only by the appended claims.

Claims

1. A resource scheduling method for a large-scale model, characterized in that, include: The decoding task request is received from the pre-filled resource pool of the large model; wherein the decoding task request includes at least a first key-value cache and a first output token, the first key-value cache and the first output token are generated by the pre-filled resource pool through pre-filling processing of the second prompt word formed by concatenating the input first prompt word with the predicted prompt word; Based on the first output token, predict the number of decoding iterations corresponding to the decoding task request; Based on the first capacity value occupied by the first key-value cache, the second capacity value occupied by the second key-value cache generated under the number of decoding iterations, and the third capacity value already occupied by the inference card in the decoding resource pool, the target capacity value of the inference card is determined. Based on the target capacity value of the inference card, a target inference card is determined, and the decoding task corresponding to the decoding task request is executed based on the target inference card.

2. The method according to claim 1, characterized in that, Before receiving the first key-value cache and the first output token from the pre-populated resource pool of the large model, the method further includes: Receive a reasoning task request; wherein the reasoning task request contains at least the first input prompt word; The predicted prompt word is concatenated after the first prompt word to obtain the concatenated second prompt word; The second prompt word is input into the pre-filled resource pool for pre-filling processing to generate the first key-value cache and the first output token. The first key-value cache and the first output token are then packaged to generate the decoding task request, which is then sent to the decoding resource pool of the large model.

3. The method according to claim 2, characterized in that, The step of packaging the first key-value cache and the first output token to generate the decoding task request, and sending the decoding task request to the decoding resource pool of the large model, includes: In response to the first key-value cache containing a first sub-key-value cache generated for the first prompt word and a second sub-key-value cache generated for the predicted prompt word, the second sub-key-value cache is removed to obtain the updated first key-value cache; The updated first key-value cache and the first output token are packaged to generate the decoding task request, and the decoding task request is sent to the decoding resource pool of the large model.

4. The method according to claim 1, characterized in that, Based on the first capacity value occupied by the first key-value cache, the second capacity value occupied by the second key-value cache generated under the number of decoding iterations, and the third capacity value already occupied by the inference card in the decoding resource pool, the target capacity value of the inference card is determined, including: Determine the iteration interval for the decoding task request to be superimposed on the inference card; wherein, the iteration interval includes the number of times the decoding task request is superimposed on the inference card to start iteration, and the number of times the decoding task request exits the inference card to terminate iteration; Based on the third capacity value already occupied by the decoding task request being processed in the inference card within the iteration interval, the second capacity value of the decoding task request at each iteration number within the iteration interval, and the first capacity value occupied by the first key-value cache, the initial capacity peak value in the iteration interval is determined. The capacity peak value closest to the termination iteration number among the initial capacity peak values ​​is determined as the target capacity value of the inference card.

5. The method according to claim 4, characterized in that, The method further includes: Delete completed requests within the iteration interval.

6. The method according to claim 1, characterized in that, The inference card includes an activated first inference card and an inactive second inference card, and the first inference card includes a first sub-inference card and a second sub-inference card. The target inference card is determined based on the target capacity value of the inference card, including: Detect whether the target capacity value of the first sub-inference card is less than the first available capacity value of the first sub-inference card; In response to the first sub-inference card's target capacity value being less than the first available capacity value, the first sub-inference card is determined as the target inference card; In response to the first sub-inference card's target capacity value being not less than the first available capacity value, it is detected whether the second sub-inference card's target capacity value is less than the second available capacity value of the second sub-inference card; If the target capacity value of the second sub-inference card is less than the second available capacity value, then the second sub-inference card is determined as the target inference card.

7. The method according to claim 6, characterized in that, Determining the target inference card based on the target capacity value of the inference card includes: In response to the first sub-inference card's target capacity value being no less than the first available capacity value, and the second sub-inference card's target capacity value being no less than the second available capacity value, the second inference card is determined as the target inference card.

8. A resource scheduling device for a large-scale model, characterized in that, include: The task receiving module is configured to receive decoding task requests from the pre-filled resource pool of the large model; wherein, the decoding task request includes at least a first key-value cache and a first output token, the first key-value cache and the first output token are generated by the pre-filled resource pool through pre-filling processing of the second prompt word formed by concatenating the input first prompt word with the predicted prompt word; The iteration count prediction module is configured to predict the number of decoding iterations corresponding to the decoding task request based on the first output token. The target capacity value determination module is configured to determine the target capacity value of the inference card based on the first capacity value occupied by the first key-value cache, the second capacity value occupied by the second key-value cache generated under the number of decoding iterations, and the third capacity value already occupied by the inference card in the decoding resource pool. The inference card determination module is configured to determine the target inference card based on the target capacity value of the inference card, so as to execute the decoding task corresponding to the decoding task request based on the target inference card.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the resource scheduling method for the large model as described in any one of claims 1 to 7.

10. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the resource scheduling method for the large model according to any one of claims 1 to 7 by executing the executable instructions.

Images