Model inference scheduling method and system, electronic device, and storage medium

Abstract

The model inference scheduling method and system, the electronic device and the storage medium provided by the present disclosure comprise a plurality of pre-filling server nodes and a plurality of decoding server nodes, and the method comprises the following steps: in response to receiving user request information, a target pre-filling server node selected from the plurality of pre-filling server nodes and a first priority are obtained in combination with a pre-filling scheduling model based on deep reinforcement learning; a pre-filling request is sent to a request queue corresponding to the first priority in the target pre-filling server node, and the target pre-filling server node is instructed to process the pre-filling request; after the pre-filling request is processed, a target decoding server node selected from the plurality of decoding server nodes and a second priority are obtained in combination with a decoding scheduling model based on deep reinforcement learning; a decoding request is sent to a request queue corresponding to the second priority in the target decoding server node, and the target decoding server node is instructed to process the decoding request; and the inference efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of model inference, in particular to a model inference scheduling method and system, an electronic device and a storage medium. BACKGROUND

[0002] With the rapid development of large language models (LLM), major manufacturers have begun to promote large-scale commercial deployment of large models to provide users with high-throughput and low-latency services. However, when deploying large models, the scheduling strategy for large model services faces many challenges as the large models change from training mode to inference mode; the existing scheduling strategy has the problem of low inference efficiency. SUMMARY

[0003] The present disclosure provides a model inference scheduling method and system, an electronic device and a storage medium to improve inference efficiency.

[0004] According to a first aspect of the present disclosure, a model inference scheduling method is provided, applied to a model inference scheduling system, the server cluster comprising a plurality of pre-filling server nodes and a plurality of decoding server nodes, the pre-filling server nodes and the decoding server nodes each being configured with a multi-level feedback queue when the model inference scheduling system is started, the plurality of request queues included in the multi-level feedback queue having different priorities, the method comprising:

[0005] In response to receiving user request information, based on the user request information and load information of the plurality of pre-filling server nodes, in combination with a pre-filling scheduling model based on deep reinforcement learning, a target pre-filling server node selected from the plurality of pre-filling server nodes and a first priority are obtained;

[0006] A pre-filling request is sent to a request queue corresponding to the first priority in the target pre-filling server node, and the target pre-filling server node is instructed to process the pre-filling request;

[0007] In response to the pre-filling request processing being completed, based on the user request information and load information of the plurality of decoding server nodes, in combination with a decoding scheduling model based on deep reinforcement learning, a target decoding server node selected from the plurality of decoding server nodes and a second priority are obtained;

[0008] A decoding request is sent to a request queue corresponding to the second priority in the target decoding server node, and the target decoding server node is instructed to process the decoding request to obtain a model inference result, wherein the decoding request includes a key-value cache generated by the target pre-filling server node.

[0009] According to a second aspect of the present disclosure, a model inference scheduling system is provided, which is configured to be applied to a model inference scheduling system, the server cluster includes a plurality of pre-filling server nodes and a plurality of decoding server nodes, the pre-filling server nodes and the decoding server nodes are each configured with a multi-level feedback queue at the start of the model inference scheduling system, the plurality of request queues included in the multi-level feedback queue have different priorities, and the system includes:

[0010] A first node selection module is configured to, in response to receiving user request information, based on the user request information and load information of the plurality of pre-filling server nodes, in combination with a pre-filling scheduling model based on deep reinforcement learning, obtain a target pre-filling server node selected from the plurality of pre-filling server nodes and a first priority;

[0011] A first scheduling module is configured to send a pre-filling request to a request queue corresponding to the first priority in the target pre-filling server node, and instruct the target pre-filling server node to process the pre-filling request;

[0012] A second node selection module is configured to, in response to the pre-filling request processing being completed, based on the user request information and load information of the plurality of decoding server nodes, in combination with a decoding scheduling model based on deep reinforcement learning, obtain a target decoding server node selected from the plurality of decoding server nodes and a second priority;

[0013] A second scheduling module is configured to send a decoding request to a request queue corresponding to the second priority in the target decoding server node, and instruct the target decoding server node to process the decoding request to obtain a model inference result, wherein the decoding request includes a key-value cache generated by the target pre-filling server node.

[0014] According to a third aspect of the present disclosure, an electronic device is provided, which includes:

[0015] At least one processor; and

[0016] A memory in communication connection with the at least one processor; wherein

[0017] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the model inference scheduling method of the first aspect.

[0018] According to a fourth aspect of the present disclosure, a non-transitory computer readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to perform the model inference scheduling method of the first aspect.

[0019] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the model inference scheduling method according to the first aspect described above.

[0020] The present disclosure provides a model inference scheduling method and system, an electronic device and a storage medium, proposes a large model inference scheduling framework based on PD separation and multi-level feedback queue, which improves inference efficiency and reduces memory occupation through the framework; and based on the framework, two scheduling algorithms based on deep reinforcement learning are used to select matched server nodes for user requests in the pre-padding and decoding stages respectively, which can improve the first word time delay and word interval of large model inference; the scheduling algorithm based on deep reinforcement learning fully utilizes user request information, server node information and the like in a large-scale server cluster for scheduling, which can overcome the construction limitations of heuristic methods in large-scale inference scheduling systems. The method can be directly used for scheduling user requests of large language models in a large-scale server cluster and deployed in a large model data center; it can also be applied to request scheduling of other types of large models after simple expansion.

[0021] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present application, nor to limit the scope of the present application. Other features of the present application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0022] The accompanying drawings are used to better understand the present scheme and do not limit the present disclosure. Among them:

[0023] Figure 1 A flowchart of a model inference scheduling method provided by an embodiment of the present disclosure is shown;

[0024] Figure 2 A flowchart of another model inference scheduling method provided by an embodiment of the present disclosure is shown;

[0025] Figure 3 A framework diagram of a large model inference scheduling system based on DRL provided by an embodiment of the present disclosure is shown;

[0026] Figure 4 A structural diagram of a model inference scheduling system provided by an embodiment of the present disclosure is shown;

[0027] Figure 5 A structural diagram of another model inference scheduling system provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0028] Exemplary embodiments of the present disclosure are described herein with reference to the accompanying drawings, which are incorporated in this specification for greater clarity and a more complete understanding of the present disclosure. Thus, it will be apparent to those of ordinary skill in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Also, descriptions of well-known functions and constructions are omitted for clarity and conciseness.

[0029] Term explanation:

[0030] LLM (Large Language Model).

[0031] Transformer, a mainstream large model structure.

[0032] Deep reinforcement learning (DRL).

[0033] Query value / Q value (Query).

[0034] TTFT, The time to first token, first token time delay.

[0035] TBT, The time between tokens, token interval.

[0036] In the related art, the large model inference architecture is a pure decoder (decoder-only) architecture based on the Transformer model. When inferring, the architecture converts a user request (also known as a prompt, which is a long string) into a token (which is an extremely short string) sequence, and uses an autoregressive mode to constantly generate tokens, i.e., concatenates the newly generated token with the old token sequence as a new input, uses the model to generate a new token, and continues until the complete answer is generated. Among them, in the initialization stage of this autoregressive process, the large model processes all tokens of the user request at one time until the first new token is generated, which is called prefill (Prefill, abbreviated as P), which is computationally intensive and has high parallelism. Then, the large model continues the calculation process of the prefill, and uses the autoregressive mode to constantly generate new tokens, which is called decoding (decode, abbreviated as D), which is input / output (Input / output, IO) intensive and has a large demand for memory bandwidth.

[0037] In the above reasoning process, all word pieces are first converted into three feature vectors including Query (Q value, query value), Key (K value, key value), and Value (V value) using part of the neural network in the model. Then, the Q value of the new word piece is calculated with the K value and V value of all word pieces in a series of calculations, which will occupy a large amount of video memory or memory (hereinafter referred to as memory) due to the following factors:

[0038] (1) The K value and V value of any word piece need to be stored in the memory after calculation, which is called KV cache (also known as key-value cache), and requires a large amount of memory. Then, when the Q value of the new word piece is calculated with the K value and V value of all word pieces, the KV cache can be directly read from the memory to avoid repeated calculation, which is called KV cache reuse.

[0039] (2) The rapid increase in the number of large model parameters not only increases the KV cache of each word piece, but also increases the length of the output word piece sequence, i.e., the number of word pieces to be stored in the KV cache, which further increases the memory requirement.

[0040] (3) Large model inference services need to handle a large number of user requests at the same time, so a large number of KV caches of user requests need to be stored at the same time.

[0041] When a large model inference framework processes a large number of user requests, it first needs to manage a large number of KV caches of user requests. Taking the large model inference framework SGLang (Structured Language Model Programs) as an example, a user request will be divided into shared prompt words and non-shared prompt words. The shared prompt word, also known as the shared prefix, is the part (a string that can be converted into a corresponding word piece sequence) located at the head of the user request prompt word and repeated with the head of other user request prompt words. The KV cache corresponding to the shared prefix can be reused for the shared prefix part of other user request, thereby avoiding repeated calculation, while the KV cache of the non-shared prompt word needs to be recalculated.

[0042] The existing model inference method has problems such as low inference efficiency and large storage occupation. In order to improve the inference efficiency and reduce the storage occupation, the model inference scheduling method provided by the embodiments of the present disclosure is proposed.

[0043] The model inference scheduling method and system, electronic device, and storage medium of the embodiments of the present disclosure are described below with reference to the accompanying drawings.

[0044] Figure 1 A flowchart of a model inference scheduling method provided by Embodiment One of the present disclosure.

[0045] It should be noted that the model inference scheduling method of the embodiments of the present application is applied to a model inference scheduling system, which includes a pre-filling server cluster and a decoding server cluster, the pre-filling server cluster includes a plurality of pre-filling server nodes, the decoding server cluster includes a plurality of decoding server nodes, the pre-filling server nodes and the decoding server nodes are configured with a multi-level feedback queue when the model inference scheduling system starts, the plurality of request queues included in the multi-level feedback queue have different priorities, and the plurality of request queues are empty when the multi-level feedback queue is just configured. The priority level of the request queue is determined by a queue priority parameter (quantum), which is used to indicate the execution time of the task in the request queue. In one example, based on the server cluster and the business characteristic information, the server cluster is divided into a pre-filling server cluster and a decoding server cluster, each server cluster contains a plurality of server nodes, such as a pre-filling server cluster containing server nodes, and a decoding server cluster containing server nodes. Note: subscript p represents a pre-filling server node, and d represents a decoding server node. The same notation is used throughout the present application.

[0046] As shown in Figure 1 , the method includes the following steps:

[0047] Step 101, in response to receiving user request information, based on the user request information and the load information of the plurality of pre-filling server nodes, in combination with a pre-filling scheduling model based on deep reinforcement learning, obtaining a target pre-filling server node selected from the plurality of pre-filling server nodes and a first priority.

[0048] In some embodiments, the user initiates a user request, i.e. a complete prompt, such as a word sequence, in response to receiving the user request information, based on the user request information and the load information of the plurality of pre-filling server nodes, in combination with a pre-filling scheduling model based on deep reinforcement learning, obtaining scheduling information in the pre-filling stage, the scheduling information in the pre-filling stage includes a target pre-filling server node selected from the plurality of pre-filling server nodes and a first priority, so that the pre-filling request can be added to the request queue corresponding to the first priority in the target pre-filling server node.

[0049] As described above, the model inference is divided into a pre-filling stage and a decoding stage, the pre-filling stage is computationally intensive, and the decoding stage is IO intensive; in this embodiment, the scheduling of the pre-filling stage and the decoding stage is separated, the pre-filling server nodes are scheduled to process user requests in the pre-filling stage; the decoding server nodes are scheduled to process the KV cache generated by the pre-filling server nodes in the decoding stage to complete decoding, that is, the PD separation (Prefill-decode Disaggregation) inference framework is adopted in this embodiment, that is, part of the server nodes only process the pre-filling stage, calculate all the KV cache of the user request information, and transmit it to another part of the server nodes that only process decoding for decoding; the architecture supports different server nodes focusing on processing a certain stage through separate design, and since the same stage of each user request information has similar computing characteristics, it is beneficial to schedule requests with more flexible granularity.

[0050] In this embodiment, based on the user request information and the load information of the plurality of pre-filling server nodes, a pre-filling scheduling model based on deep reinforcement learning is combined to select a target pre-filling server node from the plurality of pre-filling server nodes for scheduling.

[0051] On the basis of the PD separation inference framework, this embodiment adopts a scheduling algorithm of deep reinforcement learning (DRL), the main task of deep reinforcement learning is to observe the environment, such as the state (such as user request information, server node information, etc.) of the inference system, then take an action (such as selecting a server node to execute a request) based on a certain policy (the relationship between the state and the action) to interact with the environment and obtain a reward (such as the request completion time). Through the data generated by continuous interaction, the policy is continuously optimized based on the reinforcement learning algorithm, so that it can maximize the cumulative reward. DRL uses neural networks to model and optimize the policy and other modules involved in the above process, thereby improving the scheduling efficiency.

[0052] In some embodiments, the method further comprises: dividing the plurality of pre-filling server nodes into two types of server nodes, strong pre-filling server nodes and weak pre-filling server nodes, according to the configuration information of the plurality of pre-filling server nodes; the plurality of queue priority parameters of the strong pre-filling server nodes and the plurality of queue priority parameters of the weak pre-filling server nodes have the same part and different parts; dividing the plurality of decoding server nodes into two types of server nodes, strong decoding server nodes and weak decoding server nodes, according to the configuration information of the plurality of decoding server nodes; the plurality of queue priority parameters of the strong decoding server nodes and the plurality of queue priority parameters of the weak decoding server nodes have the same part and different parts.

[0053] The configuration information of the embodiment includes hardware configuration information, which can include but is not limited to information such as GPU, CPU, video memory, and memory.

[0054] That is, the plurality of pre-filling server nodes and the plurality of decoding server nodes each include two types of server nodes, and the plurality of queue priority parameters of the strong server nodes and the plurality of queue priority parameters of the weak server nodes have different parts and overlapping parts. That is, the plurality of pre-filling server nodes include pre-filling strong server nodes (P-strong server nodes) and pre-filling weak server nodes (P-weak server nodes), and the plurality of decoding server nodes include decoding strong server nodes (D-strong server nodes) and decoding weak server nodes (D-weak server nodes).

[0055] In some embodiments, the three queue priority parameters of the strong server nodes are respectively , and , and the three queue priority parameters of the weak server nodes are respectively , 2 and 4 , wherein the is the number of tokens included in the pre-filling granularity or the decoding granularity of the model inference method.

[0056] In one example, for the pre-filling server nodes, the pre-filling granularity of the large model inference algorithm used is tokens, that is, each execution pre-fills all pre-filling requests of the current batch tokens. The pre-filling server cluster is divided into 2 groups, which are P-strong server nodes and P-weak server nodes, and each pre-filling server node maintains 3 request queues of different priorities. The priority of the P-strong server node is , and the queue priority parameter quantum of the queue with the priority of is , where the queue priority parameter quantum indicates the duration of each execution of the queue request. The smaller the quantum, the shorter the task execution time of the request queue, and the higher the priority. The priority is lower by one level, and the quantum is multiplied by , the smaller the value, the smaller , and the priority of the P-weak server node is . The quantum of the three request queues of the P-strong server node is respectively: ， ， quantum of the three request queues of the P-weak server node are: ， ， 。 wherein, the value of indicates that when it is the turn to execute the pre-filling request of the request queue, the pre-filling of tokens is continuously calculated each time, and after the pre-filling is completed, the pre-filling request to be executed in the next round is selected from the highest-priority non-empty queue.

[0057] In an example, for the decoding server node, the pre-filling granularity of the large model inference algorithm used is tokens, that is, each time the execution is pre-filled with tokens of all decoding requests in the current batch. The decoding server cluster is divided into 2 groups, namely D-strong server nodes and D-weak server nodes, and each decoding server node maintains 3 request queues with different priorities (quantum). The quantum (priority from high to low) of the three request queues of the D-strong server node are: , , and the quantum (priority from high to low) of the three request queues of the D-weak server node are: , , wherein, indicates that when it is the turn to execute the decoding request of the request queue, the decoding of tokens is continuously performed each time, and after the decoding is completed, the decoding request to be executed in the next round is selected from the highest-priority non-empty queue.

[0058] The embodiment alleviates the problem that IO-intensive decoding requests are delayed by pre-filling requests and computation-intensive pre-filling requests are frequently preempted by decoding requests by designing differentiated and partially overlapping queue priority parameter levels for the request queues of the pre-filling server node and the decoding server node, that is, by using a multi-level feedback queue with differentiated and partially overlapping queue priority parameters, thereby supporting a subsequent deep reinforcement learning-based scheduling algorithm under the multi-level feedback queue and PD separation architecture to further improve inference efficiency and reduce storage occupancy.

[0059] In some embodiments, by means of a scheduling thread, based on user request information and load information of a plurality of pre-filling server nodes, a target pre-filling server node selected from the plurality of pre-filling server nodes and a first priority are obtained in combination with a pre-filling scheduling model based on deep reinforcement learning.

[0060] Step 102, sending a pre-filling request to a request queue corresponding to the first priority in the target pre-filling server node, and instructing the target pre-filling server node to process the pre-filling request.

[0061] In some embodiments, the pre-filling request is sent to the request queue corresponding to the first priority in the target pre-filling server node, that is, the pre-filling request is added to the request queue corresponding to the priority in the target pre-filling server node.

[0062] Therefore, the embodiment can improve inference efficiency by scheduling the request queue corresponding to the first priority in the target pre-filling server node.

[0063] In some embodiments, the implementation manner of instructing the target pre-filling server node to process the pre-filling request can include the following steps: traversing, by an inference thread, the plurality of request queues of the target pre-filling server node in order of priority from high to low; in response to the current request queue being empty, updating the request queue of a priority level lower than the current request queue to the current request queue; and in response to the current request queue not being empty, executing the pre-filling request in the current request queue.

[0064] In one example, for the plurality of request queues (also referred to as multi-level feedback queues) of each pre-filling server node, traversal is performed from the request queue of a high priority (low quantum) to the request queue of a low priority (high quantum); if the request queue corresponding to the priority is empty, the request queue of the current priority is exited, and the request queue of a priority level lower than the current priority is selected; and if the request queue corresponding to the priority is not empty, one batch of pre-filling requests at the head of the current request queue is executed according to the first-come-first-served principle of the request queue, such as pre-filling of one word unit of a batch of pre-filling requests. That is, requests in the request queue of a high priority are preferentially executed, and the execution time is quantum each time. In addition, specific rules can be set to adjust the request queue where the request is located, such as triggering forced service when the request queue of a low priority is not served for a long time.

[0065] In the embodiment of the present application, the inference efficiency can be further improved by selecting the target pre-filling server node and the priority by the scheduling thread and processing the pre-filling request by the inference thread.

[0066] Step 103, in response to the pre-filling request processing being completed, based on the user request information and the load information of the plurality of decoding server nodes, and in combination with the decoding scheduling model based on deep reinforcement learning, obtaining a target decoding server node selected from the plurality of decoding server nodes and a second priority.

[0067] It can be understood that the pre-filling request processing is completed, that is, all the word units contained in the user request information are calculated until the first new word unit is generated, and the pre-filling ends. The pre-filling server node will pass the calculated kv cache to the decoding server node to start decoding, and new word units are continuously generated. Therefore, after the pre-filling request processing is completed, a target decoding server node to be scheduled needs to be selected from multiple decoding server nodes. In this embodiment, the target decoding server node is selected by a decoding scheduling model based on deep reinforcement learning (also referred to as a decoding scheduling model based on DRL).

[0068] In this embodiment, based on the user request information and the load information of the multiple decoding server nodes, the decoding stage scheduling information is obtained in combination with the decoding scheduling model based on deep reinforcement learning. The decoding stage scheduling information includes the selected target decoding server node and the second priority, so that the decoding request can be added to the request queue corresponding to the second priority in the target decoding server node.

[0069] In some embodiments, the decoding stage scheduling information is obtained by a scheduling thread based on the user request information and the load information of the multiple decoding server nodes in combination with the decoding scheduling model based on deep reinforcement learning. That is, after the pre-filling request processing is completed, the decoding server node and the priority are selected by the scheduling thread using the decoding scheduling model based on DRL, and the kv cache generated by the pre-filling server node is transmitted to the selected decoding server node and added to the request queue corresponding to the selected priority, and waits for execution of decoding.

[0070] In step 104, the decoding request is sent to the request queue corresponding to the second priority in the target decoding server node, so that the target decoding server node processes the decoding request to obtain the model inference result. The decoding request includes the key value cache generated by the target pre-filling server node.

[0071] In one example, for multiple request queues of each decoding server node, the request queues are traversed from the request queue of high priority (low quantum) to the request queue of low priority (high quantum); if the request queue corresponding to the priority is empty, the request queue of the next lower priority is selected; if the request queue corresponding to the priority is not empty, one batch of decoding requests at the head of the request queue is executed according to the first-come-first-served principle of the request queue, such as decoding of one batch of pre-filling requests.

[0072] In some embodiments, when the decoding request is completed or reaches the maximum text length limit, the model inference result is returned to the user by the target decoding server node.

[0073] ​In the embodiments of the present application, the target decoding server node and priority are selected through the scheduling thread, and the decoding request is processed through the inference thread, which can further improve the inference efficiency.

[0074] The model inference scheduling method of the embodiments of the present application proposes a large model inference scheduling framework based on PD separation and multi-level feedback queue, which improves the inference efficiency and reduces the memory occupation. Based on the framework, two scheduling algorithms based on deep reinforcement learning are used to select the matched server nodes for the user requests in the pre-filling and decoding stages, which can improve the first word time delay and word interval of the large model inference. The scheduling algorithm based on deep reinforcement learning makes full use of the user request information, server node information, etc. in the large-scale server cluster for scheduling, which can overcome the construction limitations of heuristic methods in large-scale inference scheduling systems. By setting the multi-level feedback queue with differentiated and partially overlapping queue priority parameters, the problem of delay of IO-intensive decoding requests by pre-filling requests and frequent preemption of computationally intensive pre-filling requests by decoding requests can be alleviated, so that on the basis of the large model inference scheduling framework based on PD separation and multi-level feedback queue, combined with the subsequent scheduling algorithm based on deep reinforcement learning, the inference efficiency is further improved and the storage occupation is reduced. In addition, the two threads are used to process the requests for server node selection, which can further improve the inference efficiency and improve the service level of the large model inference framework. The method can be directly used for scheduling user requests for large language models in a large-scale server cluster and deployed in a large model data center. It can also be applied to the request scheduling of other types of large models after simple expansion.

[0075] Based on the above embodiments, referring to Figure 2 The second embodiment of the present application provides another model inference scheduling method, which includes the following steps:

[0076] In step 201, in response to receiving user request information, the historical key-value cache is searched based on the user request information to obtain shared prefix matching information between the user request information and each pre-filling server node.

[0077] In this embodiment, each pre-filling server node stores a historical key-value cache, and the received user request information, i.e., the complete prompt word, is used to retrieve the matching shared prefix from the historical key-value cache of each pre-filling server node to obtain the shared prefix matching information between the user request information and each pre-filling server node.

[0078] In some embodiments, the shared prefix matching information between the user request information and each pre-filling server node includes the key-value cache storage location of the shared prefix, the shared prefix length, and the remaining unmatched length.

[0079] At step 202, based on the user request information and the shared prefix matching information between the user request information and each pre-populated server node, an execution time prediction model is combined to obtain an execution estimated time of each pre-populated server node for processing the user request information.

[0080] In some embodiments, the method of obtaining the execution estimated time of each pre-populated server node for processing the user request information based on the user request information and the shared prefix matching information between the user request information and each pre-populated server node, and combining the execution time prediction model can include the following steps:

[0081] At step 2021, based on the user request information, the token sequence length of the user request is obtained.

[0082] In this embodiment, based on the user request information, the token sequence length of the user request, i.e., the user request length, is obtained.

[0083] At step 2022, based on the token sequence length of the user request, the server node type, the key-value cache storage location of the shared prefix, the shared prefix length, and the remaining unmatched length, an execution time prediction model is combined to obtain an execution estimated time of each pre-populated server node for processing the user request information.

[0084] In some embodiments, the execution time prediction model is obtained by training a neural network, and the neural network includes three hidden layers, and the activation function of the neural network includes a LeakyRelu function.

[0085] In some embodiments, the execution time prediction model uses an efficient small neural network, which includes three hidden layers with node numbers of [128, 64, 32] and uses a LeakyRelu activation function. The input of the execution time prediction model is defined to include the token sequence length of the user request, the key-value cache storage location (video memory or memory) of the shared prefix, the shared prefix length, the remaining unmatched length, and the server node type. Optionally, only the information of a number of pre-populated server nodes with the highest shared prefix matching length is taken, such as three. The output of the execution time prediction model is defined to include the execution estimated time of each pre-populated server node for processing the user request information, i.e., the execution estimated time of each pre-populated server node for executing the current pre-populated request.

[0086] The execution time prediction model considers multiple factors, including the token sequence length of the user request, and does not cause subsequent short requests to suffer from high latency due to processing long requests.

[0087] In some embodiments, a training method of an execution time prediction model is performed, including: obtaining training data collected in an underload or idle environment; training a neural network based on the training data to obtain the execution time prediction model, taking a mean squared error (MSE) as a loss function.

[0088] The training data is collected in the underload or idle environment, so that the execution time as a reference can be obtained without interference of factors such as priority preemption.

[0089] In step 203, based on the user request information, the load information of the plurality of pre-populated server nodes, and the execution estimation time of each pre-populated server node processing the user request information, a target pre-populated server node selected from the plurality of pre-populated server nodes and a first priority are obtained in combination with a pre-population scheduling model based on deep reinforcement learning.

[0090] In some embodiments, a method for obtaining a target pre-populated server node selected from a plurality of pre-populated server nodes and a first priority based on user request information, load information of the plurality of pre-populated server nodes, and execution estimation time of each pre-populated server node processing the user request information in combination with a pre-population scheduling model based on deep reinforcement learning; including:

[0091] Based on the token sequence length of the user request, the server node type, the key-value cache storage location of the shared prefix, the shared prefix length, the remaining unmatched length, the load information of the plurality of pre-populated server nodes, the execution estimation time of each pre-populated server node processing the user request information, and the request queue information of the plurality of pre-populated server nodes, a target pre-populated server node selected from the plurality of pre-populated server nodes and a first priority are obtained in combination with a pre-population scheduling model based on deep reinforcement learning.

[0092] In some embodiments, the pre-population scheduling model based on deep reinforcement learning includes a policy network and a plurality of Q networks, the input information of the policy network includes a state based on deep reinforcement learning, the state based on deep reinforcement learning is obtained based on the user request information, the load information of the plurality of pre-populated server nodes, the related data of the execution time prediction model, and the related information of the request queue of the pre-populated server node; the output information of the policy network includes an action vector based on deep reinforcement learning, the action vector includes the probability of selecting the request queue corresponding to each priority of the plurality of pre-populated server nodes.

[0093] The input information of the Q network is the same as the input information of the policy network, and the output information of the Q network is a Q vector corresponding to the action vector; the i-th element in the Q vector is the Q value of the request queue with priority z of the y-th pre-populated server node corresponding to the i-th element in the action vector; wherein, Here, " / / " represents the positive integer quotient of the division; Here, "%" represents the remainder, i.e., y is obtained by calculating the integer quotient of i and the number of request queues of the pre-filling server node, and z is obtained by calculating the remainder of i and the number of request queues of the pre-filling server node.

[0094] The input of the pre-filling scheduling model of deep reinforcement learning of the embodiment relates to user request information, load information of a plurality of pre-filling server nodes, related data of an execution time prediction model, and related information of request queues of the pre-filling server nodes, and the like, to realize optimal scheduling for service level indicators.

[0095] Pre-filling reward of the pre-filling scheduling model based on deep reinforcement learning which is defined by the following formula:

[0096]

[0097] wherein, is the pre-filling execution time, which is obtained based on the execution time prediction model, and TTFT is the time to first token, i.e., the time for completing pre-filling, which is fed back by the pre-filling server node after completing the pre-filling request.

[0098] In some embodiments, the execution time prediction model is utilized, the remaining unmatched length in the input of the execution time prediction model is set as the prompt word length, the server node type is set as the P strong server node, and the storage location of the KV cache is set as the memory, and the output is taken as the reward baseline.

[0099] In one example, the pre-filling scheduling model based on deep reinforcement learning includes one policy network and two Q networks and , and the structures of the two Q networks are the same. The input of the policy network , i.e., the state of DRL may include the input and output of the execution time prediction model, load information of a plurality of pre-filling server nodes, and request queue information. For example, the request queue information includes: a quantum greater than and closest to the length of the user request information (i.e., the length of the token sequence of the user request), and the total token length of all pre-filling requests in all request queues with a priority higher than or equal to the quantum. The output of the policy network, i.e., the action vector of DRL , is defined as The mapping relationship between the selected pre-filled server nodes and priority x is as follows: First, the output layer at the very end of the policy network is a Sigmoid function, which outputs a dimension of... ( Action vector (for the number of pre-populated server nodes) Action vector Let be a probability vector whose sum of all elements is 1. The i-th element represents the probability that the y-th pre-filled server node with priority z is selected from the queue, where y = i / / 3; z = i % 3. Then, according to the probability... Sampling is performed, and the sampling result i can be converted into the selected pre-populated server node y and priority z. For example, If the value is 10, then the output will be an action vector with a dimension of 30. This includes the probability of selecting each of the 30 request queues from 10 pre-filled server nodes; the probability in this action vector is selected through a random mechanism. If the 10th element is selected, then y = 10 / / 3 = 3, z = 10%3 = 1. Define the Q-network. and Input: i.e., the state of the DRL , and policy network Same input. Define Q network. and Output: i.e., the action vector of the DRL The corresponding Q-vector, the first in the Q-vector Each element is an action vector. The The element corresponding to the first The priority of each pre-populated server node is The q-value of the queue in reinforcement learning, also known as the action value, represents the cumulative reward obtained by the user request information being served by the request queue with priority z of the y-th pre-filled server node.

[0100] In this embodiment, the q-value in reinforcement learning, also known as the action value, is the cumulative reward obtained by serving the user's request information through the request queue with priority z of the y-th pre-filled server node.

[0101] It should be noted that the Q value is the query value in the transformer architecture, which can be translated as the query value, and the q value here is the q value in reinforcement learning, which can also be called the action value, which refers to the cumulative reward that can be obtained by taking a certain action (i.e., selecting a certain queue of a certain server node to provide service for a user), that is, the cumulative pre-filling reward. In some embodiments, the training method of the pre-filling scheduling model based on deep reinforcement learning includes: obtaining training data of the pre-filling scheduling model based on deep reinforcement learning; the training data includes pre-filling rewards; introducing random policy entropy into the pre-filling rewards of the training data to obtain training data with random policy entropy; training a policy network and multiple Q networks based on the training data with random policy entropy to obtain the pre-filling scheduling model based on deep reinforcement learning.

[0102] The embodiment introduces random policy entropy to train the scheduling model using the SAC (Soft Actor-critic, a kind of reinforcement learning algorithm) algorithm, thereby effectively improving the inference efficiency.

[0103] In some embodiments, the method of obtaining training data of the pre-filling scheduling model based on deep reinforcement learning can include the following steps:

[0104] The following operations are performed in a loop until the loop termination condition is met;

[0105] Obtain relevant data of the pre-filling stage, and based on the relevant data of the pre-filling stage, obtain the state of the policy network at the current time;

[0106] Based on the state of the policy network at the current time, obtain the action vector at the current time through the policy network, and process the user request information through the action vector at the current time to obtain the pre-filling reward at the current time;

[0107] Based on the state at the current time, the action vector at the current time, the pre-filling reward at the current time, and the state at the next time, construct a first data combination, and take the first data combination as one training sample in the training data of the pre-filling scheduling model based on deep reinforcement learning.

[0108] The training data of the pre-filling scheduling model based on deep reinforcement learning is obtained through the above loop process, which can improve the real-time performance of obtaining the training data.

[0109] In one example, the training method of the pre-filling scheduling model based on deep reinforcement learning includes the following process:

[0110] Initialize the large model inference scheduling system, the Q network of the pre-filling stage , the policy network , and the data cache ; let the current time be The state of the policy network is The action vector is , and pre-filled rewards Q value is V value The policy network is The state at the next moment is .

[0111] Data Collection: Based on the scheduling thread and inference thread described in the above embodiments, data is continuously collected by processing the pre-filled user request information. Taking the pre-filling stage as an example, this involves extracting the state from the user request information, the execution time prediction model, and the pre-filled server cluster. Using policy networks Obtain action vector .in, Methods of obtaining this include: action vectors , as a dimension The probability vector; according to the probability vector Sampling, to obtain its first One element; calculate , Then choose the first one. The priority of each pre-populated server node is The request queue.

[0112] Based on action vectors Process user request information, obtain the corresponding first-term delay (TTFT) or term interval (TBT), and calculate the pre-filling reward. The state is then updated to the state of the next moment. ,Will Store in data cache .

[0113] From data cache A batch of samples were taken from the middle. Train the pre-filled scheduling model according to the following procedure until the model converges; then, only the policy network can be used. Perform scheduling:

[0114] To prefill rewards We introduce stochastic policy entropy to construct an optimization objective function:

[0115]

[0116] in, It is the entropy function. This represents the strategy entropy coefficient.

[0117] Optimize using the following process (i.e., the SAC algorithm):

[0118] The Q value is calculated as:

[0119]

[0120] wherein, is a discount factor,

[0121]

[0122] .

[0123] The loss function of the Q network and is the mean square error (MSE):

[0124]

[0125] wherein,

[0126]

[0127] Based on the gradient , the Q network and are trained by using the gradient descent method:

[0128]

[0129] The Q network and are updated by using the soft synchronization method:

[0130] Based on the value gradient

[0131] calculated above, the optimization objective function constructed above can be converted to:

[0132]

[0133] Based on the value gradient , the strategy network is trained by using the gradient descent method:

[0134]

[0135] Through the above process, the pre-filling scheduling model based on DRL is continuously trained, and the service level is continuously improved.

[0136] ​​​Step 204, sending a pre-filling request to a request queue corresponding to the first priority in the target pre-filling server node, so that the target pre-filling server node processes the pre-filling request.

[0137] It should be noted that the implementation of the present step can refer to step 102 in the above-mentioned embodiments for details, and the principle is the same, which will not be repeated here.

[0138] Step 205, in response to the completion of the pre-filling request processing, based on the user request information and the load information of the plurality of decoding server nodes, in combination with the decoding scheduling model based on deep reinforcement learning, obtaining the target decoding server node selected from the plurality of decoding server nodes and the second priority.

[0139] In some embodiments, the structure of the decoding scheduling model based on deep reinforcement learning is the same as that of the pre-filling scheduling model based on deep reinforcement learning, and the decoding reward of the decoding scheduling model based on deep reinforcement learning is defined by the following formula:

[0140]

[0141] wherein, is the time between tokens, is the length of the new token generated by the decoding server node, which can be fed back by the decoding server node after completing the decoding request.

[0142] In one example, the decoding scheduling model based on deep reinforcement learning includes 1 policy network , 2 Q networks and The input state of the policy network is obtained based on the user request information, the load information of the plurality of decoding server nodes, the relevant data of the execution time prediction model and the relevant information of the request queue of the decoding server node, which can include: the input and output of the execution time prediction model, the load information of the plurality of decoding server nodes and the request queue information. For example, the request queue information includes: a quantum greater than and closest to the length of the user request information (i.e. the length of the token sequence of the user request), and the total token length of all decoding requests in all request queues with a priority higher than or equal to the quantum. The output action vector of the policy network and the selected decoding server node and the priority x have the following mapping relationship: first, the output layer at the end of the policy network is a Sigmoid function, outputting an action vector ( an action vector for the number of decoding server nodes an action vector is a probability vector, the sum of all elements of which is 1, and the i-th element represents the probability that the queue with priority z of the y-th decoding server node is selected, where y = i / / 3; z = i % 3. Then, sampling is performed according to the probability The sampling result i can be converted into the selected decoding server node y and the priority z.

[0143] In this embodiment, the training method of the decoding scheduling model based on deep reinforcement learning is consistent with the training method of the pre-filling scheduling model based on deep reinforcement learning, and will not be repeated. Whether to train the decoding scheduling model or the pre-filling scheduling model depends on the stage in which the user request information being processed by the scheduling thread is located. If it is in the pre-filling stage, the training data of this stage is collected to train the pre-filling scheduling model. If it is in the decoding stage, the training data of this stage is collected to train the decoding scheduling model.

[0144] In step 206, a decoding request is sent to the request queue corresponding to the second priority in the target decoding server node, so that the target decoding server node processes the decoding request to obtain a model inference result, wherein the decoding request includes the key-value cache generated by the target pre-filling server node.

[0145] It should be noted that the implementation of this step can refer to step 104 in the above embodiments for details, and the principle is the same, which will not be repeated here.

[0146] The model inference scheduling method of the embodiments of the present application, on the basis of the above embodiments, estimates the pre-filling execution time of the request by executing the time prediction model, provides decision state support for the subsequent deep reinforcement learning scheduling algorithm, and can also be used as a baseline for the deep reinforcement learning reward, thereby improving the learning efficiency and scheduling performance, and thus improving the inference efficiency. In the scheduling model based on deep reinforcement learning, the scheduling model is trained by introducing the random policy entropy and using the soft actor critic algorithm, thereby effectively improving the inference efficiency.

[0147] In order to clearly illustrate the above embodiments, specific examples will be described. Figure 3 is a framework diagram of a large model inference scheduling system based on DRL provided by the embodiments of the present application. The system represents the pre-filling scheduling model based on DRL and the decoding scheduling model based on DRL in the pre-filling and decoding stages through the scheduling model based on DRL, which is described in detail in the above embodiments. Figure 3, the execution time prediction model is used to perform time prediction according to user request information, server cluster related information, shared prefix matching information, etc., and the prediction result is transmitted to the DRL-based scheduling model. The DRL-based scheduling model selects a server node for a pre-filling request or a decoding request according to user request information, server cluster related information, and execution time prediction model related information, etc. The server cluster calls the selected server node to process the user request, and the decoding server node returns the inference result to the user. In addition, the server cluster returns the pre-filling and decoding rewards for serving the user request to the DRL-based scheduling model, and the scheduling model optimizes the model based on the above data using the SAC algorithm. The above process is repeatedly performed to train the DRL-based scheduling model and continue to provide services for users.

[0148] It should be noted that the embodiments of the present disclosure can include multiple steps, which are numbered for ease of description, but these numbers do not limit the execution time slots and execution order between the steps; these steps can be implemented in any order, and the embodiments of the present disclosure do not limit this.

[0149] Corresponding to the above-mentioned model inference scheduling method, the present disclosure also proposes a model inference scheduling system. As shown in Figure 4 , the model inference scheduling system is configured to be applied to the model inference scheduling system. The server cluster includes a plurality of pre-filling server nodes and a plurality of decoding server nodes. The pre-filling server nodes and the decoding server nodes are configured with a multi-level feedback queue when the model inference scheduling system is started. The plurality of request queues included in the multi-level feedback queue have different priorities. The system can include:

[0150] The first node selection module 411 is configured to, in response to receiving user request information, based on the user request information and the load information of the plurality of pre-filling server nodes, and in combination with the pre-filling scheduling model based on deep reinforcement learning, obtain a target pre-filling server node selected from the plurality of pre-filling server nodes and a first priority.

[0151] The first scheduling module 412 is configured to send a pre-filling request to a request queue corresponding to the first priority in the target pre-filling server node, and instruct the target pre-filling server node to process the pre-filling request.

[0152] The second node selection module 413 is configured to, in response to the pre-filling request processing being completed, based on the user request information and the load information of the plurality of decoding server nodes, and in combination with the decoding scheduling model based on deep reinforcement learning, obtain a target decoding server node selected from the plurality of decoding server nodes and a second priority.

[0153] The second scheduling module 414 is configured to send a decoding request to a request queue corresponding to the second priority in the target decoding server node, so that the target decoding server node processes the decoding request to obtain the model inference result, wherein the decoding request comprises the key-value cache generated by the target pre-filling server node.

[0154] Further, in a possible implementation of the embodiment of the present disclosure, the level of the priority of the request queue is determined by a queue priority parameter, which is used to indicate the execution duration of the task in the request queue.

[0155] Further, in a possible implementation of the embodiment of the present disclosure, as shown in the system further comprises a node type division module 415, configured to: Figure 5

[0156] According to the configuration information of the plurality of pre-filling server nodes, the plurality of pre-filling server nodes are divided into two types of server nodes, i.e., strong pre-filling server nodes and weak pre-filling server nodes; the plurality of queue priority parameters of the strong pre-filling server nodes and the plurality of queue priority parameters of the weak pre-filling server nodes have the same part and different parts.

[0157] According to the configuration information of the plurality of decoding server nodes, the plurality of decoding server nodes are divided into two types of server nodes, i.e., strong decoding server nodes and weak decoding server nodes; the plurality of queue priority parameters of the strong decoding server nodes and the plurality of queue priority parameters of the weak decoding server nodes have the same part and different parts.

[0158] Further, in a possible implementation of the embodiment of the present disclosure, the pre-filling server node stores a historical key-value cache, and the first node selection module 411 is specifically configured to:

[0159] According to the user request information, the historical key-value cache is searched to obtain shared prefix matching information between the user request information and each pre-filling server node;

[0160] Based on the user request information and the shared prefix matching information between the user request information and each pre-filling server node, and in combination with an execution time prediction model, an execution estimation time of each pre-filling server node processing the user request information is obtained.

[0161] Based on the user request information, the load information of the plurality of pre-filling server nodes, and the execution estimation time of each pre-filling server node processing the user request information, and in combination with a pre-filling scheduling model based on deep reinforcement learning, a target pre-filling server node selected from the plurality of pre-filling server nodes and a first priority are obtained.

[0162] ​Further, in a possible implementation of the embodiment of the present disclosure, the shared prefix matching information between the user request information and each pre-filling server node includes a key-value cache storage location of the shared prefix, a shared prefix length, and a remaining unmatched length.

[0163] The first node selection module 411, when obtaining the execution estimation time of each pre-filling server node for processing the user request information based on the user request information and the shared prefix matching information between the user request information and each pre-filling server node, in combination with the execution time prediction model, is specifically configured to:

[0164] obtain a token sequence length of the user request based on the user request information;

[0165] obtain the execution estimation time of each pre-filling server node for processing the user request information based on the token sequence length of the user request, the server node type, the key-value cache storage location of the shared prefix, the shared prefix length, and the remaining unmatched length, in combination with the execution time prediction model.

[0166] Further, in a possible implementation of the embodiment of the present disclosure, the first node selection module 411 is specifically configured to:

[0167] obtain the target pre-filling server node selected from the plurality of pre-filling server nodes and the first priority based on the user request information and the load information of the plurality of pre-filling server nodes, in combination with the pre-filling scheduling model based on deep reinforcement learning through a scheduling thread.

[0168] Further, in a possible implementation of the embodiment of the present disclosure, when the target pre-filling server node processes the pre-filling request, the first scheduling module 412 is specifically configured to:

[0169] traverse the plurality of request queues of the target pre-filling server node in descending order of priority through an inference thread.

[0170] in response to the current request queue being empty, update a request queue with a priority one level lower than that of the current request queue to the current request queue;

[0171] in response to the current request queue not being empty, execute the pre-filling request in the current request queue.

[0172] Further, in a possible implementation of the embodiment of the present disclosure, as shown in Figure 5 the system further includes a first model training module 416 configured to:

[0173] obtain training data collected in an under-load or no-load environment;

[0174] The neural network is trained based on the training data to obtain an execution time prediction model, wherein the neural network includes three hidden layers, and an activation function of the neural network includes a LeakyRelu function.

[0175] Further, in a possible implementation of the embodiment of the present disclosure, when the first node selection module 411 obtains the target pre-populated server node selected from the plurality of pre-populated server nodes and the first priority based on the user request information, the load information of the plurality of pre-populated server nodes, and the execution estimation time of each pre-populated server node processing the user request information, in combination with the pre-populated scheduling model based on deep reinforcement learning, the method is specifically used for:

[0176] Based on the token sequence length of the user request, the server node type, the key-value cache storage location of the shared prefix, the shared prefix length, the remaining unmatched length, the load information of the plurality of pre-populated server nodes, the execution estimation time of each pre-populated server node processing the user request information, and the request queue information of the plurality of pre-populated server nodes, in combination with the pre-populated scheduling model based on deep reinforcement learning, the target pre-populated server node selected from the plurality of pre-populated server nodes and the first priority are obtained.

[0177] Further, in a possible implementation of the embodiment of the present disclosure, the pre-populated scheduling model based on deep reinforcement learning includes a policy network and a plurality of Q networks, the input information of the policy network includes a state based on deep reinforcement learning, the state based on deep reinforcement learning is obtained based on the user request information, the load information of the plurality of pre-populated server nodes, the related data of the execution time prediction model, and the related information of the request queue of the pre-populated server node; the output information of the policy network includes an action vector based on deep reinforcement learning, the action vector includes the probability that the request queue corresponding to each priority of the plurality of pre-populated server nodes is selected;

[0178] The input information of the Q network is the same as the input information of the policy network, and the output information of the Q network is a Q vector corresponding to the action vector; the i-th element in the Q vector is an action value of the request queue with a priority of z of the y-th pre-populated server node corresponding to the i-th element in the action vector in reinforcement learning, and the action value is used to represent a cumulative reward obtained by serving the user request information by the request queue with the priority of z of the y-th pre-populated server node, wherein y is obtained by calculating the integer quotient of i and the number of request queues of the pre-populated server node, and z is obtained by calculating the remainder of i and the number of request queues of the pre-populated server node;

[0179] The pre-populated reward of the pre-populated scheduling model based on deep reinforcement learning The pre-populated reward is defined by the following first formula:

[0180]

[0181] wherein, is a pre-padding execution time, is obtained based on an execution time prediction model, TTFT is a first token time delay, used to indicate a time of completing pre-padding, and is fed back by a pre-padding server node after completing a pre-padding request.

[0182] Further, in a possible implementation manner of the embodiment of the present disclosure, the structure of the decoding scheduling model based on deep reinforcement learning is the same as that of the pre-padding scheduling model based on deep reinforcement learning, the decoding reward of the decoding scheduling model based on deep reinforcement learning is is defined by the following second formula:

[0183]

[0184] wherein, is a token interval, is a new token length generated by a decoding server node, and can be fed back by the decoding server node after completing a decoding request.

[0185] Further, in a possible implementation manner of the embodiment of the present disclosure, the system further includes a second model training module 417, configured to:

[0186] obtain training data of the pre-padding scheduling model based on deep reinforcement learning; the training data includes a pre-padding reward;

[0187] introduce a random policy entropy into the pre-padding reward of the training data, to obtain training data with the random policy entropy introduced;

[0188] train the policy network and the multiple Q networks based on the training data with the random policy entropy introduced, to obtain the pre-padding scheduling model based on deep reinforcement learning.

[0189] Further, in a possible implementation manner of the embodiment of the present disclosure, as shown in Figure 5 the second model training module 417, when obtaining the training data of the pre-padding scheduling model based on deep reinforcement learning, is specifically configured to:

[0190] perform the following operations in a loop until a loop termination condition is met;

[0191] obtain related data of a pre-padding stage, and based on the related data of the pre-padding stage, obtain a state of the policy network at a current time;

[0192] based on the state of the policy network at the current time, obtain an action vector at the current time through the policy network, and process user request information through the action vector at the current time to obtain a pre-padding reward at the current time;

[0193] Based on the state at the current moment, the action vector at the current moment, the pre-padding reward at the current moment, and the state at the next moment, a first data combination is constructed, and the first data combination is taken as one training sample in training data of the pre-padding scheduling model based on deep reinforcement learning.

[0194] Further, in a possible implementation of the embodiment of the present disclosure, the three queue priority parameters of the strong server node are respectively , and , the three queue priority parameters of the weak server node are respectively , 2 and 4 , wherein is the number of word pieces included in the pre-padding granularity or the decoding granularity of the model inference method.

[0195] It should be noted that the foregoing explanation and description of the method embodiment also apply to the device of the present embodiment, and the principle is the same, which will not be limited in the present embodiment.

[0196] The description of the features in the embodiment corresponding to the model inference scheduling system can be referred to the related description of the embodiment corresponding to the model inference scheduling method, which will not be repeated here.

[0197] Embodiments of the present application also provide an electronic device, comprising a memory and a processor, the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above model inference scheduling method embodiments.

[0198] Embodiments of the present application also provide a computer readable storage medium, which stores a computer program, wherein the computer program is configured to perform the steps in any of the above model inference scheduling method embodiments when running.

[0199] In an exemplary embodiment, the above computer readable storage medium can include, but is not limited to, a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store computer programs.

[0200] Embodiments of the present application also provide a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the steps in any of the above model inference scheduling method embodiments.

[0201] The embodiment of the present application further provides another computer program product comprising a nonvolatile computer readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the model inference scheduling method embodiments described above.

[0202] Those skilled in the art will further appreciate that the units and algorithm steps of the examples described in connection with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or both, and that the interchangeability of hardware and software methods is contemplated. Accordingly, the examples described herein are not limited to any particular hardware or software configuration, and the methods described herein can be implemented in hardware, software, or a combination thereof.

[0203] The above describes in detail a model inference scheduling method and system, an electronic device and a storage medium provided by the present application. The principles and implementation modes of the present application are described herein by applying specific examples, and the above description of the embodiments is only to help understand the method of the present application and its core idea. It should be pointed out that, for those skilled in the art, without departing from the principles of the present application, some improvements and modifications can be made to the present application, and these improvements and modifications also fall within the protection scope of the claims of the present application.

Claims

1. A model inference scheduling method, characterized in that, The application is applied to a model inference scheduling system, the model inference scheduling system includes a plurality of pre-filling server nodes and a plurality of decoding server nodes, the pre-filling server nodes and the decoding server nodes are configured with a multi-level feedback queue when the model inference scheduling system starts, a plurality of request queues included in the multi-level feedback queue have different priorities, the pre-filling server nodes store historical key-value caches, and the method comprises the following steps: In response to receiving user request information, based on the user request information and the load information of the plurality of pre-filling server nodes, in combination with a pre-filling scheduling model based on deep reinforcement learning, a target pre-filling server node selected from the plurality of pre-filling server nodes and a first priority are obtained; A pre-filling request is sent to a request queue corresponding to the first priority in the target pre-filling server node, and the target pre-filling server node processes the pre-filling request; In response to the completion of the pre-filling request processing, based on the user request information and the load information of the plurality of decoding server nodes, in combination with a decoding scheduling model based on deep reinforcement learning, a target decoding server node selected from the plurality of decoding server nodes and a second priority are obtained; A decoding request is sent to a request queue corresponding to the second priority in the target decoding server node, and the target decoding server node processes the decoding request to obtain a model inference result, wherein the decoding request includes a key-value cache generated by the target pre-filling server node; The method comprises the following steps: Based on the user request information, the historical key-value cache is searched to obtain shared prefix matching information between the user request information and each pre-filling server node; Based on the user request information and the shared prefix matching information between the user request information and each pre-filling server node, in combination with an execution time prediction model, execution estimation time of each pre-filling server node processing the user request information is obtained; Based on the user request information, the load information of the plurality of pre-filling server nodes and the execution estimation time of each pre-filling server node processing the user request information, in combination with the pre-filling scheduling model based on deep reinforcement learning, a target pre-filling server node selected from the plurality of pre-filling server nodes and a first priority are obtained.

2. The model inference scheduling method of claim 1, wherein, The priority level of the request queue is determined by a queue priority parameter, and the queue priority parameter is used to indicate the execution time of the task in the request queue, The method further comprises the following steps: According to the configuration information of the plurality of pre-filling server nodes, the plurality of pre-filling server nodes are divided into two types of server node types, strong pre-filling server nodes and weak pre-filling server nodes; there are same parts and different parts between the plurality of queue priority parameters of the strong pre-filling server nodes and the plurality of queue priority parameters of the weak pre-filling server nodes; According to the configuration information of the plurality of decoding server nodes, the plurality of decoding server nodes are divided into two types of server node types, strong decoding server nodes and weak decoding server nodes; there are same parts and different parts between the plurality of queue priority parameters of the strong decoding server nodes and the plurality of queue priority parameters of the weak decoding server nodes.

3. The model inference scheduling method of claim 2, wherein, The shared prefix matching information between the user request information and each pre-filling server node includes the key value cache storage location of the shared prefix, the shared prefix length and the remaining unmatched length; Based on the user request information and the shared prefix matching information between the user request information and each pre-filling server node, an execution estimation time of each pre-filling server node processing the user request information is obtained by combining an execution time prediction model; including: Based on the user request information, the user request token sequence length is obtained; Based on the user request token sequence length, the server node type, the key value cache storage location of the shared prefix, the shared prefix length and the remaining unmatched length, the execution estimation time of each pre-filling server node processing the user request information is obtained by combining the execution time prediction model.

4. The model inference scheduling method of claim 1, wherein, Based on the user request information and the load information of the plurality of pre-filling server nodes, a target pre-filling server node and a first priority selected from the plurality of pre-filling server nodes are obtained by combining a pre-filling scheduling model based on deep reinforcement learning; including: Through a scheduling thread, based on the user request information and the load information of the plurality of pre-filling server nodes, a target pre-filling server node and a first priority selected from the plurality of pre-filling server nodes are obtained by combining a pre-filling scheduling model based on deep reinforcement learning. The target pre-filling server node is caused to process the pre-filling request; including:

5. The model inference scheduling method of claim 2, wherein, Through an inference thread, the plurality of request queues of the target pre-filling server node are traversed in order from high to low according to the priority; In response to the current request queue being empty, a request queue with a priority one level lower than that of the current request queue is updated as the current request queue; In response to the current request queue being not empty, a pre-filling request in the current request queue is executed. The training method of the execution time prediction model, including:

6. The model inference scheduling method of claim 1 or 4, wherein, Obtaining training data collected in an under-load or empty-load environment; Based on the training data, a neural network is trained to obtain the execution time prediction model, wherein the neural network includes three hidden layers, and the activation function of the neural network includes a LeakyRelu function. ​ 7. The model inference scheduling method of claim 3, wherein, The target pre-populated server node and the first priority are obtained from the plurality of pre-populated server nodes based on the user request information, the load information of the plurality of pre-populated server nodes, and the execution estimation time of each of the pre-populated server nodes processing the user request information, in combination with the pre-populated scheduling model based on deep reinforcement learning. The target pre-populated server node and the first priority are obtained from the plurality of pre-populated server nodes based on the token sequence length of the user request, the server node type, the key-value cache storage location of the shared prefix, the shared prefix length, the remaining unmatched length, the load information of the plurality of pre-populated server nodes, the execution estimation time of each of the pre-populated server nodes processing the user request information, and the request queue information of the plurality of pre-populated server nodes, in combination with the pre-populated scheduling model based on deep reinforcement learning.

8. The model inference scheduling method of claim 1, wherein, The pre-populated scheduling model based on deep reinforcement learning includes a policy network and a plurality of Q networks, the input information of the policy network includes a deep reinforcement learning-based state, the deep reinforcement learning-based state is obtained based on the user request information, the load information of the plurality of pre-populated server nodes, the related data of the execution time prediction model, and the related information of the request queue of the pre-populated server node; the output information of the policy network includes a deep reinforcement learning-based action vector, the action vector includes the probability of selecting the request queue corresponding to each priority of the plurality of pre-populated server nodes; The input information of the Q network is the same as the input information of the policy network, and the output information of the Q network is a Q vector corresponding to the action vector; the i-th element in the Q vector is the action value of the request queue with priority z of the y-th pre-populated server node corresponding to the i-th element in the action vector in reinforcement learning, the action value is used to represent the cumulative reward obtained by the user request information being served by the request queue with priority z of the y-th pre-populated server node, wherein y is obtained by calculating the integer quotient of i and the number of request queues of the pre-populated server node, and z is obtained by calculating the remainder of i and the number of request queues of the pre-populated server node; The prefill reward of the prefill scheduling model based on deep reinforcement learning defined by the following first equation: wherein, is a pre-filling execution time, which is obtained based on the execution time prediction model, TTFT is a first word token time delay, used to indicate the time for completing the pre-filling, and is fed back by the pre-filling server node after completing the pre-filling request.

9. The model inference scheduling method of claim 7, wherein, The structure of the deep reinforcement learning based decoding scheduling model is the same as that of the deep reinforcement learning based pre-filling scheduling model, and the decoding reward of the deep reinforcement learning based decoding scheduling model is defined by a second formula as follows: defined by a second formula as follows: wherein, is a word interval, is a new word length generated by the decoding server node, which can be fed back by the decoding server node after completing the decoding request, is a pre-filled execution time.

10. The model inference scheduling method of claim 8, wherein, The training method of the pre-populated scheduling model based on deep reinforcement learning includes: Obtaining training data of the pre-populated scheduling model based on deep reinforcement learning; the training data includes the pre-populated reward; Introducing random policy entropy in the pre-populated reward of the training data to obtain training data with random policy entropy introduced; Training the policy network and the plurality of Q networks based on the training data with random policy entropy introduced to obtain the pre-populated scheduling model based on deep reinforcement learning.

11. The model inference scheduling method of claim 10, wherein, The training data of the pre-populated scheduling model based on deep reinforcement learning includes: The following operations are performed in a loop until the loop termination condition is met; obtain relevant data of a pre-filling stage, obtain a state of the policy network at a current time based on the relevant data of the pre-filling stage; obtain an action vector at the current time through the policy network based on the state of the policy network at the current time, and process the user request information through the action vector at the current time to obtain a pre-filling reward at the current time; construct a first data combination based on the state at the current time, the action vector at the current time, the pre-filling reward at the current time, and a state at a next time, and use the first data combination as one training sample in training data of the pre-filling scheduling model based on deep reinforcement learning.

12. An electronic device, comprising: comprise: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the model inference scheduling method of any one of claims 1-11.

13. A non-transitory computer-readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to enable the computer to perform the model inference scheduling method of any one of claims 1-11.

14. A computer program product, characterised in that, comprise a computer program that, when executed by a processor, implements the method of any one of claims 1-11.

Images