Model reasoning scheduling method and system, electronic equipment and storage medium

Abstract

The invention provides a model reasoning scheduling method and system, an electronic device and a storage medium, the system comprises a plurality of pre-filling server nodes and a plurality of decoding server nodes, and the method comprises the following steps: responding to received user request information, and combining a pre-filling scheduling model based on deep reinforcement learning, obtaining a target pre-filling server node selected from the plurality of pre-filling server nodes and a first priority; sending a pre-filling request to a request queue corresponding to the first priority in the target pre-filling server node, and enabling the target pre-filling server node to process the pre-filling request; after the pre-filling request is processed, obtaining a target decoding server node selected from a plurality of decoding server nodes and a second priority by combining a decoding scheduling model based on deep reinforcement learning; sending a decoding request to a request queue corresponding to the second priority in the target decoding server node, and enabling the target decoding server node to process the decoding request; and reasoning efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of model inference, in particular to a model inference scheduling method and system, an electronic device and a storage medium. BACKGROUND

[0002] With the rapid development of large language models (LLM), major manufacturers have begun to promote large-scale commercial deployment of large models to provide users with high-throughput and low-latency services. However, when deploying large models, the scheduling strategy for large model services faces many challenges as the large models change from training mode to inference mode; the existing scheduling strategy has the problem of low inference efficiency. SUMMARY

[0003] The present disclosure provides a model inference scheduling method and system, an electronic device and a storage medium to improve inference efficiency.

[0004] According to a first aspect of the present disclosure, a model inference scheduling method is provided, applied to a model inference scheduling system, the server cluster comprising a plurality of pre-filling server nodes and a plurality of decoding server nodes, the pre-filling server nodes and the decoding server nodes each being configured with a multi-level feedback queue when the model inference scheduling system is started, the plurality of request queues included in the multi-level feedback queue having different priorities, the method comprising: in response to receiving user request information, based on the user request information and load information of the plurality of pre-filling server nodes, in combination with a pre-filling scheduling model based on deep reinforcement learning, obtaining a target pre-filling server node selected from the plurality of pre-filling server nodes and a first priority; sending a pre-filling request to a request queue corresponding to the first priority in the target pre-filling server node, and instructing the target pre-filling server node to process the pre-filling request; in response to the pre-filling request processing being completed, based on the user request information and load information of the plurality of decoding server nodes, in combination with a decoding scheduling model based on deep reinforcement learning, obtaining a target decoding server node selected from the plurality of decoding server nodes and a second priority; sending a decoding request to a request queue corresponding to the second priority in the target decoding server node, and instructing the target decoding server node to process the decoding request to obtain a model inference result, wherein the decoding request includes a key-value cache generated by the target pre-filling server node.

[0005] According to a second aspect of the present disclosure, a model inference scheduling system is provided, which is configured to be applied to a model inference scheduling system, the server cluster includes a plurality of pre-filling server nodes and a plurality of decoding server nodes, the pre-filling server nodes and the decoding server nodes are both configured with a multi-level feedback queue at the start of the model inference scheduling system, the plurality of request queues included in the multi-level feedback queue have different priorities, and the system includes: a first node selection module configured to, in response to receiving user request information, based on the user request information and load information of the plurality of pre-filling server nodes, in combination with a pre-filling scheduling model based on deep reinforcement learning, obtain a target pre-filling server node selected from the plurality of pre-filling server nodes and a first priority; a first scheduling module configured to send a pre-filling request to a request queue corresponding to the first priority in the target pre-filling server node, so that the target pre-filling server node processes the pre-filling request; a second node selection module configured to, in response to the pre-filling request processing being completed, based on the user request information and load information of the plurality of decoding server nodes, in combination with a decoding scheduling model based on deep reinforcement learning, obtain a target decoding server node selected from the plurality of decoding server nodes and a second priority; a second scheduling module configured to send a decoding request to a request queue corresponding to the second priority in the target decoding server node, so that the target decoding server node processes the decoding request to obtain a model inference result, wherein the decoding request includes a key-value cache generated by the target pre-filling server node.

[0006] According to a third aspect of the present disclosure, an electronic device is provided, which includes: at least one processor; and a memory in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the model inference scheduling method of the first aspect.

[0007] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to perform the model inference scheduling method of the first aspect.

[0008] According to a fifth aspect of the present disclosure, a computer program product is provided, which includes a computer program, and the computer program, when executed by a processor, implements the model inference scheduling method of the first aspect.

[0009] The present disclosure provides a model inference scheduling method and system, an electronic device and a storage medium, proposes a large model inference scheduling framework based on PD separation and multi-level feedback queue, which improves inference efficiency and reduces memory occupation; and based on the framework, two scheduling algorithms based on deep reinforcement learning are used to select matched server nodes for user requests in the pre-padding and decoding stages respectively, which can improve the first word time delay and word interval of large model inference; the scheduling algorithm based on deep reinforcement learning fully utilizes user request information, server node information and the like in a large-scale server cluster for scheduling, which can overcome the construction limitations of heuristic methods in large-scale inference scheduling systems. The method can be directly used for scheduling user requests of large language models in a large-scale server cluster and deployed in a large model data center; it can also be applied to request scheduling of other types of large models after simple expansion.

[0010] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present application, nor to limit the scope of the present application. Other features of the present application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0011] The accompanying drawings are used to better understand the present scheme and do not constitute a limitation on the present disclosure. Among them: Figure 1 A flowchart of a model inference scheduling method provided by an embodiment of the present disclosure is shown; Figure 2 A flowchart of another model inference scheduling method provided by an embodiment of the present disclosure is shown; Figure 3 A framework diagram of a large model inference scheduling system based on DRL provided by an embodiment of the present disclosure is shown; Figure 4 A structure diagram of a model inference scheduling system provided by an embodiment of the present disclosure is shown; Figure 5 A structure diagram of another model inference scheduling system provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0012] Exemplary embodiments of the present disclosure are described below in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to help understanding, which should be considered only as exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Also, for the sake of clarity and conciseness, the description below omits the description of well-known functions and structures.

[0013] Terminology explanation: LLM (Large Language Model).

[0014] Transformer, a mainstream large model structure.

[0015] Deep reinforcement learning (DRL).

[0016] Query.

[0017] TTFT, The time to first token.

[0018] TBT, The time between tokens.

[0019] In the related art, the large model inference architecture is a pure decoder-only architecture based on the Transformer model. When inferring, the architecture converts a user request (also known as a prompt, which is a long string) into a token (which is an extremely short string) sequence, and uses an autoregressive mode to constantly generate tokens, i.e., concatenates the newly generated token with the old token sequence as a new input, uses the model to generate a new token, and continues until the complete answer is generated. Among them, in the initialization stage of the autoregressive process, the large model processes all tokens of the user request at one time until the first new token is generated, which is called prefill (Prefill, abbreviated as P), which is computationally intensive and has high parallelism. Then, the large model continues the prefill calculation process, and uses the autoregressive mode to constantly generate new tokens, which is called decoding (decode, abbreviated as D), which is input / output (Input / output, IO) intensive and requires a large amount of memory bandwidth.

[0020] In the above inference process, the model first uses part of the neural network to convert all tokens into three feature vectors, including Query (Q value, Query value), Key (K value, Key value), and Value (V value). Then, the Q value of the new token is calculated with the K value and V value of all tokens, which will occupy a large amount of video memory or memory (hereinafter referred to as memory) due to the following factors: (1) After calculating the K value and V value of any token, it needs to be stored in the memory, called KV cache (KV cache, also known as key-value cache), which requires a large amount of memory. Then, when the Q value of the new token needs to be calculated with the K value and V value of all tokens, the KV cache can be directly read from the memory to avoid repeated calculation, which is called KV cache reuse.

[0021] (2) The parameter quantity of the large model increases sharply, which not only makes the KV cache of each word element larger, but also makes the length of the output word element sequence longer, i.e., more word elements of the KV cache need to be stored, and the two factors further exacerbate the memory requirement.

[0022] (3) The large model inference service needs to process a large number of user requests at the same time, so a large number of KV caches of user requests need to be stored at the same time.

[0023] When the large model inference framework processes a large number of user requests, it first needs to manage the KV cache of a large number of user requests. Taking the large model inference framework SGLang (Structured Language Model Programs) as an example, a user request will be divided into shared prompt words and non-shared prompt words. The shared prompt word, also known as the shared prefix, is the part located at the head of the prompt word of the user request and repeated with the head of the prompt word of other user requests (a string, which can be converted into a corresponding word element sequence). The KV cache corresponding to the shared prefix can be reused for the shared prefix part of other user requests, thereby avoiding repeated calculation, while the KV cache of the non-shared prompt word needs to be recalculated.

[0024] The existing model inference method has problems such as low inference efficiency and large storage occupation. In order to improve the inference efficiency and reduce the storage occupation, the model inference scheduling method provided in the embodiments of the present disclosure is proposed.

[0025] The model inference scheduling method and system, the electronic device and the storage medium of the embodiments of the present disclosure are described below with reference to the accompanying drawings.

[0026] Figure 1 A flowchart of a model inference scheduling method provided in Embodiment One of the present disclosure.

[0027] It should be noted that the model inference scheduling method of the embodiments of the present disclosure is applied to a model inference scheduling system, which includes a pre-population server cluster and a decoding server cluster, the pre-population server cluster includes a plurality of pre-population server nodes, the decoding server cluster includes a plurality of decoding server nodes, the pre-population server nodes and the decoding server nodes are configured with a multi-level feedback queue when the model inference scheduling system starts, the priority of the plurality of request queues included in the multi-level feedback queue is different, and the plurality of request queues are empty when the multi-level feedback queue is just configured. The priority level of the request queue is determined by the queue priority parameter (quantum), which is used to indicate the execution time of the task in the request queue. In one example, based on the server cluster and the business characteristic information, the server cluster is divided into a pre-population server cluster and a decoding server cluster, each server cluster contains a plurality of server nodes, such as a pre-population server cluster containing server nodes, and a decoding server cluster containing One server node. Note: The subscript p indicates a pre-filled server node, and d indicates a decoding server node. This annotation method is used throughout this application.

[0028] like Figure 1 As shown, the method includes the following steps: Step 101: In response to receiving user request information, based on the user request information and the load information of multiple pre-filled server nodes, and combined with the pre-filled scheduling model based on deep reinforcement learning, obtain the target pre-filled server node and the first priority selected from multiple pre-filled server nodes.

[0029] In some embodiments, a user initiates a user request, i.e., a complete prompt word, such as a lexical sequence. In response to receiving the user request information, based on the user request information and the load information of multiple pre-filled server nodes, and combined with a pre-filling scheduling model based on deep reinforcement learning, scheduling information for the pre-filling stage is obtained. The scheduling information for the pre-filling stage includes a target pre-filled server node selected from the multiple pre-filled server nodes and a first priority, thereby enabling the pre-filling request to be added to the request queue corresponding to the first priority in the target pre-filled server node.

[0030] As mentioned above, model inference is divided into a prefilling stage and a decoding stage. Prefilling is computationally intensive, while decoding is I / O intensive. In this embodiment, the scheduling of the prefilling and decoding stages is separated. During the prefilling stage, prefilling server nodes are scheduled to process user requests. During the decoding stage, decoding server nodes are scheduled to process the KV cache generated by the prefilling server nodes and complete the decoding. That is, this embodiment adopts a PD (Prefill-decode Disaggregation) inference framework, where some server nodes only process the prefilling stage, calculate all KV caches of the user request information, and transmit them to other server nodes that only process the decoding stage for decoding. This architecture, through its separation design, allows server nodes with different performance levels to focus on processing a specific stage. Furthermore, since the same stage of each user request information has similar computational characteristics, it is beneficial to schedule requests with more flexible granularity.

[0031] In this embodiment, based on user request information and load information of multiple pre-filled server nodes, and combined with a pre-filled scheduling model based on deep reinforcement learning, a target pre-filled server node is selected from multiple pre-filled server nodes for scheduling.

[0032] The embodiment is based on the inference framework of PD separation, adopts a scheduling algorithm of deep reinforcement learning (DRL), and the main task of DRL is to observe the environment, such as the state (such as user request information, server node information, etc.) of the inference system, then interact with the environment based on a certain policy (policy, the relationship between the state and the action) to take action (action, such as selecting a certain server node to execute the request), and obtain a reward (reward, such as request completion time). Through the data generated by continuous interaction, the policy is continuously optimized based on the reinforcement learning algorithm, so that it can maximize the cumulative reward. DRL uses a neural network to model and optimize the policy and other modules involved in the above process, thereby improving the scheduling efficiency.

[0033] In some embodiments, the method further comprises: dividing the plurality of pre-filled server nodes into two types of server node types, strong pre-filled server nodes and weak pre-filled server nodes, according to the configuration information of the plurality of pre-filled server nodes; the plurality of queue priority parameters of the strong pre-filled server nodes and the plurality of queue priority parameters of the weak pre-filled server nodes have the same part and the different part; dividing the plurality of decoding server nodes into two types of server node types, strong decoding server nodes and weak decoding server nodes, according to the configuration information of the plurality of decoding server nodes; the plurality of queue priority parameters of the strong decoding server nodes and the plurality of queue priority parameters of the weak decoding server nodes have the same part and the different part.

[0034] The configuration information of the embodiment includes hardware configuration information, which can include but is not limited to gpu, cpu, video memory, memory, etc.

[0035] That is, the plurality of pre-filled server nodes and the plurality of decoding server nodes each include two types of server node types, and the plurality of queue priority parameters of the strong server nodes and the plurality of queue priority parameters of the weak server nodes have different parts and overlapping parts. That is, the plurality of pre-filled server nodes includes pre-filled strong server nodes (P-strong server nodes) and pre-filled weak server nodes (P-weak server nodes), and the plurality of decoding server nodes includes decoding strong server nodes (D-strong server nodes) and decoding weak server nodes (D-weak server nodes).

[0036] In some embodiments, the three queue priority parameters of the strong server nodes are , and , respectively, and the three queue priority parameters of the weak server nodes are , 2 and 4 , respectively. The number of word pieces included in the prefilling granularity or decoding granularity of the model inference method.

[0037] In one example, for a prefilling server node, the prefilling granularity of the large model inference algorithm used is word pieces, that is, all word pieces of the current batch of prefilling requests are prefilled each time of execution. The prefilling server cluster is divided into 2 groups, P-strong server nodes and P-weak server nodes, and each prefilling server node maintains 3 request queues of different priorities. The priority of the P-strong server node is , and the queue priority parameter quantum of the queue is . The queue priority parameter quantum indicates the duration of execution of the queue request each time, and the smaller the quantum, the shorter the execution time of the request queue, and the higher the priority. The priority is multiplied by quantum for each lower priority, and the higher the priority, , the smaller the value, , and the priority of the P-weak server node is . The queue priority parameter quantum of the queue is . The quantum of the three request queues of the P-strong server node is respectively: ， ， The quantum of the three request queues of the P-weak server node is respectively: ， ， 。 Wherein, The value of indicates that when it is the turn to execute the prefilling request of the request queue, the prefilling of word pieces is calculated continuously each time, and after the end, the next round of prefilling requests to be executed is selected from the highest priority non-empty queue.

[0038] In one example, for a decoding server node, the prefilling granularity of the large model inference algorithm used is word pieces, that is, all word pieces of the current batch of decoding requests are prefilled each time of execution. The decoding server cluster is divided into 2 groups, D-strong server nodes and D-weak server nodes, and each decoding server node maintains 3 request queues of different priorities (quantum). The quantum (priority from high to low) of the three request queues of the D-strong server node is respectively: , , The quantum (priority from high to low) of the three request queues of the weak server node is as follows: Among them, When it is the turn to execute the decoding request of the request queue, each time a continuous decoding of end, the decoding request to be executed in the next round is selected from the highest priority non-empty queue.

[0039] The embodiment alleviates the problem that IO-intensive decoding requests are delayed by pre-filling requests and computation-intensive pre-filling requests are frequently preempted by decoding requests by designing differentiated and partially overlapping queue priority parameter levels for the request queues of the pre-filling server node and the decoding server node, that is, by using a multi-level feedback queue with differentiated and partially overlapping queue priority parameters, thereby supporting a subsequent deep reinforcement learning-based scheduling algorithm under the multi-level feedback queue and PD separation architecture to further improve inference efficiency and reduce storage occupancy.

[0040] In some embodiments, a target pre-filling server node and a first priority are obtained by a scheduling thread based on user request information and load information of a plurality of pre-filling server nodes in combination with a deep reinforcement learning-based pre-filling scheduling model.

[0041] Step 102, a pre-filling request is sent to a request queue corresponding to the first priority in the target pre-filling server node, and the target pre-filling server node processes the pre-filling request.

[0042] In some embodiments, the pre-filling request is sent to the request queue corresponding to the first priority in the target pre-filling server node, that is, the pre-filling request is added to the request queue corresponding to the priority in the target pre-filling server node.

[0043] Therefore, the embodiment can improve inference efficiency by scheduling the request queue corresponding to the first priority in the target pre-filling server node.

[0044] In some embodiments, the implementation manner of processing the pre-filling request by the target pre-filling server node can include the following steps: traversing a plurality of request queues of the target pre-filling server node in order of priority from high to low by an inference thread; in response to the current request queue being empty, updating a request queue with a priority one level lower than the priority of the current request queue to the current request queue; and in response to the current request queue not being empty, executing the pre-filling request in the current request queue.

[0045] ​​In one example, for the plurality of request queues (also referred to as multi-level feedback queues) of each pre-filling server node, traversal is performed from the request queue with high priority (low quantum) to the request queue with low priority (high quantum); if the request queue corresponding to the priority is empty, the request queue with the next lower priority is selected; if the request queue corresponding to the priority is not empty, a batch of pre-filling requests at the head of the current request queue is executed according to the first-come-first-served principle of the request queue, such as pre-filling of one word unit of a batch of pre-filling requests. That is, the requests in the request queue with high priority are preferentially executed, and the execution time is quantum each time. In addition, specific rules can also be set to adjust the request queue where the request is located, such as triggering forced service when the request queue with low priority has not been served for a long time.

[0046] In the embodiments of the present application, the target pre-filling server node and the priority are selected by the scheduling thread, and the pre-filling request is processed by the inference thread, which can further improve the inference efficiency.

[0047] Step 103, in response to the completion of the pre-filling request processing, based on the user request information and the load information of the plurality of decoding server nodes, a target decoding server node and a second priority selected from the plurality of decoding server nodes are obtained by combining a decoding scheduling model based on deep reinforcement learning.

[0048] It can be understood that the pre-filling request processing is completed, that is, all word units contained in the user request information are calculated until the first new word unit is generated, and the pre-filling ends. The pre-filling server node will pass the calculated kv cache to the decoding server node to start decoding and continuously generate new word units. Therefore, after the pre-filling request processing is completed, a target decoding server node to be scheduled needs to be selected from the plurality of decoding server nodes. In the present embodiment, the target decoding server node is selected by a decoding scheduling model based on deep reinforcement learning (also referred to as a decoding scheduling model based on DRL).

[0049] In the present embodiment, based on the user request information and the load information of the plurality of decoding server nodes, the scheduling information in the decoding stage is obtained by combining a decoding scheduling model based on deep reinforcement learning, and the scheduling information in the decoding stage includes the selected target decoding server node and the second priority, so that the decoding request can be added to the request queue corresponding to the second priority in the target decoding server node.

[0050] ​In some embodiments, the scheduling thread obtains the scheduling information of the decoding stage based on the user request information and the load information of the plurality of decoding server nodes in combination with a decoding scheduling model based on deep reinforcement learning. That is, after the pre-filling request processing is completed, the decoding scheduling model based on DRL is used by the scheduling thread to select a decoding server node and a priority, and the KV cache generated by the pre-filling server node is transmitted to the selected decoding server node and added to the request queue corresponding to the selected priority, and the decoding is waited to be executed.

[0051] In step 104, the decoding request is sent to the request queue corresponding to the second priority in the target decoding server node, so that the target decoding server node processes the decoding request to obtain the model inference result, wherein the decoding request includes the key value cache generated by the target pre-filling server node.

[0052] In one example, for the plurality of request queues of each decoding server node, the request queues are traversed from the request queue of high priority (low quantum) to the request queue of low priority (high quantum); if the request queue corresponding to the priority is empty, the request queue of the current priority is exited and the request queue of the next lower priority is selected; if the request queue corresponding to the priority is not empty, one batch of decoding requests at the head of the request queue is executed according to the first-come-first-served principle of the request queue, such as decoding of one batch of pre-filling requests.

[0053] In some embodiments, when the decoding request is completed or reaches the maximum text length limit, the model inference result is returned to the user by the target decoding server node.

[0054] In the embodiments of the present application, the target decoding server node and the priority are selected by the scheduling thread, and the decoding request is processed by the inference thread, which can further improve the inference efficiency.

[0055] ​The model inference scheduling method of the embodiments of the present application proposes a large model inference scheduling framework based on PD separation and multi-level feedback queues, which improves inference efficiency and reduces memory occupation. Based on the framework, two scheduling algorithms based on deep reinforcement learning are used to select matching server nodes for user requests in the prefilling and decoding stages, respectively, which can improve the first word time delay and word interval of large model inference. The scheduling algorithm based on deep reinforcement learning makes full use of user request information, server node information and other information in the large-scale server cluster to overcome the construction limitations of heuristic methods in large-scale inference scheduling systems. By setting multi-level feedback queues with differentiated and partially overlapping queue priority parameters, the problem of IO-intensive decoding requests being delayed by prefilling requests and computationally intensive prefilling requests being frequently preempted by decoding requests can be alleviated, thereby further improving inference efficiency and reducing storage occupation based on the large model inference scheduling framework based on PD separation and multi-level feedback queues, combined with subsequent scheduling algorithms based on deep reinforcement learning. In addition, by processing the requests for server node selection in two threads, inference efficiency can be further improved and the service level target of the large model inference framework can be improved. The method can be directly used for scheduling user requests for large language models in a large-scale server cluster and deployed in a large model data center. It can also be applied to request scheduling of other types of large models after simple expansion.

[0056] Based on the above embodiments, referring to Figure 2 Embodiment two of the present application provides another model inference scheduling method, which includes the following steps: Step 201, in response to receiving user request information, based on the user request information, retrieving the historical key value cache to obtain shared prefix matching information between the user request information and each prefilling server node.

[0057] In this embodiment, each prefilling server node stores a historical key value cache, and the received user request information, i.e. the complete prompt word, is used to retrieve the matching shared prefix from the historical key value cache of each prefilling server node to obtain the shared prefix matching information between the user request information and each prefilling server node.

[0058] In some embodiments, the shared prefix matching information between the user request information and each prefilling server node includes the key value cache storage location of the shared prefix, the shared prefix length and the remaining unmatched length.

[0059] Step 202, based on the user request information and the shared prefix matching information between the user request information and each prefilling server node, combining an execution time prediction model to obtain an estimated execution time of each prefilling server node processing the user request information.

[0060] In some embodiments, the method of obtaining the execution estimation time of each pre-filled server node for processing the user request information based on the user request information and the shared prefix matching information between the user request information and each pre-filled server node, in combination with the execution time prediction model, can include the following steps: Step 2021, based on the user request information, obtain the token sequence length of the user request.

[0061] In this embodiment, based on the user request information, the token sequence length of the user request, i.e., the user request length, is obtained.

[0062] Step 2022, based on the token sequence length of the user request, the server node type, the key-value cache storage location of the shared prefix, the shared prefix length and the remaining unmatched length, in combination with the execution time prediction model, obtain the execution estimation time of each pre-filled server node for processing the user request information.

[0063] In some embodiments, the execution time prediction model is obtained by training a neural network, and the neural network includes 3 hidden layers, and the activation function of the neural network includes a LeakyRelu function.

[0064] In some embodiments, the execution time prediction model adopts a high-efficiency small neural network, including 3 hidden layers, with node numbers of [128, 64, 32], and the activation function uses LeakyRelu. The input of the execution time prediction model is defined to include the token sequence length of the user request, the key-value cache storage location (video memory or memory) of the shared prefix, the shared prefix length and the remaining unmatched length, and the server node type. Optionally, only the information of a number of pre-filled server nodes with the highest shared prefix matching length is taken, such as 3. The output of the execution time prediction model is defined to include the execution estimation time of each pre-filled server node for processing the user request information, i.e., the execution estimation time of each pre-filled server node for executing the current pre-filled request.

[0065] The execution time prediction model takes into account multiple factors, including the token sequence length of the user request, and will not cause subsequent short requests to suffer from higher delay due to processing long requests.

[0066] In some embodiments, the training method of the execution time prediction model includes: obtaining training data collected in an underloaded or unloaded environment; training a neural network based on the training data with a mean squared error (MSE) as a loss function to obtain the execution time prediction model.

[0067] Collecting training data in an underloaded or unloaded environment can exclude the interference of priority preemption and other factors, and obtain the execution time as a benchmark.

[0068] In step 203, based on the user request information, the load information of the plurality of pre-populated server nodes, and the execution estimation time of each pre-populated server node processing the user request information, a target pre-populated server node selected from the plurality of pre-populated server nodes and a first priority are obtained in combination with a pre-population scheduling model based on deep reinforcement learning.

[0069] In some embodiments, based on the user request information, the load information of the plurality of pre-populated server nodes, and the execution estimation time of each pre-populated server node processing the user request information, a target pre-populated server node selected from the plurality of pre-populated server nodes and a first priority are obtained in combination with a pre-population scheduling model based on deep reinforcement learning; the method comprises: Based on the token sequence length of the user request, the server node type, the key-value cache storage location of the shared prefix, the shared prefix length, the remaining unmatched length, the load information of the plurality of pre-populated server nodes, the execution estimation time of each pre-populated server node processing the user request information, and the request queue information of the plurality of pre-populated server nodes, a target pre-populated server node selected from the plurality of pre-populated server nodes and a first priority are obtained in combination with a pre-population scheduling model based on deep reinforcement learning.

[0070] In some embodiments, the pre-population scheduling model based on deep reinforcement learning comprises a policy network and a plurality of Q networks, the input information of the policy network comprises a state based on deep reinforcement learning, the state based on deep reinforcement learning is obtained based on the user request information, the load information of the plurality of pre-populated server nodes, the related data of the execution time prediction model, and the related information of the request queue of the pre-populated server node; the output information of the policy network comprises an action vector based on deep reinforcement learning, the action vector comprises the probability of selecting the request queue corresponding to each priority of the plurality of pre-populated server nodes.

[0071] The input information of the Q network is the same as the input information of the policy network, and the output information of the Q network is a Q vector corresponding to the action vector; the i-th element in the Q vector is the Q value of the request queue with priority z of the y-th pre-populated server node corresponding to the i-th element in the action vector; wherein, Here, " / / " represents the integer quotient of division; Here, "%" represents the remainder, that is, y is obtained by calculating the integer quotient of i and the number of request queues of the pre-populated server node, and z is obtained by calculating the remainder of i and the number of request queues of the pre-populated server node.

[0072] The input of the pre-filled scheduling model of deep reinforcement learning in this embodiment involves multiple factors such as user request information, load information of multiple pre-filled server nodes, relevant data of the execution time prediction model, and relevant information of the request queue of the pre-filled server nodes, so as to achieve optimal scheduling oriented towards service-level indicators.

[0073] Pre-filled rewards in a pre-filled scheduling model based on deep reinforcement learning Defined by the following formula:

[0074] in, The pre-fill execution time is obtained based on the execution time prediction model. TTFT stands for the first token delay, which is the time to complete the pre-filling process. It is fed back by the pre-filling server node after completing the pre-filling request.

[0075] In some embodiments, the execution time prediction model is used, with the remaining unmatched length in the input of the execution time prediction model set to the prompt word length, the server node type set to a P-strong server node, and the storage location of the KV cache set to memory, and the output... This serves as the baseline for rewards.

[0076] In one example, a pre-filled scheduling model based on deep reinforcement learning includes a policy network. and two Q networks and Furthermore, the two Q-networks have the same structure. Define the policy network. The input, i.e., the state of the DRL. This can include the inputs and outputs of the execution time prediction model, load information of multiple pre-filled server nodes, and request queue information. For example, request queue information includes: a quantum that is greater than and closest to the length of the user request (i.e., the length of the user's requested lexical sequence), and the total lexical length of all pre-filled requests in all request queues with a priority higher than or equal to that quantum. Define the output of the policy network: i.e., the action vector of the DRL. This action vector The mapping relationship between the selected pre-filled server nodes and priority x is as follows: First, the output layer at the very end of the policy network is a Sigmoid function, which outputs a dimension of... ( Action vector (for the number of pre-populated server nodes) Action vector Let be a probability vector whose sum of all elements is 1. The i-th element represents the probability that the y-th pre-filled server node with priority z is selected from the queue, where y = i / / 3; z = i % 3. Then, according to the probability... Sampling is performed, and the sampling result i can be converted into the selected pre-populated server node y and priority z. For example, If the value is 10, then the output will be an action vector with a dimension of 30. This includes the probability of selecting each of the 30 request queues from 10 pre-filled server nodes; the probability in this action vector is selected through a random mechanism. If the 10th element is selected, then y = 10 / / 3 = 3, z = 10%3 = 1. Define the Q-network. and Input: i.e., the state of the DRL , and policy network Same input. Define Q network. and Output: i.e., the action vector of the DRL The corresponding Q-vector, the first in the Q-vector Each element is an action vector. The The element corresponding to the first The priority of each pre-populated server node is The q-value of the queue in reinforcement learning, also known as the action value, represents the cumulative reward obtained by the user request information being served by the request queue with priority z of the y-th pre-filled server node.

[0077] In this embodiment, the q-value in reinforcement learning, also known as the action value, is the cumulative reward obtained by serving the user's request information through the request queue with priority z of the y-th pre-filled server node.

[0078] It should be noted that the Q-value in the transformer architecture is the query value, while the q-value here refers to the q-value in reinforcement learning, also known as the action value. It refers to the cumulative reward obtained by taking a certain action (i.e., selecting a queue on a server node to provide service to a user), i.e., the cumulative pre-filled reward. In some embodiments, the training method for a pre-filled scheduling model based on deep reinforcement learning includes: acquiring training data for the pre-filled scheduling model based on deep reinforcement learning; the training data includes pre-filled rewards; introducing stochastic policy entropy into the pre-filled rewards of the training data to obtain training data with stochastic policy entropy; and training a policy network and multiple Q-networks based on the training data with stochastic policy entropy to obtain the pre-filled scheduling model based on deep reinforcement learning.

[0079] The embodiment introduces random policy entropy, trains a scheduling model by using a SAC (Soft Actor-critic, a kind of reinforcement learning algorithm) algorithm, and thus effectively improves inference efficiency.

[0080] In some embodiments, the method for obtaining training data of the pre-filled scheduling model based on deep reinforcement learning can include the following steps: The following operations are repeatedly performed until a loop termination condition is met: Obtain relevant data of the pre-filling stage, and based on the relevant data of the pre-filling stage, obtain the state of the policy network at the current time; Based on the state of the policy network at the current time, obtain the action vector at the current time through the policy network, and process the user request information through the action vector at the current time to obtain the pre-filling reward at the current time; Based on the state at the current time, the action vector at the current time, the pre-filling reward at the current time, and the state at the next time, construct a first data combination, and take the first data combination as one training sample in the training data of the pre-filled scheduling model based on deep reinforcement learning.

[0081] The training data of the pre-filled scheduling model based on deep reinforcement learning is obtained through the above-mentioned loop process, which can improve the real-time performance of obtaining training data.

[0082] In one example, the training method of the pre-filled scheduling model based on deep reinforcement learning includes the following process: Initialize the large model inference scheduling system, the Q network of the pre-filling stage , the policy network , and the data buffer ; let the current time be , the state of the policy network be , the action vector be , the pre-filling reward be , the Q value be , the V value be , the policy network be , and the state at the next time be .

[0083] Data collection: based on the scheduling thread and the inference thread described in the above embodiments, the pre-filled user request information is continuously processed to collect data. Taking the pre-filling stage as an example, the state is extracted from the user request information, the execution time prediction model, and the pre-filling server cluster, and the action vector is obtained by using the policy network . Among them, The obtaining method includes: the action vector is a one-dimensional The probability vector; according to the probability vector Sampling, to obtain its first One element; calculate , Then choose the first one. The priority of each pre-populated server node is The request queue.

[0084] Based on action vectors Process user request information, obtain the corresponding first-term delay (TTFT) or term interval (TBT), and calculate the pre-filling reward. The state is then updated to the state of the next moment. ,Will Store in data cache .

[0085] From data cache A batch of samples were taken from the middle. Train the pre-filled scheduling model according to the following procedure until the model converges; then, only the policy network can be used. Perform scheduling: To prefill rewards We introduce stochastic policy entropy to construct an optimization objective function:

[0086] in, It is the entropy function. This represents the strategy entropy coefficient.

[0087] Optimize using the following process (i.e., the SAC algorithm): Calculate the Q value:

[0088] in, This is the discount factor.

[0089] .

[0090] Computing Q-networks and loss function (i.e., minimum mean square error, MSE):

[0091] in:

[0092] Based on the following gradient Q-network trained using gradient descent method and :

[0093] Updating the Q network using soft synchronization method and ( ) :

[0094] Based on the above calculated , the optimization objective function constructed above can be converted to:

[0095] Based on the value gradient of the following , the strategy network is trained using the gradient descent method :

[0096] Through the above process, the DRL-based pre-filling scheduling model is continuously trained, and the service level is continuously improved.

[0097] Step 204, send a pre-filling request to the request queue corresponding to the first priority in the target pre-filling server node, and let the target pre-filling server node process the pre-filling request.

[0098] It should be noted that the implementation of this step can refer to step 102 in the above embodiments for details, and the principle is the same, which will not be repeated here.

[0099] Step 205, in response to the completion of the pre-filling request processing, based on the user request information and the load information of the plurality of decoding server nodes, and in combination with the decoding scheduling model based on deep reinforcement learning, the target decoding server node selected from the plurality of decoding server nodes and the second priority are obtained.

[0100] In some embodiments, the structure of the decoding scheduling model based on deep reinforcement learning is the same as that of the pre-filling scheduling model based on deep reinforcement learning, and the decoding reward of the decoding scheduling model based on deep reinforcement learning is defined by the following formula:

[0101] Wherein, is the time between tokens, is the length of the new token generated by the decoding server node, which can be fed back by the decoding server node after completing the decoding request.

[0102] In one example, the deep reinforcement learning-based decoding scheduling model includes a policy network. 2 Q networks and Policy Network Input status It is obtained based on user request information, load information of multiple decoding server nodes, relevant data from the execution time prediction model, and relevant information about the request queues of the decoding server nodes. This information may include: the input and output of the execution time prediction model, load information of multiple decoding server nodes, and request queue information. For example, request queue information may include: a quantum that is greater than and closest to the length of the user request information (i.e., the length of the user's requested lexical sequence), and a priority higher than or equal to the total lexical length of all decoding requests in all request queues containing that quantum. (Policy Network) The output action vector The mapping relationship between the selected decoding server node and priority x is as follows: First, the output layer at the very end of the policy network is a sigmoid function, which outputs a dimension of... ( Action vector (number of decoding server nodes) Action vector Let be a probability vector whose sum of all elements is 1. The i-th element represents the probability that the queue with priority z of the y-th decoding server node is selected, where y = i / / 3; z = i % 3. Then, according to the probability... Sampling is performed, and the sampling result i can be converted into the selected decoding server node y and priority z.

[0103] In this embodiment, the training method for the deep reinforcement learning-based decoding scheduling model is the same as the training method for the deep reinforcement learning-based pre-filling scheduling model, and will not be described again. Specifically, whether to train the decoding scheduling model or the pre-filling scheduling model depends on the stage of the user request information being processed by the scheduling thread. If it is in the pre-filling stage, training data from that stage is collected to train the pre-filling scheduling model; if it is in the decoding stage, training data from that stage is collected to train the decoding scheduling model.

[0104] Step 206: Send a decoding request to the request queue corresponding to the second priority in the target decoding server node, so that the target decoding server node can process the decoding request and obtain the model inference result. The decoding request includes the key-value cache generated by the target pre-filled server node.

[0105] It should be noted that the specific implementation method of this step can be found in step 104 of the above embodiment, and the principle is the same, so it will not be repeated here.

[0106] The model inference scheduling method of the embodiment of the present application, on the basis of the above-mentioned embodiment, estimates the pre-filled execution time of the request by executing the time prediction model, provides decision state support for the subsequent deep reinforcement learning scheduling algorithm, and can also be used as the baseline of the deep reinforcement learning reward, thereby improving the learning efficiency and scheduling performance, and thus improving the inference efficiency. In the scheduling model based on deep reinforcement learning, the scheduling model is trained by introducing the random policy entropy and using the soft actor critic algorithm, thereby effectively improving the inference efficiency.

[0107] In order to clearly illustrate the above-mentioned embodiments, specific examples will be described. Figure 3 The framework diagram of the model inference scheduling system based on DRL provided by the embodiment of the present application is shown in the figure. The pre-filled and decoding two-stage pre-filled scheduling model based on DRL and the decoding scheduling model based on DRL are both represented by the scheduling model based on DRL, as shown in Figure 3 The execution time prediction model predicts the time according to the user request information, the server cluster related information, the shared prefix matching information, etc., and transmits the prediction result to the scheduling model based on DRL. The scheduling model based on DRL selects the server node for the pre-filled request or the decoding request according to the user request information, the server cluster related information, and the execution time prediction model related information, etc. The server cluster calls the selected server node to process the user request, and the decoding server node returns the inference result to the user. In addition, the server cluster returns the pre-filled and decoding reward for serving the user request to the scheduling model based on DRL. The scheduling model optimizes the model by using the SAC algorithm based on the above data. The above process is repeatedly performed to train the scheduling model based on DRL, and to continue to provide services for the user.

[0108] It should be noted that the embodiments of the present disclosure can include multiple steps, and in order to facilitate description, these steps are numbered, but these numbers are not a limitation on the execution time slot and execution order between the steps; these steps can be implemented in any order, and the embodiments of the present disclosure do not limit this.

[0109] Corresponding to the above-mentioned model inference scheduling method, the present disclosure also proposes a model inference scheduling system. As shown in Figure 4 The model inference scheduling system is configured to be applied to the model inference scheduling system. The server cluster includes a plurality of pre-filled server nodes and a plurality of decoding server nodes. The pre-filled server nodes and the decoding server nodes are both configured with a multi-level feedback queue when the model inference scheduling system is started. The priority of the plurality of request queues included in the multi-level feedback queue is different. The system can include: The first node selection module 411 is configured to, in response to receiving the user request information, based on the user request information and load information of a plurality of pre-filling server nodes, in combination with a pre-filling scheduling model based on deep reinforcement learning, acquire a target pre-filling server node selected from the plurality of pre-filling server nodes and a first priority; The first scheduling module 412 is configured to send a pre-filling request to a request queue corresponding to the first priority in the target pre-filling server node, so that the target pre-filling server node processes the pre-filling request; The second node selection module 413 is configured to, in response to completion of processing of the pre-filling request, based on the user request information and load information of a plurality of decoding server nodes, in combination with a decoding scheduling model based on deep reinforcement learning, acquire a target decoding server node selected from the plurality of decoding server nodes and a second priority; The second scheduling module 414 is configured to send a decoding request to a request queue corresponding to the second priority in the target decoding server node, so that the target decoding server node processes the decoding request to obtain a model inference result, wherein the decoding request includes a key-value cache generated by the target pre-filling server node.

[0110] Further, in a possible implementation manner of the embodiment of the present disclosure, the priority level of the request queue is determined by a queue priority parameter, and the queue priority parameter is used to indicate the execution time length of the task in the request queue.

[0111] Further, in a possible implementation manner of the embodiment of the present disclosure, as shown in Figure 5 The system further includes a node type division module 415 configured to: According to configuration information of the plurality of pre-filling server nodes, the plurality of pre-filling server nodes are divided into two types of server nodes, i.e., strong pre-filling server nodes and weak pre-filling server nodes; the plurality of queue priority parameters of the strong pre-filling server nodes and the plurality of queue priority parameters of the weak pre-filling server nodes have the same part and different parts; According to configuration information of the plurality of decoding server nodes, the plurality of decoding server nodes are divided into two types of server nodes, i.e., strong decoding server nodes and weak decoding server nodes; the plurality of queue priority parameters of the strong decoding server nodes and the plurality of queue priority parameters of the weak decoding server nodes have the same part and different parts.

[0112] Further, in a possible implementation manner of the embodiment of the present disclosure, the pre-filling server node stores historical key-value caches, and the first node selection module 411 is specifically configured to: Based on the user request information, the historical key-value caches are searched to obtain shared prefix matching information between the user request information and each pre-filling server node; based on the user request information and the shared prefix matching information between the user request information and each pre-population server node, in combination with an execution time prediction model, obtain execution estimation time of each pre-population server node for processing the user request information; based on the user request information, the load information of the plurality of pre-population server nodes, and the execution estimation time of each pre-population server node for processing the user request information, in combination with a pre-population scheduling model based on deep reinforcement learning, obtain a target pre-population server node selected from the plurality of pre-population server nodes and a first priority.

[0113] Further, in a possible implementation manner of the embodiment of the present disclosure, the shared prefix matching information between the user request information and each pre-population server node includes a key-value cache storage location of the shared prefix, a shared prefix length, and a remaining unmatched length; When the first node selection module 411 obtains, based on the user request information and the shared prefix matching information between the user request information and each pre-population server node, in combination with an execution time prediction model, execution estimation time of each pre-population server node for processing the user request information, the first node selection module 411 is specifically used for: based on the user request information, obtain a token sequence length of the user request; based on the token sequence length of the user request, the server node type, the key-value cache storage location of the shared prefix, the shared prefix length, and the remaining unmatched length, in combination with an execution time prediction model, obtain execution estimation time of each pre-population server node for processing the user request information.

[0114] Further, in a possible implementation manner of the embodiment of the present disclosure, the first node selection module 411 is specifically used for: based on the user request information and the load information of the plurality of pre-population server nodes, in combination with a pre-population scheduling model based on deep reinforcement learning, obtain, by a scheduling thread, a target pre-population server node selected from the plurality of pre-population server nodes and a first priority.

[0115] Further, in a possible implementation manner of the embodiment of the present disclosure, when the first scheduling module 412 causes the target pre-population server node to process the pre-population request, the first scheduling module 412 is specifically used for: traverse, by an inference thread, a plurality of request queues of the target pre-population server node in a descending order of priority; in response to the current request queue being empty, update a request queue with a priority one level lower than that of the current request queue to the current request queue; in response to the current request queue being not empty, execute a pre-population request in the current request queue.

[0116] Further, in a possible implementation manner of the embodiment of the present disclosure, as shown inFigure 5 As shown, the system further includes a first model training module 416 configured to: obtain training data collected in an underloaded or unloaded environment; train a neural network based on the training data to obtain an execution time prediction model, wherein the neural network includes three hidden layers, and an activation function of the neural network includes a LeakyRelu function.

[0117] Further, in a possible implementation of the embodiments of the present disclosure, when the first node selection module 411 is configured to obtain a target pre-populated server node and a first priority from a plurality of pre-populated server nodes based on user request information, load information of the plurality of pre-populated server nodes, and execution estimation time of each pre-populated server node processing the user request information, in combination with a pre-populated scheduling model based on deep reinforcement learning, the method further includes: obtaining a target pre-populated server node and a first priority from a plurality of pre-populated server nodes based on a token sequence length of the user request, a server node type, a shared prefix key-value cache storage location, a shared prefix length, a remaining unmatched length, load information of the plurality of pre-populated server nodes, execution estimation time of each pre-populated server node processing the user request information, and request queue information of the plurality of pre-populated server nodes, in combination with the pre-populated scheduling model based on deep reinforcement learning.

[0118] Further, in a possible implementation of the embodiments of the present disclosure, the pre-populated scheduling model based on deep reinforcement learning includes a policy network and a plurality of Q networks, input information of the policy network includes a state based on deep reinforcement learning, the state based on deep reinforcement learning is obtained based on user request information, load information of a plurality of pre-populated server nodes, related data of an execution time prediction model, and related information of a request queue of a pre-populated server node; output information of the policy network includes an action vector based on deep reinforcement learning, the action vector includes a probability of a request queue corresponding to each priority of the plurality of pre-populated server nodes being selected; input information of the Q networks is the same as input information of the policy network, and output information of the Q networks is a Q vector corresponding to the action vector; an i-th element in the Q vector is an action value of a request queue with a priority of z of an i-th pre-populated server node corresponding to an i-th element in the action vector in reinforcement learning, the action value is used to represent a cumulative reward obtained by the user request information being served by the request queue with the priority of z of the i-th pre-populated server node, wherein y is obtained by calculating an integer quotient of i and a number of request queues of the pre-populated server nodes, and z is obtained by calculating a remainder of i and the number of request queues of the pre-populated server nodes; a pre-populated reward of the pre-populated scheduling model based on deep reinforcement learning is defined by the following first formula:

[0119] wherein, is a pre-padding execution time, is obtained based on an execution time prediction model, TTFT is a first word element time delay, used to indicate a time for completing pre-padding, and is fed back by a pre-padding server node after completing a pre-padding request.

[0120] Further, in a possible implementation of the embodiment of the present disclosure, the structure of the decoding scheduling model based on deep reinforcement learning is the same as that of the pre-padding scheduling model based on deep reinforcement learning, and the decoding reward of the decoding scheduling model based on deep reinforcement learning is is defined by the following second formula:

[0121] wherein, is a word element interval, is a new word element length generated by a decoding server node, and can be fed back by the decoding server node after completing a decoding request.

[0122] Further, in a possible implementation of the embodiment of the present disclosure, the system further includes a second model training module 417, configured to: obtain training data of the pre-padding scheduling model based on deep reinforcement learning; the training data includes a pre-padding reward; introduce random policy entropy into the pre-padding reward of the training data to obtain training data with introduced random policy entropy; train the policy network and the multiple Q networks based on the training data with introduced random policy entropy to obtain the pre-padding scheduling model based on deep reinforcement learning.

[0123] Further, in a possible implementation of the embodiment of the present disclosure, as shown in Figure 5 the second model training module 417, when obtaining the training data of the pre-padding scheduling model based on deep reinforcement learning, is specifically configured to: perform the following operations in a loop until a loop termination condition is met; obtain related data of a pre-padding stage, and obtain a state of the policy network at a current time based on the related data of the pre-padding stage; obtain an action vector at the current time through the policy network based on the state of the policy network at the current time, and process user request information through the action vector at the current time to obtain a pre-padding reward at the current time; Based on the state at the current moment, the action vector at the current moment, the pre-padding reward at the current moment, and the state at the next moment, a first data combination is constructed, and the first data combination is taken as one training sample in training data of the pre-padding scheduling model based on deep reinforcement learning. Further, in a possible implementation of the embodiment of the present disclosure, the three queue priority parameters of the strong server node are respectively , and , the three queue priority parameters of the weak server node are respectively , 2 and 4 , wherein is the number of word pieces included in the pre-padding granularity or the decoding granularity of the model inference method.

[0124] It should be noted that the foregoing explanation and description of the method embodiment also apply to the device of the present embodiment, and the principle is the same. In the present embodiment, the device is not limited.

[0125] The description of the features in the embodiment corresponding to the model inference scheduling system can be referred to the related description of the embodiment corresponding to the model inference scheduling method, which will not be repeated here.

[0126] Embodiments of the present application also provide an electronic device, comprising a memory and a processor, the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above model inference scheduling method embodiments.

[0127] Embodiments of the present application also provide a computer readable storage medium, which stores a computer program, wherein the computer program is configured to perform the steps in any of the above model inference scheduling method embodiments when running.

[0128] In an example embodiment, the above computer readable storage medium can include, but is not limited to, a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store computer programs.

[0129] Embodiments of the present application also provide a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the steps in any of the above model inference scheduling method embodiments.

[0130] The embodiment of the present application further provides another computer program product comprising a nonvolatile computer readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the model inference scheduling method embodiments.

[0131] Those skilled in the art can further understand that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been described in the above description in a general manner. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0132] The above describes in detail a model inference scheduling method and system, an electronic device and a storage medium provided by the present application. The principles and implementation modes of the present application are described herein by applying specific examples, and the above description of the examples is only used to help understand the method of the present application and its core idea. It should be pointed out that, for those skilled in the art, without departing from the principles of the present application, some improvements and modifications can be made to the present application, and these improvements and modifications also fall within the protection scope of the claims of the present application.

Claims

1. A model inference scheduling method, characterized in that, The method is applied to a model inference scheduling system, the model inference scheduling system comprises a plurality of pre-filling server nodes and a plurality of decoding server nodes, the pre-filling server nodes and the decoding server nodes are configured with a multi-level feedback queue at the start of the model inference scheduling system, a plurality of request queues included in the multi-level feedback queue have different priorities, and the method comprises the following steps: In response to receiving user request information, based on the user request information and load information of the plurality of pre-filling server nodes, a target pre-filling server node selected from the plurality of pre-filling server nodes and a first priority are obtained by combining a pre-filling scheduling model based on deep reinforcement learning; A pre-filling request is sent to a request queue corresponding to the first priority in the target pre-filling server node, and the target pre-filling server node processes the pre-filling request; In response to the completion of the pre-filling request processing, based on the user request information and load information of the plurality of decoding server nodes, a target decoding server node selected from the plurality of decoding server nodes and a second priority are obtained by combining a decoding scheduling model based on deep reinforcement learning; A decoding request is sent to a request queue corresponding to the second priority in the target decoding server node, and the target decoding server node processes the decoding request to obtain a model inference result, wherein the decoding request comprises a key-value cache generated by the target pre-filling server node.

2. The model inference scheduling method of claim 1, wherein, The priority level of the request queue is determined by a queue priority parameter, and the queue priority parameter is used to indicate the execution time of the task in the request queue, The method further comprises: According to the configuration information of the plurality of pre-filling server nodes, the plurality of pre-filling server nodes are divided into two types of server nodes: strong pre-filling server nodes and weak pre-filling server nodes; the plurality of queue priority parameters of the strong pre-filling server nodes and the plurality of queue priority parameters of the weak pre-filling server nodes have the same part and the different part; According to the configuration information of the plurality of decoding server nodes, the plurality of decoding server nodes are divided into two types of server nodes: strong decoding server nodes and weak decoding server nodes; the plurality of queue priority parameters of the strong decoding server nodes and the plurality of queue priority parameters of the weak decoding server nodes have the same part and the different part.

3. The model inference scheduling method of claim 2, wherein, The pre-filling server node stores a historical key-value cache, Based on the user request information and the load information of the plurality of pre-filling server nodes, a target pre-filling server node selected from the plurality of pre-filling server nodes and a first priority are obtained by combining a pre-filling scheduling model based on deep reinforcement learning; comprising: Based on the user request information, the historical key-value cache is searched to obtain shared prefix matching information between the user request information and each pre-filling server node; based on the user request information and the shared prefix matching information between the user request information and each of the pre-populated server nodes, and in combination with an execution time prediction model, obtain an execution estimation time of each of the pre-populated server nodes for processing the user request information; based on the user request information, load information of the plurality of pre-populated server nodes, and the execution estimation time of each of the pre-populated server nodes for processing the user request information, and in combination with the pre-populated scheduling model based on deep reinforcement learning, obtain a target pre-populated server node selected from the plurality of pre-populated server nodes and a first priority.

4. The model inference scheduling method of claim 3, wherein, The shared prefix matching information between the user request information and each of the pre-populated server nodes includes a key-value cache storage location of the shared prefix, a shared prefix length, and a remaining unmatched length. The execution estimation time of each of the pre-populated server nodes for processing the user request information is obtained based on the user request information and the shared prefix matching information between the user request information and each of the pre-populated server nodes, and in combination with an execution time prediction model. Based on the user request information, obtain the token sequence length of the user request; Based on the token sequence length of the user request, the server node type, the key-value cache storage location of the shared prefix, the shared prefix length, and the remaining unmatched length, and in combination with the execution time prediction model, obtain the execution estimation time of each of the pre-populated server nodes for processing the user request information.

5. The model inference scheduling method according to claim 1 or 3, characterized in that, The target pre-populated server node selected from the plurality of pre-populated server nodes and the first priority are obtained based on the user request information and load information of the plurality of pre-populated server nodes, and in combination with a pre-populated scheduling model based on deep reinforcement learning. It includes: Through a scheduling thread, the target pre-populated server node selected from the plurality of pre-populated server nodes and the first priority are obtained based on the user request information and load information of the plurality of pre-populated server nodes, and in combination with a pre-populated scheduling model based on deep reinforcement learning.

6. The model inference scheduling method of claim 2, wherein, The target pre-populated server node is caused to process the pre-populated request; it includes: Through an inference thread, in order of the priority from high to low, traverse a plurality of request queues of the target pre-populated server node; In response to the current request queue being empty, update a request queue with a priority one level lower than that of the current request queue to the current request queue; In response to the current request queue not being empty, execute a pre-populated request in the current request queue.

7. The model inference scheduling method of claim 3 or 4, wherein, The training method of the execution time prediction model includes: Obtain training data collected in an underloaded or unloaded environment; Based on the training data, train a neural network to obtain the execution time prediction model, wherein the neural network includes three hidden layers, and the activation function of the neural network includes a LeakyRelu function.

8. The model inference scheduling method of claim 4, wherein, The target pre-populated server node and the first priority are obtained from the plurality of pre-populated server nodes based on the user request information, the load information of the plurality of pre-populated server nodes, and the execution estimation time of each of the pre-populated server nodes processing the user request information, in combination with the pre-populated scheduling model based on deep reinforcement learning. The target pre-populated server node and the first priority are obtained from the plurality of pre-populated server nodes based on the token sequence length of the user request, the server node type, the key-value cache storage location of the shared prefix, the shared prefix length, the remaining unmatched length, the load information of the plurality of pre-populated server nodes, the execution estimation time of each of the pre-populated server nodes processing the user request information, and the request queue information of the plurality of pre-populated server nodes, in combination with the pre-populated scheduling model based on deep reinforcement learning.

9. The model inference scheduling method of claim 3, wherein, The pre-populated scheduling model based on deep reinforcement learning includes a policy network and a plurality of Q networks, the input information of the policy network includes a deep reinforcement learning-based state, the deep reinforcement learning-based state is obtained based on the user request information, the load information of the plurality of pre-populated server nodes, the related data of the execution time prediction model, and the related information of the request queue of the pre-populated server node; the output information of the policy network includes a deep reinforcement learning-based action vector, the action vector includes the probability of selecting the request queue corresponding to each priority of the plurality of pre-populated server nodes; The input information of the Q network is the same as the input information of the policy network, and the output information of the Q network is a Q vector corresponding to the action vector; the i-th element in the Q vector is the action value of the request queue with priority z of the y-th pre-populated server node corresponding to the i-th element in the action vector in reinforcement learning, the action value is used to represent the cumulative reward obtained by the user request information being served by the request queue with priority z of the y-th pre-populated server node, wherein y is obtained by calculating the integer quotient of i and the number of request queues of the pre-populated server node, and z is obtained by calculating the remainder of i and the number of request queues of the pre-populated server node; The prefill reward of the prefill scheduling model based on deep reinforcement learning defined by the following first equation: wherein, is a pre-filling execution time, which is obtained based on the execution time prediction model, TTFT is a first word token time delay, used to indicate the time for completing the pre-filling, and is fed back by the pre-filling server node after completing the pre-filling request.

10. The model inference scheduling method of claim 8, wherein, The structure of the deep reinforcement learning based decoding scheduling model is the same as that of the deep reinforcement learning based pre-filling scheduling model, and the decoding reward of the deep reinforcement learning based decoding scheduling model is defined by a second formula as follows: defined by a second formula as follows: wherein, is a word interval, is a new word length generated by the decoding server node, which can be fed back by the decoding server node after completing the decoding request.

11. The model inference scheduling method of claim 9, wherein, The training method of the pre-populated scheduling model based on deep reinforcement learning includes: Obtaining training data of the pre-populated scheduling model based on deep reinforcement learning; the training data includes the pre-populated reward; Introducing random policy entropy in the pre-populated reward of the training data to obtain training data with random policy entropy introduced; Training the policy network and the plurality of Q networks based on the training data with random policy entropy introduced to obtain the pre-populated scheduling model based on deep reinforcement learning.

12. The model inference scheduling method of claim 11, wherein, The training data of the pre-populated scheduling model based on deep reinforcement learning includes: The following operations are performed in a loop until the loop termination condition is met; obtain relevant data of a pre-filling stage, obtain a state of the policy network at a current time based on the relevant data of the pre-filling stage; obtain an action vector at the current time through the policy network based on the state of the policy network at the current time, and process the user request information through the action vector at the current time to obtain a pre-filling reward at the current time; construct a first data combination based on the state at the current time, the action vector at the current time, the pre-filling reward at the current time, and a state at a next time, and use the first data combination as one training sample in training data of the pre-filling scheduling model based on deep reinforcement learning.

13. An electronic device, comprising: comprise: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the model inference scheduling method of any one of claims 1-12.

14. A non-transitory computer-readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to enable the computer to perform the model inference scheduling method of any one of claims 1-12.

15. A computer program product, characterised in that, comprise a computer program that, when executed by a processor, implements the method of any one of claims 1-12.

Images