A request scheduling method and apparatus

Abstract

This invention discloses a request scheduling method and apparatus, relating to the field of large language model technology. A specific embodiment of the invention includes: receiving text processing requests or multimodal processing requests; acquiring key performance indicators of the large language model system in real time; selecting a target text scheduling strategy from multiple text scheduling strategies for text processing requests based on the key performance indicators; selecting a target multimodal scheduling strategy from multiple multimodal scheduling strategies for multimodal processing requests; and then, based on the target text scheduling strategy or the target multimodal scheduling strategy, invoking the large language model to perform inference operations for the processing request. This invention, by constructing separate encoding, pre-filling, and decoding structures and dynamically scheduling instances according to system performance indicators, improves the resource scheduling flexibility of the large language model system, enhances system resource utilization and large language model processing performance, and improves user experience.

Description

Technical Field

[0001] This invention relates to the field of large model technology, and in particular to a request scheduling method and apparatus. Background Technology

[0002] Currently, systems running large language models have been widely applied in various business systems due to the strong reasoning capabilities of large language models.

[0003] In existing large language model systems, text requests are handled by scheduling pre-filling and decoding tasks using static resource configuration, while multimodal requests consisting of text and images are typically executed in a serial manner.

[0004] Existing methods for request scheduling lack dynamic scheduling, resulting in poor flexibility and low performance in existing large language model systems. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a request scheduling method and apparatus applied to a large language model system. This method can determine whether a request type is a text processing request or a multimodal processing request; acquire key performance indicators of the large language model system in real time; for text processing requests, select a target text scheduling strategy from multiple text scheduling strategies based on the key performance indicators; for multimodal processing requests, select a target multimodal scheduling strategy from multiple multimodal scheduling strategies based on the key performance indicators; and, based on the target text scheduling strategy or target multimodal scheduling strategy, combined with the key performance indicators, invoke the large language model to perform inference operations for the processing request. By constructing separate encoding, pre-filling, and decoding structures and dynamically scheduling instances based on system performance indicators, embodiments of the present invention improve the resource scheduling flexibility of the large language model system, enhance system resource utilization and large language model processing performance, and improve user experience.

[0006] To achieve the above objectives, according to one aspect of the present invention, a request scheduling method is provided, applied to a large language model system, comprising: receiving a processing request; determining that the request type of the processing request is a text processing request or a multimodal processing request; acquiring key performance indicators of the large language model system in real time; for the text processing request, selecting a target text scheduling strategy from a variety of text scheduling strategies based on the key performance indicators; for the multimodal processing request, selecting a target multimodal scheduling strategy from a variety of multimodal scheduling strategies based on the key performance indicators; and, based on the target text scheduling strategy or the target multimodal scheduling strategy, combined with the key performance indicators, invoking the large language model to perform inference operations for the processing request.

[0007] Optionally, the step of selecting a target text scheduling strategy for processing the text processing request from multiple text scheduling strategies based on the key performance indicators includes: if the performance of the large language model is predicted to fail to meet the preset service level target based on the key performance indicators, selecting a pre-filling-decoding separation scheduling strategy as the target text scheduling strategy for processing the text processing request from multiple text scheduling strategies.

[0008] Optionally, the step of selecting a target multimodal scheduling strategy for processing the multimodal processing request from multiple multimodal scheduling strategies based on the key performance indicators includes: selecting a pre-filling-encoding-decoding separation scheduling strategy as the target multimodal scheduling strategy for processing the multimodal processing request from multiple multimodal scheduling strategies based on the key performance indicators.

[0009] Optionally, a pre-filling-decoding separation scheduling strategy is constructed; the construction of the pre-filling-decoding separation scheduling strategy includes: constructing multiple elastic instance pools; using a first elastic instance pool to store instances that process pre-filling requests; using a second elastic instance pool to store instances that process decoding requests; using a third elastic instance pool to store instances that are scheduled to process decoding requests and are also processing pre-filling requests; and using a fourth elastic instance pool to store instances that are scheduled to process pre-filling requests and are also processing decoding requests; wherein each of the instances is stateless and can process either pre-filling requests or decoding requests.

[0010] Optionally, the step of invoking the large language model to perform inference operations for the processing request based on the target text scheduling strategy and in conjunction with the key performance indicators includes: for the pre-filling stage, if it is determined according to the key performance indicators that the instance executing the pre-filling request cannot meet the first service level target, triggering the conversion of the instance executing the decoding request into the instance executing the pre-filling request; for the decoding stage, if it is determined according to the key performance indicators that the instance executing the decoding request cannot meet the second service level target, triggering the conversion of the instance executing the pre-filling request into the instance executing the decoding request.

[0011] Optionally, triggering the conversion of the instance processing the decoding request to the instance processing the pre-filling request includes: reducing the number of instances in the second elastic instance pool and increasing the number of instances in the fourth elastic instance pool; after the instance completes the decoding request task, reducing the number of instances in the fourth elastic instance pool and increasing the number of instances in the first elastic instance pool; keeping the number of instances in the third elastic instance pool unchanged; and / or, triggering the conversion of the instance processing the pre-filling request to the instance processing the decoding request includes: reducing the number of instances in the first elastic instance pool and increasing the number of instances in the third elastic instance pool; after the instance completes the pre-filling request task, slowing down the number of instances in the third elastic instance pool; increasing the number of instances in the second elastic instance pool; and keeping the number of instances in the fourth elastic instance pool unchanged.

[0012] Optionally, the step of invoking the large language model to perform inference operations for the processing request based on the target text scheduling strategy and the key performance indicators further includes: for the pre-filling stage, evaluating the queuing latency of each instance in the first elastic instance pool that processes the pre-filling request; selecting the candidate instance with the shortest queuing latency to execute the pre-filling request based on the queuing latency; and for the decoding stage, selecting a candidate instance from the first elastic instance pool that has completed the pre-filling task to continue executing the decoding request.

[0013] Optionally, the pre-filling-decoding-encoding separation scheduling strategy is constructed; the construction of the pre-filling-decoding-encoding separation scheduling strategy includes: dividing the execution flow of the multimodal processing request into different stages corresponding to encoding requests, pre-filling requests, and decoding requests; using the instance to process one of the encoding requests, pre-filling requests, and decoding requests; evaluating the batch number of images processed for image encoding and the batch number of words processed for text pre-filling and decoding according to the key performance indicators; and determining the processing priority order of encoding requests, pre-filling requests, and decoding requests in each iteration.

[0014] Optionally, determining the processing priority order of encoding requests, pre-padding requests, and decoding requests includes: obtaining the decoding task corresponding to each decoding request and determining the decoding task as the highest priority; obtaining the pre-padding task that is currently partially executing a pre-padding request and determining the pre-padding task as the second highest priority; obtaining the encoding task corresponding to the encoding request for the image and determining the encoding task as a normal priority.

[0015] Optionally, the hybrid expert architecture corresponding to the large language model is divided into an attention network and an expert network; if it is determined that any instance has an abnormality and needs to be recovered, the attention network parameters are obtained from the normally running instance and loaded to restore the instance service; at the same time, the key parameters of the expert network are asynchronously obtained and stored in the expert pool so as to load the expert network based on the expert pool in the future; and / or, if it is determined that the decoding request processed by the large language model has an abnormality, the key-value pair cache of the decoding request is obtained; according to the key performance indicators, a recalculation operation or a backup operation of the key-value pair cache is triggered to switch the instance executing the decoding request.

[0016] To achieve the above objectives, according to a second aspect of the present invention, a request scheduling apparatus is provided, comprising: a request receiving module, an indicator acquisition module, a strategy determination module, and an inference module; wherein, The request receiving module is used to receive a processing request and determine whether the request type of the processing request is a text processing request or a multimodal processing request. The indicator acquisition module is used to predict the key performance indicators of the large language model system. The strategy determination module is used to select a target text scheduling strategy for processing the text processing request from a variety of text scheduling strategies based on the key performance indicators, and to select a target multimodal scheduling strategy for processing the multimodal processing request from a variety of multimodal scheduling strategies based on the key performance indicators. The inference module is used to invoke the large language model to perform inference operations for the processing request based on the target text scheduling strategy or the target multimodal scheduling strategy and in conjunction with the key performance indicators.

[0017] Optionally, the request scheduling device is configured to select a target text scheduling strategy for processing the text processing request from multiple text scheduling strategies based on the key performance indicators, including: if the performance of the large language model is predicted to fail to meet the preset service level target based on the key performance indicators, selecting a pre-filling-decoding separation scheduling strategy as the target text scheduling strategy for processing the text processing request from multiple text scheduling strategies.

[0018] Optionally, the request scheduling device is configured to select a target multimodal scheduling strategy for processing the multimodal processing request from a variety of multimodal scheduling strategies based on the key performance indicators, including: selecting a pre-filling-encoding-decoding separation scheduling strategy as the target multimodal scheduling strategy for processing the multimodal processing request from a variety of multimodal scheduling strategies based on the key performance indicators.

[0019] Optionally, a scheduling device is requested to construct a pre-filling-decoding separation scheduling strategy; the construction of the pre-filling-decoding separation scheduling strategy includes: constructing multiple elastic instance pools; using a first elastic instance pool to store instances that process pre-filling requests; using a second elastic instance pool to store instances that process decoding requests; using a third elastic instance pool to store instances that are scheduled to process decoding requests and are also processing pre-filling requests; and using a fourth elastic instance pool to store instances that are scheduled to process pre-filling requests and are also processing decoding requests; wherein each of the instances is stateless and can process either pre-filling requests or decoding requests.

[0020] Optionally, a request scheduling device is configured to, based on the target text scheduling strategy and in conjunction with the key performance indicators, invoke the large language model to perform inference operations for the processing request, including: for the pre-filling stage, if it is determined according to the key performance indicators that the instance executing the pre-filling request cannot meet the first service level target, triggering the conversion of the instance executing the decoding request into the instance executing the pre-filling request; for the decoding stage, if it is determined according to the key performance indicators that the instance executing the decoding request cannot meet the second service level target, triggering the conversion of the instance executing the pre-filling request into the instance executing the decoding request.

[0021] Optionally, the request scheduling device, configured to trigger the conversion of the instance processing the decoding request into the instance processing the pre-filling request, includes: reducing the number of instances in the second elastic instance pool and increasing the number of instances in the fourth elastic instance pool; after the instance completes the decoding request task, reducing the number of instances in the fourth elastic instance pool and increasing the number of instances in the first elastic instance pool; keeping the number of instances in the third elastic instance pool unchanged; and / or, triggering the conversion of the instance processing the pre-filling request into the instance processing the decoding request includes: reducing the number of instances in the first elastic instance pool and increasing the number of instances in the third elastic instance pool; after the instance completes the pre-filling request task, slowing down the number of instances in the third elastic instance pool; increasing the number of instances in the second elastic instance pool; and keeping the number of instances in the fourth elastic instance pool unchanged.

[0022] Optionally, the request scheduling device, configured to invoke the large language model to perform inference operations for the processing request based on the target text scheduling strategy and the key performance indicators, further includes: for the pre-filling stage, evaluating the queuing latency of each instance in the first elastic instance pool that processes the pre-filling request; selecting the candidate instance with the shortest queuing latency to execute the pre-filling request based on the queuing latency; and for the decoding stage, selecting a candidate instance from the first elastic instance pool that has completed the pre-filling task to continue executing the decoding request.

[0023] Optionally, a scheduling device is requested to construct the pre-filling-decoding-encoding separation scheduling strategy; the construction of the pre-filling-decoding-encoding separation scheduling strategy includes: dividing the execution flow of the multimodal processing request into different stages corresponding to encoding requests, pre-filling requests, and decoding requests; using the instance to process one of the encoding requests, pre-filling requests, and decoding requests; evaluating the batch number of images processed for image encoding and the batch number of words processed for text pre-filling and decoding according to the key performance indicators; and determining the processing priority order of encoding requests, pre-filling requests, and decoding requests in each iteration.

[0024] Optionally, a request scheduling device is used to determine the processing priority order of encoding requests, pre-filling requests, and decoding requests, including: obtaining the decoding task corresponding to each decoding request and determining the decoding task as the highest priority; obtaining the pre-filling task that is partially executing the pre-filling request and determining the pre-filling task as the second highest priority; obtaining the encoding task corresponding to the encoding request for the image and determining the encoding task as a normal priority.

[0025] Optionally, a request scheduling device is configured to divide the hybrid expert architecture corresponding to the large language model into an attention network and an expert network; if it is determined that any instance operation is abnormal and needs to be restored, the attention network parameters are obtained from the normally operating instance and loaded to restore the instance service; simultaneously, key parameters of the expert network are asynchronously obtained and stored in an expert pool for subsequent loading of the expert network based on the expert pool; and / or, if it is determined that the decoding request processed by the large language model is abnormal, the key-value pair cache of the decoding request is obtained; according to the key performance indicators, a recalculation operation or a backup operation of the key-value pair cache is triggered to switch the instance executing the decoding request.

[0026] To achieve the above objectives, according to a third aspect of the present invention, a request-scheduled electronic device is provided, characterized in that it includes: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform any of the methods described above for request scheduling.

[0027] To achieve the above objectives, according to a fourth aspect of the present invention, a computer-readable medium is provided having a computer program stored thereon, characterized in that, when the program is executed by a processor, it implements the method described in any of the above-described request scheduling methods.

[0028] To achieve the above objectives, according to a fifth aspect of the present invention, a computer program product is provided, comprising a computer program, characterized in that, when executed by a processor, the computer program implements any of the above-described request scheduling methods.

[0029] One embodiment of the above invention has the following advantages or beneficial effects: it can determine whether the request type of the processing request is a text processing request or a multimodal processing request; it can obtain the key performance indicators of the large language model system in real time; for text processing requests, it selects a target text scheduling strategy from multiple text scheduling strategies based on the key performance indicators; for multimodal processing requests, it selects a target multimodal scheduling strategy from multiple multimodal scheduling strategies based on the key performance indicators; and further, based on the target text scheduling strategy or the target multimodal scheduling strategy, combined with the key performance indicators, it calls the large language model to perform inference operations for the processing request. This embodiment of the invention improves the resource scheduling flexibility of the large language model system by constructing separate encoding, pre-filling, and decoding structures and dynamically scheduling different instances according to system performance indicators. While meeting service level objectives, it improves the system's resource utilization and the processing performance of the large language model; it also improves the response speed of the large language model, thereby enhancing the user experience of using the large language model.

[0030] The further effects of the aforementioned unconventional alternative methods will be explained below in conjunction with specific implementation methods. Attached Figure Description

[0031] The accompanying drawings are provided to better understand the invention and are not intended to unduly limit the scope of the invention. Wherein: Figure 1 This is a flowchart illustrating a request scheduling method provided in an embodiment of the present invention; Figure 2 is a schematic diagram of a request scheduling architecture provided by an embodiment of the present invention; Figure 3 This is a schematic diagram of another request scheduling architecture provided by an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a request scheduling device provided in an embodiment of the present invention; Figure 5 This is an exemplary system architecture diagram in which embodiments of the present invention can be applied; Figure 6 This is a schematic diagram of the structure of a computer system suitable for implementing terminal devices or servers of the present invention. Detailed Implementation

[0032] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0033] It should be noted that the collection, use, storage, sharing and transfer of user personal information involved in the technical solution of the present invention all comply with the provisions of relevant laws and regulations, and require notification to users and obtaining their consent or authorization. When applicable, user personal information is subjected to de-identification and / or anonymization and / or encryption technical processing.

[0034] like Figure 1 As shown, this embodiment of the invention provides a request scheduling method applied to a large language model system. The method may include the following steps: Step S101: Receive a processing request and determine whether the request type of the processing request is a text processing request or a multimodal processing request.

[0035] Step S102: Obtain the key performance indicators of the large language model system in real time.

[0036] Step S103: For the text processing request, select a target text scheduling strategy from multiple text scheduling strategies based on the key performance indicators; for the multimodal processing request, select a target multimodal scheduling strategy from multiple multimodal scheduling strategies based on the key performance indicators.

[0037] Step S104: Based on the target text scheduling strategy or the target multimodal scheduling strategy, and in conjunction with the key performance indicators, call the large language model to perform inference operations for the processing request.

[0038] Specifically, a processing request is a request that requires processing using a large language model. The request includes information input to the large language model. If the information is plain text, the request type is determined to be a text processing request; if the information is a fusion of text and images, the request type is determined to be a multimodal processing request. For example, for an e-commerce platform, a processing request inputting the large language model could be general information, which is a text processing request; however, if the processing request inputting the large language model includes uploaded item images and item descriptions, the request to generate detailed item descriptions is a multimodal processing request.

[0039] Furthermore, key performance indicators (KPIs) of the large language model system are acquired in real time. These KPIs are related to the large language model's operational metrics and system metrics, and include one or more of the following: TTFT (Time To First Token), TPOT (Time Per Output Token), GPU (Graphics Processing Unit) memory usage, the number and length of requests, and the token generation interval. TTFT refers to the time from request submission to the generation of the first token, and is an important indicator of the large language model service's response speed. TPOT refers to the time required for the large language model to complete a task, including computation time and waiting time.

[0040] This invention improves the inference processing performance of large language models by performing dynamic scheduling based on key performance indicators to determine whether the system operation meets the Service Level Objective (SLO). The SLO can be a service quality standard set between the service provider and the service user, including response time and availability.

[0041] The following describes the request scheduling methods for text processing requests and multimodal processing requests, respectively. Specifically, based on the target text scheduling strategy or the target multimodal scheduling strategy, and in conjunction with the key performance indicators, the large language model is invoked to perform inference operations for the processing requests.

[0042] 1) For text processing requests.

[0043] The step of selecting a target text scheduling strategy for processing the text processing request from multiple text scheduling strategies based on the key performance indicators includes: when the performance of the large language model is predicted to be unable to meet the preset service level target based on the key performance indicators, selecting a pre-filling-decoding separation scheduling strategy as the target text scheduling strategy for processing the text processing request from multiple text scheduling strategies.

[0044] Specifically, the various text scheduling strategies may include a pre-fill-decode serial strategy and a pre-fill-decode separate scheduling strategy; preferably, when the performance of the large language model is predicted to be unable to meet the preset service level target based on the key performance indicators, the pre-fill-decode separate scheduling strategy (i.e., the target text scheduling strategy) is selected from the various text scheduling strategies as the target text scheduling strategy for processing the text processing request.

[0045] Specifically, a prefill-decode separation scheduling strategy (i.e., target text scheduling strategy) is constructed. This construction includes: constructing multiple elastic instance pools; using a first elastic instance pool to store instances that process prefill requests; using a second elastic instance pool to store instances that process decoding requests; using a third elastic instance pool to store instances that are scheduled to process decoding requests and are also processing prefill requests; and using a fourth elastic instance pool to store instances that are scheduled to process prefill requests and are also processing decoding requests. Each instance is stateless and can process either prefill requests or decoding requests.

[0046] In this embodiment of the invention, instances that process pre-filling requests and instances that process decoding requests are separated and processed independently in text processing. That is, each instance is stateless and can process either pre-filling requests or decoding requests. This improves the flexibility of instance scheduling.

[0047] Furthermore, four types of elastic instance pools are constructed: the first elastic instance pool stores instances that process pre-fill requests (represented by P), the second elastic instance pool stores instances that process decoding requests (represented by D), the third elastic instance pool stores instances that are scheduled to process decoding requests and are currently processing pre-fill requests (represented by P->D), and the fourth elastic instance pool stores instances that are scheduled to process pre-fill requests and are currently processing decoding requests (represented by D->P). By constructing these four types of elastic instance pools, when scheduling and switching instances to execute different requests, it is only necessary to remove the instance from its original pool and move it to the new pool to achieve zero-wait-time instance scheduling, avoiding the overhead of instance restart or model reloading in traditional systems.

[0048] Furthermore, the instance scheduling strategy based on stateless instances and elastic instance pools can dynamically schedule instances based on whether key performance indicators (SLOs) are met. Specifically, based on the target text scheduling strategy and the key performance indicators, the large language model is invoked to perform inference operations for the processing request. This includes: for the pre-filling stage, if the instance executing the pre-filling request cannot meet the first service level objective based on the key performance indicators, the instance executing the decoding request is converted into the instance executing the pre-filling request; for the decoding stage, if the instance executing the decoding request cannot meet the second service level objective based on the key performance indicators, the instance executing the pre-filling request is converted into the instance executing the decoding request. The first service level objective is, for example, the TTFT requirement value; the second service level objective is, for example, the TPOT threshold. This instance scheduling method triggers the conversion of decoding instances to pre-filled instances during the pre-filling phase if existing instances are predicted to be unable to meet TTFT requirements. Conversely, during the decoding phase, when resource insufficiency, an average token generation interval exceeding the TPOT threshold, or idle pre-filled instances are detected, the conversion of pre-filled instances (i.e., those handling pre-filling requests) to decoding instances (i.e., those handling decoding requests) is triggered to handle sudden traffic surges. This strategy ensures the system maintains high efficiency under varying load conditions while meeting SLO requirements.

[0049] Further, triggering the conversion of the instance processing the decoding request to the instance processing the pre-filling request includes: reducing the number of instances in the second elastic instance pool and increasing the number of instances in the fourth elastic instance pool; after the instance completes the decoding request task, reducing the number of instances in the fourth elastic instance pool and increasing the number of instances in the first elastic instance pool; keeping the number of instances in the third elastic instance pool unchanged; and / or, triggering the conversion of the instance processing the pre-filling request to the instance processing the decoding request includes: reducing the number of instances in the first elastic instance pool and increasing the number of instances in the third elastic instance pool; after the instance completes the pre-filling request task, slowing down the number of instances in the third elastic instance pool; increasing the number of instances in the second elastic instance pool; and keeping the number of instances in the fourth elastic instance pool unchanged.

[0050] Specifically, taking the number of instances to be converted as an example, the step of triggering the conversion of the instance processing the decoding request into the instance processing the pre-filling request includes: decrementing the number of instances in the second elastic instance pool by one and incrementing the number of instances in the fourth elastic instance pool by one; after the instance completes the decoding request task, decrementing the number of instances in the fourth elastic instance pool by one and incrementing the number of instances in the first elastic instance pool by one; keeping the number of instances in the third elastic instance pool unchanged; and / or, the step of triggering the conversion of the instance processing the pre-filling request into the instance processing the decoding request includes: decrementing the number of instances in the first elastic instance pool by one and incrementing the number of instances in the third elastic instance pool by one; after the instance completes the pre-filling request task, decrementing the number of instances in the third elastic instance pool by one; incrementing the number of instances in the second elastic instance pool by one; keeping the number of instances in the fourth elastic instance pool unchanged.

[0051] Specifically, since the first elastic instance pool stores instances that process pre-fill requests (represented by P), the second elastic instance pool stores instances that process decoding requests (represented by D), the third elastic instance pool stores instances that are scheduled to process decoding requests and are processing pre-fill requests (represented by P->D), and the fourth elastic instance pool stores instances that are scheduled to process pre-fill requests and are processing decoding requests (represented by D->P), when performing instance conversion, the number of instances in the original elastic instance pool can be directly increased or decreased, and the number of instances in the third or fourth elastic instance pool can be increased or decreased, thereby achieving dynamic instance scheduling without affecting the requests being processed.

[0052] Furthermore, the embodiment of the present invention further includes, based on the target text scheduling strategy and the key performance indicators, calling the large language model to perform inference operations for the processing request, which further includes: for the pre-filling stage, evaluating the queuing latency of each instance in the first elastic instance pool that processes the pre-filling request; selecting the candidate instance with the shortest queuing latency to execute the pre-filling request based on the queuing latency; and for the decoding stage, selecting a candidate instance from the first elastic instance pool that has completed the pre-filling task to continue executing the decoding request. For request scheduling, the system can adopt a two-level architecture of global and local, maximizing the SLO compliance rate through multi-dimensional optimization. At the global level, the scheduler implements a greedy strategy of prioritizing minimum load and considers the limitations of SLO constraints. Therefore, for prefill requests in the prefill phase, the scheduler evaluates the expected queuing latency of each instance in the Prefill pool (i.e., the first elastic instance pool), selects the candidate instance with the lowest latency, and verifies whether it meets the SLO requirements through the TTFT prediction model. For decoding requests in the decoding phase, priority can be given to continuing processing by the original prefill instance (i.e., selecting a candidate instance from the first elastic instance pool that has finished the prefill task to continue executing the decoding request), in order to avoid the transmission overhead of the key-value cache (KV Cache) and ensure that new requests do not cause TPOT to exceed the limit. Furthermore, at the local scheduling level, each instance can employ a refined queue management strategy. For example, key-value cache (KV Cache) transmission requests are placed in an independent migration queue and processed sequentially according to the First Come First Service (FCFS) principle. Additionally, inference computation requests for large language models can adopt a scheme combining chunked prefilling and continuous batching, which not only ensures the priority of decoding requests but also improves the overall utilization of computing resources in the large language model system.

[0053] By employing a targeted text scheduling strategy for text processing requests, this architecture not only overcomes the limitations of the pre-filled-decoder architecture in dynamic load management and resource utilization optimization, but also ensures high quality and reliability of online services, providing users with a superior service experience. Furthermore, this architecture can flexibly address the complex needs of hybrid online-offline deployment scenarios, achieving a dual improvement in performance and efficiency through refined resource management and scheduling strategies.

[0054] 2) For multimodal processing requests.

[0055] Specifically, the multimodal processing request is a request to process the fusion of text and images.

[0056] The step of selecting a target multimodal scheduling strategy for processing the multimodal processing request from multiple multimodal scheduling strategies based on the key performance indicators includes: selecting a pre-filling-encoding-decoding separation scheduling strategy as the target multimodal scheduling strategy for processing the multimodal processing request from multiple multimodal scheduling strategies based on the key performance indicators.

[0057] Specifically, various multimodal scheduling strategies may include the EP-D strategy (i.e., the encoding and pre-filling phases are executed together, while the decoding phase is executed separately), the ED-P strategy (i.e., the encoding and decoding phases are executed together, while the pre-filling phase is executed separately), or the EPD strategy (the encoding phase, pre-filling phase, and decoding phase are separated, i.e., the pre-filling-encoding-decoding separation scheduling strategy).

[0058] In this embodiment of the invention, preferably, a pre-filling-encoding-decoding separation scheduling strategy is selected as the target multimodal scheduling strategy for processing the multimodal processing request. The selection method employs binary search to identify the optimal deployment configuration among the various multimodal scheduling strategies, while also considering request characteristic data, real-time system load, computation time at each stage, and latency and throughput performance metrics (e.g., maximum batch size in the encoding stage, maximum number of tokens for pre-filling / decoding input, etc.) to select the optimal deployment configuration. For example, the optimal deployment configuration is the pre-filling-encoding-decoding separation scheduling strategy, to maximize resource utilization while satisfying the SLO (Solution Time Limit).

[0059] Specifically, the pre-filling-decoding-encoding separation scheduling strategy is constructed. This includes: dividing the execution flow of the multimodal processing request into different stages corresponding to encoding requests, pre-filling requests, and decoding requests; processing one of the encoding requests, pre-filling requests, and decoding requests using the instance; evaluating the batch number of images processed for image encoding and the batch number of words processed for text pre-filling and decoding based on the key performance indicators; and determining the processing priority order of encoding requests, pre-filling requests, and decoding requests in each iteration.

[0060] The prefill-decode-encode separation scheduling strategy enables stage-level scheduling. Specifically, it divides multimodal processing requests into three stages: encoding, prefill, and decoding. Scheduling and batch processing optimization are performed based on instances and stages respectively. This strategy avoids the performance bottleneck caused by coarse-grained inference merging stages, allowing for finer control over execution time and batch size. It effectively improves system throughput while ensuring latency control, meeting SLO requirements. Specifically, it first sets the maximum batch size for image encoding and the token budget for large language models based on the user-defined SLO. Specifically, during system startup, binary search can be used for automatic testing to determine the maximum number of concurrent images that image encoding can support (i.e., the number of batch images processed for image encoding) and the maximum word length limit that the large language model can handle (i.e., the number of batch words processed for text pre-filling and decoding). That is, based on these key performance indicators, the number of batch images processed for image encoding and the number of batch words processed for text pre-filling and decoding are evaluated, providing a configuration basis for subsequent scheduling to ensure that the execution time of batch processing tasks in each subsequent iteration is less than the SLO of TPOT. Furthermore, in each iteration, all running decoding requests can first be added to the batch, and then it can be checked whether there are any chunked prefill tasks that have already been partially computed. If so, these tasks are added to the batch; otherwise, it is checked whether there are any encoding tasks that need to be computed. If so, these tasks are added to the batch. This strategy can complete requests in the pre-filling stage as quickly as possible, reducing the TTFT (first token delay) of requests; and the encoding stage of new requests will only be processed when no requests are in the pre-filling stage. That is, determining the processing priority order of encoding requests, pre-padding requests, and decoding requests includes: obtaining the decoding task corresponding to each decoding request and determining the decoding task as the highest priority; obtaining the pre-padding task that is currently partially executing the pre-padding request and determining the pre-padding task as the second highest priority; obtaining the encoding task corresponding to the encoding request for the image and determining the encoding task as the normal priority.

[0061] In the architecture of the pre-fill-decode-encode separation scheduling strategy, each instance only executes a portion of the subtasks in the three stages, while the remaining stages are migrated to other instances for processing through requests. This avoids resource waste and interference, enabling the system to dynamically adjust resource allocation based on real-time load, thereby improving resource utilization efficiency and system flexibility. Furthermore, during system operation, the optimal separation strategy can be automatically selected based on request detection to achieve a dynamic trade-off between throughput and latency, significantly improving the overall processing capacity and resource utilization efficiency of the system while meeting SLO requirements.

[0062] Furthermore, the pre-fill-decode-encode separation scheduling strategy (target multimodal scheduling strategy) can also support a multi-stream parallel scheduling mechanism, which allows concurrent execution of tasks from different types (text type, image type). Specifically, a dual-stream parallel model is adopted, in which the visual model and the language model are assigned to different execution streams. The visual stream is used for computationally intensive image encoding tasks, while the language stream is used for pre-fill and decoding operations for language (i.e., text) generation. By isolating task workloads into different streams, the system enables concurrent execution of heterogeneous stages from different types of requests, improving the system's resource utilization and task processing capabilities.

[0063] Furthermore, embodiments of the present invention provide an efficient fault handling mechanism for large language model inference, which may include optimization strategies such as rapid request migration and rapid instance recovery, enabling large language models to quickly restore services when nodes fail, reducing downtime and business losses.

[0064] Specifically, for the rapid migration of requests processed by decoding instances, strategies such as key-value cache (KV Cache) recalculation and key-value cache (KV Cache) backup can be adopted. Key-value cache recalculation is suitable when computing resources are sufficient, while key-value cache backup can reduce the computational overhead of recalculation when communication bandwidth allows. The request migration strategy can be dynamically selected by comprehensively considering indicators such as request length, request priority, hardware communication bandwidth, and hardware computing power. For example, for high-priority requests, key-value cache backup is triggered when the inference length reaches a preset length threshold. The preset length threshold is selected based on the key-value cache transmission time being lower than the recalculation time. A unified storage abstraction management system can also be established, including the storage size available for key-value cache on each machine and the communication bandwidth between machines, to determine the optimal location for request backup and the migration overhead. For requests processed by pre-filled instances, since the key-value cache generated during the pre-filling stage is relatively small, key-value cache recalculation can be directly adopted. Through the recalculation operation, the optimal recalculation instance can be determined based on request priority, and instances that meet SLO requirements and performance can be selected for deployment. If an anomaly is detected in the decoding request processed by the large language model, the key-value pair cache of the decoding request is retrieved; based on the key performance indicators, a recalculation operation or a backup operation of the key-value pair cache is triggered to switch the instance executing the decoding request.

[0065] Furthermore, for the recovery of failed instances, this embodiment of the invention provides an instance recovery strategy. Specifically, the hybrid expert architecture corresponding to the large language model is divided into an attention network (attention or attention+gate) and an expert network (performing expert computation). The expert network is a MoE model, where MoE (Mixture of Experts) is a technique to improve model efficiency and performance by assigning tasks to different expert networks. Since the number of parameters in the attention network is a small fraction of the total number of parameters, it can be quickly transmitted and recovered. However, the number of parameters in the expert network is the vast majority of the total number of parameters, thus requiring greater transmission overhead. To reduce the instance recovery time, the recovery can be divided into two steps: attention and expert, and an "expert pool" is set up. The instance recovery process is as follows: Step 1) The failed node obtains the parameters of the attention part from the nearest non-failed node (i.e., a normally functioning instance). Step 2) A larger batch of requests or dimensions is selected for the calculation of the attention part, while the parameters of the "hot" expert are transmitted and recovered. The transmitted expert is stored in the "expert pool". Step 3) After the gate calculation is complete, update the "hot" experts to the experts most frequently requested. Step 4) Perform expert calculations on some requests. After step 1), repeat steps 2) and 3) until the experts in the expert pool can support enough requests for expert calculations, then proceed to step 4). If there are not enough requests to perform calculations in step 4), return to step 2) and continue repeating steps 2) and 3), and so on. During parameter transmission, a global transmission topology can also be constructed, and the fastest transmission path is found through optimal path search, further accelerating instance recovery time.

[0066] That is, the hybrid expert architecture corresponding to the large language model is divided into an attention network and an expert network; when it is determined that any instance has an abnormality and needs to be restored, the attention network parameters are obtained from the normally running instance and loaded to restore the instance service; at the same time, the key parameters of the expert network are asynchronously obtained and stored in the expert pool so as to load the expert network based on the expert pool in the future.

[0067] The embodiments of the present invention ensure high service availability through an efficient fault handling mechanism (including rapid request migration and rapid instance recovery), solving the problems of long fault recovery time and impact on service continuity and availability in existing systems.

[0068] Furthermore, the request scheduling method of this embodiment can be executed using multiple layers of components. Specifically, it can include a preprocessing layer, a scheduling layer, a resource layer, and a fault recovery framework. The preprocessing layer includes a performance predictor and an analyzer. The scheduling layer integrates multiple strategies (dynamic pre-filling-decoding separation scheduling strategy, hybrid encoding-pre-filling-decoding scheduling strategy, unified online / offline scheduling strategy, etc.). The resource layer can be composed of multiple heterogeneous instance pools. Specifically, for the dynamic pre-filling-decoding separation scheduling strategy, the predictor can dynamically switch between pre-filling and decoding instances based on key performance indicator information and the workload status of multiple instance types. For the hybrid encoding-pre-filling-decoding scheduling strategy, the analyzer can determine the optimal encoding-pre-filling-decoding separation strategy for multimodal requests. Simultaneously, the unified online / offline scheduling strategy schedules key-value pair caches and image caches based on the online or offline attributes of the request. These caches can be offloaded and routed between distributed instances, while the fault recovery framework ensures high availability of the entire service.

[0069] like Figure 2 As shown in the figure, this embodiment of the invention provides a schematic diagram of a request scheduling structure, including: a performance predictor 201, a global scheduler 202, an instance scheduler 203, a request scheduler 204, and a real-time performance monitor 205.

[0070] Specifically, Figure 2 The schematic diagram of the request scheduling shown is applied to the pre-filling-decoding separation scheduling strategy.

[0071] Among them, the performance predictor 201 is used to acquire and predict the key performance indicators of the large language model system in real time; The global scheduler 202 is used to select a target text scheduling strategy for processing the text processing request from a variety of text scheduling strategies based on the key performance indicators.

[0072] Instance scheduler 203 is configured to store instances (P) that process pre-fill requests using a first elastic instance pool; instances (D) that process decoding requests using a second elastic instance pool; instances (P->D) that are scheduled to process decoding requests and are currently processing pre-fill requests using a third elastic instance pool; and instances (D->P) that are scheduled to process pre-fill requests and are currently processing decoding requests using a fourth elastic instance pool. Each instance is stateless and can process either pre-fill requests or decoding requests. During the pre-filling phase, if the key performance indicators determine that the instance processing the pre-fill request cannot meet the first service level objective, the instance processing the decoding request is converted to the instance processing the pre-fill request. During the decoding phase, if the key performance indicators determine that the instance processing the decoding request cannot meet the second service level objective, the instance processing the pre-fill request is converted to the instance processing the decoding request.

[0073] The request scheduler 204 is used to evaluate the queuing latency of each instance in the first elastic instance pool that processes pre-fill requests based on the request queue and the key-value pair cache transmission queue for the pre-filling phase; select the candidate instance with the shortest queuing latency to execute the pre-fill request based on the queuing latency; and select the candidate instance that has finished the pre-filling task from the first elastic instance pool to continue executing the decoding request for the decoding phase, wherein the execution engine can be used to execute the pre-fill request or the decoding request.

[0074] The real-time performance monitor 205 is used to obtain key performance indicators of the large language model system in real time, such as receiving reported performance data from the request scheduler, forming a loop, and calling the large language model to perform inference operations for text processing requests through a dynamic pre-filling-decoding separation scheduling strategy coordinated by various components.

[0075] like Figure 3 As shown in the figure, this embodiment of the invention provides a schematic diagram of a request scheduling structure, including: a performance predictor 301, a multimodal scheduling strategy set 302, and a scheduler 303.

[0076] Specifically, Figure 3 The schematic diagram of the request scheduling shown is applied to the pre-filling-encoding-decoding separation scheduling strategy.

[0077] The performance predictor 301 is used to acquire and predict the key performance indicators of the large language model system in real time; Scheduler 303 is used to select a pre-filling-encoding-decoding separation scheduling strategy as the target multimodal scheduling strategy for processing the multimodal processing request from a variety of multimodal scheduling strategies (i.e., multimodal scheduling strategy set 302) based on the key performance indicators. The construction of the pre-filling-decoding-encoding separation scheduling strategy includes: dividing the execution flow of the multimodal processing request into different stages corresponding to encoding requests, pre-filling requests, and decoding requests; processing one of the encoding requests, pre-filling requests, and decoding requests using the instance; evaluating the batch number of images processed for image encoding and the batch number of words processed for text pre-filling and decoding based on the key performance indicators; and determining the processing priority order of encoding requests, pre-filling requests, and decoding requests in each iteration. Specifically, determining the processing priority order of encoding requests, pre-filling requests, and decoding requests includes: obtaining the decoding task corresponding to each decoding request and determining the decoding task as the highest priority; obtaining the pre-filling task that is currently partially executing a pre-filling request and determining the pre-filling task as the second highest priority; obtaining the encoding task corresponding to the encoding request for the image and determining the encoding task as a normal priority.

[0078] The multimodal scheduling strategy set 302 includes various scheduling strategies such as ED-P (i.e., aggregate execution of the encoding and decoding stages, with the pre-filling stage executed separately), EP-D (i.e., aggregate execution of the encoding and pre-filling stages, with the decoding stage executed separately), and EPD (encoding stage, pre-filling stage, and decoding stage are separated, i.e., pre-filling-encoding-decoding separation scheduling strategy).

[0079] Figure 3 The encoding instance represents the instance that handles the encoding request (processing the image cache), the pre-fill instance represents the instance that handles the pre-fill request (processing the image cache and key-value pair cache), and the decoding instance represents the instance that handles the decoding request (processing the key-value pair cache); the dashed lines between the three types of instances represent the asynchronous transmission of their output processing results.

[0080] It is understandable that the pre-fill-encode-decode separation scheduling strategy executed for multimodal requests can also be applied to processing text requests, that is, the pre-fill-encode-decode separation scheduling strategy includes the pre-fill-decode separation scheduling strategy.

[0081] like Figure 4 As shown, this embodiment of the invention provides a request scheduling device 400, including: a request receiving module 401, an indicator acquisition module 402, a strategy determination module 403, and an inference module 404; wherein, The request receiving module 401 is used to receive a processing request and determine whether the request type of the processing request is a text processing request or a multimodal processing request. The indicator acquisition module 402 is used to predict the key performance indicators of the large language model system; The strategy determination module 403 is used to select a target text scheduling strategy for processing the text processing request from a variety of text scheduling strategies based on the key performance indicators; and to select a target multimodal scheduling strategy for processing the multimodal processing request from a variety of multimodal scheduling strategies based on the key performance indicators. The inference module 404 is used to invoke the large language model to perform inference operations for the processing request based on the target text scheduling strategy or the target multimodal scheduling strategy and in conjunction with the key performance indicators.

[0082] This invention also provides a request-scheduled electronic device, comprising: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided in any of the above embodiments.

[0083] This invention also provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the method provided in any of the above embodiments.

[0084] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements any of the above-described request scheduling methods.

[0085] Figure 5 An exemplary system architecture 500 is shown that can be applied to the request scheduling method or request scheduling apparatus of the present invention.

[0086] like Figure 5 As shown, system architecture 500 may include terminal devices 501, 502, and 503, a network 504, and a server 505. Network 504 serves as the medium for providing communication links between terminal devices 501, 502, and 503 and server 505. Network 504 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0087] Users can use terminal devices 501, 502, and 503 to interact with server 505 via network 504 to receive or send messages, etc. Various client applications can be installed on terminal devices 501, 502, and 503, such as e-commerce client applications, web browser applications, search applications, instant messaging tools, and email clients.

[0088] Terminal devices 501, 502, and 503 can be various electronic devices with displays and supporting various client applications, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0089] Server 505 can be a server that provides various services, such as a backend management server that supports client applications used by users through terminal devices 501, 502, and 503. The backend management server can process received processing requests and feed back the processing results output by the large language model to the terminal devices.

[0090] It should be noted that the request scheduling method provided in this embodiment of the invention is generally executed by server 505, and correspondingly, the request scheduling device is generally set in server 505.

[0091] It should be understood that Figure 5 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0092] The following is for reference. Figure 6 It shows a schematic diagram of the structure of a computer system 600 suitable for implementing a terminal device of the present invention. Figure 6 The terminal device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0093] like Figure 6 As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 602 or programs loaded from storage section 608 into random access memory (RAM) 603. The RAM 603 also stores various programs and data required for the operation of the system 600. The CPU 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0094] The following components are connected to I / O interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to I / O interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 610 as needed so that computer programs read from it can be installed into storage section 608 as needed.

[0095] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 609, and / or installed from removable medium 611. When the computer program is executed by central processing unit (CPU) 601, it performs the functions defined above in the system of this invention.

[0096] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0097] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0098] The modules and / or units described in the embodiments of the present invention can be implemented in software or hardware. The described modules and / or units can also be housed in a processor; for example, a processor can be described as including a request receiving module, an indicator acquisition module, a strategy determination module, and an inference module. The names of these modules do not necessarily limit the module itself; for example, the request receiving module can also be described as "a module that determines whether the request type of the processing request is a text processing request or a multimodal processing request."

[0099] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs, which, when executed by the device, cause the device to include: the ability to determine whether a processing request is a text processing request or a multimodal processing request; real-time acquisition of key performance indicators of the large language model system; for text processing requests, selecting a target text scheduling strategy from multiple text scheduling strategies based on the key performance indicators; for multimodal processing requests, selecting a target multimodal scheduling strategy from multiple multimodal scheduling strategies based on the key performance indicators; and, based on the target text scheduling strategy or the target multimodal scheduling strategy, combined with the key performance indicators, invoking the large language model to perform inference operations for the processing request. The embodiments of the present invention, by constructing separate encoding, pre-filling, and decoding structures and dynamically scheduling instances according to system performance indicators, improve the resource scheduling flexibility of the large language model system, enhance system resource utilization and large language model processing performance, and improve user experience.

[0100] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A request scheduling method, characterized in that, Applications to large language model systems include: Receive a processing request and determine whether the request type is a text processing request or a multimodal processing request; Real-time acquisition of key performance indicators of the large language model system; For the text processing request, based on the key performance indicators, a target text scheduling strategy is selected from multiple text scheduling strategies to process the text processing request. For the multimodal processing request, based on the key performance indicators, a target multimodal scheduling strategy is selected from multiple multimodal scheduling strategies to process the multimodal processing request; Based on the target text scheduling strategy or the target multimodal scheduling strategy, and in conjunction with the key performance indicators, the large language model is invoked to perform inference operations for the processing request.

2. The method according to claim 1, characterized in that, The step of selecting a target text scheduling strategy for processing the text processing request from multiple text scheduling strategies based on the key performance indicators includes: If, based on the aforementioned key performance indicators, the performance of the large language model is predicted to fail to meet the preset service level target, then... From a variety of text scheduling strategies, the pre-filling-decoding separation scheduling strategy is selected as the target text scheduling strategy for processing the text processing request.

3. The method according to claim 1, characterized in that, The step of selecting a target multimodal scheduling strategy for processing the multimodal processing request from multiple multimodal scheduling strategies based on the key performance indicators includes: Based on the key performance indicators, the pre-filling-encoding-decoding separation scheduling strategy is selected from a variety of multimodal scheduling strategies as the target multimodal scheduling strategy for processing the multimodal processing request.

4. The method according to claim 2, characterized in that, Further includes: Construct a pre-filling-decoding separation scheduling strategy; The construction of the pre-filling-decoding separation scheduling strategy includes: Construct multiple elastic instance pools; use the first elastic instance pool to store instances that process pre-fill requests; use the second elastic instance pool to store instances that process decoding requests; use the third elastic instance pool to store instances that are scheduled to process decoding requests and are also processing pre-fill requests; use the fourth elastic instance pool to store instances that are scheduled to process pre-fill requests and are also processing decoding requests. Each of the aforementioned instances is stateless and can handle either pre-filled requests or decoding requests.

5. The method according to claim 4, characterized in that, The step of invoking the large language model to perform inference operations for the processing request based on the target text scheduling strategy and the key performance indicators includes: For the pre-filling phase, if it is determined from the key performance indicators that the instance executing the pre-filling request cannot meet the first service level target, the instance executing the decoding request is converted into the instance executing the pre-filling request. During the decoding phase, if the instance executing the decoding request cannot meet the second service level objective based on the key performance indicators, the instance executing the pre-fill request is converted into the instance executing the decoding request.

6. The method according to claim 5, characterized in that, The step of triggering the conversion of the instance processing the decoding request into the instance processing the pre-filling request includes: reducing the number of instances in the second elastic instance pool and increasing the number of instances in the fourth elastic instance pool; after the instance completes the decoding request task, reducing the number of instances in the fourth elastic instance pool and increasing the number of instances in the first elastic instance pool; and keeping the number of instances in the third elastic instance pool unchanged. And / or, The triggering of converting the instance processing the pre-fill request into the instance processing the decoding request includes: reducing the number of instances in the first elastic instance pool and increasing the number of instances in the third elastic instance pool; after the instance completes the pre-fill request task, slowing down the number of instances in the third elastic instance pool; increasing the number of instances in the second elastic instance pool; and keeping the number of instances in the fourth elastic instance pool unchanged.

7. The method according to claim 5, characterized in that, The step of invoking the large language model to perform inference operations for the processing request based on the target text scheduling strategy and the key performance indicators further includes: For the pre-filling phase, the queuing latency of each instance in the first elastic instance pool that processes the pre-filling request is evaluated; based on the queuing latency, the candidate instance with the shortest queuing latency is selected to execute the pre-filling request. For the decoding phase, candidate instances that have completed the pre-filling task are selected from the first elastic instance pool to continue executing the decoding request.

8. The method according to claim 3, characterized in that, Further includes: Construct the pre-filling-decoding-encoding separation scheduling strategy; The construction of the pre-filling-decoding-encoding separation scheduling strategy includes: The execution flow of the multimodal processing request is divided into different stages corresponding to encoding request, pre-filling request and decoding request; The instance is used to process one of the following requests: an encoding request, a pre-filling request, and a decoding request. Based on the key performance indicators, evaluate the number of batch images for image encoding and the number of batch tokens for text pre-filling and decoding; and determine the processing priority order of encoding requests, pre-filling requests, and decoding requests in each iteration.

9. The method according to claim 8, characterized in that, The process priority order for determining the encoding request, pre-filling request, and decoding request includes: Obtain the decoding task corresponding to each decoding request, and determine the decoding task as the highest priority; Obtain the pre-fill task that is currently partially executing the pre-fill request, and determine the pre-fill task as the second highest priority; Obtain the encoding task corresponding to the encoding request for the image, and assign the encoding task a normal priority.

10. The method according to claim 1, characterized in that, Further includes: The hybrid expert architecture corresponding to the large language model is divided into an attention network and an expert network; If it is determined that any instance is running abnormally and needs to be restored, the attention network parameters are obtained from the normally running instance and loaded to restore the instance service; at the same time, the key parameters of the expert network are asynchronously obtained and stored in the expert pool, so as to load the expert network based on the expert pool in the future. And / or, If an anomaly is detected in the decoding request processed by the large language model, the key-value pair cache of the decoding request is retrieved; based on the key performance indicators, a recalculation operation or a backup operation of the key-value pair cache is triggered to switch the instance executing the decoding request.

11. A request scheduling device, characterized in that, include: The system comprises a request receiving module, an indicator acquisition module, a strategy determination module, and an inference module; among which, The request receiving module is used to receive a processing request and determine whether the request type of the processing request is a text processing request or a multimodal processing request. The indicator acquisition module is used to predict the key performance indicators of the large language model system. The strategy determination module is used to select a target text scheduling strategy for processing the text processing request from a variety of text scheduling strategies based on the key performance indicators, and to select a target multimodal scheduling strategy for processing the multimodal processing request from a variety of multimodal scheduling strategies based on the key performance indicators. The inference module is used to invoke the large language model to perform inference operations for the processing request based on the target text scheduling strategy or the target multimodal scheduling strategy and in conjunction with the key performance indicators.

12. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-10.

13. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-10.

14. A computer program product, comprising a computer program, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-10.

Images