A large language model service request scheduling method and system

Abstract

The application provides a large language model service request scheduling method and system, belonging to the technical field of large language models and service computing; a large language model is fine-tuned through construction of an instruction fine-tuning data set and a response text length is predicted, a relationship model between the response time of the fine-tuned large language model and the response text length is constructed to obtain a response time approximate solution; service requests of the large language model are processed in batches and a batch scheduling strategy is set; an error processing mechanism is introduced, the predicted response text length is compared with the response text length output by the large language model after receiving the service request, and the service response state of the large language model is determined according to the comparison result. The application can fully utilize the task characteristics in the service response process on the basis of ensuring accurate prediction of the service response time, so as to realize effective and orderly scheduling of the large language model service response.

Description

Technical Field

[0001] This invention belongs to the field of large language model and service computing technology, and particularly relates to a method and system for scheduling large language model service requests. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Large Language Models (LLMs) are deep learning models trained on large amounts of text data, enabling them to generate natural language text or understand the meaning of language text. LLMs and their derived intelligent services have been widely applied in various fields, such as dialogue generation, machine translation, text classification, and content creation. To ensure service quality and reduce resource costs, these LLM services are typically deployed in cloud environments. However, because the service quality of LLM services is closely related to hardware conditions such as computing resources, the response time of LLM service requests under limited computing power is prone to fluctuations, severely impacting user experience and application performance. Furthermore, existing static batch processing scheduling systems for LLM services often struggle to accurately predict task size, typically employing a First-Come, First-Served (FCFS) strategy to process requests with fixed batch sizes. This can easily lead to inefficient resource allocation and even problems such as task queue blocking and service interruptions, severely restricting the large-scale application of LLM services.

[0004] To address these issues, some research focuses on improving the efficiency of large language models themselves, such as through model compression and quantization techniques to reduce the time and resources required for model inference. However, these methods often require retraining or tuning the model, which is costly and struggles to solve problems like queue blocking caused by concurrent requests. Other work focuses on optimizing scheduling algorithms, such as employing more complex queue management strategies and resource pre-allocation strategies; however, these methods often struggle to adapt to the complexity and diversity of large language model service tasks, and their prediction accuracy and resource utilization still have significant room for improvement. Furthermore, existing technologies typically lack precise assessment mechanisms for request complexity and resource requirements, leading to inaccurate resource allocation strategies that impact service efficiency and stability. Therefore, there is an urgent need for a more efficient, accurate, and reliable large model service response time prediction and task scheduling system to improve system resource utilization in cloud environments, reduce average request response time, and enhance user experience.

[0005] It is evident that existing methods for scheduling large language model service requests in cloud environments still have significant shortcomings. These shortcomings stem from inherent technical issues within the methods themselves, such as:

[0006] (1) Existing methods for scheduling large language model service requests in cloud environments only consider simple queue models or resource utilization, while ignoring the internal complexity and dynamism of large language models. However, for different types of request tasks, the attention mechanism and parameter update process within the large language model will affect the execution time of the task. These factors are generally ignored in existing methods, which leads to insufficient accuracy in predicting the response time of large language model service response scheduling, and thus affects the quality of request scheduling.

[0007] (2) Existing cloud-based methods for scheduling large language model service requests, while considering the length of the request input, typically model using simple linear relationships, failing to effectively capture the non-linear relationship between task complexity and response time. However, for some complex natural language inference tasks, the response time may exhibit a non-linear relationship with the input length. Therefore, the simple linear models constructed by existing cloud-based methods for scheduling large language model service requests still struggle to accurately predict response time.

[0008] (3) Existing cloud-based methods or systems for scheduling large language model service requests, even if they can accurately predict task response times, still require effective scheduling strategies to optimize the execution order of request tasks in order to minimize average response time and improve system throughput. However, commonly used static batch processing scheduling algorithms are often inefficient when handling large language model service request tasks, unable to effectively cope with response time fluctuations caused by requests of different sizes, and may even result in long processing wait times, leading to low resource utilization and difficulty in dynamically optimizing according to the size of different requests, ultimately affecting the overall system performance and user experience. Summary of the Invention

[0009] To overcome the shortcomings of the prior art, the present invention provides a method and system for scheduling large language model service requests, which can make full use of the task characteristics in the service response process to achieve effective and orderly scheduling of large language model service responses, while ensuring accurate prediction of service response time.

[0010] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0011] The first aspect of this invention provides a method for scheduling large language model service requests.

[0012] A method for scheduling large language model service requests includes:

[0013] Build a fine-tuning dataset for instructions;

[0014] The dataset is fine-tuned according to the instructions, and the parameters of the large language model are fine-tuned using an efficient parameter fine-tuning method; the length of the response text is predicted based on the fine-tuned large language model.

[0015] A relationship model is constructed between the response time of the fine-tuned large language model and the length of the resulting response text. Based on the relationship model, an approximate solution for the response time of the large language model is obtained.

[0016] Based on the known approximate solution of the response time of the large language model, the service requests of the large language model are processed in batches, and a batch scheduling strategy is set; according to the set batch scheduling strategy, the service requests are sent to the large language model.

[0017] An error handling mechanism is introduced to compare the predicted response text length with the actual response text length output by the large language model after receiving the service request, as well as the response text length generated in each batch. The service response status of the large language model is determined based on the comparison results.

[0018] Furthermore, the instruction fine-tuning dataset contains a series of data pairs, which include instruction requirements, input text, and corresponding response text length labels.

[0019] Furthermore, an efficient parameter fine-tuning method is used to fine-tune the parameters of the large language model. Specifically, a low-rank adaptive method is used to optimize the parameters in the large language model.

[0020] Furthermore, the response time of the large language model is the total time required for the user to generate a complete response; where the total time required for the user to generate a complete response includes the sum of the time from when the user submits the request to when the complete text is generated.

[0021] Furthermore, the batch scheduling strategy is set based on three conditions, specifically, the three conditions include the request length, response length, and response time of the service request sent by the user.

[0022] Furthermore, the batch scheduling strategy is deployed in the service scheduler of the large language model, and the service scheduler is equipped with scheduling strategy options; a request expectation is set, and the set request expectation is used as the scheduling strategy option in the service scheduler.

[0023] Furthermore, the service response status of the large language model is determined based on the obtained comparison results, including: setting an upper limit value for identifiers and a text length threshold. When the output identifier of any batch of service requests is greater than the set upper limit value for identifiers, and the difference between the current maximum generated text length and the minimum generated text length corresponding to the generated request in the batch is greater than the set text length threshold, the inference service of that batch is terminated.

[0024] A second aspect of the present invention provides a large language model service request scheduling system.

[0025] A large language model service request scheduling system includes:

[0026] The instruction fine-tuning dataset generation module is configured to: build instruction fine-tuning datasets;

[0027] The response text length prediction module is configured to: fine-tune the dataset according to the instructions, fine-tune the parameters of the large language model using an efficient parameter fine-tuning method, and predict the response text length based on the fine-tuned large language model;

[0028] The relational model building module is configured to: build a relational model between the response time of the fine-tuned large language model and the length of the resulting response text, and obtain an approximate solution for the response time of the large language model based on the relational model;

[0029] The batch processing module is configured to: based on the known approximate solution of the response time of the large language model, process the service requests of the large language model in batches and set a batch scheduling strategy; and send service requests to the large language model according to the set batch scheduling strategy.

[0030] The service request scheduling module is configured to introduce an error handling mechanism, which compares the predicted response text length with the actual response text length output by the large language model after receiving the service request and the response text length currently generated in each batch, and determines the service response status of the large language model based on the comparison results.

[0031] A third aspect of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps of a large language model service request scheduling method as described in the first aspect of the present invention.

[0032] A fourth aspect of the present invention provides an electronic device including a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of a large language model service request scheduling method as described in the first aspect of the present invention.

[0033] The above one or more technical solutions have the following beneficial effects:

[0034] (1) This invention constructs an instruction fine-tuning dataset and, based on this dataset, employs an efficient parameter fine-tuning method to fine-tune the parameters of a large language model; then, it predicts the length of the response text based on the fine-tuned large language model. This invention leverages the powerful semantic understanding and modeling capabilities of the large language model itself, enabling accurate prediction of the length of the output text at a relatively low cost, thus providing a foundation for response time prediction. Therefore, this invention, based on the fine-tuned large language model, can better focus on the internal complexity and dynamics of the model, thereby achieving accurate prediction of the response time of the large language model service.

[0035] (2) Based on the preliminary prediction of the response text length of the large language model, this invention constructs a relationship model between the fine-tuned response time of the large language model and the obtained response text length, and obtains an approximate solution for the response time of the large language model based on the obtained relationship model. This relationship model can effectively capture the nonlinear relationship between task complexity and response time, thus providing a basis for service scheduling. Therefore, the relationship model constructed by this invention can provide further assurance for predicting response time.

[0036] (3) Based on the known approximate solution for the response time of the large language model, this invention processes service requests from the large language model in batches and sets a batch scheduling strategy. According to the set batch scheduling strategy, service requests are sent to the large language model. This service request scheduling mode can achieve efficient batch processing operations and effectively cope with response time fluctuations caused by requests of different sizes. Therefore, this invention can dynamically optimize according to the scale of different requests, fully utilize the task characteristics in the service response process, and achieve effective and orderly scheduling of service responses from the large language model; thereby improving the overall system performance and user experience.

[0037] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0038] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0039] Figure 1 This is a flowchart of a large language model service request scheduling method according to Embodiment 1 of the present invention.

[0040] Figure 2 This is a schematic diagram of the arrival of large language model service requests and task scheduling in Embodiment 1 of the present invention.

[0041] Figure 3This is a flowchart of the instruction fine-tuning dataset construction and large language model fine-tuning in Embodiment 1 of the present invention.

[0042] Figure 4 This is an example diagram of a data sample in Embodiment 1 of the present invention.

[0043] Figure 5 This is a flowchart of the large language model service response time prediction in Embodiment 1 of the present invention.

[0044] Figure 6 This is a flowchart of the iterative algorithm in Embodiment 1 of the present invention.

[0045] Figure 7 This is an overall architecture diagram of the large language model service request scheduling system in the cloud environment in Embodiment 2 of the present invention. Detailed Implementation

[0046] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0047] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations of the present invention.

[0048] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0049] The overall approach of this invention is as follows: This invention provides a method for scheduling requests to a large language model service. First, the large language model is fine-tuned using a dataset fine-tuning instruction. Leveraging the powerful semantic understanding and modeling capabilities of the large language model, the length of the output text can be accurately predicted at a relatively low cost. Then, by establishing a relationship model between response time and output length, an approximate solution for the large language model service response time is obtained. Based on this, requests with similar request sizes are batched, and request expectations are introduced as a request scheduling strategy option for the service scheduler. This request scheduling strategy effectively reduces the average response time and prevents batches with longer service times from waiting in the queue for extended periods. Simultaneously, this invention designs an error handling mechanism to promptly address prediction deviations occurring in the system, thereby effectively ensuring the service quality of the large language model. The method provided by this invention does not conflict with other existing inference acceleration technologies and can effectively supplement existing large language model inference acceleration toolkits, further improving the overall performance of large language model services.

[0050] Example 1

[0051] This embodiment discloses a method for scheduling service requests for large language models.

[0052] like Figure 1 As shown, a method for scheduling large language model service requests includes:

[0053] Step S1: Construct the instruction fine-tuning dataset;

[0054] Step S2: Fine-tune the dataset according to the instructions, and fine-tune the parameters of the large language model using an efficient parameter fine-tuning method; predict the length of the response text based on the fine-tuned large language model;

[0055] Step S3: Construct a relationship model between the response time of the fine-tuned large language model and the length of the obtained response text, and obtain an approximate solution for the response time of the large language model based on the relationship model;

[0056] Step S4: Based on the known approximate solution of the response time of the large language model, process the service requests of the large language model in batches and set a batch scheduling strategy; send service requests to the large language model according to the set batch scheduling strategy.

[0057] Step S5: Introduce an error processing mechanism to compare the predicted response text length with the actual response text length output by the large language model after receiving the service request and the response text length currently generated in each batch, and determine the service response status of the large language model based on the comparison results.

[0058] Based on the above steps, this invention can effectively and orderly schedule the service response of large language models by fully utilizing the task characteristics during the service response process while ensuring accurate prediction of service response time. To facilitate understanding of the technical solution of this invention, the specific implementation steps are further explained and described below.

[0059] It should be noted that the solution provided in this embodiment is implemented using a cloud server cluster as the platform. This cloud platform supports the deployment of multiple large language model services such as ChatGLM3-6B, Qwen1.5-7B, and LLaMA2-7B. Furthermore, this cloud platform is an experimental environment composed of six cloud servers, each equipped with an NVIDIA GeForce RTX 3090 GPU and 24GB of video memory, and supports on-demand elastic expansion. Figure 2 As shown, the large language model in the cloud environment (i.e. Figure 2The large language model service described above addresses concurrent user requests by employing an efficient batch scheduling strategy to intelligently batch requests, optimize task execution order, and dynamically adjust batch sizes to adapt to real-time load changes. Ultimately, this improves the service quality of the large language model, reduces average response time, and ensures the stability of the large language model service. This embodiment performs the following operations in the aforementioned cloud environment.

[0060] Step S1: Construct the instruction fine-tuning dataset.

[0061] Based on the target response text length of the large language model, a fine-tuning dataset is constructed; among them, the instruction fine-tuning dataset contains a series of data pairs, which include instruction requirements, input text and corresponding response text length labels.

[0062] Furthermore, such as Figure 3 As shown, the instruction fine-tuning dataset for fine-tuning the large language model in this embodiment includes two parts; one part consists of 15,000 samples modified from the public dataset Alpaca; the other part consists of 5,000 samples constructed using the generation capabilities of the large language model based on seed samples. Specifically, the seed samples are simulated samples defined based on real data samples that can reflect the characteristics and distribution of real data, and have the characteristics of being fundamental and representative.

[0063] Furthermore, the Alpaca dataset modification includes cleaning and normalizing the instructions, as well as performing quality control on the input-output pairs to improve the quality and consistency of the instruction fine-tuning dataset. The selection of seed samples is based on considerations of instruction diversity and coverage, striving to encompass different types of instructions and tasks. Specifically, seed samples are constructed using the generative capabilities of the large language model through transformation, expansion, and combination to ensure the diversity and scale of the dataset. Simultaneously, for the generated samples, to effectively evaluate the various possibilities of the large language model's response text length, this embodiment selects to sample each instruction 5 times, and uses the maximum length among the 5 samples as the target length for each instruction, and uses the target length as the length of the large language model's output text, i.e.:

[0064] L = max{L1,L2,L3,L4,L5};

[0065] Here, L represents the length of the output text of the large language model. Each sample data includes the instruction requirement, input text, and corresponding response text length labels. The dataset covers texts of different lengths, styles, and domains, thus ensuring the model's generalization ability.

[0066] like Figure 4As shown, this dataset contains sample data in an "instruction-input-output" format; where the instruction is the task to be performed by the large language model, the input is the user request, and the output is the length of the response text predicted by the large language model. During instruction-supervised fine-tuning, the content corresponding to the instruction is concatenated with the content corresponding to the input to form the human instruction, and the content corresponding to the output is the response answer from the large language model.

[0067] Step S2: Fine-tune the dataset according to the established instructions, and fine-tune the parameters of the large language model using an efficient parameter fine-tuning method; predict the length of the response text based on the fine-tuned large language model.

[0068] Based on the constructed fine-tuning dataset, the large language model is fine-tuned using instructions. The parameters of the large language model are optimized by employing the Low-Rank Adaptation (LoRA) method to improve the accuracy of the large language model in predicting the length of the response text. For example... Figure 3 As shown, this can be achieved through the following steps:

[0069] Step S2-1: Load the pre-trained large language model. To reduce the number of training parameters and accelerate training, this embodiment uses the LoRA fine-tuning method based on LLaMA Factory. LLaMA Factory, or Large Language Model Factory, is an open-source fine-tuning framework that provides developers with a simple and efficient tool to quickly adapt to specific task requirements and improve model performance based on a pre-trained model. This fine-tuning method adds a small number of trainable parameters to the pre-trained model, rather than directly fine-tuning all parameters of the large model. This significantly reduces computational resource consumption and memory usage, enabling efficient fine-tuning of large language models even in resource-constrained environments and rapid adaptation to specific tasks. Specifically, the calculation formula for the LoRA fine-tuning method is as follows:

[0070] h = W0x + ΔWx;

[0071] Where h represents the model output and x represents the model input. W0 represents the pre-trained weight matrix, and ΔW represents the additional matrix added in the side path, which can be further refined to ΔW = BA. During initialization, matrix A is randomly Gaussian initialized, and matrix B is initialized to 0; this ensures that only the left-hand side of the matrix is ​​effective in the initial stage. During training, the pre-trained weight matrix is ​​frozen, and only the ΔW part is iteratively optimized.

[0072] Step S2-2: Before fine-tuning the instructions, set the necessary parameters for fine-tuning; these parameters include the learning rate, number of training epochs, and calculation type. The specific settings for these necessary parameters are shown in Table 1.

[0073] Table 1. Necessary parameters for fine-tuning

[0074] Learning rate 5e-5 Number of training rounds 3 Calculation type bf16 Batch size 2 Validation set ratio 0.1 Learning rate regulator Cosine

[0075] The learning rate regulator, Cosine, is a learning rate adjustment strategy based on the cosine function. The bf16 computation type is an efficient data format with short computation time and low storage requirements. After setting the parameters, the instruction fine-tuning dataset constructed in step S1 is input into the large language model for instruction fine-tuning.

[0076] In steps S2-3, during the instruction fine-tuning process, the large language model predicts the length of the response text based on the input text.

[0077] The trainable parameters introduced by the LoRA method are updated using the backpropagation algorithm. Simultaneously, the AdamW optimizer is employed to adjust the learning rate and other hyperparameters, continuously optimizing the performance of the large language model. Backpropagation starts with the loss value of the output layer and uses the chain rule to backpropagate the gradient of the loss with respect to the parameters of each layer, updating the model parameters based on the gradients to reduce the loss. Its core is to guide the adjustment of model parameters by calculating gradients. The AdamW optimizer is a commonly used parameter optimization algorithm; its specific adjustment and calculation process is encapsulated as a method that can be directly called. Its principle is to adjust the learning rate and other hyperparameters by combining an adaptive learning rate adjustment strategy and setting hyperparameters such as weight decay according to certain rules. The large language model is periodically evaluated, and based on the evaluation results, hyperparameters and data preprocessing strategies are adjusted until the large language model's performance reaches the expected effect. After model training, the trained LoRA-related weight files are merged with the base model weights and exported to generate a new model. The fine-tuned large language model is then saved. The saved large language model can be deployed and used to predict the response text length.

[0078] This embodiment, by fine-tuning a large language model and leveraging its powerful semantic understanding and modeling capabilities, can accurately predict the length of the model's output text at a relatively low cost, providing a foundation for response time prediction. By establishing a relationship model between response time and response text length, the response time of the large language model service can be obtained, thus providing a basis for service scheduling.

[0079] Step S3: Construct a relationship model between the response time of the fine-tuned large language model and the length of the obtained response text, and obtain an approximate solution for the response time of the large language model based on the relationship model.

[0080] The response time of a large language model is the total time required for the user to generate a complete response; this total time includes the time from when the user submits the request to when the complete text is generated. By establishing the relationship between the response time of the large language model and the length of the resulting response text, an approximate solution to the response time of the large language model can be obtained. Figure 5 As shown, this can be achieved through the following steps:

[0081] Step S3-1: Collect sample data for large language model request services. Each sample includes server resource information such as user input text length, model response text length, CPU / GPU utilization, and memory utilization. The sample label is the response time of the large model. Response time is the time elapsed from when the user submits a request to when the model returns the complete text.

[0082] Step S3-2: Clean and preprocess the collected data, such as handling outliers and missing values, and performing standardization.

[0083] Step S3-3: Construct decision tree regression, support vector regression, XGBoost regression, LightGBM regression, and KNN regression models respectively to train the preprocessed data, obtaining multiple independent prediction models. During this process, 5-fold cross-validation is used to avoid overfitting and improve the model's generalization ability; model parameters are then fine-tuned using a grid search method until the optimal hyperparameter combination is obtained, maximizing the prediction accuracy of the large language model.

[0084] Step S3-4: Evaluate the above large language model. Use an independent test dataset to evaluate the prediction accuracy of the trained large language model. Specifically, through experimental result analysis, select three high-performing models as the prediction base models; at the same time, use the average prediction value of the above three base models as the final response time prediction result to improve the stability and accuracy of the prediction.

[0085] The performance evaluation results of the five cross-validations are averaged to obtain the final evaluation result. Specifically, in this embodiment, the root mean square error (RMSE) is selected as the model performance evaluation index. The evaluation results of the model are shown in Table 2.

[0086] Table 2 Evaluation Results of the Large Language Model

[0087] Decision Tree Regression 3.32 Support Vector Regression 3.28 KNN returns 2.37 XGBoost Returns 1.66 LightGBM returns 1.31

[0088] Based on the above results, this embodiment selects KNN regression, XGBoost regression, and LightGBM regression, which have better performance, as the prediction base models. Simultaneously, the average prediction value of these three base models is used as the final response time prediction result to further improve the accuracy and stability of the large language model prediction. This integrated large language model will serve as the final prediction model for predicting new input data.

[0089] Step S4: Based on the known approximate solution of the response time of the large language model, process the service requests of the large language model in batches and set a batch scheduling strategy; send the service requests to the large language model according to the set batch scheduling strategy.

[0090] After obtaining an approximate solution for the response time, the large language model service requests are batched. The batching scheduling strategy is set based on three conditions: the request length, response length, and response time of the user's service request. The batching scheduling strategy is deployed in the large language model service scheduler, which has scheduling strategy options. Then, request expectations are set, and these expectations are used as the scheduling strategy options in the service scheduler. The batching scheduling strategy can be implemented through the following steps:

[0091] Step S4-1: Based on the approximate response time solution T obtained in the above steps... r Requests entering the service queue are processed in batches. In this embodiment, the specific batching strategy uses the K-Means clustering method, specifically considering the request length L. r Expected response length L output And the approximate solution T of the predicted response time r Constructing a three-dimensional feature vector (L r ,L output ,T r Using the K-Means algorithm, requests are divided into K groups. The number of groups, K, can be dynamically adjusted based on historical data analysis and system load; the number of requests within each group, B... k The number of requests can be dynamically determined based on the clustering results. To avoid excessively large request numbers within certain groups, which could affect scheduling efficiency, a maximum number of requests N within a group can be set to control the maximum size of the scheduling batch. The maximum number of requests N is set based on the system's maximum throughput.

[0092] Step S4-2, Introduce Request Expectation E i As a scheduling strategy option. Request expectation E i Predicted response time T from requests within the group r and the waiting time W for the request r The weighted calculation determines this, that is:

[0093]

[0094] Where α and β represent weight coefficients, satisfying 0≤α≤1, 0≤β≤1, and α+β=1. The weights can be dynamically adjusted according to the actual situation; the higher the request expectation, the higher the scheduling priority, that is, the system prioritizes processing batches with higher request expectations.

[0095] Step S4-3: Initialize the batch size B of the large language model. o Simultaneously, requests within each group are packaged into batches and processed in batches. Based on the expected request values ​​of each batch, E... i The scheduling priority is set; based on the scheduling priority of each batch, the service scheduler selects batches for processing in priority order. When an LLM (Large Language Model) instance completes batch processing and becomes idle, the service scheduler continues to select a target batch from the waiting queue according to the scheduling priority, and then schedules it to an LLM instance.

[0096] Step S4-4: System Evaluation. The performance of the scheduling system is evaluated using metrics such as average response time and average waiting time. Specifically, by monitoring and analyzing these metrics, the service quality of large language model service request scheduling can be effectively assessed, thereby continuously optimizing the batching strategy and scheduling algorithm to achieve optimal system performance.

[0097] Through the above process, this embodiment constructs an efficient batch processing scheduling method. By batching requests with similar request sizes and introducing request expectations as scheduling strategy options for the service scheduler, it can comprehensively consider request processing time and waiting time. On the one hand, it prevents batches with long service times from waiting in the queue for a long time, and on the other hand, it effectively reduces the average response time of requests and improves the overall service quality of the large language model.

[0098] Step S5: Introduce an error processing mechanism to compare the predicted response text length with the actual response text length output by the large language model after receiving the service request and the response text length currently generated in each batch, and determine the service response status of the large language model based on the comparison results.

[0099] Although the accuracy of the system's prediction of response text length and response time can be significantly improved after fine-tuning the large language model and establishing a model relating response time to response text length, prediction biases can still occur. To address this, this embodiment introduces an error handling mechanism to capture and promptly optimize handling of anomalies, thereby improving system robustness. By continuously monitoring the batch response status of the large language model service, the system automatically triggers the error handling mechanism upon detecting any anomalies, collecting real-time data on request processing anomalies and initiating corresponding optimization measures. This error handling mechanism enables the capture and timely optimization of anomalies, enhancing system robustness.

[0100] In actual implementation, the system monitors and predicts the response text length L in real time. output Actual response text length The current length L of the generated text in the batch MAX The differences between them are used to determine the service response status of the large language model. Specifically, an upper limit for identifiers and a text length threshold are set. When, in any batch of service requests, the output identifier of a request exceeds the set upper limit for identifiers, and the difference between the current maximum generated text length and the minimum generated text length corresponding to the already generated requests in that batch exceeds the set text length threshold, the inference service for that batch is terminated. Preferably, the upper limit for identifiers and the text length threshold are set to δ and D, respectively. When more than δ requests output [end] identifiers in a batch, and the current maximum generated text length L... MAX The minimum generated text length L of the δ requests already generated in the batch. δ When the difference exceeds the text length threshold D, the system determines it as a prediction error, terminates the inference service for that batch, and marks the set R of requests that have not completed inference. F .

[0101] Furthermore, abnormal requests are collected in real time and packaged into batches; through iterative processing, new abnormal request batches R are formed. F(r) When the percentage of requests x that have been processed within the same batch exceeds P, the inference service for that batch is terminated, and the set R of requests that have not been inferred is marked. F(r-x) Where P represents the threshold that can be set in the exception handling mechanism.

[0102] To continuously improve the service quality of the large language model, this embodiment employs an iterative optimization mechanism to continuously optimize the parameters and model in the above steps. Based on server load and actual response time, parameters such as batch size, batching strategy, and weighting coefficients are dynamically adjusted to achieve optimal scheduling. This process is iteratively executed until all requests are completed. Furthermore, after each iteration, the system generates a set R of completed requests. F(x) Feedback is sent to the user; specifically, the implementation process of the iterative algorithm is as follows: Figure 6 As shown. Due to the inherent response length limitation of large language model services, requests exceeding the maximum response length will be automatically completed, avoiding infinite iterations. Through the above error handling mechanism, this invention can effectively avoid scheduling costs caused by errors in response text length prediction, ensuring stable system operation and high-quality service even in the face of complex requests and abnormal response text lengths. Furthermore, this invention does not conflict with other existing inference acceleration technologies, effectively supplementing existing large language model inference acceleration toolkits, enhancing their functionality and performance, and further improving the overall performance of large model services; this complementarity provides new directions and possibilities for future research and development in related fields.

[0103] Furthermore, to verify the overall effectiveness of the large language model service request scheduling method provided by this invention, the following two verification experiments were conducted in this embodiment:

[0104] Experiment 1 verifies the performance of fine-tuned large language models in predicting response text length. Specifically, the comparison models selected in this embodiment include the un-fine-tuned large language model and other prediction models. For the un-fine-tuned large language model, the effectiveness of fine-tuning is directly verified by comparing the performance difference between the fine-tuned model and the same un-fine-tuned baseline large language model in predicting response text length. For other prediction models, comparing the performance of the fine-tuned model with other models used for predicting text length helps to analyze the relative advantages and disadvantages of different prediction models in solving the same task. The selected models include decision trees, Bi-LSTM, and Bi-LSTM+attention.

[0105] This embodiment constructs and enhances features based on user prompts, and predicts text length by mining the relationship between user input and response text length. In the comparative experiment, Acc±25 and Acc±50 were selected as evaluation metrics, representing the accuracy of the prediction results within 25 and 50 ranges above and below the label value, respectively. The experimental results are shown in Table 3; the experiment used ChatGLM3-6B as the baseline large language model.

[0106] Table 3 Comparison Results of Experiment 1

[0107] Untuned ChatGLM3-6B 31％ 52％ Decision Tree 11％ 19％ Bi-LSTM 16％ 34％ Bi-LSTM+attention 21％ 38％ The finely tuned ChatGLM3-6B 43％ 66％

[0108] Analysis of the experimental results shows that the fine-tuned ChatGLM3-6B significantly outperforms the untuned ChatGLM3-6B, achieving improvements of 12% and 14% in Acc±25 and Acc±50 respectively, indicating that fine-tuning effectively improves the model's prediction accuracy. Furthermore, compared to decision trees, Bi-LSTM, and Bi-LSTM+attention, the fine-tuned ChatGLM3-6B and the untuned ChatGLM3-6B demonstrate that the large language model performs better in both Acc±25 and Acc±50 metrics, indicating that its accuracy in predicting response text length is significantly higher than these traditional models.

[0109] Experiment 2 verifies the performance of the large-model service scheduling method and error handling mechanism proposed in this invention. The following experimental setup was performed before the verification in this embodiment:

[0110] (1) An experimental environment was built based on 6 cloud servers in the cloud platform, of which 2 servers were used to deploy the finely tuned large model and the service request scheduler to ensure request processing capacity.

[0111] (2) Design a request generator to simulate concurrent requests from different services in the experiment. The request generator can be dynamically adjusted according to the set parameters (such as the number of requests, the request interval, the request type, etc.) to simulate the load situation in real scenarios; among them, the request types cover a variety of scenarios such as text dialogue, comparative analysis and translation.

[0112] (3) Set experimental evaluation indicators, mainly including: Average Response Time (ART), which is the average time required from submitting a request to receiving a result; and Average Wait Time (AWT), which is the average time required from submitting a request to the start of processing the request.

[0113] (4) Selection of comparison method: This embodiment selects commonly used static batch processing scheduling algorithms as the benchmark, including the naive first-come-first-served scheduling method, the shortest job first method, and the priority scheduling method, so as to better evaluate the relative performance advantages of the scheduling method proposed in this embodiment. The first-come-first-served scheduling method focuses on the arrival order of requests; the shortest job first method focuses on the size of requests; and the priority scheduling method focuses on the priority of requests, which is determined by the request expectation of a single request.

[0114] Furthermore, this embodiment selected a validation set of 3000 sample datasets generated from modified Alpaca datasets and seed samples, covering requests of different lengths and types. Experimental results are shown in Table 4:

[0115] Table 4 Comparison Results of Experiment 2

[0116]

[0117]

[0118] Analysis of the experimental results shows that the scheduling method and error handling mechanism of the present invention significantly reduce the average response time and average waiting time, indicating that by effectively constructing and scheduling request batches, the overall efficiency and service quality of large model services can be improved, providing strong support for the feasibility of further promoting the application of this method in practice.

[0119] Example 2

[0120] This embodiment discloses a large language model service request scheduling system.

[0121] A large language model service request scheduling system includes:

[0122] The instruction fine-tuning dataset generation module is configured to: build instruction fine-tuning datasets;

[0123] The response text length prediction module is configured to: fine-tune the dataset according to the instructions, fine-tune the parameters of the large language model using an efficient parameter fine-tuning method, and predict the response text length based on the fine-tuned large language model;

[0124] The relational model building module is configured to: build a relational model between the response time of the fine-tuned large language model and the length of the resulting response text, and obtain an approximate solution for the response time of the large language model based on the relational model;

[0125] The batch processing module is configured to: based on the known approximate solution of the response time of the large language model, process the service requests of the large language model in batches and set a batch scheduling strategy; and send service requests to the large language model according to the set batch scheduling strategy.

[0126] The service request scheduling module is configured to introduce an error handling mechanism, which compares the predicted response text length with the actual response text length output by the large language model after receiving the service request and the response text length currently generated in each batch, and determines the service response status of the large language model based on the comparison results.

[0127] like Figure 7As shown, the large language model service request scheduling system provided by this invention is placed in the control chain layer of the entire cloud environment and is used to control the large language model service scheduling strategy. The control chain layer interacts with the application layer and the application access layer upwards in sequence; and with the cloud service cluster layer downwards. Among them, the user-visible layer is the topmost layer, which is the application access layer.

[0128] Example 3

[0129] The purpose of this embodiment is to provide a computer-readable storage medium.

[0130] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a large language model service request scheduling method as described in Embodiment 1 of this disclosure.

[0131] Example 4

[0132] The purpose of this embodiment is to provide an electronic device.

[0133] An electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of a large language model service request scheduling method as described in Embodiment 1 of this disclosure.

[0134] The steps and methods involved in the apparatuses of Embodiments 2, 3, and 4 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0135] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0136] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A method for scheduling service requests for a large language model, characterized in that, include: Build a fine-tuning dataset for instructions; Based on the instructions, the dataset is fine-tuned, and the parameters of the large language model are fine-tuned using an efficient parameter fine-tuning method. Predict the length of the response text based on the fine-tuned large language model; A relationship model is constructed between the response time of the fine-tuned large language model and the length of the resulting response text. Based on this relationship model, an approximate solution for the response time of the large language model is obtained. Specifically, sample data of large language model request services is collected. Each sample includes server operating resource information, including the length of the user input text, the length of the model response text, CPU / GPU utilization, and memory utilization. The sample label is the response time of the large model, which is the time elapsed from when the user submits a request to when the model returns the complete text. The collected data is then cleaned and preprocessed. Decision tree regression, support vector regression, XGBoost regression, LightGBM regression, and KNN regression models were constructed and trained on the preprocessed data to obtain multiple independent prediction models. Based on the performance of the multiple independent prediction models, KNN regression, XGBoost regression, and LightGBM regression were selected as the prediction base models. At the same time, the average prediction value of the above three prediction base models was used as the final response time prediction result. Based on the known approximate solution of the response time of the large language model, the service requests of the large language model are processed in batches, and a batch scheduling strategy is set; according to the set batch scheduling strategy, the service requests are sent to the large language model. The batch scheduling strategy is set based on three conditions: the request length, response length, and response time of the service request sent by the user. Setting the batch scheduling strategy involves comprehensively considering the request length, expected response length, and approximate predicted response time. The service requests are grouped using K-Means clustering, and request expectations are introduced as scheduling strategy options. The batch size of the large language model is initialized. Simultaneously, requests within each group are packaged into batches and processed in batches. Scheduling priorities are set according to the request expectations of each batch. Based on the scheduling priorities of each batch, the service scheduler selects batches for processing in priority order. An error handling mechanism is introduced to compare the predicted response text length with the actual response text length output by the large language model after receiving the service request and the response text length currently generated in each batch. The service response status of the large language model is determined by the comparison results. The service response status of the large language model is determined based on the obtained comparison results, including: setting the upper limit of the identifier and the text length threshold as δ and D, respectively; when more than δ requests in the batch output the end identifier, and the current maximum generated text length L... MAX The minimum generated text length L of the δ requests already generated in the batch. δ When the difference exceeds the text length threshold D, the system determines it as a prediction error, terminates the inference service for that batch, and marks the set R of requests that have not completed inference. F ; For abnormal requests, they are collected in real time and packaged into batches; through iterative processing, new abnormal request batches are formed. F(r) When the percentage of requests x that have been processed within the same batch exceeds P, the inference service for that batch is terminated, and the set R of requests that have not been inferred is marked. F(r-x) Where P represents the threshold that can be set in the exception handling mechanism; The above process is executed iteratively until all requests are completed; and, after each iteration, the system generates a set R of completed requests. F(x) Feedback was sent to the user.

2. The method for scheduling large language model service requests as described in claim 1, characterized in that, The instruction fine-tuning dataset contains data pairs, which include instruction requirements, input text, and corresponding response text length labels.

3. The method for scheduling large language model service requests as described in claim 1, characterized in that, An efficient parameter fine-tuning method is used to fine-tune the parameters of the large language model. Specifically, a low-rank adaptive method is used to optimize the parameters in the large language model.

4. The method for scheduling large language model service requests as described in claim 1, characterized in that, The response time of the large language model is the total time required for the user to generate a complete response; where the total time required for the user to generate a complete response includes the sum of the time from when the user submits the request to when the complete text is generated.

5. The method for scheduling large language model service requests as described in claim 1, characterized in that, The batch scheduling strategy is deployed in the service scheduler of the large language model. The service scheduler has scheduling strategy options. Request expectations are set, and the set request expectations are used as the scheduling strategy options in the service scheduler.

6. A large language model service request scheduling system, characterized in that, The method for scheduling large language model service requests according to any one of claims 1-5 includes: The instruction fine-tuning dataset generation module is configured to: build instruction fine-tuning datasets; The response text length prediction module is configured to: fine-tune the dataset according to the instructions, fine-tune the parameters of the large language model using an efficient parameter fine-tuning method, and predict the response text length based on the fine-tuned large language model; The relational model building module is configured to: build a relational model between the response time of the fine-tuned large language model and the length of the resulting response text, and obtain an approximate solution for the response time of the large language model based on the relational model; The batch processing module is configured to: based on the known approximate solution of the response time of the large language model, process the service requests of the large language model in batches and set a batch scheduling strategy; and send service requests to the large language model according to the set batch scheduling strategy. The service request scheduling module is configured to introduce an error handling mechanism, which compares the predicted response text length with the actual response text length output by the large language model after receiving the service request and the response text length currently generated in each batch, and determines the service response status of the large language model based on the comparison results.

7. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of a large language model service request scheduling method as described in any one of claims 1-5.

8. An electronic device, comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the large language model service request scheduling method as described in any one of claims 1-5.

Images