Large language model job scheduling method and system based on reply length prediction
By predicting the response length of large language models using an embedding model and an LSTM regressor, and combining resource-aware scheduling and dynamic scaling, the problem of resource allocation imbalance caused by the uncertainty of LLM response length is solved, thereby improving system throughput and service stability.
Patent Information
- Application Number
- CN202511712202.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-20
- Publication Date
- 2026-03-03
AI Technical Summary
The reasoning process of Large Language Models (LLM) is computationally intensive, consumes a lot of GPU memory, and the uncertainty of response length makes resource scheduling difficult. Existing scheduling methods lack accurate prediction capabilities, which causes long response requests to block the execution of short requests, frequently triggering preemption operations and affecting system throughput and service quality.
Dense semantic embeddings are extracted using the Embedding model, and temporal feature sequences are constructed by combining multi-turn or single-turn dialogue scenarios. LSTM regressors are used to predict response lengths, and a resource-aware intelligent scheduling algorithm is built to dynamically expand and re-verify, prioritize scheduling short jobs, ensure that KV cache resources do not exceed limits, and avoid frequent preemption.
It improved system throughput, reduced short job waiting latency, reduced request header blocking, and enhanced service stability and the adaptability of scheduling strategies.
Smart Images
Figure CN121597797A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of service computing technology, and in particular to a large language model job scheduling method and system based on response length prediction. Background Technology
[0002] Large Language Models (LLMs), a natural language processing technology based on the Transformer architecture, have been widely applied in online service scenarios such as intelligent customer service and code generation, becoming a key technology in the field of service computing, thanks to their powerful text generation and context understanding capabilities. However, the LLM inference process is characterized by computational intensity, high memory consumption, and sensitivity to response latency. Furthermore, the length of the response to a user request is highly uncertain. This uncertainty makes it difficult for the system to accurately predict the resource consumption of a single request, thus posing challenges to resource scheduling and service quality.
[0003] To address these challenges, existing technologies typically deploy LLMs as elastic services on remote server clusters and provide on-demand access capabilities through application programming interfaces (APIs). This model supports dynamic scaling, multi-tenant isolation, and other features to improve resource utilization. Furthermore, inference system-level optimization strategies (e.g., continuous batch processing, KV caching) are widely adopted to improve hardware resource utilization by dynamically merging requests or avoiding redundant computations of historical context.
[0004] However, existing technologies still have significant drawbacks: on the one hand, mechanisms based on dynamic scaling suffer from cold start latency, making it difficult to handle high-concurrency requests; on the other hand, existing scheduling methods (such as first-come, first-served or simple sorting based on input length) lack the ability to accurately predict response lengths, failing to introduce resource-aware intelligent decision-making at the scheduling level. This leads to long-response requests easily blocking the execution of short requests, causing Header-of-Order (HOL) blocking issues, and frequently triggering preemption operations when memory is scarce, resulting in additional context switching overhead and limiting system throughput and service quality. Summary of the Invention
[0005] This application provides a large language model job scheduling method and system based on response length prediction. The intelligent scheduling mechanism based on response length prediction effectively improves system throughput and service stability through semantic awareness and dynamic expansion.
[0006] On the one hand, this application provides a large language model job scheduling method based on response length prediction, the method comprising: Receive user request text and determine whether the current dialogue scenario is a single-turn dialogue scenario or a multi-turn dialogue scenario; If the current dialogue scenario is a multi-turn dialogue scenario, then the semantic similarity between the current request text and the historical dialogue context is calculated by combining the current request text with the historical request of each turn, and a temporal feature sequence is constructed based on the semantic similarity and the corresponding historical response length. If the current dialogue scenario is a single-turn dialogue scenario, then based on the current user request text, the top-k most semantically similar historical requests are retrieved from the local cache, and an equivalent temporal feature sequence is constructed based on the response length and similarity of the top-k historical requests. The Embedding model is used to extract dense semantic embeddings from the user request text, and the temporal feature sequence is input into a pre-trained Long Short-Term Memory (LSTM) network regressor to predict the length of the LLM response tokens corresponding to the current request. Based on the predicted response term length, a resource-aware intelligent scheduling algorithm is constructed. The predicted response term length is used as a priority. By simulating the operation process of the job to be scheduled, it is determined that the job will not cause preemption. This algorithm is used to determine the set of requests participating in batch processing in each round of scheduling. A dynamic scaling and revalidation algorithm is constructed. If the predicted response term length is lower than the actual length, the response term length is binned according to the predicted response term length, the job budget is dynamically increased, and a virtual job is constructed for revalidation.
[0007] On the other hand, this application provides a large language model job scheduling system based on response length prediction, the system comprising: The judgment module is used to receive user request text and determine whether the current dialogue scenario is a single-turn dialogue scenario or a multi-turn dialogue scenario; The first construction module is used to calculate the semantic similarity between the current request and each round of historical requests by combining the current request text and the historical dialogue context if the current dialogue scenario is a multi-turn dialogue scenario, and to construct a temporal feature sequence based on the semantic similarity and the corresponding historical response length. The second construction module is used to, if the current dialogue scenario is a single-turn dialogue scenario, recall the top-k most semantically similar historical requests in the local cache based on the current user request text, and construct an equivalent temporal feature sequence based on the response length and similarity of the top-k historical requests. The prediction module is used to extract dense semantic embeddings from the user request text using the Embedding model, and input the temporal feature sequence into a pre-trained Long Short-Term Memory (LSTM) regressor to predict the length of the LLM response tokens corresponding to the current request. The simulation module is used to construct a resource-aware intelligent scheduling algorithm based on the predicted response word length. It uses the predicted response word length as a priority and simulates the operation process of the job to be scheduled to determine that the job will not cause preemption. It is used to determine the set of requests participating in batch processing in each round of scheduling. The update module is used to build a dynamic expansion and re-verification algorithm. If the predicted response token length is lower than the actual length, the response tokens are binned according to the predicted response token length, the job budget is dynamically increased, and a virtual job is built for re-verification.
[0008] Thirdly, this application provides an apparatus comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the technical solution of the topology optimization design method integrating subjective preferences as described above.
[0009] Fourthly, this application provides a storage medium storing a computer program, which, when executed by a processor, implements the steps of the technical solution of the topology optimization design method integrating subjective preferences as described above.
[0010] As can be seen from the technical solution provided in this application, on the one hand, by using an embedding model to extract dense semantic embeddings and constructing temporal feature sequences in combination with multi-turn or single-turn dialogue scenarios, and then predicting the response length using an LSTM regressor, the semantic relevance and length evolution trend of requests can be captured, thereby providing more reliable load estimation for subsequent scheduling and alleviating resource allocation imbalance caused by length uncertainty. On the other hand, by using the response token length as a feature for job priority, the system can prioritize scheduling short jobs. Combined with resource-aware simulation verification, it ensures that KV cache resources do not exceed limits at any time, which helps reduce the waiting latency of short jobs, reduce request header blocking, and thus improve the overall system throughput. On the third hand, based on the constructed dynamic expansion algorithm, when the predicted response token length is lower than the actual length, the budget is dynamically adjusted and re-verified, thereby dealing with prediction errors, avoiding frequent preemption caused by insufficient budget, improving the adaptability of the scheduling strategy, and thus ensuring service stability. In summary, the technical solution of this application, based on the intelligent scheduling mechanism of response length prediction, effectively improves system throughput and service stability through semantic awareness and dynamic expansion. Attached Figure Description
[0011] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0012] Figure 1 This is a system framework diagram of the large language model job scheduling method based on response length prediction provided in the embodiments of this application; Figure 2 This is a schematic diagram of the large language model job scheduling method based on response length prediction provided in the embodiments of this application; Figure 3 This is a flowchart of the prediction and training process of the job response length prediction model based on Embedding provided in the embodiments of this application; Figure 4 This is a flowchart of the resource-aware intelligent scheduling algorithm based on predicted response length provided in the embodiments of this application; Figure 5 This is a flowchart of the dynamic expansion and re-verification algorithm provided in the embodiments of this application; Figure 6 This is a schematic diagram of the structure of a large language model job scheduling system based on response length prediction provided in an embodiment of this application; Figure 7 This is a schematic diagram of the device structure provided in the embodiments of this application. Detailed Implementation
[0013] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.
[0014] In this specification, adjectives such as "first" and "second" are used only to distinguish one element or action from another, without necessarily requiring or implying any actual such relationship or order. Where circumstances permit, reference to an element or component or step (etc.) should not be construed as being limited to only one of the elements, components, or steps, but may be one or more of the elements, components, or steps, etc.
[0015] For ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn to actual scale.
[0016] like Figure 1The diagram shown is a system framework diagram of a large language model job scheduling method based on response length prediction provided in this application embodiment. After a user request arrives at the model service instance, temporal features are constructed according to the request type and the response length is predicted. Subsequently, the user request and its corresponding request length are added to the job priority request management module. Then, during the model inference process, jobs with excessively short predictions are dynamically expanded and re-verified, and new inference batches are constructed for jobs to be scheduled based on load conditions. Through job priority management and intelligent scheduling, this application significantly reduces job latency while avoiding frequent job preemption.
[0017] To address the aforementioned problems in the prior art, this application provides a large language model job scheduling method based on response length prediction, the specific implementation process of which is as follows: Figure 2 As shown, steps S201 to S206 are detailed below: Step S201: Receive the user request text and determine whether the current dialogue scenario is a single-turn dialogue scenario or a multi-turn dialogue scenario.
[0018] In this embodiment, the user request text refers to the user submitting a textual request as a large model inference task, waiting for the model to respond. The request is in the form of a dialogue, meaning the user sends a request to receive a model response, followed by a multi-round interaction process, which can include tasks such as text translation, entity recognition, and code generation. Multi-round dialogue scenarios and single-round dialogue scenarios are distinguished based on the number of interaction rounds. If the request only contains the user's request and has no historical dialogue, it is a single-round dialogue scenario; otherwise, it is a multi-round dialogue scenario.
[0019] Step S202: If the current dialogue scenario is a multi-turn dialogue scenario, then combine the current request text with the historical dialogue context to calculate the semantic similarity between the current request and the historical requests of each turn, and construct a temporal feature sequence based on the semantic similarity and the corresponding historical response length.
[0020] Specifically, as an embodiment of this application, if the current dialogue scenario is a multi-turn dialogue scenario, then by combining the current request text with the historical dialogue context, the semantic similarity between the current request and the historical requests of each turn is calculated, and based on the semantic similarity and the corresponding historical response length, a temporal feature sequence can be constructed as follows: based on the current turn user request text Corresponding semantic embedding vector With historical request texts Corresponding semantic embedding vector Historical response texts Corresponding semantic embedding vector Calculating semantic similarity includes using the following formula: and Similar input : , Calculate using the following formula and Output similarity : , in, Represents the cosine similarity function; the input length of each round of historical interactions. Output length Input similarity Output similarity Concatenate into a four-dimensional feature vector , constitutes a time-series feature sequence .
[0021] Step S203: If the current dialogue scenario is a single-turn dialogue scenario, then based on the current user request text, recall the top-k most semantically similar historical requests in the local cache, and construct an equivalent temporal feature sequence based on the response length and similarity of the top-k historical requests.
[0022] For single-turn dialogues, since there is no historical dialogue sequence, the construction of the temporal feature sequence differs from that in multi-turn dialogue scenarios. Specifically, if the current dialogue scenario is a single-turn dialogue scenario, based on the current user request text, the top-k semantically most similar historical requests are retrieved from the local cache, and an equivalent temporal feature sequence is constructed based on the response length and similarity of these top-k historical requests. This can be achieved by pre-caching completed historical single-turn dialogue samples in a local vector database, with each sample including the request text. and its corresponding semantic embedding vector Corresponding reply text and its corresponding semantic embedding vector and the known token length and When the current single-round history request text is received At that time, use the Embedding model to generate Embedded vector And perform vector similarity retrieval in the local cache to recall the top-k historical single-round request samples that are semantically most similar to the current request; request each recalled sample... The current single-round history request text is calculated according to the following formula. With the recall sample request Input similarity Output similarity : , , in, Indicates cosine similarity; Using the known data from this sample and Calculated and Construct four-dimensional feature vectors ; Analyze the top-k feature vectors corresponding to the top-k historical single-round request samples according to input similarity Sort from high to low to form an equivalent time-series feature sequence. .
[0023] Step S204: The Embedding model is used to extract dense semantic embeddings from the user request text, and the temporal feature sequence is input into the pre-trained Long Short-Term Memory (LSTM) network regressor to predict the length of the LLM response tokens corresponding to the current request.
[0024] As one embodiment of this application, the dense semantic embedding extracted from the user request text using the Embedding model can be as follows: Figure 2 As shown, specifically, extracting dense semantic embeddings from user request text using the Embedding model can be achieved by splitting the user interaction history dialogue into several rounds, each round containing the historical request text. and corresponding historical reply text ,in, , For the current round of user requests; for the text of the current round of user requests. Historical request texts for each round and LLM history reply text Input each element into the pre-trained embedding model to obtain the corresponding semantic embedding vector. , and ; Obtain the word length of each text, denoted as and This serves as a statistical characteristic of the length of historical dialogues.
[0025] The temporal feature sequence (including the temporal feature sequence constructed in multi-turn dialogue scenarios and the equivalent temporal feature sequence constructed in single-turn dialogue scenarios) is input into a pre-trained Long Short-Term Memory (LSTM) regressor to predict the length of the LLM response tokens corresponding to the current request. By extracting similarity and length features, the prediction model can perceive similarity information and length trends, achieving more accurate predictions.
[0026] Step S205: Based on the predicted response term length, a resource-aware intelligent scheduling algorithm is constructed. The predicted response term length is used as a priority. By simulating the operation process of the job to be scheduled, it is determined that the job will not cause preemption. This algorithm is used to determine the set of requests participating in batch processing in each round of scheduling.
[0027] As one embodiment of this application, a resource-aware intelligent scheduling algorithm is constructed based on the predicted response token length. Using the predicted response token length as a priority, the algorithm simulates the execution process of the jobs to be scheduled to determine that the jobs will not trigger preemption. The set of requests participating in batch processing in each round of scheduling can be: requests to the job request pool are prioritized according to the predicted response token length; shorter response token lengths result in higher priority and are scheduled first. During each round of scheduling, for each candidate request, the priority is determined based on the currently occupied key-value cache (KV) in the system. The system uses cache resources, the state of currently running requests, and the input length and predicted response token length of those requests to perform a security simulation to simulate the occupancy of key-value cache resources by candidate requests during future iterations. During the simulation, the number of active requests is dynamically tracked, and at the end of each request's execution, a check is performed to determine whether the total key-value cache resource usage at any given time exceeds a preset maximum capacity limit if the candidate request is added to the current batch. A candidate request is only included in the current scheduling batch if the key-value cache resource usage does not exceed the capacity limit throughout the entire simulation. A maximum consecutive failure threshold is set for each scheduling iteration. When multiple consecutive candidate requests are rejected due to insufficient resources, if the maximum consecutive failure threshold is reached, the current scheduling process is terminated, the selected request is locked, and the system enters the inference execution phase.
[0028] In the above embodiments, a resource-aware intelligent scheduling algorithm can be constructed based on the predicted response lexical length. Figure 4 The example dynamically constructs a batch description of jobs to be scheduled, specifically including steps S2051 to S2054, which are detailed below: Step S2051: Obtain the requests from the job request pool, sort them by predicted response length as priority, and obtain the priority job list:
[0029] Where Q is the priority queue, This is the input request text. This is the length of the input request. The length of the predicted response tokens is denoted by m. Shorter tokens have higher priority. The total queue size is m.
[0030] Step S2052: Obtain the current system status, including the remaining KV cache capacity of the current system. Job priority queue ,as well as The state of the request that is currently being inferred and executed is represented as:
[0031] in, Indicates a request The remaining generation length, There are t requests in the reasoning process. by The shorter the remaining length, the higher the priority.
[0032] Step S2053: Simulate the inference process of each request to be scheduled, confirm that scheduling the request will not cause any request to be preempted, and add it to the batch to be scheduled.
[0033] The reasoning process of simulating each scheduled request can be represented as the scheduled job:
[0034] Will Add to ,get:
[0035] Check the key-value usage at the end of each job in Q until a request is detected. .
[0036] For each The following inequality constraints must be satisfied when the request ends:
[0037] in, For homework Batch size at the end, After verification Decrease and update :
[0038]
[0039] If and only if scheduling This will not cause job preemption. Add to batch. If scheduling fails, Move to end.
[0040] Step S2054: Repeat step S2053 until... Or the number of simulation failures exceeds Then, the batch will be submitted for inference.
[0041] Step S206: Construct a dynamic expansion and revalidation algorithm. If the predicted response token length is lower than the actual length, bin the responses according to the predicted response token length, dynamically increase the job budget, and construct a virtual job for revalidation.
[0042] As one implementation of this application, a dynamic scaling and re-verification algorithm is constructed. If the predicted response token length is lower than the actual length, binning is performed based on the predicted response token length, and the job budget is dynamically increased and a virtual job is constructed for re-verification. This can be achieved as follows: If the model response length is found to exceed the original prediction value during the actual generation process, binning is performed based on the number of generated tokens (e.g., divided into intervals such as [0–16], [16–32], [32–128], [128–256], etc.), with each interval corresponding to a preset scaling increment (e.g., 16, 32, 128, 256). When the generated length is detected to be close to the prediction limit, a virtual job is automatically constructed (i.e., the input length is 0, and the predicted generated length is the current generated length plus the corresponding interval scaling value), and the security simulation is re-executed. If the simulation after budget expansion still meets the resource constraints, the current request continues to be executed; otherwise, it is preempted and the allocated resources are released. It should be noted that the "current request" in "continue to execute the current request" refers to a request that can continue to be executed safely after the predicted response token length is shorter than the actual length and after scaling verification. The specific implementation plan for step S206 can be as follows: Figure 5 As shown, this is mainly because the prediction model has unavoidable errors. If the predicted response length is too long, the request will end early and release resources to the waiting request; if the predicted length is too short, it may cause the previous scheduling algorithm to fail. To address this, a dynamic scaling and re-verification algorithm is constructed to automatically scale the job budget. This can be implemented through steps S2061 to S2063, as detailed below: Step S2061: Obtain the remaining KV cache space and the queue of requests that have been inferred. Based on the current generation length and the expected generation length, filter out the requests that have reached the limit and have not yet been completed, and obtain the queue:
[0043] Step S2062: For The request in the middle queries the budget extension table T to obtain the bucketed extended budget based on the predicted response length. A new queue is obtained:
[0044] Step S2063: Construct a virtual job for the request with the expanded budget and call the simulation run algorithm in step S2053 for re-verification. This virtual job is represented as follows:
[0045] Once the verification is successful, the job can be retained and the extended predicted response length can be updated; otherwise, the job will be preempted. This ensures the effectiveness of the scheduling algorithm and prevents frequent preemption, thereby further improving and reducing the average job latency.
[0046] From the above appendix Figure 2 As illustrated by the example of large language model job scheduling based on response length prediction, on the one hand, by employing an embedding model to extract dense semantic embeddings and constructing temporal feature sequences in conjunction with multi-turn or single-turn dialogue scenarios, and then using an LSTM regressor to predict response length, the semantic relevance and length evolution trend of requests can be captured. This provides more reliable load estimation for subsequent scheduling, alleviating resource allocation imbalances caused by length uncertainty. On the other hand, by using response token length as a feature for job priority, the system can prioritize scheduling shorter jobs. Combined with resource-aware simulation verification, this ensures that KV cache resources do not exceed limits at any time, helping to reduce waiting latency for short jobs, reduce request header blocking, and thus improve the overall system throughput. Thirdly, based on the constructed dynamic scaling algorithm, the budget is dynamically adjusted and re-verified when the predicted response token length is lower than the actual length. This can cope with prediction errors, avoid frequent preemption caused by insufficient budget, improve the adaptability of the scheduling strategy, and thus ensure service stability. In summary, the technical solution of this application, based on an intelligent scheduling mechanism for response length prediction, effectively improves system throughput and service stability through semantic awareness and dynamic scaling.
[0047] Please see the appendix Figure 6 This application provides a large language model job scheduling system based on response length prediction. The system may include a judgment module 601, a first construction module 602, a second construction module 603, a prediction module 604, a simulation module 605, and an update module 606, as detailed below: The judgment module 601 is used to receive the user request text and determine whether the current dialogue scenario is a single-turn dialogue scenario or a multi-turn dialogue scenario. The first construction module 602 is used to calculate the semantic similarity between the current request and the historical requests in each round if the current dialogue scenario is a multi-turn dialogue scenario, and to construct a temporal feature sequence based on the semantic similarity and the corresponding historical response length. The second construction module 603 is used to recall the top-k most semantically similar historical requests in the local cache based on the current user request text if the current dialogue scenario is a single-turn dialogue scenario, and to construct an equivalent temporal feature sequence based on the response length and similarity of the top-k historical requests. Prediction module 604 is used to extract dense semantic embeddings from user request text using the Embedding model, and input the temporal feature sequence into a pre-trained Long Short-Term Memory (LSTM) network regressor to predict the length of LLM response tokens corresponding to the current request. The simulation module 605 is used to construct a resource-aware intelligent scheduling algorithm based on the predicted response term length. It uses the predicted response term length as a priority and determines that the job will not cause preemption by simulating the running process of the job to be scheduled. It is used to determine the set of requests participating in batch processing in each round of scheduling. The update module 606 is used to build a dynamic expansion and revalidation algorithm. If the predicted response token length is lower than the actual length, the response tokens are binned according to the predicted response token length, the job budget is dynamically increased, and a virtual job is built for revalidation.
[0048] From the above appendix Figure 6 As illustrated in the example of the large language model job scheduling system based on response length prediction, on the one hand, by employing an embedding model to extract dense semantic embeddings and constructing temporal feature sequences in conjunction with multi-turn or single-turn dialogue scenarios, and then using an LSTM regressor to predict response length, the system can capture the semantic relevance and length evolution trend of requests, thereby providing more reliable load estimation for subsequent scheduling and alleviating resource allocation imbalances caused by length uncertainty. On the other hand, by using response token length as a feature for job priority, the system can prioritize scheduling shorter jobs. Combined with resource-aware simulation verification, it ensures that KV cache resources do not exceed limits at any time, helping to reduce waiting latency for short jobs, reduce request header blocking, and thus improve the overall system throughput. Thirdly, based on the constructed dynamic expansion algorithm, when the predicted response token length is lower than the actual length, the budget is dynamically adjusted and re-verified, thereby addressing prediction errors, avoiding frequent preemption caused by insufficient budget, improving the adaptability of the scheduling strategy, and thus ensuring service stability. In summary, the technical solution of this application, based on an intelligent scheduling mechanism for response length prediction, effectively improves system throughput and service stability through semantic awareness and dynamic expansion.
[0049] Figure 7 This is a schematic diagram of the structure of a device provided in one embodiment of this application. For example... Figure 7 As shown, the device 7 in this embodiment mainly includes: a processor 70, a memory 71, and a computer program 72 stored in the memory 71 and executable on the processor 70, such as a program for a large language model job scheduling method based on response length prediction. When the processor 70 executes the computer program 72, it implements the steps in the above-described embodiment of the large language model job scheduling method based on response length prediction, for example... Figure 2The steps S201 to S206 are shown. Alternatively, when the processor 70 executes the computer program 72, it implements the functions of each module / unit in the above-described device embodiments, for example... Figure 6 The functions of the judgment module 601, the first construction module 602, the second construction module 603, the prediction module 604, the simulation module 605, and the update module 606 are shown.
[0050] For example, the computer program 72 of the large language model job scheduling method based on response length prediction mainly includes: receiving user request text and determining whether the current dialogue scenario is a single-turn dialogue scenario or a multi-turn dialogue scenario; if the current dialogue scenario is a multi-turn dialogue scenario, then combining the current request text with the historical dialogue context, calculating the semantic similarity between the current request and each round of historical requests, and constructing a temporal feature sequence based on the semantic similarity and the corresponding historical response length; if the current dialogue scenario is a single-turn dialogue scenario, then based on the current user request text, recalling the top-k most semantically similar historical requests in the local cache. An equivalent temporal feature sequence is constructed based on the response lengths and similarities of the top-k historical requests. A dense semantic embedding is extracted from the user request text using an embedding model, and the temporal feature sequence is input into a pre-trained Long Short-Term Memory (LSTM) regressor to predict the response token length of the current request using a Large Language Model (LLM). Based on the predicted response token length, a resource-aware intelligent scheduling algorithm is constructed. Using the predicted response token length as a priority, the algorithm simulates the execution process of the scheduled job to determine if the job will trigger preemption, thus determining the set of requests participating in batch processing in each round of scheduling. The computer program 72 can be divided into one or more modules / units, which are stored in memory 71 and executed by processor 70 to complete this application. One or more modules / units can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program 72 in device 7.For example, computer program 72 can be divided into the functions of a judgment module 601, a first construction module 602, a second construction module 603, a prediction module 604, a simulation module 605, and an update module 606 (a module in the virtual device). The specific functions of each module are as follows: The judgment module 601 is used to receive the user request text and determine whether the current dialogue scenario is a single-turn dialogue scenario or a multi-turn dialogue scenario; the first construction module 602 is used to, if the current dialogue scenario is a multi-turn dialogue scenario, combine the current request text with the historical dialogue context to calculate the semantic similarity between the current request and each round of historical requests, and construct a temporal feature sequence based on the semantic similarity and the corresponding historical response length; the second construction module 603 is used to, if the current dialogue scenario is a single-turn dialogue scenario, based on the current user request text, recall the top-k most semantically similar historical requests in the local cache, and according to T The equivalent temporal feature sequence is constructed from the response lengths and similarities of op-k historical requests; the prediction module 604 is used to extract dense semantic embeddings from the user request text using an embedding model, and input the temporal feature sequence into a pre-trained Long Short-Term Memory (LSTM) regressor to predict the response token length of the current request corresponding to the Large Language Model (LLM); the simulation module 605 is used to construct a resource-aware intelligent scheduling algorithm based on the predicted response token length, using the predicted response token length as a priority, and simulating the operation of the scheduled job to determine that the job will not trigger preemption, and to determine the set of requests participating in batch processing in each round of scheduling; the update module 606 is used to construct a dynamic expansion and re-verification algorithm, if the predicted response token length is lower than the actual length, binning is performed according to the predicted response token length, the job budget is dynamically increased, and a virtual job is constructed for re-verification.
[0051] Device 7 may include, but is not limited to, processor 70 and memory 71. Those skilled in the art will understand that... Figure 7 This is merely an example of device 7 and does not constitute a limitation on device 7. It may include more or fewer components than shown, or combine certain components, or different components. For example, the device may also include input / output devices, network access devices, buses, etc.
[0052] The processor 70 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.
[0053] The memory 71 can be an internal storage unit of the device 7, such as a hard disk or RAM of the device 7. The memory 71 can also be an external storage device of the device 7, such as a plug-in hard disk, Smart MediaCard (SMC), Secure Digital (SD) card, or Flash Card equipped on the device 7. Furthermore, the memory 71 can include both internal and external storage units of the device 7. The memory 71 is used to store computer programs and other programs and data required by the device. The memory 71 can also be used to temporarily store data that has been output or will be output.
[0054] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed. That is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above-described device can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.
[0055] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0056] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0057] In the embodiments provided in this application, it should be understood that the disclosed apparatus / device and method can be implemented in other ways. For example, the apparatus / device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.
[0058] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0059] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0060] If integrated modules / units are implemented as software functional units and sold or used as independent products, they can be stored in a storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program for the large language model job scheduling method based on response length prediction can be stored in a storage medium. When executed by a processor, this computer program can implement the steps of the various method embodiments described above, namely, receiving user request text and determining whether the current dialogue scenario is a single-turn or multi-turn dialogue scenario; if the current dialogue scenario is a multi-turn dialogue scenario, then combining the current request text with the historical dialogue context, calculating the semantic similarity between the current request and each round of historical requests, and constructing a temporal feature sequence based on the semantic similarity and the corresponding historical response length; if the current dialogue scenario is a single-turn... In dialogue scenarios, based on the current user request text, the top-k semantically most similar historical requests are retrieved from the local cache. An equivalent temporal feature sequence is constructed based on the response length and similarity of these top-k historical requests. An embedding model is used to extract dense semantic embeddings from the user request text, and the temporal feature sequence is input into a pre-trained Long Short-Term Memory (LSTM) regressor to predict the response token length of the current request using a Large Language Model (LLM). Based on the predicted response token length, a resource-aware intelligent scheduling algorithm is constructed. Using the predicted response token length as a priority, the algorithm simulates the execution process of the scheduled job to determine if the job will trigger preemption, thus determining the set of requests participating in batch processing in each round of scheduling. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or some intermediate form. Storage media can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the contents of storage media can be appropriately added to or removed according to the requirements of legislation and patent practice in a jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, storage media may not include electrical carrier signals and telecommunication signals.
[0061] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application. The specific embodiments described above further illustrate the purpose, technical solutions, and beneficial effects of this application. It should be understood that the above descriptions are merely specific embodiments of this application and are not intended to limit the protection scope of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.
Claims
1. A large language model job scheduling method based on response length prediction, characterized in that, include: Receive user request text and determine whether the current dialogue scenario is a single-turn dialogue scenario or a multi-turn dialogue scenario; If the current dialogue scenario is a multi-turn dialogue scenario, then the semantic similarity between the current request text and the historical dialogue context is calculated by combining the current request text with the historical request of each turn, and a temporal feature sequence is constructed based on the semantic similarity and the corresponding historical response length. If the current dialogue scenario is a single-turn dialogue scenario, then based on the current user request text, the top-k most semantically similar historical requests are retrieved from the local cache, and an equivalent temporal feature sequence is constructed based on the response length and similarity of the top-k historical requests. The Embedding model is used to extract dense semantic embeddings from the user request text, and the temporal feature sequence is input into a pre-trained Long Short-Term Memory (LSTM) network regressor to predict the length of the LLM response tokens corresponding to the current request. Based on the predicted response term length, a resource-aware intelligent scheduling algorithm is constructed. The predicted response term length is used as a priority. By simulating the operation process of the job to be scheduled, it is determined that the job will not cause preemption. This algorithm is used to determine the set of requests participating in batch processing in each round of scheduling. A dynamic scaling and revalidation algorithm is constructed. If the predicted response term length is lower than the actual length, the response term length is binned according to the predicted response term length, the job budget is dynamically increased, and a virtual job is constructed for revalidation.
2. The large language model job scheduling method based on response length prediction as described in claim 1, characterized in that, The method of extracting dense semantic embeddings from user request text using the Embedding model includes: The user interaction history is broken down into several rounds, each containing the historical request text and the corresponding historical response text. ,in, , For the current round of user requests; For the current round of user request text Historical request texts for each round and LLM history reply text Input each element into the pre-trained embedding model to obtain the corresponding semantic embedding vector. , and ; Obtain the word length of each text, denoted as . and This serves as a statistical characteristic of the length of historical dialogues.
3. The large language model job scheduling method based on response length prediction as described in claim 2, characterized in that, If the current dialogue scenario is a multi-turn dialogue scenario, then by combining the current request text with the historical dialogue context, the semantic similarity between the current request and each historical request is calculated, and based on the semantic similarity and the corresponding historical response length, a temporal feature sequence is constructed, including: Based on the user request text of the current round Corresponding semantic embedding vector With historical request texts Corresponding semantic embedding vector Historical response texts Corresponding semantic embedding vector Calculating semantic similarity includes: The following formula is used to calculate the... With the Input similarity : ， The following formula is used to calculate the... With the Output similarity : ， in, Represents the cosine similarity function; The input length of each round of historical interactions Output length Input similarity Output similarity Concatenate into a four-dimensional feature vector , constitutes a time-series feature sequence .
4. The large language model job scheduling method based on response length prediction as described in claim 2, characterized in that, If the current dialogue scenario is a single-turn dialogue scenario, then based on the current user request text, the top-k semantically most similar historical requests are retrieved from the local cache, and an equivalent temporal feature sequence is constructed based on the response length and similarity of the top-k historical requests, including: Completed historical single-turn dialogue samples are pre-cached in a local vector database, with each sample including the request text. and its corresponding semantic embedding vector Corresponding reply text and its corresponding semantic embedding vector and known lexical lengths and ; When the current single-round history request text is received At that time, the Embedding model is used to generate the... Embedded vector And perform vector similarity retrieval in the local cache to recall the top-k historical single-round request samples that are most semantically similar to the current request; For each recalled sample request The current single-round history request text is calculated according to the following formula. Input similarity with the requested sample Output similarity : ， ， in, Indicates cosine similarity; Using the known data from this sample and Calculated and Construct four-dimensional feature vectors ; The top-k feature vectors corresponding to the top-k historical single-round request samples are sorted from high to low according to input similarity to form an equivalent time-series feature sequence. .
5. The large language model job scheduling method based on response length prediction as described in claim 1, characterized in that, The resource-aware intelligent scheduling algorithm is constructed based on the predicted response term length. Using the predicted response term length as a priority, the algorithm simulates the execution process of the job to be scheduled to determine that the job will not trigger preemption. This algorithm is used to determine the set of requests participating in batch processing in each round of scheduling, including: The priority of requests to the job request pool is determined according to the predicted response token length. The shorter the response token length, the higher the priority and the priority is given to scheduling. During each round of scheduling, for each candidate request, a security simulation is performed based on the key-value cache resources currently occupied in the system, the status of the currently running request, and the input length and predicted response token length of the request to simulate the candidate request's occupation of the key-value cache resources during future iterations. The number of active requests is dynamically tracked during the simulation, and a check is performed at the end of each request's execution: if the candidate request is added to the current running batch, whether the total key-value cache resource usage of the system at any time exceeds the preset maximum capacity limit; A candidate request will only be included in the current scheduling batch if the usage of key-value cache resources does not exceed the capacity limit throughout the entire simulation process. Set a threshold for the maximum number of consecutive failures in a single scheduling iteration; When multiple candidate requests are rejected due to insufficient resources, if the maximum consecutive failure count threshold is reached, the current scheduling process is terminated, the selected request is locked, and the inference execution phase begins.
6. The large language model job scheduling method based on response length prediction as described in claim 5, characterized in that, Construct dynamic scaling and re-verification algorithms, including: If the model response length exceeds the original prediction value during the actual generation process, binning is performed based on the number of generated tokens, with each interval corresponding to a preset expansion increment; When the generated length is detected to be close to the prediction limit, a virtual job is automatically constructed and the security simulation is re-executed; If the simulation after expanding the budget still meets the resource constraints, the current request will continue to be executed; otherwise, it will be preempted and the allocated resources will be released.
7. A large language model job scheduling system based on response length prediction, characterized in that, include: The judgment module is used to receive user request text and determine whether the current dialogue scenario is a single-turn dialogue scenario or a multi-turn dialogue scenario; The first construction module is used to calculate the semantic similarity between the current request and each round of historical requests by combining the current request text and the historical dialogue context if the current dialogue scenario is a multi-turn dialogue scenario, and to construct a temporal feature sequence based on the semantic similarity and the corresponding historical response length. The second construction module is used to, if the current dialogue scenario is a single-turn dialogue scenario, recall the top-k most semantically similar historical requests in the local cache based on the current user request text, and construct an equivalent temporal feature sequence based on the response length and similarity of the top-k historical requests. The prediction module is used to extract dense semantic embeddings from the user request text using the Embedding model, and input the temporal feature sequence into a pre-trained Long Short-Term Memory (LSTM) regressor to predict the length of the LLM response tokens corresponding to the current request. The simulation module is used to construct a resource-aware intelligent scheduling algorithm based on the predicted response word length. It uses the predicted response word length as a priority and simulates the operation process of the job to be scheduled to determine that the job will not cause preemption. It is used to determine the set of requests participating in batch processing in each round of scheduling. The update module is used to build a dynamic expansion and re-verification algorithm. If the predicted response token length is lower than the actual length, the response tokens are binned according to the predicted response token length, the job budget is dynamically increased, and a virtual job is built for re-verification.
8. An apparatus comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 6.
9. A storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 6.