Large language model reasoning calculation service energy consumption optimization scheduling method based on task length prediction

Abstract

The invention discloses a task length prediction-based large language model reasoning calculation service energy consumption optimization scheduling method, which comprises the following steps of: firstly, reasoning by taking an Alpaca-52k instruction data set as an input source and a large language model (such as Llama3-8B), counting the number of output tokens of the large language model, and labeling each piece of input data; then, a Qwen2-1. 5B large language model is finely tuned by using an Alpaca-52k instruction data set and the response length of Llama3-8B reasoning, and computing resources of the prediction method are reduced on the premise that the prediction performance is guaranteed; then, the response length of the Llama3-8B reasoning task is predicted through the fine-tuned Qwen2-1. 5B model, task balanced sorting scheduling is carried out according to the response length so as to improve the large language model reasoning speed, and finally, a deep reinforcement learning power selection algorithm is used to reduce the calculation power as much as possible on the premise that the large language model reasoning task time delay is met so as to improve the large language model reasoning efficiency. Therefore, the energy consumption of large language model reasoning calculation is reduced.

Description

Technical Field

[0001] The present invention belongs to the field of parallel computing task scheduling and relates to a large language model reasoning computing service energy consumption optimization scheduling method based on task length prediction. Background Art

[0002] Currently, large language models are capable of handling complex language tasks such as long text generation, contextual understanding, and multi-turn conversations. The generated text is typically fluent and grammatically correct. By fine-tuning large language models, they can be quickly adapted to specific tasks, reducing the reliance on large amounts of labeled data. Their strong generalization capabilities enable them to perform well on unseen tasks, greatly expanding their application scenarios. However, both training and inference of large language models require enormous computing resources and energy, resulting in high training costs and application overhead. First, from a computing resource perspective, the massive number of parameters in large models (often reaching tens of billions or even trillions) places extremely high demands on hardware infrastructure. First, there is the high computational complexity caused by large-scale matrix operations, especially when processing high-dimensional tensors; second, the high memory access cost caused by frequent reads of the key-value (KV) cache; and third, the high graphics memory usage required to store model parameters and intermediate state. These factors collectively constitute significant obstacles to achieving the goals of low latency, high throughput, and low energy consumption.

[0003] Based on different optimization directions, current research on large-model inference acceleration can be roughly divided into three categories: data level, model level, and system level. However, the autoregressive generation mechanism commonly used in large language models also introduces significant efficiency bottlenecks. Specifically, the autoregressive algorithm predicts the next token based on historical output through iteration. This sequential generation characteristic leads to a large uncertainty in the inference length, making it difficult to estimate the computational workload of the task in advance. This poses significant challenges to task allocation and resource scheduling. This uncertainty can lead to low utilization of computing resources, especially in application scenarios that require real-time response. Summary of the Invention

[0004] Purpose of the invention: The present invention proposes an energy consumption optimization scheduling method for large language model inference computing services based on task length prediction. By predicting the response length of large language model inference tasks, the computing energy consumption is reduced as much as possible while meeting the real-time requirements of large language model inference.

[0005] Technical solution: To achieve the purpose of the present invention, the technical solution adopted by the present invention is: a computing service energy consumption optimization scheduling method based on large language model reasoning task length prediction, the method comprising the following steps:

[0006] (1) First, the Alpaca-52k instruction dataset is used as the input source, and the large language model to be predicted (such as Llama3-8B) is used for inference. The number of its output tokens is counted, and each input data is labeled to create a dataset.

[0007] (2) Fine-tune the Qwen2-1.5B large language model using the Alpaca-52k instruction dataset and Llama3-8B inference response length labels in (1) to predict the response length of the large language model inference task;

[0008] (3) Using the Qwen2-1.5B model fine-tuned in (2), the response length of the Llama3-8B inference task is predicted, and the tasks are sorted in descending order and the round-robin algorithm is used to allocate the tasks to achieve load balancing.

[0009] (4) Based on the inference tasks sorted in descending order after load balancing in (3), task balanced sorting and scheduling are performed to improve the inference speed of large language models;

[0010] (5) After the balanced sorting of tasks based on (4), the power selection algorithm of deep reinforcement learning is used to reduce the computing power as much as possible while meeting the latency of the large language model inference task, so as to reduce the energy consumption of the large language model inference calculation.

[0011] Furthermore, in step (1), in order to fine-tune the large language model, the corresponding dataset must be prepared first. However, there is no such data in the current public dataset, so it needs to be made by yourself. An automatic annotation framework is established for the response length prediction task of the Llama3-8B model. Using the Alpaca-52k instruction dataset as the input source, the response text is generated through model reasoning, and the number of tokens is recorded as a label. The data is then cleaned to remove abnormal data, and a stratified sampling strategy is implemented using the diversity enhancement module. Based on the response length distribution, 10 quantile intervals are divided to ensure that the proportion of each subset sample is controlled between 8% and 12%, significantly improving the model generalization ability. Finally, 27,000 samples are selected, with an average response length of 273, a minimum length of 2, and a maximum length of 1745.

[0012] Furthermore, in step (2), the prompt words for Qwen2-1.5B fine-tuning are designed: "<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\n" constitute the fixed template of the Qwen2 large language model; then, questions that require predicting the response length are added; "Before responding to the question, you have to predict the length of the question response. Print the estimated number of words in your response in the first line.\n" constitute the prompt words, allowing the model to only predict the response length of the question and only need to output a number; finally, the supervisory signal in step (1), that is, the label of the response length, is added to constitute the fine-tuning target.

[0013] Since the Qwen2-1.5B lightweight large language model was selected, the full-parameter fine-tuning method, which has the best fine-tuning results, was directly used. Furthermore, to further accelerate the fine-tuning of the Qwen2-1.5B model, a curriculum learning approach based on the Bayesian Fisher Information Quantity (BI) was employed. Specifically, based on the concept of curriculum learning, it mimics the "step-by-step" principle in human education. Training data is graded by difficulty, allowing the model to first learn simple examples and gradually transition to complex examples, similar to a child learning letters before forming words and sentences. Through phased training, curriculum learning helps the model build a solid foundation. Curriculum learning first requires categorizing the training data into multiple difficulty levels, for example, based on sample complexity, noise level, or task difficulty. Furthermore, the model's parameter fit must also be considered. The Fisher Information Quantity (FIQ) reflects the richness of information about the parameters in the data. If a small change in a parameter significantly alters the data distribution, the data provides a large amount of information, indicating a high FIQ. Conversely, if a parameter change has little effect on the data distribution, the FIQ is low. Therefore, the Fisher information of the samples in the training set under the model parameters can be calculated to divide the training data.

[0014] When the model has multiple unknown parameters θ=(θ1,θ2,...,θ d ), Fisher information I ij (θ) is a d×d symmetric matrix:

[0015]

[0016] However, the data distribution P(X|θ) is usually unknown (especially for complex models such as neural networks), and it is impossible to calculate the expectation analytically. d , the complexity of storing and computing matrices is O(d 2 ) or even O(d 3 )(find the inverse time), which is almost impossible in deep learning. Empirical Fisher information The above problem is solved by replacing the theoretical expectation with the sample mean and calculating directly based on the observed data:

[0017]

[0018] In addition, this problem can be considered in the Bayesian framework. The training data obeys the prior distribution of B(X|θ) in the training data set and the posterior distribution of P(X|θ) under large model parameters, and the BayesFisher information I Bayes The matrix is ​​defined as the sum of the information content of the posterior distribution and the information content of the prior distribution:

[0019] I Bayes =I Prior +I Data , (3)

[0020] Among them, I Prior is the Fisher information of the prior distribution, I Data It is the Fisher information of the posterior distribution. Based on this, according to the BayesFisher information of the training samples, simple samples are first used to speed up the convergence of the model, and then the data is gradually increased during the training process to reduce the total training fine-tuning time.

[0021] In the training parameter settings, the batch size (batch_size) is set to 8, which balances the GPU memory limitation and the stability requirements of gradient updates. The AdamW optimizer is used for parameter updates. This optimizer effectively improves the regularization effect of the traditional Adam optimizer through the decoupled weight decay mechanism. The initial learning rate is set to 1×10 -4 , this value refers to the fine-tuning convention of pre-trained language models to ensure the rationality of the parameter update step size; in addition, a linear learning rate schedule is designed, and the total number of training steps is calculated by the formula: total number of steps = (total amount of data × number of training rounds) / batch size. In particular, no warm-up phase is set (warmup steps = 0), and the learning rate is increased from the initial value of 1×10 -4 Initially, it decreases linearly to zero as the training progresses. This monotonically decreasing strategy helps fine-tune the parameters in the later stages of training.

[0022] Furthermore, in step (3), the response length of the Llama3-8B inference task is predicted based on the fine-tuned Qwen-1.5B model proposed in step (2). The predicted response lengths are then sorted in descending order. A round-robin algorithm is then used to divide the input data into multiple small batches based on a data parallel strategy. Each small batch is then assigned to different computing nodes for parallel processing. During inference, batches processed independently by different nodes are then merged to achieve fine-grained load balancing.

[0023] Furthermore, in step (4), a task-balanced sorting method based on deep reinforcement learning is designed, and the large language model inference task is divided into prefill and decode stages. The prefill stage is the initial processing link of the LLM inference task. Its main goal is to perform full calculations on the input sequence (such as the prompt provided by the user), generate and cache the key and value vectors of all positions, and lay the foundation for subsequent decoding. For example, when the input is "Why is the sky blue?", the model needs to segment it into multiple tokens and process the entire sequence in parallel through the Transformer layer, and then generate the first token. The decode stage uses an autoregressive method to generate output tokens one by one. Every time a new token is generated, it needs to read the historical KV cache and calculate the associated weight through the self-attention mechanism to predict the probability of the next token. This process repeats until the terminator is generated or the maximum length is reached. The prefill stage can fully utilize the parallel computing power of the GPU to process the entire input sequence at one time. It has high computational density and significant peak memory usage. The decode stage has a small amount of single-step computation, but it needs to frequently access the KV cache in the memory. The prefill and decode stages correspond to the "context initialization" and "sequence generation" of LLM reasoning, respectively. The two have complementary requirements for GPU resources. Therefore, how to allocate reasonable computing resources to the inference task based on the response length becomes the key to improving the inference speed. Since the large language model inference framework vllm adopts the FCFS strategy internally and does not consider the response length of the task, the prefill and decode stages of the large language model inference task do not call the GPU resources evenly, which slows down the inference speed. Therefore, the inference tasks can be re-sorted based on the descending order to balance the call of GPU resources by the large language model inference task. After sorting the task response length in descending order based on the prediction in step (2), starting from the first task, select a task every M tasks until the end of the round, and then start the second round of selection from the second task. Repeat this process until all tasks have been selected.

[0024] This heuristic algorithm enables parallel execution of tasks of varying lengths. Using continuous batching, longer tasks enter the decoding phase simultaneously, while shorter tasks finish decoding early and enter the prefill phase for the next task. The core of this heuristic algorithm lies in the selection of the interval number M, whose value directly determines the effectiveness of this balancing process. Therefore, deep reinforcement learning is used to explore an appropriate value for M and accelerate inference for large language models. The state space, action space, and reward function for reinforcement learning are designed as follows.

[0025] State space: The state is the basis of the agent's decision-making. In order to explore the appropriate M value, the agent must first observe the state of the task. Therefore, the predicted values ​​of the task response length are sorted in descending order and the predicted values ​​are sorted in descending order.

[0026] Action Space: Since the goal of this reinforcement learning is to explore the value of M, but the number of inference tasks in each batch is not uniform, the value of M should change accordingly. Assuming the number of inference tasks in a batch is N, let M = N / a, where a∈{10,20,30,40,50,60,70,80,90,100}. Therefore, the action space of this reinforcement learning is [10,20,30,40,50,60,70,80,90,100].

[0027] Reward function: To guide the agent to explore and find the appropriate action a, the reward function is the inference speed (token / s) of the batch task. When the inference speed is low, the agent is penalized, and when the inference speed is high, the agent is rewarded. Because the inference speed fluctuates between 900-1200 (token / s), in order to reflect the difference in rewards, it is mapped to e speed / 200 range to better guide the agent's learning.

[0028] Because large language models have slow inference speeds and long overall training times, the SAC (Soft Actor-Critic) algorithm was adopted to maximize data utilization. Furthermore, a training environment for reinforcement learning was developed. Specifically, this environment reads 12,000 samples from the Alpaca instruction dataset as the training dataset, train_set. It then randomly selects a certain amount of training data. After the agents interact, it randomly selects another amount of training data, repeating this process until there are no more samples in train_st. Furthermore, the actor and critic for deep reinforcement learning use a three-layer fully connected neural network with 128 hidden layer parameters. During testing, computation time is negligible. The overall training process of the task sorting and balanced scheduling algorithm based on deep reinforcement learning is as follows: first, initialize the parameters of the training round number Epoch, batch size batch_size, experience pool threshold minimal_size, Actor network learning rate actor_lr, Critic network learning rate critic_lr, Actor network parameter θ, Critic1 network parameter ω1 and Critic2 network parameter ω2; then reset the training environment, read the training set train_set, obtain the training data train_data randomly selected from the environment, use the fine-tuned Qwen2-1.5B model to predict the response length length of the training data, sort it in descending order according to the predicted length length and normalize it to obtain the environment state s t ; Then the Actor network is based on s t , calculate the probability distribution of each action and get a t ; Then, let the Llama3-8B model reason, record the reasoning speed, and calculate the reward r t , and get the next batch of training data and the next state s returned by the training environment t+1 , and then the quaternion {s t ,a t ,r t ,s t+1} put into experience pool B; when the number of experience pool B reaches minimal_size, start updating the network parameters, that is, sampling batch_size samples from experience pool B, and calculating the output value1 of the Critic1 network and the output value2 of the Critic2 network based on this sample and selecting the smaller value m, then calculate the policy entropy entropy output by the Actor network, and use this to calculate the temporal difference residual δ to calculate the loss update network of the Critic1 network and the Critic2 network; then use the new Critic1 network and Critic2 network parameters ω1 and ω2 and the policy entropy entropy of the Actor network to update the Actor network parameters θ; finally, after training, save the Actor network parameters for online decision scheduling.

[0029] Furthermore, in step (5), a power selection algorithm based on deep reinforcement learning is designed. Given N large language model inference tasks {τ1,…,τ N}, the length of each task is unknown, and the fine-tuned Qwen-1.5B in step (2) is used to predict the response length. The predicted length of each task is e i , and the unified deadline of all tasks is D. According to the real-time requirement, each task must be completed before the deadline D. In addition, these tasks are non-preemptive, that is, once a task starts running in a card, it will continue to run until completion.

[0030] The large language model inference task is executed on M GPUs (RTX 3090) {G1,…,G M}, and the GPU supports DVFS technology, that is, under the GPU Boost intelligent acceleration framework, the core frequency can be dynamically adjusted in 1MHz steps. And users can use the NVIDIA system management interface (nvidia-smi) to query GPU device clock, temperature, power and other information, and set the frequency and power of the GPU device. In the established model, the GPU device can set L power levels {p1,…,p L}, and its corresponding Llama3-8B large language model inference speed under the vllm framework is {s1,…,s L}(token / s). It should be noted that due to the different inference tasks, the inference speed at each power level will fluctuate. The problem modeling is carried out as follows:

[0031] Task allocation constraint: Because a task only needs to be assigned to one GPU and all tasks must be assigned, define a binary variable x ik , when its value is 1, it means that the task τ i Assigned to G k , so the constraint is expressed as:

[0032]

[0033] ●Power selection constraint: Considering the system-level DVFS, that is, each GPU executes all assigned tasks at a power level, define the binary variable y kl When its value is 1, it means GPUG k The task is executed at power level l, so the constraint is expressed as:

[0034]

[0035] Real-time constraint: Each real-time task τ i The execution must be completed before the deadline D, so the constraint is expressed as:

[0036]

[0037] Objective equation: The energy consumption required to complete all inference tasks is expressed as:

[0038]

[0039] ●Original problem: Minimize the total energy consumption while satisfying all constraints:

[0040]

[0041] However, reducing power consumption comes at the expense of performance. Furthermore, because the response length of large language model inference tasks is unknown, it's impossible to develop a fine-grained power adjustment strategy to strike a good balance between performance and energy consumption. Based on a fine-tuned Qwen-1.5B model, we predict the response length of large language model inference tasks and, combined with the performance released by each GPU power level, fine-grainedly adjust GPU power to balance GPU computing performance and power consumption. Based on this, we design a power selection algorithm based on deep reinforcement learning to explore the optimal balance between GPU computing performance and energy consumption. The design of its action space, state space, and reward function is shown below:

[0042] Action space: The goal of this reinforcement learning is to balance GPU performance and energy consumption, so the action of the agent is to select the appropriate power level l. Six power levels are set [350, 325, 300, 275, 250, 225]. In fact, more power levels can be set, but in reality, it is not possible to make more fine-grained power level selections to reduce more energy consumption. Based on experimental experience, these 6 power levels were finally selected, of which 350W is the upper limit of power. If the power is lower than 225W, the increased execution time will lead to higher energy consumption, which is not worth the loss. Therefore, 225W is set as the lower limit of power. To further improve the performance of the algorithm, the lowest inference speed can be calculated based on the predicted task response length and the deadline D:

[0043]

[0044] The power level is selected accordingly, and then reinforcement learning is used to adjust the power level. Depending on the situation, the power level can be increased, decreased, or kept unchanged. Therefore, the action space is [0, 1, 2], representing increasing the power level, keeping it unchanged, and decreasing the power level respectively.

[0045] State space: To allow the agent to select the appropriate power level l, the inference tasks are first sorted in descending order according to the predicted response length. Then, the predicted minimum inference speed is calculated according to formula (6). Finally, the normalized features are embedded to form the state space. It is hoped that the agent can explore the optimal mapping relationship between the minimum processing speed and the power level according to different task conditions.

[0046] Reward function: To guide the agent in selecting the appropriate power level l, the reward function is defined as the total energy consumption of completing the task. When the task execution time t is less than the deadline D, a certain reward is given based on the total energy consumption of completing the task, and the reward decreases as the total energy consumption increases:

[0047] reward i =-αtp l , (10)

[0048] where p l is the power of the lth level, and α is the normalization coefficient. However, if the time t to complete the task is greater than the deadline D, it violates the real-time constraint, so it is penalized:

[0049] reward i =-1. (11)

[0050] Similarly, the training environment required for reinforcement learning must be written. This environment first reads 12,000 samples from the Alpaca instruction dataset as the training dataset, train_set. It then randomly selects a certain amount of training data and generates a random deadline based on the selected amount of training data, ensuring that the GPU, running at the highest power level, can complete the task before the deadline. Feature embedding is then performed and input into the Actor network, which makes decisions. After interacting with the agent, a certain amount of training data is randomly selected again. This process is repeated until there are no more samples in train_set, and the next round of training begins. The Actor and Critic use a three-layer fully connected neural network with 128 hidden layer parameters. During the testing phase, computation time is negligible. The overall training process of the task sorting and balanced scheduling algorithm based on deep reinforcement learning is as follows: first, initialize the parameters of the training round number Epoch, batch size batch_size, experience pool threshold minimal_size, Actor network learning rate actor_lr, Critic network learning rate critic_lr, Actor network parameter θ, Critic1 network parameter ω1 and Critic2 network parameter ω2, then reset the training environment, read the Llama3-8B response length dataset train_set, obtain the training data train_data randomly selected in the environment, generate a random deadline D according to train_data, use the fine-tuned Qwen2-1.5B model to predict the response length length of the training data, sort in descending order, and calculate And normalize to get the environment state s t ; Then the Actor network is based on s t , calculate the probability distribution of each action and get a t , and predict the minimum inference speed to adjust the power level; then, let the Llama3-8B model infer, record the power level and energy consumption, and calculate the reward r t , and get the next batch of training data and the next state s returned by the training environment t+1 , and then the quaternion {s t ,a t ,r t ,s t+1} put into experience pool B; when the number of experience pool B reaches minimal_size, start updating the network parameters, that is, sampling batch_size samples from experience pool B, and calculating the output value1 of the Critic1 network and the output value2 of the Critic2 network based on this sample and selecting the smaller value m, then calculate the policy entropy output by the actor network. Based on this, the temporal difference residual δ is calculated to calculate the loss of the Critic1 and Critic2 networks and update the network. The new Critic1 and Critic2 network parameters ω1 and ω2 and the policy entropy of the actor network are then used to update the actor network parameters θ. Finally, the trained actor network is saved for online decision-making and scheduling.

[0051] Through the above steps, we have realized the preparation of the dataset for fine-tuning the response length prediction of the large language model inference task, data enhancement, fine-tuning of the response length prediction of the Llama3-8B model inference task based on Qwen2-1.5B, and computing services such as task allocation load balancing, GPU computing resource balancing, and energy consumption optimization scheduling.

[0052] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the method for optimizing energy consumption scheduling of a large language model inference computing service based on task length prediction is implemented.

[0053] A computer-readable storage medium stores computer instructions, which, when executed by a processor, implement the energy consumption optimization scheduling method for large language model reasoning computing services based on task length prediction.

[0054] Beneficial effects: Compared with the prior art, the technical solution of the present invention has the following beneficial effects: 1) Producing a Llama3-8B response length dataset, and performing data enhancement through cleaning and sampling techniques; 2) Fine-tuning the Qwen2-1.5B model to predict the response length of the Llama3-8B model inference task, reducing the absolute value of the prediction error to 57 tokens, and increasing the probability of a prediction error less than 50 by 6%, thereby performing token-level fine-grained load balancing and improving the overall inference speed by up to 6.7%; 3) Using a task sorting and balancing method based on deep reinforcement learning, balancing the GPU computing power resources and IO access resources in large model parallel inference, reducing the waste of computing resources, and improving the overall performance of the system by 8.5%; 4) Based on the response length prediction and computing resource balancing of large language model inference tasks, as well as the performance released at various power levels, the energy consumption of computing services can be reduced by up to 26.6% while meeting real-time requirements.

[0055] Explanation of the attached table

[0056] Table 1 is a performance comparison table of the response length of various large language models and fine-tuned Qwen2-1.5B prediction of Llama3-8B reasoning tasks in Example 2 of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 It is a Qwen2-1.5B model fine-tuning template designed in an embodiment of the present invention;

[0058] Figure 2 This is a schematic diagram of response length prediction and fine-grained load balancing for a large language model inference task according to Example 1 of the present invention;

[0059] Figure 3 This is a schematic diagram of balanced task sorting based on the predicted large language model inference task response length in Example 1 of the present invention;

[0060] Figure 4 This is a comparison chart of the inference speed and inference time of the task sorting and balancing algorithm based on deep reinforcement learning in Example 2 of the present invention, with the number of tasks configured as 1000, 2000, 3000, 4000, and 5000 under the VLLM framework;

[0061] Figure 5 This is a comparison chart of the inference speed of Example 2 of the present invention, with 2, 4, and 8 GPUs configured and 1000, 2000, 3000, 4000, and 5000 tasks generated, using a polling algorithm based on task response length prediction and a task sorting and balancing algorithm based on deep reinforcement learning;

[0062] Figure 6 This is a comparison chart of Example 2 of the present invention, where the number of tasks is configured as 3000, 4000, and 5000, corresponding to randomly generated schedulable delays, and the service energy consumption is calculated using a power selection algorithm based on deep reinforcement learning in the VLLM framework;

[0063] Figure 7 This is a comparison chart of the service energy consumption calculated using a polling algorithm based on task response length prediction and a task sorting and power selection algorithm based on deep reinforcement learning in Example 2 of the present invention, with the number of tasks configured as 10,000 and the number of GPUs as 2, 4, and 8, corresponding to randomly generated schedulable delays.

[0064] Figure 8 It is a schematic diagram of the overall framework of the present invention. DETAILED DESCRIPTION

[0065] The following describes the embodiments of the present invention in detail with reference to the accompanying drawings and examples, so that the present invention can fully understand how to apply technical means to solve technical problems and achieve technical effects, and thus implement the invention accordingly. It should be noted that, as long as no conflict exists, the various embodiments of the present invention and the various features of the embodiments can be combined with each other, and the resulting technical solutions are all within the scope of protection of the present invention.

[0066] Example 1: Energy consumption optimization scheduling method for large language model inference computing service based on task length prediction,

[0067] Figure 1 It is a Qwen2-1.5B model fine-tuning template designed in an embodiment of the present invention;

[0068] Figure 2 This is a schematic diagram of response length prediction and fine-grained load balancing for a large language model inference task according to Example 1 of the present invention;

[0069] Figure 3 This is a diagram of task balancing sorting based on the predicted large language model inference task response length in Example 1 of the present invention. Figure 1 、 Figure 2 and Figure 3 , explaining each step in detail:

[0070] Step (1): In order to fine-tune the large language model, the corresponding dataset must be prepared first. However, there is no such data in the current public dataset, so it needs to be made by yourself. An automatic annotation framework is established for the response length prediction task of the Llama3-8B model. Using the Alpaca-52k instruction dataset as the input source, the response text is generated through model reasoning, and the number of tokens is recorded as a label. The data is then cleaned to remove abnormal data, and a stratified sampling strategy is implemented using the diversity enhancement module. Based on the response length distribution, 10 quantile intervals are divided to ensure that the proportion of each subset sample is controlled between 8% and 12%, significantly improving the model generalization ability. Finally, 27,000 samples are selected, with an average response length of 273, a minimum length of 2, and a maximum length of 1745.

[0071] Step (2), such as Figure 1As shown in the figure, the prompt words for Qwen2-1.5B fine-tuning are designed: "<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\n" constitute the fixed template of the Qwen2 large language model; then the question that needs to predict the response length is added; "Before responding to the question, you have to predict the length of the question response. Print the estimated number of words in your response in the first line.\n" constitutes the prompt word, so that the model only predicts the response length of the question and only needs to output a number; finally, the supervisory signal in step (1), that is, the label of the response length, is added to constitute the fine-tuning target.

[0072] Since the Qwen2-1.5B lightweight large language model was selected, the full-parameter fine-tuning method, which has the best fine-tuning results, was directly used. Furthermore, to further accelerate the fine-tuning of the Qwen2-1.5B model, a curriculum learning approach based on the Bayesian Fisher Information Quantity (BI) was employed. Specifically, based on the concept of curriculum learning, it mimics the "step-by-step" principle in human education. Training data is graded by difficulty, allowing the model to first learn simple examples and gradually transition to complex examples, similar to a child learning letters before forming words and sentences. Through phased training, curriculum learning helps the model build a solid foundation. Curriculum learning first requires categorizing the training data into multiple difficulty levels, for example, based on sample complexity, noise level, or task difficulty. Furthermore, the model's parameter fit must also be considered. The Fisher Information Quantity (FIQ) reflects the richness of information about the parameters in the data. If a small change in a parameter significantly alters the data distribution, the data provides a large amount of information, indicating a high FIQ. Conversely, if a parameter change has little effect on the data distribution, the FIQ is low. Therefore, the Fisher information of the samples in the training set under the model parameters can be calculated to divide the training data.

[0073] When the model has multiple unknown parameters θ=(θ1,θ2,...,θ d ), Fisher information I ij (θ) is a d×d symmetric matrix:

[0074]

[0075] However, the data distribution P(X|θ) is usually unknown (especially for complex models such as neural networks), and it is impossible to calculate the expectation analytically. d , the complexity of storing and computing matrices is O(d 2 ) or even O(d 3 )(find the inverse time), which is almost impossible in deep learning. Empirical Fisher information The above problem is solved by replacing the theoretical expectation with the sample mean and calculating directly based on the observed data:

[0076]

[0077] In addition, this problem can be considered in the Bayesian framework. The training data obeys the prior distribution of B(X|θ) in the training data set and the posterior distribution of P(X|θ) under large model parameters, and the BayesFisher information I Bayes The matrix is ​​defined as the sum of the information content of the posterior distribution and the information content of the prior distribution:

[0078] I Bayes =I Prior +I Data , (3)

[0079] Among them, I Prior is the Fisher information of the prior distribution, I Data It is the Fisher information of the posterior distribution. Based on this, according to the BayesFisher information of the training samples, simple samples are first used to speed up the convergence of the model, and then the data is gradually increased during the training process to reduce the total training fine-tuning time.

[0080] In the training parameter settings, the batch size (batch_size) is set to 8, which balances the GPU memory limitation and the stability requirements of gradient updates. The AdamW optimizer is used for parameter updates. This optimizer effectively improves the regularization effect of the traditional Adam optimizer through the decoupled weight decay mechanism. The initial learning rate is set to 1×10 -4 , this value refers to the fine-tuning convention of pre-trained language models to ensure the rationality of the parameter update step size; in addition, a linear learning rate schedule is designed, and the total number of training steps is calculated by the formula: total number of steps = (total amount of data × number of training rounds) / batch size. In particular, no warm-up phase is set (warmup steps = 0), and the learning rate is increased from the initial value of 1×10 -4 Initially, it decreases linearly to zero as the training progresses. This monotonically decreasing strategy helps fine-tune the parameters in the later stages of training.

[0081] After fine-tuning training is completed, save the Qwen2-1.5B model to predict the response length of the Llama3-8B inference task, providing a basis for subsequent optimization scheduling.

[0082] Step (3), such as Figure 2 As shown in the figure, the fine-tuned Qwen-1.5B model proposed in step (2) is used to predict the response length of the Llama3-8B inference task. The predicted response lengths are then sorted in descending order. Then, a round-robin algorithm is used to divide the input data into multiple small batches based on a data parallel strategy. Each small batch is assigned to a different computing node for parallel processing. During inference, batches processed by different nodes are independent and the results are finally merged to achieve fine-grained load balancing.

[0083] Step (4) designs a task-balanced sorting method based on deep reinforcement learning. The large language model reasoning task is divided into prefill and decode stages. The prefill stage is the initial processing link of the LLM reasoning task. Its main goal is to perform full calculations on the input sequence (such as the prompt provided by the user), generate and cache the key and value vectors of all positions, and lay the foundation for subsequent decoding. For example, when the input is "Why is the sky blue?", the model needs to segment it into multiple tokens and process the entire sequence in parallel through the Transformer layer, and then generate the first token. The decode stage uses an autoregressive method to generate output tokens one by one. Every time a new token is generated, it needs to read the historical KV cache and calculate the associated weight through the self-attention mechanism to predict the probability of the next token. This process repeats until the terminator is generated or the maximum length is reached. The prefill stage can fully utilize the parallel computing power of the GPU to process the entire input sequence at one time. The computational density is high and the peak memory usage is significant. The decode stage has a small amount of single-step computation, but it needs to frequently access the KV cache in the memory. The prefill and decode stages correspond to the "context initialization" and "sequence generation" of LLM reasoning, respectively. The two have complementary requirements for GPU resources. Therefore, how to allocate reasonable computing resources to the inference task based on the response length becomes the key to improving the inference speed. Since the large language model inference framework vllm adopts the FCFS strategy internally and does not consider the response length of the task, the prefill and decode stages of the large language model inference task do not call the GPU resources evenly, which slows down the inference speed. Therefore, the inference tasks can be re-sorted based on the descending order to balance the call of GPU resources by the large language model inference task. After sorting the task response length in descending order based on the prediction in step (2), starting from the first task, select a task every M tasks until the end of the round, and then start the second round of selection from the second task. Repeat this process until all tasks have been selected.

[0084] This heuristic algorithm enables parallel execution of tasks of varying lengths. Using continuous batching, longer tasks enter the decoding phase simultaneously, while shorter tasks finish decoding early and enter the prefill phase for the next task. The core of this heuristic algorithm lies in the selection of the interval number M, whose value directly determines the effectiveness of this balancing process. Therefore, deep reinforcement learning is used to explore an appropriate value for M and accelerate inference for large language models. The state space, action space, and reward function for reinforcement learning are designed as follows.

[0085] State space: The state is the basis of the agent's decision-making. In order to explore the appropriate M value, the agent must first observe the state of the task. Therefore, the predicted values ​​of the task response length are sorted in descending order and the predicted values ​​are sorted.

[0086] Action Space: Since the goal of this reinforcement learning is to explore the value of M, but the number of inference tasks in each batch is not uniform, the value of M should change accordingly. Assuming the number of inference tasks in a batch is N, let M = N / a, where a∈{10,20,30,40,50,60,70,80,90,100}. Therefore, the action space of this reinforcement learning is [10,20,30,40,50,60,70,80,90,100].

[0087] Reward function: To guide the agent to explore and find the appropriate action a, the reward function is the inference speed (token / s) of the batch task. When the inference speed is low, the agent is penalized, and when the inference speed is high, the agent is rewarded. Because the inference speed fluctuates between 900-1200 (token / s), in order to reflect the difference in rewards, it is mapped to e speed / 200 range to better guide the agent's learning.

[0088] Because large language models have slow inference speeds and long overall training times, the SAC (Soft Actor-Critic) algorithm was adopted to maximize data utilization. Furthermore, a training environment was developed for reinforcement learning. Specifically, this environment reads 12,000 samples from the Alpaca instruction dataset as the training dataset, train_set. It then randomly selects a certain amount of training data. After the agents interact, it randomly selects another amount of training data, repeating this process until there are no more samples in train_set. Furthermore, the actor and critic for deep reinforcement learning use a three-layer fully connected neural network with 128 hidden layer parameters. During testing, computation time is negligible. The overall training process of the task sorting and balanced scheduling algorithm based on deep reinforcement learning is as follows: first, initialize the parameters of the training round number Epoch, batch size batch_size, experience pool threshold minimal_size, Actor network learning rate actor_lr, Critic network learning rate critic_lr, Actor network parameter θ, Critic1 network parameter ω1 and Critic2 network parameter ω2; then reset the training environment, read the training set train_set, obtain the training data train_data randomly selected from the environment, use the fine-tuned Qwen2-1.5B model to predict the response length length of the training data, sort it in descending order according to the predicted length length and normalize it to obtain the environment state s t; Then the Actor network is based on s t , calculate the probability distribution of each action and get a t ; Then, let the Llama3-8B model reason, record the reasoning speed, and calculate the reward r t , and get the next batch of training data and the next state s returned by the training environment t+1 , and then the quaternion {s t ,a t ,r t ,s t+1} put into experience pool B; when the number of experience pool B reaches minimal_size, start updating the network parameters, that is, sampling batch_size samples from experience pool B, and calculating the output value1 of the Critic1 network and the output value2 of the Critic2 network based on this sample and selecting the smaller value m , then calculate the policy entropy entropy output by the Actor network, and use this to calculate the temporal difference residual δ to calculate the loss update network of the Critic1 network and the Critic2 network; then use the new Critic1 network and Critic2 network parameters ω1 and ω2 and the policy entropy entropy of the Actor network to update the Actor network parameters θ; finally, after the training is completed, save the Actor network parameters for online decision scheduling.

[0089] In step (5), a power selection algorithm based on deep reinforcement learning is designed. Given N large language model inference tasks {v1,…,τ N}, the length of each task is unknown, and the fine-tuned Qwen-1.5B in step (2) is used to predict the response length. The predicted length of each task is e i , and the unified deadline of all tasks is D. According to the real-time requirement, each task must be completed before the deadline D. In addition, these tasks are non-preemptive, that is, once a task starts running in a card, it will continue to run until completion.

[0090] The large language model inference task is executed on M GPUs (RTX 3090) {G1,…,G M}, and the GPU supports DVFS technology, that is, under the GPU Boost intelligent acceleration framework, the core frequency can be dynamically adjusted in 1MHz steps. And users can use the NVIDIA system management interface (nvidia-smi) to query GPU device clock, temperature, power and other information, and set the frequency and power of the GPU device. In the established model, the GPU device can set L power levels {p1,…,p L}, and its corresponding Llama3-8B large language model inference speed under the vllm framework is {s1,…,s L}(token / s). It should be noted that due to the different inference tasks, the inference speed at each power level will fluctuate. The problem modeling is carried out as follows:

[0091] ●Task allocation constraint: Because a task only needs to be assigned to one GPU, and to ensure that all tasks are assigned, define a binary variable x ik , when its value is 1, it means that the task τ i Assigned to G k , so the constraint is expressed as:

[0092]

[0093] Power selection constraint: Considering the system-level DVFS, that is, each GPU executes all assigned tasks at a power level, define the binary variable y kl When its value is 1, it means GPUG k The task is executed at power level l, so the constraint is expressed as:

[0094]

[0095] Real-time constraint: Each real-time task τ i The execution must be completed before the deadline D, so the constraint is expressed as:

[0096]

[0097] Objective equation: The energy consumption required to complete all inference tasks is expressed as:

[0098]

[0099] Original problem: Minimize the total energy consumption while satisfying all constraints:

[0100]

[0101] However, reducing power consumption comes at the expense of performance. Furthermore, because the response length of large language model inference tasks is unknown, it's impossible to develop a fine-grained power adjustment strategy to strike a good balance between performance and energy consumption. Based on a fine-tuned Qwen-1.5B model, we predict the response length of large language model inference tasks and, combined with the performance released by each GPU power level, fine-grainedly adjust GPU power to balance GPU computing performance and power consumption. Based on this, we design a power selection algorithm based on deep reinforcement learning to explore the optimal balance between GPU computing performance and energy consumption. The design of its action space, state space, and reward function is shown below:

[0102] Action space: The goal of this reinforcement learning is to balance GPU performance and energy consumption, so the action of the agent is to select the appropriate power level l. Six power levels are set [350, 325, 300, 275, 250, 225]. In fact, more power levels can be set, but in reality, it is not possible to make more fine-grained power level selections to reduce more energy consumption. Based on experimental experience, these 6 power levels were finally selected, of which 350W is the upper limit of power. If the power is lower than 225W, the increased execution time will lead to higher energy consumption, which is not worth the loss. Therefore, 225W is set as the lower limit of power. To further improve the performance of the algorithm, the lowest inference speed can be calculated based on the predicted task response length and the deadline D:

[0103]

[0104] The power level is selected accordingly, and then reinforcement learning is used to adjust the power level. Depending on the situation, the power level can be increased, decreased, or kept unchanged. Therefore, the action space is [0, 1, 2], representing increasing the power level, keeping it unchanged, and decreasing the power level respectively.

[0105] State space: To allow the agent to select the appropriate power level l, the inference tasks are first sorted in descending order according to the predicted response length. Then, the predicted minimum inference speed is calculated according to formula (6). Finally, the normalized features are embedded to form the state space. It is hoped that the agent can explore the optimal mapping relationship between the minimum processing speed and the power level according to different task conditions.

[0106] Reward function: To guide the agent in selecting the appropriate power level l, the reward function is defined as the total energy consumption of completing the task. When the task execution time t is less than the deadline D, a certain reward is given based on the total energy consumption of completing the task, and the reward decreases as the total energy consumption increases:

[0107] reward i =-αtp l , (10)

[0108] where p l is the power of the lth level, and α is the normalization coefficient. However, if the time t to complete the task is greater than the deadline D, it violates the real-time constraint, so it is penalized:

[0109] reward i =-1. (11)

[0110] Similarly, the training environment required for reinforcement learning must be written. This environment first reads 12,000 samples from the Alpaca instruction dataset as the training dataset, train_set. It then randomly selects a certain amount of training data and generates a random deadline based on the selected amount of training data, ensuring that the GPU, running at the highest power level, can complete the task before the deadline. Feature embedding is then performed and input into the Actor network, which makes decisions. After interacting with the agent, a certain amount of training data is randomly selected again. This process is repeated until there are no more samples in train_set, and the next round of training begins. The Actor and Critic use a three-layer fully connected neural network with 128 hidden layer parameters. During the testing phase, computation time is negligible. The overall training process of the task sorting and balanced scheduling algorithm based on deep reinforcement learning is as follows: first, initialize the parameters of the training round number Epoch, batch size batch_size, experience pool threshold minimal_size, Actor network learning rate actor_lr, Critic network learning rate critic_lr, Actor network parameter θ, Critic1 network parameter ω1 and Critic2 network parameter ω2, then reset the training environment, read the Llama3-8B response length dataset train_set, obtain the training data train_data randomly selected in the environment, generate a random deadline D according to train_data, use the fine-tuned Qwen2-1.5B model to predict the response length length of the training data, sort in descending order, and calculate And normalize to get the environment state s t ; Then the Actor network is based on s t , calculate the probability distribution of each action and get a t , and predict the minimum inference speed to adjust the power level; then, let the Llama3-8B model infer, record the power level and energy consumption, and calculate the reward r t , and get the next batch of training data and the next state s returned by the training environment t+1 , and then the quaternion {s t ,a t ,r t ,s t+1} put into experience pool B; when the number of experience pool B reaches minimal_size, start updating the network parameters, that is, sampling batch_size samples from experience pool B, and calculating the output value1 of the Critic1 network and the output value2 of the Critic2 network based on this sample, and select the smaller value m, then calculate the policy entropy output by the actor network. Based on this, the temporal difference residual δ is calculated to calculate the loss of the Critic1 and Critic2 networks and update the network. The new Critic1 and Critic2 network parameters ω1 and ω2 and the policy entropy of the actor network are then used to update the actor network parameters θ. Finally, the trained actor network is saved for online decision-making and scheduling.

[0111] Through the above steps, we have realized the preparation of the dataset for fine-tuning the response length prediction of the large language model inference task, data enhancement, fine-tuning of the response length prediction of the Llama3-8B model inference task based on Qwen2-1.5B, and computing services such as task allocation load balancing, GPU computing resource balancing, and energy consumption optimization scheduling.

[0112] Example 2:

[0113] Table 1 is a performance comparison table of the response length of various large language models and fine-tuned Qwen2-1.5B prediction of Llama3-8B reasoning tasks in Example 2 of the present invention.

[0114] Table 1: Using Qwen2-1.5B as the base model,

[0115]

[0116] Table 1 and Figures 4 to 7 This is the experimental result of the present invention.

[0117] The response length of the Llama3-8B model inference task is predicted, and the Alpaca-52k instruction dataset is used as the input source. After the response length of the Llama3-8B model's inference output statistics and data augmentation, a 27,000-sample training set and a 2,000-sample test set are produced. The temperature is set to 0.6 to balance the creativity and stability of the generated text. The kernel sampling strategy of top_p=0.9 is used to dynamically screen high-probability token sets with a cumulative probability of more than 90%; max_tokens is set to 4096, that is, the upper limit of the response length of each inference task is 4096 tokens. Then, the Transformers framework is used to fully fine-tune the Qwen2-1.5B model, and the training set is divided into two parts according to the BayesFisher information of the sample. The part with less information is used for fine-tuning training first, and then the part with more information is used for fine-tuning training. In the training parameter settings, the batch size (batch_size) is set to 8, which balances the GPU memory limitation and the stability requirements of gradient updates. The AdamW optimizer is used for parameter updates. This optimizer effectively improves the regularization effect of the traditional Adam optimizer through the decoupled weight decay mechanism. The initial learning rate is set to 1×10 -4 , this value refers to the fine-tuning convention of pre-trained language models to ensure the rationality of the parameter update step size; in addition, a linear decay learning rate schedule is designed, and the total number of training steps is calculated by the formula: total number of steps = (total amount of data × number of training rounds) / batch size. In particular, no warmup phase is set (warmup steps = 0), and the learning rate is increased from the initial value of 1×10 -4Initially, the training progresses linearly toward zero. This monotonically decreasing strategy facilitates fine-tuning of parameters in the later stages of training. Finally, the fine-tuned Qwen2-1.5B model is saved and tested on the test set. The mean error (Mean Error) between the predicted and actual values ​​is calculated, as well as the probability that the absolute value of the error is less than 50 (Error<50) and less than 100 (Error<100). The experimental results are shown in Table 1. The results show that the fine-tuned LLM generally outperforms the PI LLM in terms of prediction performance, despite its large parameter size. Among them, the native Llama3-8B model has the worst performance, with an average error of 204 and a probability of error <50 of only 36%. The Llama3-8B model fine-tuned by Lora has an average error of 63, an error <50 probability increased to 56%, and an error <100 probability of 81%. The fully fine-tuned Qwen2-1.5B model has better prediction performance than the Lora-fine-tuned Llama3-8B model. Ablation experiments show that after data augmentation, the average prediction error is reduced to 57, and the probability of error <50 is increased to 63%, further improving the prediction performance.

[0118] Figure 4 This is a comparison chart of the inference speed and inference time of the task sorting and balancing algorithm based on deep reinforcement learning in Example 2 of the present invention, with the number of tasks configured as 1000, 2000, 3000, 4000, and 5000. The VLLM framework uses a task sorting and balancing algorithm based on deep reinforcement learning. Random refers to the inference speed of the original order of tasks, or the random sorting, while Sort_reverse refers to the inference speed of descending sorting based on the predicted task response length. DRL_balance is a task sorting and balancing scheduling algorithm based on deep reinforcement learning. The results show that compared with Sort_reverse, the DRL_balance algorithm accelerates by an average of 11.9%, and compared with Random, the DRL_balance algorithm accelerates by an average of 8.5%. As the number of tasks increases proportionally, the inference time also increases proportionally. However, under the same number of inferences, the DRL_balance algorithm takes the least time.

[0119] Figure 5This is a comparison chart of the inference speed of Example 2 of the present invention, which configures the number of GPUs to be 2, 4, and 8, and generates 1000, 2000, 3000, 4000, and 5000 tasks. It uses a polling algorithm based on task response length prediction and a task sorting and balancing algorithm based on deep reinforcement learning. Among them, Worst represents the lower limit of the inference speed of this experiment. The Random algorithm is based on the original or random sorting of the test set, and according to the number of inference tasks, it is evenly distributed to each graphics card. Round_robin uses a polling algorithm to distribute tasks based on the predicted task response length. The DRL_rr algorithm, based on Round_robin, uses deep reinforcement learning to balance task sorting, balances the scheduling of large language model inference tasks on different types of computing resources, and further improves the inference speed. The experimental results show that regardless of the number of inference tasks and the number of GPUs, the DRL_rr algorithm maintains its leading position, with an average acceleration of 6.7% compared to the Random algorithm. It's also worth noting that the performance of the Random algorithm is unstable. Its random nature causes performance fluctuations, with a lower bound of worst, while the DRL_rr algorithm can maintain near-optimal performance. Furthermore, while the round_robin algorithm achieves fine-grained load balancing based on the response length of the inference task, it doesn't balance the GPU's internal computing resources, resulting in a consistent difference in inference speed compared to DRL_rr.

[0120] Figure 6 This is a comparison chart of Example 2 of the present invention, in which the number of tasks is configured as 3000, 4000, and 5000, corresponding to randomly generated schedulable delays, and the service energy consumption is calculated using a power selection algorithm based on deep reinforcement learning under the vllm framework. Conserve refers to the energy consumption of running the large language model inference task at the highest power without considering the voltage and frequency adjustment technology, and Optimal refers to the global optimal solution of the test. By traversing all power levels and their execution times, the minimum energy consumption that meets the real-time constraints is selected. These two values ​​represent the upper and lower levels of the problem, and are compared with the power level selection algorithm DRL_er based on deep reinforcement learning proposed in this patent to evaluate its performance. The experimental results show that as the delay increases, the energy consumption required to execute the task decreases, and at the lowest power, it is reduced by up to 21%. Secondly, under different numbers of large language model push tasks and different delays, the quality of the solution obtained by the DRL_er algorithm is close to the optimal solution Optimal, with an average difference of 3.3%;

[0121] Figure 7This chart compares the service energy consumption of Example 2 of the present invention, using a round-robin algorithm based on task response length prediction and a deep reinforcement learning-based task ranking and power selection algorithm under the VLLM framework, with randomly generated schedulable delays. The baseline approach uses multi-GPU load balancing based solely on the number of tasks, without considering DVFS, and using the highest power for inference within the VLLM framework. The optimal approach uses the energy consumption of executing tasks at the minimum power level that meets real-time constraints by traversing all GPUs, all power levels, and their execution times. These two values ​​represent the upper and lower bounds of the problem. The RR_DRL_er algorithm combines the round-robin algorithm based on task response length prediction and the deep reinforcement learning-based power selection algorithm for comparison. Experimental results show that regardless of the number of GPUs, the RR_DRL_er algorithm's solution quality approaches the optimal solution, with an average difference of 3.7%. When the number of GPUs is 8, it can achieve energy savings of up to 26.6%.

[0122] Although the embodiments disclosed herein are as described above, the contents described herein are merely embodiments for facilitating understanding of the present invention and are not intended to limit the present invention. Any person skilled in the art may make any modifications and variations in the form and details of the embodiments without departing from the spirit and scope of the present invention. However, the scope of patent protection of the present invention shall remain subject to the scope defined by the appended claims.

Claims

1. A method for optimizing energy consumption scheduling of large language model inference computing services based on task length prediction, characterized in that: The steps include: Step (1) First, the Alpaca-52k instruction dataset is used as the input source, the large language model to be predicted is inferred, the number of its output tokens is counted, and each input data is labeled to create a dataset; Step (2): Fine-tune the Qwen2-1.5B large language model using the Alpaca-52k instruction dataset and Llama3-8B inference response length labels in step (1) to predict the response length of the large language model inference task; Step (3), using the Qwen2-1.5B model fine-tuned in step (2), predict the response length of the Llama3-8B inference task, sort the tasks in descending order, and use the polling algorithm to allocate tasks to achieve load balancing; Step (4) performs task balancing and sorting based on the inference tasks sorted in descending order after load balancing in step (3) to improve the inference speed of the large language model; Step (5): After the tasks are balanced and sorted based on step (4), the power selection algorithm of deep reinforcement learning is used to reduce the computing power while meeting the latency of the large language model inference task, so as to reduce the energy consumption of the large language model inference calculation.

2. The method for optimizing energy consumption scheduling of large language model reasoning computing services based on task length prediction according to claim 1 is characterized in that: In step (1), for the response length prediction task of the Llama3-8B model, an automated annotation framework is established. The Alpaca-52k instruction dataset is used as the input source, and the response text is generated through model reasoning. The number of tokens is recorded as a label. Then the data is cleaned to remove abnormal data, and a stratified sampling strategy is implemented using the diversity enhancement module. Based on the response length distribution, 10 quantile intervals are drawn to ensure that the proportion of samples in each subset is controlled between 8% and 12%, which significantly improves the generalization ability of the model. Finally, 27,000 samples are selected, and the average response length of the samples is 273, the minimum length is 2, and the maximum length is 1745.

3. The method for optimizing energy consumption scheduling of large language model reasoning computing services based on task length prediction according to claim 2 is characterized in that: In step (2), design the prompt words for Qwen2-1.5B fine-tuning: "<|im_start|>system\n You are a helpful assistant.<|im_end|>\n<|im_start|>user\n" to form a fixed template for the Qwen2 large language model; then add the question that needs to predict the length of the response; "Before responding to the question, you have to predict the length of the question response. Print the estimated number of words in your response in the first line.\n" constitutes a prompt word, so that the model only predicts the response length of the question and only needs to output a number; finally, the supervisory signal in step (1), that is, the label of the response length, is added to form the fine-tuning target. Since the Qwen2-1.5B lightweight large language model is selected, the full parameter fine-tuning method with the best fine-tuning effect is directly used. In order to further accelerate the fine-tuning speed of the Qwen2-1.5B model, a course learning method based on Bayes Fisher information is adopted. Specifically, based on the concept of course learning, it imitates the "step-by-step" principle in human education, and classifies the training data according to difficulty. The model first learns simple samples and gradually transitions to complex samples. Fisher information reflects the richness of information about parameters in the data. If a small change in the parameter leads to a significant change in the data distribution, it means that the data provides a large amount of information and the Fisher information is high. Conversely, if the parameter change has little effect on the data distribution, the Fisher information is low. The Fisher information of the samples in the training set under the model parameters is calculated to divide the training data. When the model has multiple unknown parameters θ=(θ1,θ2,...,θ d ), Fisher information I ij (θ) is a d×d symmetric matrix: The data distribution P(X|θ) is usually unknown, and the expectation cannot be calculated analytically. Moreover, for high-dimensional parameters θ∈R d , the complexity of storing and computing matrices is O(d 2 ) or even O(d 3 )(find the inverse time), is almost impossible in deep learning, and experience The above problem is solved by replacing the theoretical expectation with the sample mean and calculating directly based on the observed data: In addition, considering this problem in the Bayesian framework, the training data obeys the prior distribution of B(X|θ) in the training data set and the posterior distribution of P(X|θ) under large model parameters, while the BayesFisher information I Bayes The matrix is ​​defined as the sum of the information content of the posterior distribution and the information content of the prior distribution: I Bayes =I Prior +I Data , (3) Among them, I Prior is the Fisher information of the prior distribution, I Data is the Fisher information of the posterior distribution. Based on this, according to the BayesFisher information of the training samples, simple samples are first used to speed up the convergence of the model, and then the data is gradually increased during the training process to reduce the total training fine-tuning time. In the training parameter settings, the batch size (batch_size) is set to 8, which balances the GPU memory limitation and the stability requirement of gradient update. The AdamW optimizer is used for parameter update. This optimizer effectively improves the regularization effect of the traditional Adam optimizer through the decoupled weight decay mechanism. The initial learning rate is set to 1×10 -4 , this value refers to the fine-tuning convention of the pre-trained language model to ensure the rationality of the parameter update step size; in addition, a linear decay learning rate schedule is designed, and the total number of training steps is calculated by the formula: total number of steps = (total amount of data × number of training rounds) / batch size. No warm-up phase is set (warmup steps = 0), and the learning rate is increased from the initial value of 1×10 -4 At the beginning, it decreases linearly to zero as the training progresses. This monotonically decreasing strategy helps fine-tune the parameters in the later stages of training. After fine-tuning training is completed, save the Qwen2-1.5B model to predict the response length of the Llama3-8B inference task, providing a basis for subsequent optimization scheduling.

4. The method for optimizing energy consumption scheduling of large language model inference computing services based on task length prediction according to claim 1 is characterized in that: In step (3), the response length of the inference task of Llama3-8B is predicted based on the fine-tuned Qwen-1.5B model proposed in step (2), and then sorted in descending order according to the predicted response length. Then, a polling algorithm is used to divide the input data into multiple small batches based on the data parallel strategy, and each small batch is assigned to different computing nodes for parallel processing. During inference, the batches processed by different nodes are independent, and the results are finally merged to achieve fine-grained load balancing.

5. The method for optimizing energy consumption scheduling of large language model inference computing services based on task length prediction according to claim 1 is characterized in that: In step (4), a task balanced sorting method based on deep reinforcement learning is designed. The large language model reasoning task is divided into prefill and decode stages. The prefill stage is the initial processing link of the LLM reasoning task. The main goal is to perform full calculations on the input sequence, generate and cache the Key and Value vectors of all positions, and lay the foundation for subsequent decoding. The prefill stage can make full use of the parallel computing capabilities of the GPU to process the entire input sequence at one time. The computing density is high and the peak memory usage is significant. The prefill and decode stages correspond to the "context initialization" and "sequence generation" of LLM reasoning respectively. The two have complementary characteristics in terms of GPU resource requirements. After the task response lengths predicted in step (2) are sorted in descending order, starting from the first task, a task is selected every M tasks until the round ends, and then the second round of selection starts from the second task. The process is repeated until all tasks are selected. Use deep reinforcement learning to explore the appropriate M value and accelerate the reasoning speed of large language model reasoning. The state space, action space, and reward function of reinforcement learning are designed as follows: State space: The state is the basis of the agent's decision-making. In order to explore the appropriate M value, the agent first observes the state of the task, sorts the predicted values ​​of the task response length in descending order, and then calculates the predicted values. Action space: Since the goal of reinforcement learning is to explore the value of M, the number of reasoning tasks in each batch is not the same, so the value of M should change accordingly. Assuming that the number of reasoning tasks in this batch is N, let M = N / a, where a∈{10,20,30,40,50,60,70,80,90,100}, so the action space of reinforcement learning is [10,20,30,40,50,60,70,80,90,100]. Reward function: To guide the agent to explore and find the appropriate action a, the reward function is the inference speed (token / s) of the batch task. When the inference speed is low, the agent is penalized, and when the inference speed is high, the agent is rewarded. Because the inference speed fluctuates between 900-1200 (token / s), in order to reflect the difference in rewards, it is mapped to e speed / 200 range, better guide the agent to learn, The SAC (Soft Actor-Critic) algorithm is used. In addition, the training environment required for reinforcement learning is also written. Specifically, the environment reads 12,000 samples from the Alpaca instruction dataset as the training dataset train_set, and then randomly selects a certain amount of training data. After the intelligent agent interacts, a certain amount of training data is randomly selected again, and this process is repeated until there are no samples in train_set. In addition, the Actor and Critic of deep reinforcement learning use a 3-layer fully connected neural network with a hidden layer parameter of 128. In the test phase, the computing time is negligible. The overall training process of the task sorting and balanced scheduling algorithm based on deep reinforcement learning is as follows: first, the initial The parameters training round number Epoch, batch size batch_size, experience pool threshold minimal_size, Actor network learning rate actor_lr, Critic network learning rate critic_lr, Actor network parameter θ, Critic1 network parameter ω1 and Critic2 network parameter ω2 are optimized; then the training environment is reset, the training set train_set is read, and the training data train_data randomly selected from the environment is obtained. The fine-tuned Qwen2-1.5B model is used to predict the response length length of the training data. The environment state s is obtained by sorting in descending order according to the predicted length length and normalizing it. t ; Then the Actor network is based on s t , calculate the probability distribution of each action and get a t ; Then, let the Llama3-8B model infer, record the inference speed, and calculate the reward r t , and get the next batch of training data and the next state s returned by the training environment t+1 , and then the quaternion {s t ,a t ,r t ,s t+1 } put into experience pool B; when the number of experience pool B reaches minimal_size, start updating the network parameters, that is, sampling batch_size samples from experience pool B, and calculating the output value1 of the Critic1 network and the output value2 of the Critic2 network based on this sample and selecting the smaller value m , then calculate the policy entropy entropy output by the Actor network, and use this to calculate the temporal difference residual δ to calculate the loss update network of the Critic1 network and the Critic2 network; then use the new Critic1 network and Critic2 network parameters ω1 and ω2 and the policy entropy entropy of the Actor network to update the Actor network parameters θ; finally, after the training is completed, save the Actor network parameters for online decision scheduling.

6. The method for optimizing energy consumption scheduling of large language model inference computing services based on task length prediction according to claim 1 is characterized in that: In step (5), a power selection algorithm based on deep reinforcement learning is designed. Given N large language model inference tasks {τ1,…,τ N }, the length of each task is unknown, and the fine-tuned Qwen-1.5B in step (2) is used to predict the response length. The predicted length of each task is e i , and the unified deadline of all tasks is D. According to the real-time requirement, each task must be completed before the deadline D. In addition, these tasks are non-preemptive, that is, once a task starts running in a card, it will continue to run until it is completed. The large language model inference task is executed on M GPUs (RTX 3090) {G1,…,G M }, and the GPU supports DVFS technology, that is, under the GPU Boost intelligent acceleration framework, the core frequency can be dynamically adjusted in 1MHz steps. Users use the NVIDIA system management interface (nvidia-smi) to query the GPU device clock, temperature, and power information, and set the frequency and power of the GPU device. In the established model, the GPU device sets L power levels {p1,…,p L }, and its corresponding Llama3-8B large language model inference speed under the vllm framework is {s1,…,s L }(token / s), and model the problem accordingly: Task allocation constraint: Because a task only needs to be assigned to one GPU and all tasks must be assigned, define a binary variable x ik , when its value is 1, it means that the task τ i Assigned to G k , so the constraint is expressed as: Power selection constraint: Considering the system-level DVFS, that is, each GPU executes all assigned tasks at a power level, define the binary variable y kl When its value is 1, it means GPUG k The task is executed at power level l, so the constraint is expressed as: Real-time constraint: Each real-time task τ i The execution must be completed before the deadline D, so the constraint is expressed as: Objective equation: The energy consumption required to complete all inference tasks is expressed as: Original problem: Minimize the total energy consumption while satisfying all constraints: Based on the fine-tuned Qwen-1.5B model, we predict the response length of large language model inference tasks. We also fine-tune the GPU power based on the performance released at each power level to balance GPU computing performance and power consumption. Based on this, we design a power selection algorithm based on deep reinforcement learning to explore the optimal balance between GPU computing performance and energy consumption. The design of its action space, state space, and reward function is shown below: Action Space: The goal of this reinforcement learning approach is to balance GPU performance and energy consumption. Therefore, the agent's action is to select an appropriate power level l. Six power levels are set: [350, 325, 300, 275, 250, 225]. In practice, more power levels are set, with 225W as the lower limit. First, the lowest inference speed is calculated based on the predicted task response length (length) and the deadline D: The power level is selected accordingly, and then reinforcement learning is used to adjust the power level. Depending on the situation, the power level can be increased, decreased, or kept unchanged. Therefore, the action space is [0,1,2], representing increasing the power level, keeping it unchanged, and decreasing the power level respectively. State space: In order for the agent to select the appropriate power level l, the reasoning tasks are first sorted in descending order according to the predicted response length. Then, the predicted minimum reasoning speed is calculated according to formula (6). Finally, the normalized features are embedded to form the state space. It is hoped that the agent can explore the optimal mapping relationship between the minimum processing speed and the power level according to different task conditions. Reward function: To guide the agent in selecting the appropriate power level l, the reward function is defined as the total energy consumption of completing the task. When the task execution time t is less than the deadline D, a certain reward is given based on the total energy consumption of completing the task, and the reward decreases as the total energy consumption increases: reward i =-αtp l , (10) where p l is the power of the lth level, α is the normalization coefficient. If the time t to complete the task is greater than the deadline D, it violates the real-time constraint and is penalized: reward i =-1. (11) First, you need to write the training environment required for reinforcement learning. That is, the environment first reads 12,000 samples from the Alpaca instruction dataset as the training dataset train_set, then randomly selects a certain amount of training data, and generates a random deadline according to the selected amount of training data, and ensures that the GPU can complete the execution at the highest power level before its deadline. Then, feature embedding is performed and input into the Actor network to make decisions. After interacting with the intelligent agent, a certain amount of training data is randomly selected again. This process is repeated until there are no samples in train_set, and the next round of training begins. The Actor and Critic use a 3-layer fully connected neural network with a hidden layer parameter of 128. In the test phase, task sorting and balanced scheduling based on deep reinforcement learning are performed. The overall training process of the algorithm is as follows: first, initialize the parameters training round number Epoch, batch size batch_size, experience pool threshold minimal_size, Actor network learning rate actor_lr, Critic network learning rate critic_lr, Actor network parameter θ, Critic1 network parameter ω1 and Critic2 network parameter ω2, then reset the training environment, read the Llama3-8B response length dataset train_set, obtain the training data train_data randomly selected from the environment, generate a random deadline D based on train_data, use the fine-tuned Qwen2-1.5B model to predict the response length length of the training data, sort in descending order, and calculate And normalize to get the environment state s t ; Then the Actor network is based on s t , calculate the probability distribution of each action and get a t , and predict the minimum inference speed to adjust the power level; then, let the Llama3-8B model infer, record the power level and energy consumption, and calculate the reward r t , and get the next batch of training data and the next state s returned by the training environment t+1 , and then the quaternion {s t ,a t ,r t ,s t+1 } put into experience pool B; when the number of experience pool B reaches minimal_size, start updating the network parameters, that is, sampling batch_size samples from experience pool B, and calculating the output value1 of the Critic1 network and the output value2 of the Critic2 network based on this sample and selecting the smaller value m , then calculate the policy entropy enyropy output by the Actor network, and use the temporal difference residual δ to calculate the loss update network of the Critic1 network and the Critic2 network; then use the new Critic1 network and Critic2 network parameters ω1 and ω2 and the policy entropy enyropy of the Actor network to update the Actor network parameters θ. Finally, save the trained Actor network for online decision scheduling. Through the above steps, we have realized the preparation of the dataset for fine-tuning the response length prediction of the large language model inference task, data enhancement, fine-tuning of the response length prediction of the Llama3-8B model inference task based on Qwen2-1.5B, and computing services such as task allocation load balancing, GPU computing resource balancing, and energy consumption optimization scheduling.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the energy consumption optimization scheduling method for large language model reasoning computing services based on task length prediction is implemented as described in any one of claims 1 to 6 above.

8. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the computer instruction is executed by the processor, the energy consumption optimization scheduling method for large language model reasoning computing services based on task length prediction is implemented as described in any one of claims 1-6.

Images