Distributed large model inference method based on token awareness and lyapunov optimization

Abstract

The present application relates to the field of distributed large model inference, and particularly relates to a distributed large model inference method based on token perception and Lyapunov optimization. The scheme comprises: after receiving a user input prompt, predicting the output token length of the corresponding task through a length perception semantic module, calculating the corresponding total amount of work, based on the token length prediction result, the real-time state of the device and the long-term virtual queue state, generating a final unloading decision through the Lyapunov guided unloading optimization module iterative unloading algorithm; the client unloads the task to the corresponding cloud edge device according to the final unloading decision, and the device executes the large model inference; the scheduler updates the virtual queue length according to the actual task execution time and the device computing power utilization rate, and provides input for the next time slot decision. The present application is suitable for inference systems utilizing distributed large models.

Description

Technical Field

[0001] This invention relates to the field of distributed large model reasoning, and specifically to a distributed large model reasoning method based on token awareness and Lyapunov optimization. Background Technology

[0002] Large-scale models, with their trillions of tokens trained and billions of parameters, have demonstrated transformative capabilities in natural language understanding, logical reasoning, and complex task processing, and have been widely applied in real-world scenarios such as intelligent customer service, content generation, and decision support. However, the deployment of large-scale models faces significant challenges: limited by the massive number of parameters and computational requirements, traditional deployment methods often rely on cloud servers, but centralized cloud deployment leads to a significant increase in response latency for time-sensitive tasks (such as real-time voice interaction and instant question answering), making it difficult to meet user experience requirements.

[0003] To address this issue, cloud-edge-device collaborative deployment solutions have emerged, deploying small-scale models on edge servers and large-scale models in the cloud. This achieves efficient collaboration through task offloading, with lightweight tasks handled at the edge and complex tasks handled in the cloud. However, this solution still faces three core challenges in practical applications: 1) Inference time fluctuations caused by autoregressive architecture: Mainstream large models use a transformer architecture, requiring token-by-token output generation during inference. The length of the output token is significantly affected by the semantic complexity of the input prompts and the task type, with fluctuations reaching several times or even tens of times, leading to uncontrollable inference time and severely impacting task scheduling efficiency. 2) Dynamically random deployment environment: In real-world scenarios, user request volume and the number of active clients fluctuate dramatically over time, and unstable network bandwidth and latency further exacerbate the uncertainty of task scheduling. Existing solutions are mostly optimized based on static environment assumptions and cannot cope with long-term randomness. 3) Resource constraints differences among heterogeneous devices: Cloud-edge-device systems consist of devices with different computing capabilities and communication characteristics, and these devices face long-term computational load constraints. Existing resource allocation schemes often ignore equipment heterogeneity, resulting in some equipment being overloaded and others being idle, leading to low overall system efficiency. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide a distributed large model reasoning method based on token awareness and Lyapunov optimization, which achieves low latency, high reliability and efficient resource utilization for large model reasoning in dynamic heterogeneous environments.

[0005] This invention achieves the above objectives by adopting the following technical solution: This invention provides a distributed large-model inference method based on token awareness and Lyapunov optimization, comprising:

[0006] S1. After receiving user input prompts, the length-aware semantic module predicts the length of the output token for the corresponding task and calculates the corresponding total workload. The length-aware semantic module is based on a fine-tuned pre-trained language model and enhances the sensitivity of token length-related semantics through a feature recalibration mechanism to achieve accurate prediction of the output token length.

[0007] S2. Based on the token length prediction result, the real-time status of the device and the long-term virtual queue status, the final unloading decision is generated by the Lyapunov guided unloading optimization module iteratively unloading algorithm.

[0008] S3. The client offloads the task to the corresponding cloud-edge device according to the final offload decision, and the device performs large model inference; the scheduler updates the virtual queue length according to the actual task execution time and device computing power utilization, providing input for the next time slot decision.

[0009] Furthermore, step S1 specifically includes:

[0010] S101. The user input prompt is segmented into words and converted into a token sequence processed by a pre-trained language model. An initial semantic vector is generated through the embedding layer.

[0011] S102. For the initial semantic vector, the feature weights are dynamically adjusted through three steps: compression, activation, and recalibration to generate a recalibrated feature vector that highlights the semantic information related to the token length.

[0012] S103. Input the recalibrated feature vector into the output layer, generate the predicted output token length through linear transformation and activation function, and calculate the computing power requirements of the pre-filling stage and the decoding stage for the corresponding task based on the predicted output token length to obtain the total workload.

[0013] Furthermore, compression, excitation, and recalibration specifically include:

[0014] Compression: Global features are aggregated through adaptive average pooling and adaptive max pooling to generate descriptors containing global statistics and salient features;

[0015] Incentive: Input the descriptor into a two-layer fully connected network, and generate an attention weight vector through nonlinear transformation to quantify the importance of each feature for token length prediction;

[0016] Recalibration: The weight vector is applied to the initial semantic vector through element-wise multiplication to generate a recalibrated feature vector.

[0017] Furthermore, step S2 specifically includes:

[0018] S201. Collect status information in each time slot t;

[0019] Task Status: Collection of Active Tasks Workload of each task Delay sensitivity Precision sensitivity ;

[0020] Device status: Computing power of cloud-edge-device servers Current network speed Cloud-edge-device speed Cloud-edge-device speed ;

[0021] Virtual queue status: Virtual queue length for each device ;

[0022] Long-term computing power threshold ;

[0023] S202, Define the virtual queue update rules;

[0024] Define the virtual queue update formula:

[0025]

[0026] In the formula, , Unload decision variables for binary;

[0027] when When the queue length increases, it indicates that the current computing power utilization of device j exceeds the limit and subsequent task allocation needs to be reduced.

[0028] S203, Iterative unloading algorithm;

[0029] The long-term user experience quality optimization is transformed into a time-slot-by-time integer nonlinear programming problem, which is solved efficiently by an iterative offloading algorithm with damping and congestion control.

[0030] Furthermore, step S203 specifically includes:

[0031] Constructing the cost matrix:

[0032] In the k-th iteration, construct the task-equipment cost matrix. Each element in the cost matrix It consists of base costs and congestion penalties;

[0033] The base cost includes communication latency. , For the amount of task data, Calculate latency for network protocol latency. , This indicates the task that arrives first in the same time slot, and the Lyapunov item;

[0034] Optimal allocation solution:

[0035] Substituting the cost matrix into an integer linear programming problem, we can solve for the optimal unloading decision in the current iteration as follows:

[0036] Each task is assigned to only one device;

[0037] Load-aware updates:

[0038] Using damping factor To smoothly update device-sensing load and avoid iterative oscillations, the following method is used:

[0039] ;

[0040] Convergence criteria:

[0041] Repeat the steps of constructing the cost matrix, solving for the optimal allocation, and sensing the load update until the offloading decision is made in two iterations. and The difference is less than the threshold, or the maximum number of iterations has been reached. Output the final unloading decision .

[0042] The beneficial effects of this invention are as follows:

[0043] This invention achieves accurate token length prediction through a length-aware semantic module and optimizes long-term user experience quality and system stability in dynamic heterogeneous environments through a Lyapunov guided unloading optimization module, ultimately achieving low-latency, high-precision, and resource-efficient cloud-edge-device large-scale collaborative inference. Attached Figure Description

[0044] Figure 1 This is a flowchart of the distributed large model inference method based on token awareness and Lyapunov optimization provided by the present invention. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0046] This invention provides a distributed large-model inference method based on token awareness and Lyapunov optimization, such as... Figure 1 As shown, it specifically includes:

[0047] S1. After receiving user input prompts, the length-aware semantic module predicts the length of the output token for the corresponding task and calculates the corresponding total workload. The length-aware semantic module is based on a fine-tuned pre-trained language model and enhances the sensitivity of token length-related semantics through a feature recalibration mechanism to achieve accurate prediction of the output token length.

[0048] S101. Segment the user input prompt into words, convert it into a token sequence that can be processed by the pre-trained language model ModernBERT, and generate an initial semantic vector through the embedding layer.

[0049] S102. For the initial semantic vector, the feature weights are dynamically adjusted through three steps: compression, activation, and recalibration to generate a recalibrated feature vector that highlights the semantic information related to the token length.

[0050] Compression: Global features are aggregated through adaptive average pooling and adaptive max pooling to generate descriptors containing global statistics and salient features, as follows:

[0051]

[0052] In the formula, z represents the initial semantic vector, and s represents the descriptor with dimension . This forms a bottleneck structure to reduce computational cost, which represents the feature information dimension after adaptive average pooling and adaptive max pooling. The hidden layer information dimension is much smaller than the input. The advantage of doing this is that it allows us to aggregate high-dimensional feature information while saving computational overhead.

[0053] Incentive: The descriptor is input into a two-layer fully connected network, and an attention weight vector e is generated through a nonlinear transformation to quantify the importance of each feature for token length prediction, as follows:

[0054]

[0055] in, , For FC layer weights, The sigmoid function normalizes e to [0,1].

[0056] Recalibration: The weight vector e is applied to the initial semantic vector z through element-wise multiplication to generate a recalibrated feature vector, as follows:

[0057]

[0058] In the formula, This step involves recalibrating the feature vector, which amplifies features related to token length and suppresses irrelevant features.

[0059] S103. Input the recalibrated feature vector into the output layer, generate the predicted output token length through linear transformation and activation function, and calculate the computing power requirements of the pre-filling stage and the decoding stage for the corresponding task based on the predicted output token length to obtain the total workload.

[0060] S2. Based on the token length prediction result, the real-time status of the device and the long-term virtual queue status, the final unloading decision is generated by the Lyapunov guided unloading optimization module iteratively unloading algorithm.

[0061] S201. Collect status information in each time slot t;

[0062] Task Status: Collection of Active Tasks Workload of each task Delay sensitivity Precision sensitivity ;

[0063] Device status: Computing power of cloud-edge-device servers Current network speed Cloud-edge-device speed Cloud-edge-device speed ;

[0064] Virtual queue status: Virtual queue length for each device ;

[0065] Long-term computing power threshold ;

[0066] S202, Define the virtual queue update rules;

[0067] Define the virtual queue update formula:

[0068]

[0069] In the formula, , Unload decision variables for binary;

[0070] when When the queue length increases, it indicates that the current computing power utilization of device j exceeds the limit and subsequent task allocation needs to be reduced.

[0071] S203, Iterative unloading algorithm;

[0072] The long-term user experience quality optimization is transformed into a time-slot-by-time integer nonlinear programming problem, which is efficiently solved using an iterative offloading algorithm with damping and congestion control. Specifically, this includes:

[0073] Constructing the cost matrix:

[0074] In the k-th iteration, construct the task-equipment cost matrix. Each element in the cost matrix (The cost of assigning task i to device j) consists of the base cost and the congestion penalty;

[0075] The base cost includes communication latency. , For the amount of task data, Calculate latency for network protocol latency. , Indicates the tasks arriving earlier in the same time slot, and the Lyapunov term; virtual queue length. Corresponding penalties; congestion penalties: based on the device-aware load of the previous iteration. Penalize devices that are overloaded to prevent overload: , This is the congestion penalty coefficient. This represents the congestion penalty value for this iteration. This indicates the previous round of sensing load.

[0076] Optimal allocation solution:

[0077] Substituting the cost matrix into an integer linear programming problem, we can solve for the optimal unloading decision in the current iteration as follows:

[0078] Each task is assigned to only one device;

[0079] Load-aware updates:

[0080] Using damping factor To smoothly update device-sensing load and avoid iterative oscillations, the following method is used:

[0081] ;

[0082] This formula is used to calculate the current sensed load update. It uses a damping factor to allocate the contribution of historical sensed load and current allocated workload to the current sensed load, and then updates the value of the current sensed load.

[0083] Convergence criteria:

[0084] Repeat the steps of constructing the cost matrix, solving for the optimal allocation, and sensing the load update until the offloading decision is made in two iterations. and The difference is less than the threshold, or the maximum number of iterations has been reached. Output the final unloading decision .

[0085] S3. The client offloads the task to the corresponding cloud-edge device according to the final offload decision, and the device performs large model inference; the scheduler updates the virtual queue length according to the actual task execution time and device computing power utilization, providing input for the next time slot decision.

[0086] The invention will now be described in conjunction with specific scenarios.

[0087] Client: There are 10 clients that receive large model requests. The large model inference requests cover four categories: text understanding, logical reasoning, long text generation, and complex multi-turn dialogue. The request rate can fluctuate by up to 10 times per second. Different requests have significantly different requirements for latency (≤300ms-1000ms) and accuracy (≥0.75-0.95).

[0088] Edge servers: There are 5 edge servers with relatively weak computing power, deploying the LLaMA-7B model.

[0089] Cloud servers: There are 3 cloud servers with strong computing power, deploying the LLaMA-70B model.

[0090] Scheduler: A cloud-based central scheduler that integrates a length-aware semantic module and a Lyapunov-guided offloading optimization module, enabling real-time resource allocation and scheduling based on the state of the scenario.

[0091] Step 1: Token Length Prediction (executed by the length-aware semantic module). After the scheduler receives 10 concurrent client requests, it segments the input prompt for each request into a token sequence that ModernBERT can process, generating an initial semantic vector; then, after feature recalibration, it passes through another layer of network to obtain the output predicted length.

[0092] Step 2: Task offloading decision (Lyapunov-guided offloading optimization module). The scheduler collects task status, device status, and virtual queue status, and updates the virtual queue; then, an iterative offloading algorithm is used to solve the problem: a cost matrix is ​​constructed, an integer linear programming problem is solved based on this matrix, and the device load is smoothed according to the damping factor. The process is repeated iteratively to obtain the final allocation strategy.

[0093] Step 3: Each computing device runs model inference according to the assigned request, completes the task, and returns data such as computing latency and actual load to the central scheduler, providing a basis for decision-making for the scheduling task of the next time slot.

[0094] In terms of token length prediction accuracy, the L1 loss of the length-aware semantic module of this invention is 91.85, which is 0.2%, 17.3%, 16.15%, and 92.62% lower than LoRA (92.07), LSTM (107.79), Transformer (106.69), and Qwen2.5-7B (176.93), respectively. Moreover, LAS only requires 0.09M trainable parameters, which is 99% lower than LoRA (8.75M), significantly reducing the computational cost.

[0095] In terms of long-term user experience quality and system stability, this invention, through its Lyapunov-guided offloading optimization module, significantly outperforms the baseline scheme in terms of Lyapunov rewards (combining latency, accuracy, and queue stability, with higher values ​​being better) under different cloud-edge-device server configurations (N=4 / 15 / 20, U=6 / 15 / 20). For example, when N=4 and U=15, the LOO reward reaches 36602, which is an improvement of 113.8%, 169.9%, 14.1%, and 100.9% compared to Greedy-Accuracy (-265297), Greedy-Delay (13565), DiffusionRL (32077), and TransformerPPO (-4050226), respectively. At the same time, the long-term average computing power utilization of all devices is ≤ the threshold, the virtual queue length converges to a stable value, and there is no device overload phenomenon.

[0096] In terms of balancing latency and accuracy, the average response latency for latency-sensitive tasks is 382ms, a 16.2% reduction compared to DiffusionRL (456ms) and a 37.6% reduction compared to TransformerPPO (612ms). The average inference accuracy for accuracy-sensitive tasks reaches 0.89, a 23.6% improvement compared to Greedy-Delay (0.72) and an 85.4% improvement compared to edge local execution (0.48). The overall system resource utilization reaches 82.3%, a 45.1% improvement compared to Greedy-Accuracy (56.7%).

[0097] This invention effectively solves the problems of uncontrollable token length, difficulty in adapting to dynamic environments, and resource constraints of heterogeneous devices in large model inference in cloud-edge-device collaborative scenarios, and greatly improves the efficiency, stability and user experience of inference.

[0098] The above description is merely a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and environments, and can be altered within the scope of the concept described herein through the above teachings or related technologies or knowledge. Modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims.

Claims

1. A distributed large model inference method based on token-aware and Lyapunov optimization, characterized in that, Comprise: S1, after receiving user input prompt, predict the output token length of the corresponding task through the length-aware semantic module, calculate the corresponding total work, the length-aware semantic module is based on the fine-tuned pre-trained language model, and the token length related semantic sensitivity is enhanced through the feature recalibration mechanism to realize the accurate prediction of the output token length; S2, based on the token length prediction result, the real-time state of the device and the long-term virtual queue state, the Lyapunov guided offloading optimization module is used to generate the final offloading decision through the iterative offloading algorithm; S3, the client offloads the task to the corresponding cloud edge device according to the final offloading decision, and the device executes the large model inference; The scheduler updates the virtual queue length according to the actual task execution time and the device computing power utilization rate, and provides input for the next time slot decision; Step S2 specifically comprises: S201, collect state information at each time slot t; Task status: active task set , workload of each task , delay sensitivity , precision sensitivity ; Device status: computing power of cloud edge server , current network rate , cloud edge rate , cloud edge rate ; Virtual queue status: the length of the virtual queue corresponding to each device ; Long-term hash power threshold ; S202, define virtual queue update rule; Define the virtual queue update formula: ； In the formula, , is a binary offload decision variable; When the queue length increases, it indicates that the current computing power utilization of device j is over-standard, and the subsequent task allocation needs to be reduced; S203, iterative offloading algorithm; Convert the long-term user experience quality optimization into a per-time-slot integer nonlinear programming problem, and solve it efficiently through the iterative offloading algorithm with damping and congestion control; Construct the cost matrix: In the kth iteration, construct the task-device cost matrix Each element in the cost matrix is composed of a base cost and a congestion penalty; The base cost includes communication latency , for the task data volume, for the network protocol latency, computation latency , denotes the simultaneous slot first-arrival task, and Lyapunov term; Optimal allocation solution: Substitute the cost matrix into the integer linear programming problem to solve the optimal offloading decision of the current iteration, as follows: each task is assigned to only one device; Perception load update: Adopting damping factor The device-aware load is smoothed to avoid iterative oscillations in the following way: ； Convergence judgment: repeating the steps of building the cost matrix, solving for the optimal allocation, and updating the perceived load until the unloading decisions of two iterations differ by less than a threshold value, or a maximum number of iterations is reached with the difference is less than a threshold value, or a maximum number of iterations is reached outputting a final unloading decision .

2. The distributed large model inference method based on token-aware and Lyapunov optimization according to claim 1, wherein, Step S1 specifically comprises: S101, tokenize the user input prompt, convert it into a token sequence processed by the pre-trained language model, and generate an initial semantic vector through the embedding layer; S102, for the initial semantic vector, adjust the feature weights through three steps of compression, excitation and recalibration to generate a recalibrated feature vector, highlighting the semantic information related to the token length; S103, input the recalibrated feature vector into the output layer, generate the predicted output token length through linear transformation and activation function, and calculate the pre-padding stage computing power requirement and decoding stage computing power requirement based on the predicted output token length, to obtain the total work.

3. The distributed large model inference method based on token-aware and Lyapunov optimization according to claim 2, characterized in that, Compression, excitation and recalibration specifically comprise: Compression: aggregate global features through adaptive average pooling and adaptive max pooling to generate a descriptor containing global statistics and significant features; Excitation: input the descriptor into a two-layer fully connected network to generate an attention weight vector through nonlinear transformation, which quantifies the importance of each feature to token length prediction; Recalibration: apply the weight vector to the initial semantic vector through element multiplication to generate a recalibrated feature vector.

Images