Distributed large model reasoning method based on token perception and Lyapunov optimization

Abstract

The invention relates to the field of distributed large model reasoning, in particular to a distributed large model reasoning method based on token perception and Lyapunov optimization. According to the scheme, the method comprises the steps that after a user input prompt is received, the output token length of a corresponding task is predicted through a length perception semantic module, the corresponding total work amount is calculated, and a final unloading decision is generated through a Lyapunov guiding unloading optimization module iteration unloading algorithm based on a token length prediction result, the equipment real-time state and the long-term virtual queue state; the client side unloads the task to the corresponding cloud side end equipment according to the final unloading decision, and the equipment executes large model reasoning; and the scheduler updates the length of the virtual queue according to the actual task execution time and the equipment computing power utilization rate, and provides input for the next time slot decision. The method is suitable for an inference system using a distributed large model.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of distributed large model inference, in particular to a distributed large model inference method based on token perception and Lyapunov optimization. BACKGROUND

[0002] Large models have exhibited transformative capabilities in natural language understanding, logical reasoning, and complex task processing due to their trillion token training volume and tens of billions of parameters. They have been widely applied to practical scenarios such as intelligent customer service, content generation, and decision support. However, the deployment of large models faces significant challenges. Due to the huge parameter volume and computational requirements, traditional deployment methods rely on cloud servers. However, centralized deployment in the cloud significantly increases the response delay of time-sensitive tasks (such as real-time voice interaction and instant question answering), making it difficult to meet user experience requirements.

[0003] To address this issue, a cloud-edge-end collaborative deployment scheme has emerged, which deploys small-scale models on edge servers and large-scale models on the cloud. Through task offloading, it achieves efficient collaboration by processing lightweight tasks on edge servers and complex tasks on the cloud. However, this scheme still faces three major challenges in practical applications: 1) Inference time fluctuations caused by autoregressive architecture: The transformer architecture is widely used in mainstream large models, which requires generating output tokens one by one. The length of output tokens is significantly affected by the semantic complexity of input prompts and task types, with fluctuations ranging from several times to tens of times. This makes the inference time uncontrollable and severely affects task scheduling efficiency. 2) Dynamic and random deployment environment: In actual scenarios, user request volume and active client number fluctuate dramatically over time, and communication conditions such as network bandwidth and delay are unstable, further exacerbating the uncertainty of task scheduling. Existing solutions are based on static environment assumptions and cannot handle long-term time dimension randomness. 3) Resource constraint differences of heterogeneous devices: Cloud-edge-end systems consist of devices with different computing capabilities and communication characteristics, and these devices have long-term computing load constraints. Existing resource allocation schemes often ignore device heterogeneity, leading to overloading of some devices and idling of others, resulting in low overall system efficiency. SUMMARY

[0004] The present application aims to overcome the shortcomings of the prior art and provide a distributed large model inference method based on token perception and Lyapunov optimization, achieving low latency, high reliability, and resource-efficient utilization of large model inference in dynamic heterogeneous environments.

[0005] The present application achieves the above-mentioned objectives by adopting the following technical solutions. The present application provides a distributed large model inference method based on token perception and Lyapunov optimization, comprising:

[0006] S1, after receiving the user input prompt, the length-aware semantic module is used to predict the output token length of the corresponding task, and the total amount of work corresponding to the task is calculated. The length-aware semantic module is based on a fine-tuned pre-trained language model, and a feature recalibration mechanism is used to enhance the sensitivity of token length-related semantics, so as to realize accurate prediction of the output token length;

[0007] S2, based on the token length prediction result, the real-time state of the device and the long-term virtual queue state, the Lyapunov guided offloading optimization module is used to iteratively generate a final offloading decision;

[0008] S3, the client offloads the task to the corresponding cloud-edge device according to the final offloading decision, and the device executes the large model inference; the scheduler updates the virtual queue length according to the actual task execution time and the device computing power utilization rate, and provides input for the next time slot decision.

[0009] Further, step S1 specifically includes:

[0010] S101, the user input prompt is segmented, converted into a token sequence processed by a pre-trained language model, and an initial semantic vector is generated through an embedding layer;

[0011] S102, for the initial semantic vector, the feature weights are dynamically adjusted through compression, excitation and recalibration to generate a recalibrated feature vector, highlighting the semantic information related to the token length;

[0012] S103, the recalibrated feature vector is input into the output layer, the predicted output token length is generated through linear transformation and activation function, and the pre-filled stage computing power requirement and the decoding stage computing power requirement of the corresponding task are calculated based on the predicted output token length, and the total amount of work is obtained.

[0013] Further, the compression, excitation and recalibration specifically include:

[0014] Compression: global features are aggregated through adaptive average pooling and adaptive max pooling to generate a descriptor containing global statistics and significant features;

[0015] Excitation: the descriptor is input into a two-layer fully connected network to generate an attention weight vector through nonlinear transformation, which quantifies the importance of each feature to token length prediction;

[0016] Recalibration: the weight vector is applied to the initial semantic vector through element multiplication to generate a recalibrated feature vector.

[0017] Further, step S2 specifically includes:

[0018] S201, collect state information at each time slot t;

[0019] Task state: active task set , workload of each task , delay sensitivity , precision sensitivity ;

[0020] Device status: computing power of cloud-edge-server , current network rate , cloud-edge rate , cloud-edge rate ;

[0021] Virtual queue status: virtual queue length corresponding to each device ;

[0022] Long-term computing power threshold ;

[0023] S202, define virtual queue update rule;

[0024] Define virtual queue update formula:

[0025]

[0026] In the formula, , is a binary offloading decision variable;

[0027] When , the queue length increases, prompting that the current computing power utilization rate of device j is over-standard, and subsequent task allocation needs to be reduced;

[0028] S203, iterative offloading algorithm;

[0029] Convert long-term user experience quality optimization into a per-slot integer nonlinear programming problem, and efficiently solve it through an iterative offloading algorithm with damping and congestion control.

[0030] Further, step S203 specifically includes:

[0031] Construct a cost matrix:

[0032] In the kth iteration, construct a task-device cost matrix , each element in the cost matrix is composed of a basic cost and a congestion penalty;

[0033] The basic cost includes communication delay , is the task data volume, is the network protocol delay, and the calculation delay , represents the simultaneous slot first-arrival task, and the Lyapunov term;

[0034] Solution for optimal allocation:

[0035] Substituting the cost matrix into an integer linear programming problem, we can solve for the optimal unloading decision in the current iteration as follows:

[0036] Each task is assigned to only one device;

[0037] Load-aware updates:

[0038] Using damping factor To smoothly update device-sensing load and avoid iterative oscillations, the following method is used:

[0039] ;

[0040] Convergence criteria:

[0041] Repeat the steps of constructing the cost matrix, solving for the optimal allocation, and sensing the load update until the offloading decision is made in two iterations. and The difference is less than the threshold, or the maximum number of iterations has been reached. Output the final unloading decision .

[0042] The beneficial effects of this invention are as follows:

[0043] This invention achieves accurate token length prediction through a length-aware semantic module and optimizes long-term user experience quality and system stability in dynamic heterogeneous environments through a Lyapunov guided unloading optimization module, ultimately achieving low-latency, high-precision, and resource-efficient cloud-edge-device large-scale collaborative inference. Attached Figure Description

[0044] Figure 1 This is a flowchart of the distributed large model inference method based on token awareness and Lyapunov optimization provided by the present invention. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0046] This invention provides a distributed large-model inference method based on token awareness and Lyapunov optimization, such as... Figure 1 As shown, it specifically includes:

[0047] S1, after receiving the user input prompt, predicting the output token length of the corresponding task through the length-aware semantic module, calculating the corresponding total amount of work, the length-aware semantic module is based on the fine-tuned pre-trained language model, and the token length related semantic sensitivity is enhanced through the feature recalibration mechanism to realize the accurate prediction of the output token length;

[0048] S101, performing word segmentation on the user input prompt, converting it into a token sequence that can be processed by the pre-trained language model ModernBERT, and generating an initial semantic vector through an embedding layer;

[0049] S102, for the initial semantic vector, dynamically adjusting the feature weight through three steps of compression, excitation and recalibration to generate a recalibrated feature vector, highlighting the semantic information related to the token length;

[0050] Compression: aggregate global features through adaptive average pooling and adaptive max pooling to generate a descriptor containing global statistics and significant features, as follows:

[0051]

[0052] where z represents the initial semantic vector, s represents the descriptor, and the dimension is , forming a bottleneck structure to reduce the amount of calculation, which represents the feature information dimension after adaptive average pooling and adaptive max pooling much smaller than the input hidden layer information dimension This has the advantage of aggregating high-dimensional feature information while saving computational overhead.

[0053] Excitation: input the descriptor into a two-layer fully connected network to generate an attention weight vector e through nonlinear transformation, which quantifies the importance of each feature to token length prediction, as follows:

[0054]

[0055] where , is the FC layer weight, is the sigmoid function, which normalizes e to [0,1].

[0056] Recalibration: apply the weight vector e to the initial semantic vector z through element multiplication to generate a recalibrated feature vector, as follows:

[0057]

[0058] where represents the recalibrated feature vector, this step amplifies the features related to the token length and suppresses irrelevant features.

[0059] S103, input the recalibration feature vector into the output layer, generate the predicted output token length through linear transformation and activation function, and calculate the pre-filling stage computing power requirement and the decoding stage computing power requirement of the corresponding task based on the predicted output token length to obtain the total workload.

[0060] S2, based on the token length prediction result, the real-time state of the device and the long-term virtual queue state, generate the final offloading decision through the Lyapunov guided offloading optimization module iterative offloading algorithm;

[0061] S201, collect state information at each time slot t;

[0062] Task state: active task set , workload of each task , delay sensitivity , precision sensitivity ;

[0063] Device state: computing power of cloud edge server , current network rate , cloud edge rate , cloud edge rate ;

[0064] Virtual queue state: virtual queue length corresponding to each device ;

[0065] Long-term computing power threshold ;

[0066] S202, define the virtual queue update rule;

[0067] Define the virtual queue update formula:

[0068]

[0069] In the formula, , is a binary offloading decision variable;

[0070] When , the queue length increases, prompting that the current computing power utilization rate of the device j is out of standard, and the subsequent task allocation needs to be reduced;

[0071] S203, iterative offloading algorithm;

[0072] Convert the long-term user experience quality optimization into a time-slot-by-time-slot integer nonlinear programming problem, and efficiently solve it through the iterative offloading algorithm with damping and congestion control, which specifically includes:

[0073] Construct the cost matrix:

[0074] In the kth iteration, construct the task-device cost matrix , each element in the cost matrix (cost of assigning task i to device j) consists of a base cost and a congestion penalty;

[0075] The base cost contains the communication delay , , the task data volume, , the network protocol delay, and the computation delay , represents the first-arrival task in the same time slot, and the Lyapunov term; the virtual queue length corresponding penalty; the congestion penalty: the device-aware load based on the last round of iteration , increase the penalty for devices with too high load to avoid overloading: , is the congestion penalty coefficient, represents the congestion penalty value of this round of iteration, represents the last round of aware load.

[0076] Optimal allocation solution:

[0077] Substitute the cost matrix into the integer linear programming problem to solve the optimal offloading decision of the current iteration in the following way:

[0078] Each task is only assigned to one device;

[0079] Aware load update:

[0080] A damping factor is used to smooth the update of the device-aware load to avoid iteration oscillation in the following way:

[0081] ;

[0082] This formula is used to calculate the current aware load update formula, which uses the damping factor to allocate the contribution of the historical aware load and the current allocated workload to the current aware load to update the value of the current aware load.

[0083] Convergence judgment:

[0084] Repeat the steps of constructing the cost matrix, solving the optimal allocation, and updating the aware load until the offloading decisions of two iterations and differ by less than a threshold value, or the maximum number of iterations is reached , output the final offloading decision .

[0085] S3, the client offloads tasks to the corresponding cloud edge device according to the final offloading decision, and the device performs large model inference; the scheduler updates the virtual queue length according to the actual task execution time and the device computing power utilization rate, and provides input for the next time slot decision.

[0086] The application will be described below in combination with specific scenarios.

[0087] Clients: There are 10 clients that receive large model requests, and the initiated large model inference requests cover four categories of text understanding, logical reasoning, long text generation, and complex multi-round dialogue. The request rate fluctuates by 10 times per second, and different requests have significant differences in delay (≤300ms-1000ms) and precision (≥0.75-0.95) requirements.

[0088] Edge servers: There are 5 edge servers with weak computing power, and LLaMA-7B model is deployed.

[0089] Cloud servers: There are 3 cloud servers with strong computing power, and LLaMA-70B model is deployed.

[0090] Scheduler: A cloud central scheduler integrates a length-aware semantic module and a Lyapunov-guided offloading optimization module, which can allocate resources and schedule in real time according to the state of the scene.

[0091] Step one: Token length prediction (length-aware semantic module execution). After receiving 10 concurrent requests from the scheduler, the input prompt of each request is tokenized, converted into a token sequence that can be processed by ModernBERT, and an initial semantic vector is generated. Then, after feature recalibration, the output predicted length is obtained through a layer of network.

[0092] Step two: Task offloading decision (Lyapunov-guided offloading optimization module). The scheduler collects task state, device state, and virtual queue state, and updates the virtual queue. Then, an iterative offloading algorithm is used to solve: a cost matrix is constructed, an integer linear programming problem is solved based on this, and the device load is smoothed according to the damping factor, and the final allocation strategy is obtained through repeated iteration.

[0093] Step three: Each computing device runs model inference according to the request obtained from the allocation, completes the task, and returns the calculation delay, actual load, and other data to the central scheduler, providing decision basis for the scheduling task of the next time slot.

[0094] In terms of token length prediction accuracy, the L1 loss of the length-aware semantic module of the application is 91.85, which is reduced by 0.2%, 17.3%, 16.15% and 92.62% compared with LoRA (92.07), LSTM (107.79), Transformer (106.69) and Qwen2.5-7B (176.93) respectively, and the LAS only needs 0.09M trainable parameters, which is reduced by 99% compared with LoRA (8.75M), greatly reducing the calculation cost.

[0095] In terms of long-term user experience quality and system stability, the Lyapunov-guided offloading optimization module of the application has a Lyapunov reward (comprehensive delay, accuracy and queue stability, the higher the better) that is significantly better than the baseline scheme under different cloud edge server number configurations (N=4 / 15 / 20, U=6 / 15 / 20). For example, when N=4 and U=15, the LOO reward is 36602, which is increased by 113.8%, 169.9%, 14.1% and 100.9% compared with Greedy-Accuracy (-265297), Greedy-Delay (13565), DiffusionRL (32077) and TransformerPPO (-4050226) respectively; at the same time, the long-term average computing power utilization rate of all devices is ≤ threshold value, the virtual queue length converges to a stable value, and there is no device overload phenomenon.

[0096] In terms of delay and accuracy balance, the average response delay of delay-sensitive tasks is 382ms, which is reduced by 16.2% and 37.6% compared with DiffusionRL (456ms) and TransformerPPO (612ms) respectively; the average inference accuracy of accuracy-sensitive tasks is 0.89, which is increased by 23.6% and 85.4% compared with Greedy-Delay (0.72) and edge local execution (0.48) respectively; the overall resource utilization rate of the system is 82.3%, which is increased by 45.1% compared with Greedy-Accuracy (56.7%).

[0097] The application effectively solves the problems of uncontrollable token length, difficulty in adapting to dynamic environment, and resource constraints of heterogeneous devices in the large model inference in the cloud edge cooperation scene, and greatly improves the efficiency, stability and user experience of inference.

[0098] The foregoing is considered as illustrative only of the principles of the application. Further, since numerous modifications and changes will readily occur to those skilled in the art, it is not desired to limit the application to the exact construction and operation described. Accordingly, all such variations are intended to be included within the scope of the present application as defined in the claims below and their equivalents.

Claims

1. A distributed large-model inference method based on token awareness and Lyapunov optimization, characterized in that, include: S1. After receiving user input prompts, the length-aware semantic module predicts the length of the output token for the corresponding task and calculates the corresponding total workload. The length-aware semantic module is based on a fine-tuned pre-trained language model and enhances the sensitivity of token length-related semantics through a feature recalibration mechanism to achieve accurate prediction of the output token length. S2. Based on the token length prediction result, the real-time status of the device and the long-term virtual queue status, the final unloading decision is generated by the Lyapunov guided unloading optimization module iteratively unloading algorithm. S3. The client uninstalls the task to the corresponding cloud-edge-device based on the final uninstallation decision, and the device performs large model inference. The scheduler updates the virtual queue length based on the actual task execution time and the device's computing power utilization, providing input for the next time slot decision.

2. The distributed large-model inference method based on token awareness and Lyapunov optimization as described in claim 1, characterized in that, Step S1 specifically includes: S101. The user input prompt is segmented into words and converted into a token sequence processed by a pre-trained language model. An initial semantic vector is generated through the embedding layer. S102. For the initial semantic vector, the feature weights are dynamically adjusted through three steps: compression, activation, and recalibration to generate a recalibrated feature vector that highlights the semantic information related to the token length. S103. Input the recalibrated feature vector into the output layer, generate the predicted output token length through linear transformation and activation function, and calculate the computing power requirements of the pre-filling stage and the decoding stage for the corresponding task based on the predicted output token length to obtain the total workload.

3. The distributed large-model inference method based on token awareness and Lyapunov optimization as described in claim 2, characterized in that, Compression, excitation, and recalibration specifically include: Compression: Global features are aggregated through adaptive average pooling and adaptive max pooling to generate descriptors containing global statistics and salient features; Incentive: Input the descriptor into a two-layer fully connected network, and generate an attention weight vector through nonlinear transformation to quantify the importance of each feature for token length prediction; Recalibration: The weight vector is applied to the initial semantic vector through element-wise multiplication to generate a recalibrated feature vector.

4. The distributed large-model inference method based on token awareness and Lyapunov optimization as described in claim 1, characterized in that, Step S2 specifically includes: S201. Collect status information in each time slot t; Task Status: Collection of Active Tasks Workload of each task Delay sensitivity Precision sensitivity ; Device status: Computing power of cloud-edge-device servers Current network speed Cloud-edge-device speed Cloud-edge-device speed ; Virtual queue status: Virtual queue length for each device ; Long-term computing power threshold ; S202, Define the virtual queue update rules; Define the virtual queue update formula: ； In the formula, , Unload decision variables for binary; when When the queue length increases, it indicates that the current computing power utilization of device j exceeds the limit and subsequent task allocation needs to be reduced. S203, Iterative unloading algorithm; The long-term user experience quality optimization is transformed into a time-slot-by-time integer nonlinear programming problem, which is solved efficiently by an iterative offloading algorithm with damping and congestion control.

5. The distributed large-model inference method based on token awareness and Lyapunov optimization according to claim 1, characterized in that, Step S203 specifically includes: Constructing the cost matrix: In the k-th iteration, construct the task-equipment cost matrix. Each element in the cost matrix It consists of base costs and congestion penalties; Base cost includes communication latency , For the amount of task data, Calculate latency for network protocol latency. , This indicates the task that arrives first in the same time slot, and the Lyapunov item; Optimal allocation solution: Substituting the cost matrix into an integer linear programming problem, we can solve for the optimal unloading decision in the current iteration as follows: Each task is assigned to only one device; Load-aware updates: Using damping factor To smoothly update device-sensing load and avoid iterative oscillations, the following method is used: ； Convergence criteria: Repeat the steps of constructing the cost matrix, solving for the optimal allocation, and sensing the load update until the offloading decision is made in two iterations. and The difference is less than the threshold, or the maximum number of iterations has been reached. Output the final unloading decision .

Images