An edge large model task offloading and scheduling method for multi-user quality of experience

Abstract

The application provides a multi-user quality of experience oriented edge large model task offloading and scheduling method, which comprises the following steps: a user terminal initiates a large language model inference task request to an edge server; the edge server acquires the system state at the current moment in real time, and constructs a nonlinear streaming user experience quality evaluation model; a target function is constructed, and the target function is constrained based on the system state at the current moment; the constrained target function is solved to obtain an optimal offloading strategy of the current task; the task is distributed to the corresponding edge server based on the optimal offloading strategy, the system is decoupled and scheduled in the pre-filling stage and the decoding stage; the actual parameters of the task are recorded, the prior evaluation parameters in the heuristic search algorithm are fine-tuned based on the actual parameters, and the edge large model task offloading and scheduling are completed; and the application maximizes the global multi-user comprehensive satisfaction degree of the system in the environment where the edge resources are limited and the network state dynamically changes.

Description

Technical Field

[0001] This invention belongs to the field of edge computing technology, specifically relating to a method for offloading and scheduling large edge model tasks for improving user experience quality across multiple users. Background Technology

[0002] In recent years, large language models (LLMs) have demonstrated remarkable capabilities. However, LLMs have a massive number of parameters, and the inference process is extremely computationally intensive (e.g., GPU / NPU computing power) and memory resources. Traditional centralized cloud deployments face problems such as high transmission latency and network congestion. Therefore, offloading large model inference capabilities to edge computing nodes closer to users (such as edge servers), i.e., edge large model inference, has become an important development trend for the collaborative optimization of communication and computing. In scenarios where multiple users access edge servers, the system not only needs to solve the problem of "where should the task be offloaded (end, edge, cloud)?", but also the problem of "how to schedule computing resources within the edge server" to cope with the significant differences in resource requirements between the prefill and decoding stages unique to large language models.

[0003] Existing technology, patent publication number CN116431326A, discloses a multi-user dependency task offloading method based on edge computing and deep reinforcement learning. This method models the subtasks of mobile applications as directed acyclic graphs (DAGs) and establishes task execution latency and energy consumption models based on local, edge, and cloud environments. It utilizes deep reinforcement learning (DRL) to make offloading decisions with the optimization objective of minimizing the weighted sum of total task latency and total energy consumption. Patent publication number CN121523893A discloses an asymmetric partitioning scheduling system and method for heterogeneous GPU clusters. This method, targeting heterogeneous GPU clusters, proposes bucketing based on request token length and employs an asymmetric partitioning strategy to partition and schedule model layers, thereby reducing pipeline bubbling and improving the overall system throughput. The publication number CN121523853A discloses a hierarchical adaptive prefilling chunk scheduling method and system for large language model inference. This method delves into the internals of large model inference and allocates prefilled chunks of different sizes to different partitions of the model based on the computation and memory access characteristics of different layers. Under time budget constraints, these chunks are executed in batches with decoding tasks to improve system throughput and reduce tail latency.

[0004] The aforementioned existing technologies only focus on "total completion time," the "system throughput" or "resource utilization" of the underlying hardware. Large-scale model inference uses streaming output, and the user's actual experience is highly dependent on "first-to-last response time (TTFT)" and "time per output token (TPOT)." More importantly, user satisfaction with these two metrics exhibits a non-linear, step-like decline (e.g., satisfaction drops sharply after exceeding a certain threshold), and existing technologies lack precise mathematical modeling of multi-dimensional streaming user experience metrics (QoUE). Furthermore, in resource-constrained environments, there is a lack of mechanisms to guarantee a "bottom line" for multi-user experience.

[0005] In summary, existing technical solutions cannot accurately depict the unique nonlinear experience patterns of large-scale model streaming generation, are difficult to achieve joint optimization of offloading decisions and underlying inference scheduling in edge scenarios with limited computing power and dynamically changing states, and are even less able to maximize the overall global multi-user satisfaction (QoUE) of the system while ensuring the bottom line of user experience for all users. Summary of the Invention

[0006] To address the problems existing in the prior art, this invention proposes a method for offloading and scheduling edge large-scale model tasks oriented towards multi-user user experience quality. This method includes: constructing a multi-user mobile edge computing system, including user terminals, base stations, edge servers, and cloud servers; user terminals initiating large language model inference task requests to edge servers via base stations; edge servers acquiring the current system state in real time and constructing a nonlinear streaming user experience quality evaluation model; constructing an objective function based on the nonlinear streaming user experience quality evaluation model; constraining the objective function based on the current system state; solving the constrained objective function using a heuristic search algorithm based on the large language model inference task request to obtain the optimal offloading strategy for the current task; allocating the task to the corresponding edge server based on the optimal offloading strategy, with the system decoupling the scheduling of the edge servers during the pre-filling and decoding phases; and recording the actual parameters of the task, fine-tuning the prior evaluation parameters in the heuristic search algorithm based on the actual parameters to complete the offloading and scheduling of edge large-scale model tasks.

[0007] The beneficial effects of this invention are:

[0008] This invention can maximize the overall multi-user satisfaction (QoUE) of the system while ensuring the basic service quality of all access users in environments with limited edge resources and dynamically changing network conditions, thereby significantly improving the reliability and availability of edge large model inference services. Attached Figure Description

[0009] Figure 1 This is a diagram of the overall framework structure of the present invention. Detailed Implementation

[0010] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0011] This invention aims to solve the technical problems of misaligned evaluation indicators, isolated optimization perspectives, and lack of bottom-line guarantee for user experience in existing edge computing task offloading and large model inference scheduling technologies. It proposes an edge large model task offloading and scheduling method oriented towards the user experience quality of multiple users.

[0012] Specifically, the purpose of this invention is to construct a nonlinear experience evaluation model that fits the characteristics of streaming output: Overcoming the shortcomings of traditional evaluation indicators such as "total completion time" or "system throughput", this invention aims to establish a comprehensive user experience quality (QoUE) mathematical model based on first-word response time (TTFT) and time per output token (TPOT) for the streaming output characteristics of large language models, so as to accurately characterize the nonlinear step-like decline law of user subjective satisfaction.

[0013] This invention achieves joint decision-making for network offloading and fine-grained large-scale model scheduling: breaking the bottleneck of the isolation between communication network status and underlying computing resources in the existing technology, this invention aims to achieve the coordinated linkage between macro-level task offloading decision-making across "end-edge-cloud" and micro-level resource scheduling in the prefill / decode stage within the edge server, thereby achieving end-to-end fine-grained optimization.

[0014] Establish a scheduling guarantee mechanism based on strict constraints of "experience bottom line": In response to the service quality avalanche problem that is prone to occur in edge scenarios with limited computing power and high concurrency, this invention aims to introduce strict bottom line threshold constraints (i.e., anti-timeout mechanism) to ensure that no user's inference request falls into a state of "complete dissatisfaction" in the process of pursuing overall system efficiency.

[0015] The embodiments of the present invention rely on a multi-user mobile edge computing architecture, such as... Figure 1 As shown, the system mainly consists of three layers: a terminal device layer that generates large model inference tasks, an edge server cluster deployed on the base station side (which integrates a system state awareness and QoUE evaluation module, a heuristic offloading decision-maker, and a micro-decoupling scheduler), and a cloud server layer with abundant computing power.

[0016] A method for offloading and scheduling large-scale edge computing tasks for multi-user experience quality is disclosed. The method includes: constructing a multi-user mobile edge computing system, including user terminals, base stations, edge servers, and cloud servers; user terminals initiating large language model inference task requests to edge servers via base stations; edge servers acquiring the current system state in real time and constructing a nonlinear streaming user experience quality evaluation model; constructing an objective function based on the nonlinear streaming user experience quality evaluation model; constraining the objective function based on the current system state; solving the constrained objective function using a heuristic search algorithm based on the large language model inference task request to obtain the optimal offloading strategy for the current task; allocating the task to the corresponding edge server based on the optimal offloading strategy, with the system decoupling the scheduling of edge servers during the pre-filling and decoding phases; and recording the actual parameters of the task and fine-tuning the prior evaluation parameters in the heuristic search algorithm based on the actual parameters to complete the offloading and scheduling of the large-scale edge computing tasks.

[0017] In this embodiment, the large language model inference task request includes the token length of the input prompt and the maximum number of tokens expected to be generated; and personalized experience baseline thresholds are assigned to users with different priorities.

[0018] The “length of the input Prompt token” and “maximum number of tokens expected to be generated” mentioned above are standard underlying configuration parameters for edge servers when accepting inference requests.

[0019] The length of the input prompt token refers to the length of the original text input by the user after it has been segmented by the model. Specifically, after the edge server receives the text request sent by the terminal, it will first call the tokenizer that comes with the large language model to encode the original text and convert it into a sequence of digital tokens. The accurate value can be obtained by directly counting the length of the sequence.

[0020] The maximum number of tokens expected to be generated is a constraint that limits the length of a single response from a large model. It is obtained as follows: In the data message that the user terminal initiates the inference request, it usually carries a configuration parameter field. The edge server can directly extract the value of this field by parsing the request message. If the user request does not explicitly carry this field, the system will automatically read it and assign it a preset system default maximum length value (such as 1024, 2048, etc.).

[0021] The allocation logic and formula for the "personalized experience baseline threshold" are as follows: this threshold is calculated and allocated through a scaling mechanism that combines a baseline value with priority weights. The system first pre-sets a set of "baseline experience thresholds" applicable to ordinary users, specifically including the upper limit of the baseline satisfaction period. The benchmark is completely unsatisfactory and the bottom line is not satisfactory. and the benchmark maximum tolerance generation interval Next, the system parses the user identifier carried in the request to determine its priority level, and assigns a corresponding "priority penalty coefficient" to different priorities. .

[0022] To reflect the level of priority protection, users with higher priority levels have smaller coefficient values. The minimum threshold for a user's personalized experience is the product of the baseline value and this coefficient, calculated using the following formula:

[0023] Personalized satisfaction period upper limit:

[0024]

[0025] Complete dissatisfaction with personalization is the bottom line:

[0026]

[0027] Personalized maximum tolerance generation interval:

[0028]

[0029] In this embodiment, it is assumed that the system sets a baseline value. Second, Second, Seconds / Token, and set a high priority coefficient. When a high-priority user initiates a request, the personalized threshold can be calculated by substituting the values ​​into the formula. Second, Second, Seconds / Token. Through this linear scaling calculation, the algorithm is subject to stricter constraints in subsequent resource scheduling, thereby forcing the system to prioritize the inference experience of high-priority users.

[0030] In this embodiment, the constraints on the objective function are: , and The total computational and memory requirements allocated to each node shall not exceed the physical limit; among which For the first character's response time, To be completely dissatisfied with the bottom line; Let i be the average generation time of the i-th task. The maximum tolerance generation interval; Let be the variance of the generation time for the i-th task. This represents the maximum time-consuming variance.

[0031] The physical upper limit refers to the actual specifications of the AI ​​acceleration hardware (such as GPUs or NPUs) installed on the edge server. For example:

[0032] Physical memory limit: The available physical video memory of a single edge server node is typically configured between 32GB and 160GB (e.g., using a single 80GB video memory card or a combination of two 32GB video memory cards), mainly to meet the pre-filling and decoding needs of large models with different parameter scales such as 7B to 30B.

[0033] Physical upper limit of computing power: The peak computing power of a single edge server node (such as FP16 Tensor computing power) is typically between 300 TOPS and 2000 TOPS.

[0034] In actual macro-level offloading decisions, the system estimates the GPU memory and computing power required by the new task. Only if the total resource requirements of the edge server remain within the upper limit of this hardware specification after the addition of the new task will the system retain the server as a candidate offloading node; otherwise, bottom-line constraint pruning will be triggered directly.

[0035] The specific process of baseline constraint pruning includes the following sub-steps:

[0036] 1. Dynamic Resource Increment Calculation: Extract the current memory usage, allocated computing power, and currently executing task sequence of the target edge server in real time, and calculate the sum of static memory (model weight loading) and dynamic incremental memory (KV cache) required after a new task is added.

[0037] 2. Multi-dimensional Constraint Consistency Verification: The estimated total system resource requirements are compared and verified with the physical memory and computing power limits of the edge server; simultaneously, it is estimated whether the addition of a new task will cause the experience indicators of the existing queued tasks on the server to fall below the personalized experience threshold, i.e., whether it will cause the existing tasks to malfunction. or The situation.

[0038] 3. Candidate solution space reduction: If any of the above physical resource constraints or experience bottom line constraints fail the verification, the edge server is determined to be in a resource-limited unreachable state, and it is directly pruned and removed from the candidate unload path set of the current inference task.

[0039] In this embodiment, the heuristic search algorithm used to solve the constrained objective function includes: traversing local terminal processing and edge servers. And candidate offloading paths for cloud servers, estimating transmission latency and queue processing latency under different paths; and distributing tasks to edge servers. If this allocation scheme causes the server The existing internal queuing task Squeezed to greater than This could cause the average interval of the original Decode task to be greater than [a certain value]. If this is deemed a violation of the hard-to-implement user experience constraints, the server will be... Prune and remove nodes directly from the candidate solutions; among the remaining candidate nodes that satisfy the constraints, use a greedy strategy to calculate the marginal QoUE gain allocated to each node, and select the node with the largest gain to issue an unload command; if all edge nodes are pruned, the task is unloaded to the cloud server.

[0040] The estimation of transmission delay and queue processing delay under different paths includes estimation through three parts: communication transmission, queuing waiting, and internal node execution.

[0041] Transmission delay and queuing delay estimates include both transmission delay and queuing delay.

[0042] Transmission delay: The system estimates the transmission time for uploading Prompt data and downloading Token data based on Shannon's theorem and the real-time available uplink and downlink wireless bandwidth.

[0043] Queuing delay: If a task is assigned to an edge node, the system will calculate the sum of the estimated remaining processing time of all preceding tasks in the current waiting queue of that node and use it as the queuing delay of the new task.

[0044] Node-internal execution latency estimation: Traditional offloading methods typically treat the execution latency of a large model as an indivisible, whole "black box" parameter. To accurately evaluate the streaming experience, the total execution latency is forcibly decoupled at the estimation level into two physical stages: estimation through communication transmission, queuing, and node-internal execution. 1. Transmission latency and queuing latency estimation.

[0045] Transmission delay: The system estimates the transmission time for uploading Prompt data and downloading Token data based on Shannon's theorem and the real-time available uplink and downlink wireless bandwidth.

[0046] Specifically, transmission delay prediction model ( The system estimates uplink and downlink transmission times based on real-time acquired wireless channel conditions. Assume the user... The submitted task input Prompt data size is The maximum expected data volume is The estimated transmission delay is calculated as follows:

[0047]

[0048] in, and According to Shannon's theorem, the edge controllers are respectively... Real-time calculated uplink and downlink available wireless transmission rates from the current user to the candidate node ( To allocate bandwidth, (This refers to the signal-to-noise ratio). If the candidate node is a cloud server, the round-trip time (RTT) of the fiber optic backhaul in the core network must also be added.

[0049] Queuing delay: If a task is assigned to an edge node, the system will calculate the total estimated remaining processing time of all preceding tasks in the current waiting queue of that node and use it as the queuing delay of the new task.

[0050] Queue delay prediction model ( When a task is assigned to a candidate edge server When a task is completed, it needs to enter the server's waiting queue. The queuing delay depends on the remaining processing time of tasks already in the queue. The system's estimated queuing delay formula is:

[0051]

[0052] in, For the target node The set of all currently queued prior tasks. Preceding tasks for the system The remaining execution time.

[0053] 2. Node internal execution latency estimation.

[0054] Traditional unloading methods typically reduce the execution delay of large models ( This is treated as an indivisible, whole "black box" parameter. To accurately evaluate the streaming experience, the total execution latency is forcibly decoupled into two physical stages at the estimation level:

[0055]

[0056] Specifically, this includes: large-scale model inference execution latency prediction model ( For large language model computation features, the system decouples the execution latency into two stages for prediction: prefill and decoding.

[0057] Prefill delay prediction (computation-intensive):

[0058]

[0059] in, For the number of model parameters, For the length of the input token, This represents the upper limit of the floating-point computing power of candidate nodes. This is a hardware-related computational efficiency constant.

[0060] Decoding latency estimation (memory-intensive):

[0061]

[0062] in, The maximum number of tokens expected to be generated. The step time required to generate a single token.

[0063] Since the Decode phase is primarily limited by video memory bandwidth, the step time for generating a single token is:

[0064]

[0065] in This represents the memory bandwidth of the candidate node.

[0066] Prefill latency estimation: This stage is computationally intensive. The system mainly estimates latency based on the floating-point computing power limit (FLOPS) of the candidate nodes, the number of model parameters, and the length of the input Prompt token.

[0067] Decode latency estimation: This stage is memory-intensive. The system mainly estimates the generation time per step based on the upper limit of the candidate node's memory bandwidth and the maximum number of tokens expected to be generated.

[0068] 3. Final streaming index prediction mapping.

[0069] Based on the above breakdown, the system no longer estimates the total completion time in a rough manner, but instead reconstructs it into a prediction of a streaming indicator: the predicted first-to-last-time response (TTFT) is: uplink transmission delay + queuing delay + prefill delay + downlink transmission delay of a single token.

[0070] The Predicted Generation Interval (TPOT) is the single-step decoding delay plus the downlink transmission delay of a single token.

[0071] In this embodiment, the specific calculation logic for the marginal QoUE gain is as follows: When the system evaluates the newly arriving inference task... Assigned to a candidate edge server that meets the constraints At that time, this allocation action will not only affect the task It brings benefits to the user experience, but it also consumes server space. Queuing and computing resources cause server The increased waiting time for existing queued tasks has led to a certain degree of decrease in the QoUE of those tasks. Therefore, the allocation to nodes is defined... Marginal QoUE gain For: New task The predicted experience quality obtained at this node is minus the total experience quality loss caused by waiting for new tasks within that node. The specific calculation formula is as follows:

[0072]

[0073] in: For new tasks If assigned to a node The overall quality of the predictive experience that can be obtained; For nodes All pre-order tasks currently being queued for execution; For nodes There is a mission in the Central Plains Predicted experience quality before new tasks are added; To hypothesize a new task After joining, the task was affected by the reallocation of queuing resources. Predict new values ​​for experience quality.

[0074] In this embodiment, the specific optimization process of the Greedy Strategy is as follows: After calculating the marginal gain of all unpruned candidate nodes (including local terminals, edge servers, and cloud servers), the system uses the Greedy Strategy to make a single-step optimal decision. The specific operation is as follows:

[0075] The system traverses all candidate node sets. Find the target node that provides the maximum marginal QoUE gain. Its mathematical expression is:

[0076]

[0077] Decision issuance and degradation determination: If the calculated maximum marginal gain This indicates that the allocation has a positive contribution to the overall system experience, and the system will issue the unload command to the node. If the overall load of the edge cluster is too high, causing all candidate edge nodes to... (That is, the addition of a new task will cause a serious deterioration in the overall experience), then the greedy strategy will trigger a degradation mechanism, directly reverting the task to a lower level. The system is offloaded to a cloud server with ample computing power but high transmission latency for backup processing.

[0078] In this embodiment, the present invention proposes an edge large-scale model task offloading and scheduling method oriented towards multi-user experience quality, the method comprising:

[0079] Step S1: Real-time Awareness of Concurrent Requests from Multiple Users and Network Status. When multiple end users simultaneously initiate large language model inference requests to the edge network, the status awareness module within the edge server collects the current system status in real time. This includes: the available uplink / downlink wireless channel transmission rates for each user device, the available GPU memory capacity and computing load status of each server node within the edge cluster. Simultaneously, it analyzes the characteristics of each inference task (such as the length of the input Prompt token and the maximum number of tokens expected to be generated) and assigns personalized experience baseline thresholds (including the upper limit of the satisfaction period) to users with different priorities. Completely dissatisfied with the bottom line Maximum tolerance generation interval ).

[0080] In this embodiment, the first-letter response time (TTFT) satisfaction evaluation function : Set up user The time when the request was initiated was The time when the first token was received was ,but .Will Modeled as a nonlinear piecewise function:

[0081]

[0082] in, arrive For the perfect satisfaction range; arrive Introducing attenuation coefficient and curvature parameters To reflect the accelerating decline in satisfaction; To completely dissatisfy the threshold bottom line.

[0083] Satisfaction evaluation function for each output token time (TPOT) : Set up user Received the The and the first The time difference between the tokens is Calculate the average time taken for its generation process. and variance The satisfaction function is constructed as follows:

[0084]

[0085] In the formula: and The maximum average interval and maximum jitter that the user can tolerate; and These are the weighting coefficients, and It emphasizes the fundamental role of generation speed.

[0086] Step S2: Construct a nonlinear streaming user experience quality (QoUE) evaluation model. For the characteristics of streaming output, the system internally constructs a multi-dimensional QoUE calculation model: 1. Calculate the first-word response time (TTFT) satisfaction. If the task prediction ,but ;like ,but It exhibits polynomial or exponential acceleration in decay; if predicted If it falls below the bottom line, it is determined that the price has fallen below the bottom line. 2. Calculate the satisfaction level for each output token time (TPOT). It consists of a penalty term for the average generation time and the time variance (jitter). 3. Finally, the overall QoUE estimate for this task is defined as... .

[0087] Overall Quality of User Experience (QoUE): Two metrics are combined using a product approach to ensure neither has a significant weakness.

[0088]

[0089] in, This refers to the set of all users currently connected to the edge computing system. The comprehensive user experience quality for large model inference services in the i-th task, in terms of multiple dimensions. Satisfaction with the first character's response time. Satisfaction rate for each output token over time. Let be the response time of the first character of the i-th task. Let i be the average generation time of the i-th task. Let be the variance of the generation time for the i-th task.

[0090] Step S3: Macro-level offloading decision based on heuristic search for bottom-line experience constraints. The offloading decision maker of the edge server aims to maximize the total QoUE of all connected users in the system, and uses a heuristic search algorithm to solve the problem: 1. Candidate solution prediction: Traverse local terminal processing, edge server... And candidate offloading paths such as cloud servers, estimating transmission latency and queue processing latency under different paths. 2. Bottom-line constraint pruning: Assuming tasks are assigned to edge servers. If it is anticipated that this allocation will cause server issues... The existing internal queuing task Squeezed to greater than This could cause the average interval of the original Decode task to be greater than [a certain value]. If this is deemed a violation of the hard-to-implement user experience constraints, the server will be... 3. Heuristic optimization: Among the remaining candidate nodes that satisfy the constraints, a greedy strategy is used to calculate the marginal QoUE gain allocated to each node, and the node with the largest gain is selected to issue the unloading command. If all edge nodes are pruned, a degradation mechanism is triggered, and the task is unloaded to the cloud server.

[0091] Step S4: Preemptive Timeout Prevention and Smooth Decoding Co-scheduling within Edge Nodes. After a task is distributed to a specific edge server, its internal micro-decoupled scheduler allocates computing power between the prefill and decoding stages: 1. Preemptive Timeout Prevention Mechanism: The scheduler monitors the task queue in real time. When it detects that the waiting time of a task in the queue has exceeded... and approaching At that time, the task The system is in a precipitous decline phase. At this point, the system triggers a preemptive power switch, temporarily suspending some tasks in the Decode phase and allocating all released computing resources to the Prefill phase of the timed-out task, ensuring its completion. 1. Complete pre-filling and output the first token before the bottom line. 2. Stable decoding mechanism: When no task faces timeout risk (i.e. all new task prediction TTFTs are in the safe range), the scheduler will use most of the computing power to maintain a constant and maximum continuous batching scale to execute the Decode stage, so as to minimize the variance of the token generation rate and ensure the smoothness of streaming text output.

[0092] The waiting time has exceeded and approaching Waiting time for unprocessed tasks Approaching the upper limit of the satisfaction period At that time, the system determines The gradient is about to begin to drop sharply; at this point, a forced preemption is triggered, selecting the task with the current average generation time from the currently executing Decode task queue. With maximum tolerance generation interval Decode tasks that still have a safe time margin are paused, and GPU computing resources are allocated to the prefill phase of these tasks, until the bottom line is completely unsatisfactory. Output the first token.

[0093] To minimize Ensuring smooth streaming output includes: This represents the variance of the time taken for each output token when the model generates text character by character. Physically, it reflects the jitter (i.e., the stuttering sensation caused by fluctuating output speed) during the streaming speech generation process. During the large model decoding stage, if the scale of continuous batching processed by the underlying GPU fluctuates drastically, the computation time for each round of speech generation will vary greatly, resulting in a severe "pulsating" stuttering experience for the user on screen. Therefore, this invention, under normal conditions without triggering emergency preemption, deliberately maintains a stable continuous batch size through a scheduling algorithm. This ensures that the underlying computational overhead remains constant, thereby reducing the variance of the generation time. This mechanism, through the stable allocation of underlying micro-computing power, directly eliminates fluctuations in the surface generation rate, ensuring that the text generated by the large model can be presented smoothly and at a uniform speed, fundamentally guaranteeing the continuity and subjective experience quality of streaming reading in multi-concurrency scenarios.

[0094] Step S5: Policy Closure and Prediction Model Parameter Update. After the task is completed, the system collects the actual generated TTFT and TPOT and calculates the true QoUE score. Using the deviation between the true value and the predicted value in Step S3, the system dynamically corrects the environmental delay coefficient and attenuation factor in the heuristic search algorithm to adapt to the highly dynamic and fluctuating mobile computing network environment.

[0095] In this embodiment, in step S2, and In addition to multiplication, the integration method can also be based on the specific business focus, by configuring dynamic weight coefficients and using mathematical forms such as logarithmic weighted summation to calculate the overall experience satisfaction.

[0096] In this embodiment, in step S3, in addition to using a greedy heuristic search algorithm, when the network node scale is extremely large and the system state is extremely complex, the method of the present invention can also use other search optimization strategies such as multi-agent deep reinforcement learning (MADRL) or genetic algorithm to select candidate nodes.

[0097] Offline parameter optimization mechanism: Before deployment at edge base stations, the offloading and scheduling logic proposed in this invention can be integrated as an independent algorithm module into a general edge computing offline simulation environment. By replaying historical mobile network trajectories and inference loads, the optimal tuning of system prediction parameters can be completed offline, thereby reducing the cost of deployment in the actual network.

[0098] The above-described embodiments further illustrate the purpose, technical solution, and advantages of the present invention. It should be understood that the above-described embodiments are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made to the present invention within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for offloading and scheduling large edge model tasks for multi-user experience quality, characterized in that, include: Construct a multi-user mobile edge computing system, including user terminals, base stations, edge servers, and cloud servers; The user terminal initiates a request for a large language model inference task to the edge server through the base station; The edge server acquires the current system status in real time and constructs a non-linear streaming user experience quality evaluation model. An objective function is constructed based on a nonlinear streaming user experience quality evaluation model; the objective function is constrained based on the current system state. Based on the large language model inference task request, a heuristic search algorithm is used to solve the constrained objective function to obtain the optimal unloading strategy for the current task. Based on the optimal unloading strategy, the task is assigned to the corresponding edge server, and the system decouples the scheduling of the edge server in the pre-filling stage and the decoding stage. The system records the actual parameters of the task and fine-tunes the prior evaluation parameters in the heuristic search algorithm based on the actual parameters to complete the unloading and scheduling of the edge large model task.

2. The edge large-scale model task offloading and scheduling method for multi-user experience quality as described in claim 1, characterized in that, The large language model inference task request includes the token length of the input prompt, the maximum number of tokens expected to be generated, and assigns personalized experience baseline thresholds to users with different priorities.

3. The edge large-scale model task offloading and scheduling method for multi-user experience quality as described in claim 1, characterized in that, The current system status includes: the available uplink / downlink wireless channel transmission rates of each user device, the available GPU memory capacity and computing load status of each server node in the edge cluster.

4. The edge large-scale model task offloading and scheduling method for multi-user experience quality as described in claim 1, characterized in that, The objective function is constructed as follows: ； ； in, This refers to the set of all users currently connected to the edge computing system. The comprehensive user experience quality for large model inference services in the i-th task, in terms of multiple dimensions. Satisfaction with the first character's response time. Satisfaction rate for each output token over time. Let be the response time of the first character of the i-th task. Let i be the average generation time of the i-th task. Let be the variance of the generation time for the i-th task.

5. The edge large-scale model task offloading and scheduling method for multi-user experience quality as described in claim 1, characterized in that, The constraints of the objective function are: , and The total computing power and memory requirements allocated to each node shall not exceed the physical limit. in For the first character's response time, To be completely dissatisfied with the bottom line; Let i be the average generation time of the i-th task. The maximum tolerance generation interval; Let be the variance of the generation time for the i-th task. This represents the maximum time-consuming variance.

6. The edge large-scale model task offloading and scheduling method for multi-user experience quality as described in claim 1, characterized in that, Solving the constrained objective function using a heuristic search algorithm includes: traversing local terminal processing and edge servers. And candidate offloading paths for cloud servers, estimating transmission latency and queue processing latency under different paths; and distributing tasks to edge servers. If this allocation scheme causes the server The existing internal queuing task Squeezed to greater than This could cause the average interval of the original Decode task to be greater than [a certain value]. If this is deemed a violation of the hard-to-implement user experience constraints, the server will be... Prune and remove nodes directly from the candidate solutions; among the remaining candidate nodes that satisfy the constraints, use a greedy strategy to calculate the marginal QoUE gain allocated to each node, and select the node with the largest gain to issue an unload command; if all edge nodes are pruned, the task is unloaded to the cloud server.

7. The edge large-scale model task offloading and scheduling method for multi-user experience quality as described in claim 1, characterized in that, The system decouples the scheduling of edge servers in the pre-filling and decoding phases, including preemptive timeout prevention scheduling and smooth decoding scheduling.

8. The edge large-scale model task offloading and scheduling method for multi-user experience quality as described in claim 7, characterized in that, Preemptive timeout prevention scheduling includes: Real-time monitoring of the queue; when the waiting time for unprocessed tasks approaches... And towards During evolution, the system determines Gradient descent is fastest; at this point, a forced preemption is triggered, pausing the execution of part of the Decode task, allocating GPU computing resources to the Prefill phase of this task, and... First output the request token.

9. The edge large-scale model task offloading and scheduling method for multi-user experience quality as described in claim 7, characterized in that, Smooth decoding scheduling includes: when the system does not trigger preemption, executing decoding tasks by maintaining a stable continuous batch size, and minimizing... This is the goal, thereby ensuring the smoothness of the streaming decoding output.

Images