Large model reasoning optimization method for SLO perception in edge heterogeneous computing power network

Abstract

The invention discloses a large model reasoning optimization method for SLO perception in an edge heterogeneous computing power network, aiming at an edge heterogeneous computing power computing cluster scene providing large model reasoning service, requests are strategically scheduled to make full use of heterogeneous computing power of a cluster, so that SLO heterogeneous requests are met to the maximum extent; the method comprises the following steps: firstly, separating a pre-filling stage and a decoding stage of a reasoning process to different nodes, and respectively adopting a data parallel deployment strategy and an assembly line parallel deployment strategy to ensure that the computing power of equipment is fully utilized; then, according to heterogeneous computing power characteristics of decoding nodes, a distributed assembly line non-uniform deployment strategy is realized to improve the model reasoning throughput; finally, round rewards are collected in offline exploration, a large model is used for assisting in encoding of potential rewards, a decoder is trained to decode agency rewards in each step, a reinforcement learning PPO algorithm is assisted in training a scheduler, effective scheduling requests are achieved, the resource utilization rate is increased, meanwhile, cluster energy consumption is reduced, and the throughput of large model reasoning service is increased.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, specifically to a large model inference optimization method for SLO perception in edge heterogeneous computing networks. Background Technology

[0002] With the rapid development of Large Language Model (LLM) technology, it has demonstrated outstanding capabilities in intelligent chat and question answering, text generation, and logical reasoning. However, the scale of parameters and computational demands are also growing exponentially. Traditional cloud computing paradigms face network latency bottlenecks in scenarios with extremely high real-time requirements, such as autonomous driving. Furthermore, due to privacy concerns regarding sensitive user data, deploying LLM on edge devices is becoming increasingly crucial for achieving low-latency responses and reducing reliance on cloud infrastructure. However, unlike the homogeneous computing resources of cloud data centers, edge network clusters often exhibit heterogeneity, primarily in computing, storage, and network transmission capabilities. A single device in an edge cluster may not be able to deploy a complete large model. These heterogeneities and resource constraints of edge devices pose significant challenges to efficient LLM inference. Existing research mainly focuses on model compression, deploying quantization and small-scale LLMs for inference on resource-constrained edge devices. However, this severely impacts LLM performance, leading to low-quality inference results.

[0003] The inference process of LLM can be divided into a prefill phase and a decoding phase. The prefill phase calculates and generates a key-value cache (KV cache) based on user input prompts, which is computationally intensive. The decoding phase generates output tokens iteratively based on the KV cache, requiring frequent memory access and communication. With the increasing application of large-scale models, such as chatbots, programming assistants, and document summarization tools, LLM inference tasks are exhibiting diverse characteristics. Different types of requests typically have different service level objectives (SLOs), which are essentially highly related to the prefill and decoding phases of large-scale model inference. Document summarization requires higher prefill throughput to process long documents, and chatbots require lower per-token decoding latency to match human reading speed. Therefore, meeting the SLO plays a crucial role in ensuring a good user experience.

[0004] Traditional inference frameworks (such as vLLM and SGLang) deploy both the Prefill and Decode phases on the same GPU, leading to computational interference and coupling between the two phases. Recent PD separation strategies aim to improve inference throughput and resource utilization by separating these two phases for parallel computation on different GPUs or devices. Existing PD separation inference frameworks (such as Mooncake and Dynamo) attempt to separate the Prefill and Decode phases of the inference process and process them in parallel on different devices to improve system throughput. While this effectively eliminates phase interference, it faces issues such as weight redundancy and transmission overhead in edge environments. Furthermore, the inference process follows a fixed inference pattern and lacks strategic scheduling to determine whether to perform PD separation or PD coupling computation for heterogeneous SLO requests. Considering the heterogeneity of different SLO requests and the communication overhead of KV Cache transmission during PD separation computation, from the perspectives of overall inference latency, throughput, and cluster energy consumption, a complete PD separation strategy would increase transmission overhead between distributed edge nodes and cause high energy consumption.

[0005] Therefore, in order to achieve efficient inference by deploying LLM on resource-constrained edge computing network devices, satisfy different types of SLO requests while maximizing overall inference throughput and reducing cluster energy consumption costs, there is an urgent need for a distributed large model inference optimization method that can sense different SLO requests and make full use of the computing power of edge heterogeneous network clusters. Summary of the Invention

[0006] This invention addresses the problems existing in the aforementioned background technology by disclosing a SLO-aware large model inference optimization method in edge heterogeneous computing power networks. Faced with user requests of different SLO requirement types, in resource-constrained and heterogeneous edge network computing clusters, a distributed deployment of LLM is implemented. Requests are strategically scheduled to achieve both PD separation and PD coupling, allocating appropriate workloads to each device to fully utilize its computing power. This maximizes the fulfillment of request SLOs while improving overall inference throughput and reducing cluster energy consumption. Specifically, this invention includes the following steps:

[0007] Step 1: Based on the performance of the edge cluster nodes, all nodes are divided into two categories: high-computing-power nodes and low-computing-power nodes, forming two node groups. The Prefill and Decode stages of the inference process are separated and performed on different node groups. The nodes in the high-computing-power node group use a data parallel strategy, while the nodes in the low-computing-power node group use a distributed pipeline parallel strategy. Different types of requests are classified according to their SLO characteristics, and three request scheduling strategies are designed based on the request SLO characteristics to ensure full utilization of the device's computing power.

[0008] 1) Grouping of heterogeneous cluster nodes.

[0009] For a group of edge cluster devices The total number of devices is m. Assuming that all devices in the cluster are single-GPU devices, first, based on the total number of model parameters W to be deployed and the FLOPS (floating-point operations per second) computing power F of cluster node i, the process is as follows: i The cluster is divided into high-computing-power node groups. and low computing power node groups Where a+b=m.

[0010] 2) Design of parallel and scheduling strategies.

[0011] In high-computing-power node groups In this system, each node can deploy a complete model and employs a data-parallel strategy for inference. This means that the inference computation for each request is executed by only one node, and different requests can be assigned to different nodes for independent inference computation, maximizing parallelism. In low-computing-power node groups... The system employs a distributed pipelined parallel inference strategy, dividing the model into S stages layer by layer, and satisfying the following conditions: Each stage is assigned to different nodes, with each node deploying only a portion of the model layers and corresponding weights. High-performance node groups... and low computing power node groups A waiting queue Q is set up in each of the following: H and Q L Used to store requests for computation to be performed, enabling batch processing of these requests. Node group and Due to differences in computational characteristics, the batch size is set to the hyperparameter B for each batch. H and B L .

[0012] For each time slot t∈{1,…,T}, the request R arriving... t The number of tokens in its input context is The number of output tokens is Different requests have different SLO attributes, which are used separately. This represents the maximum latency during the Prefill phase and the average maximum latency for generating each token during the Decode phase, corresponding to the two important SLO metrics: Time To First Token (TTFT) and Time Per Output Token (TPOT). Based on the different SLO characteristics of different request types, request types are categorized according to SLO requirements, and three request scheduling strategies are designed as follows:

[0013] x tk∈{0,1},k∈{HPHD,HPLD,LPLD}

[0014] Where x tk Represents the request R under time slot t. t Whether to select a certain scheduling strategy, where k represents one of the three specific strategies. The HPHD strategy indicates that in high-computing-power node groups... Prefill and Decode computations are completed in the middle, and the HPLD strategy indicates that they are performed in the high-computing-power node group respectively. and low computing power node groups Prefill and Decode computations are completed in the middle, and the LPLD strategy indicates that in low-computing-power node groups The Prefill and Decode calculations are completed in the middle.

[0015] To optimize KV Cache transmission under the HPLD strategy, the calculated KV Cache is asynchronously transmitted to the corresponding model layer on the pipeline of the low-computing-power node group after each model layer is completed during the computation process. This achieves overlap between Prefill transmission and computation, which can effectively reduce the transmission latency of KV Cache under the HPLD strategy.

[0016] 3) Modeling of computation delay and waiting delay in each node group during the two-stage PD process.

[0017] Assuming pipelined inference is blocking, then in high-computing-power node groups... Node H in i and in low computing power node groups The computation latency of executing Prefill batch processing in the distributed pipeline can be calculated using the total computational cost of tokens and the computing power of nodes, as shown in the following formula:

[0018]

[0019] Where α is the computational cost per token (FLOPs / token) during full prefill computation, which is related to the model size but not to the token size. i Represents node H i The computing power (FLOP / s) is considered. At this point, the large model is evenly divided into layers and distributed to each node of the pipeline. The latency calculation of the pipeline batch prefill takes the worst case into account. Since the context length of the prefill requests scheduled to the pipeline is similar, the pipeline stage with the largest computational latency can be regarded as the latency in the worst case.

[0020] If node group Node H in i Simultaneous processing of B HIf there are multiple batch Decode requests, each request is executed alternately based on the token. Therefore, the total batch processing latency for Decode is related to the longest output sequence in the batch. However, if in a node group... In the pipeline, due to the heterogeneity of the nodes, the computation time is uneven, resulting in "bubbles". The decoding computation latency of a single token in the pipeline can be regarded as the sum of the computation latency of each stage and the maximum bubble latency. Therefore, the decoding computation latency calculation in the two cases is as follows:

[0021]

[0022] Where γ is the computational cost per token during the complete Decode computation. Due to batch processing requirements, queues are set up. And because of the three designed request scheduling strategies, requests in the Prefill and Decode phases may end up in one of two waiting queues. Therefore, the formula for calculating the queuing delay of a request in the Prefill or Decode computation phase based on its node group is as follows:

[0023]

[0024] Where q t ∈{Q H Q L} represents the target queue to which the Prefill or Decode request is scheduled, n H (t) represents Q under time slot t. H The number of prefill or decode tasks currently in the queue, and similarly, n L (t) represents Q under time slot t. L The number of Prefill or Decode tasks currently in the queue.

[0025] Based on the above calculation delay and queuing waiting delay, request R can be calculated. t The total end-to-end delay is shown in the following formula:

[0026]

[0027] 4) Energy consumption modeling.

[0028] Assume the idle power consumption of node i is The maximum power consumption during operation is The actual power consumption of a node in time slot t is proportional to the load. The formula for calculating the total energy consumption of all devices in the cluster during this time period is as follows:

[0029]

[0030] U tiThe computing resource utilization rate of a node is defined as the ratio of the amount of computing power allocated to node i per unit time to the device's computing power. This indicates the range of power consumption fluctuations at the node.

[0031] 5) Optimize the target definition.

[0032] In conclusion, let z t ∈{0,1} represents R t Whether the task is completed within the time frame T, λ1 and λ2 represent the weighting coefficients of throughput and energy cost, respectively. To satisfy the request SLO constraint, maximize throughput, and minimize energy cost, the final optimization problem is modeled as follows:

[0033]

[0034] st (TTFT delay constraint)

[0035] (TPOT delay constraint)

[0036] (Total Delay Constraint)

[0037] Step 2: Based on the heterogeneous computing power and energy consumption characteristics of the decoding nodes, implement a distributed pipeline non-uniform deployment strategy to further optimize model inference throughput. In low-computing-power node groups... To achieve a smooth pipeline, computation and communication should overlap as much as possible, and the computation time of each node should be similar to avoid frequent idle waiting by nodes due to pipeline inconsistencies, which would lead to wasted computing power and low resource utilization. At the same time, to achieve an efficient pipeline, nodes with higher computing power should be allocated more model layers for computation, but power consumption will be correspondingly higher. However, excessively pursuing low power consumption will result in wasted computing resources. Therefore, a trade-off must be struck between energy cost and performance. The specific steps are as follows:

[0038] 1) Considering the pipeline primarily handles the Decode stage computations, and each sequence in the input batch consists of a single token, the computational cost of each node can be assumed to be only related to the number of classification layers. Assume the deployed large model has M layers, and after uniform stratification in step 1, each low-computing-power node has the same number of layers, thus the FLOPs computation α of each layer is also approximately the same. The goal here is to determine the number of layers s allocated to each node. i This approach aims to maximize the energy efficiency of pipelined inference through non-uniform hierarchical deployment. Since the dimensionality of intermediate computation results is fixed, and given stable batch size and network bandwidth, communication latency also tends to stabilize. Let T be the computation time of node i. i We hope that node Li The computational load C is made as equal as possible across nodes to ensure similar computation times for each node. Energy consumption is also considered to achieve a trade-off between energy efficiency and performance. Therefore, the overall optimization objective for efficient pipelined inference in low-computing-power node groups is defined as follows:

[0039]

[0040] Wherein, the parameter λ∈[0,1] represents the trade-off between control performance and energy consumption, and T max =max j T i Calculate the delay for the maximum stage in a stable pipeline.

[0041] 2) To address this optimization objective, a heuristic hierarchical method based on computing power allocation and combined with energy efficiency correction is designed. The proportion is determined according to the computing power of each low-computing-power node, let r... i This represents the proportion of node i's computing power to the total computing power of all nodes. Additionally, the energy efficiency factor η of a node is defined. i The energy efficiency factor is denoted as per watt of computing power, where it is positively correlated with the computing power of each node and negatively correlated with the peak power consumption of the node. Our goal is to reward high-energy-efficiency nodes (i.e., higher η values) and penalize low-energy-efficiency nodes during allocation. Let w be the stratification ratio of node i obtained after adjusting the energy efficiency factor. i Using the aforementioned basic computing power allocation ratio r i and the energy efficiency factor correction term w i The formula for calculating the layer allocation of node i is:

[0042]

[0043] The parameter γ controls the allocation according to the computing power ratio, β reflects the influence of the energy efficiency correction term, and δ represents the accuracy correction coefficient for node i. Finally, verification and iterative adjustment are performed, and a threshold T is set. thres =0.05 indicates that the difference in computation delay between nodes should be less than this value, and the computation time T for each node should be less than this value. i And find the minimum time T min Then, if there exists a node T i >T min ·(1+T thres If a layer fails to migrate, it attempts to migrate one of the layers to the node with the shortest migration time. Due to pipeline characteristics, the migration process is essentially a loading and unloading process of the first and last model layers at each intermediate node. This process is repeated until convergence is achieved, satisfying the constraint T. i <T min ·(1+T thres ), To make the computation latency of each node as close as possible.

[0044] Step 3: Design a request scheduler and collect request and cluster state trajectories and round-based rewards during exploration. Use a large model to assist in encoding potential rewards and train a decoder to decode the proxy rewards at each step to assist the reinforcement learning PPO algorithm in training the scheduler. This enables the scheduling direction designed in Step 1 to be determined based on the SLO characteristics of the request, so as to maximize the overall inference throughput and minimize the total energy consumption of the cluster devices while satisfying the request SLO.

[0045] Since throughput and energy consumption are long-term metrics in the optimization objectives, the scheduler is trained using round-based reinforcement learning. This means that multiple state-action pairs are explored to form a trajectory before the reward value for that round is obtained and used to train the reinforcement learning model. Intuitively, this means determining the overall reward value based on throughput and energy consumption performance over a period of time. However, round-based rewards are sparse and difficult to learn. Therefore, we consider using LLM (Limited Least Merge Model) to intelligently decompose the sparse round rewards into "proxy rewards" for each step, thereby simulating the effect of real dense rewards and reducing the difficulty of reinforcement learning. The specific steps are as follows:

[0046] 1) Definition of state space, action space, and turn-based reward.

[0047] State space S t Defined as:

[0048]

[0049] in These respectively indicate that the Prefill and Decode tasks are included. and The number of nodes in each node group's waiting queue. R represents the resource utilization rate of all devices in the cluster per unit time. t This indicates the description of the incoming request.

[0050] Action Space A t Defined as:

[0051] A t =k∈{HPHD,HPLD,LPLD}

[0052] The actions that the scheduler may take correspond to the three scheduling strategies designed in step 1. Based on the cluster state information in the environment and the potential SLO requirements in the task description, different scheduling strategies can be adopted in a targeted manner.

[0053] Round-based rewards are defined as follows: within a specified time period T, the overall cluster throughput, energy consumption cost, and SLO violation status are collected, and throughput rewards, energy consumption penalties, and SLO violation penalties are awarded. As shown in the following formula:

[0054]

[0055] Where, λ T , λ E , λ V These represent the weighting coefficients for throughput, energy consumption cost, and SLO violation rate, respectively. t ∈{0,1} represents the request R under time slot t. t Whether the corresponding SLO constraints are satisfied.

[0056] 2) Use an LLM-assisted reinforcement learning training scheduler.

[0057] To guide the LLM in understanding the task and outputting potential rewards, standard cue words are designed for the current task scenario. These cue words consist of a templated environment description and specific task instructions. The environment description primarily explains the current optimization problem to the LLM and the task description that requires compressing and extracting state information. The task instructions in the cue words mainly describe the requirement for the LLM to generate a potential reward encoding function φ, where the input to function φ is a specific state s. t and action a t The output is a 3-dimensional potential reward vector z with explicit semantics. t =φ(s) t ,a t The vector is defined as [z1, z2, z3], where each dimension represents the performance of the input state and action at the corresponding level. z1 represents load balancing, z2 represents energy efficiency, and z3 represents SLO urgency. Feature engineering is performed using prior knowledge from LLM to filter out a large amount of redundant information irrelevant to the final goal.

[0058] The process of generating the potential reward encoding function requires self-verification, i.e., LLM. First, n candidate response outputs are generated, each including a latent reward encoding function. These candidate responses are then used as partial context for a cue word, prompting the LLM to summarize an improved response output ξ. The latent reward encoding function φ is extracted from the response ξ and used with pre-collected random state-action pairs. Verify φ and provide error feedback to the LLM until φ is executable.

[0059] Next, we define the reward decoding model f. ψ It receives potential reward z t As input, output a scalar value r t =f ψ (z tThe specific reward value is represented by ), which uses multi-dimensional information about potential rewards to analyze and derive the agent reward for a specific state-action pair, thereby solving the problem of contribution allocation for round-based rewards and improving the training performance of reinforcement learning by alleviating the sparse feedback problem.

[0060] Finally, a policy network π is built into the scheduler. θ To make scheduling decisions, inference tests are conducted on the current system using a mixed request dataset with different SLO requirements. The scheduler is used as an agent to explore the experimental environment. After each round, the round containing multiple state-action pairs is stored as a trajectory τ in the experience pool. In the process, the reward R(τ) for that round is obtained according to the definition of round-based rewards, and then the experience pool is... The trajectories in the data are sampled to obtain a batch of trajectories. Estimate the loss of the potential reward-assisted analysis agent's reward contribution allocation using the following loss function, and update the reward decoder f. ψ Parameters:

[0061]

[0062] Where η is the reward decoder f ψ The learning rate of training, This represents the gradient of the loss function. Update the reward decoder f. ψ The parameters are then used to predict the agent reward r. t =f ψ (z t ), and used in the reinforcement learning PPO algorithm to apply scheduling strategy π θ Optimize the training.

[0063] Repeat the above process until completion. Each round involves simultaneous training of the reward decoder f. ψ and the policy network π under the scheduler θ Value Network V ξ Finally, the trained scheduler is deployed to the cluster entry master node to schedule hybrid heterogeneous SLO requests to complete the inference service.

[0064] Compared with existing technologies, the advantages of this method are:

[0065] 1. In the deployment scenario of large model inference service in edge computing cluster, the heterogeneity of computing power and request SLO of cluster devices are considered at the same time. Request tasks with different SLO requirements are scheduled to be computed on appropriate nodes. At the same time, the idea of ​​PD separation is introduced to comprehensively consider the computing characteristics of different inference stages and the computing power characteristics of different devices in order to make full use of edge computing power and realize efficient and robust large model inference service in heterogeneous edge computing power cluster.

[0066] 2. During the training of the scheduler, the LLM-assisted reinforcement learning training method was used. The sparse round rewards were intelligently decomposed into agent rewards for each step through LLM, thereby simulating the effect of real dense rewards, reducing the difficulty of reinforcement learning and improving the training performance of reinforcement learning. Attached Figure Description

[0067] Figure 1 This is a flowchart of the present invention;

[0068] Figure 2 This is a system architecture diagram of the present invention;

[0069] Figure 3 This is a flowchart of the scheduler training process of the present invention. Detailed Implementation

[0070] like Figure 2 As shown, the specific steps of this invention are as follows:

[0071] Step 1: Based on the performance of the edge cluster nodes, all nodes are divided into two categories: high-computing-power nodes and low-computing-power nodes, forming two node groups. The Prefill and Decode stages of the inference process are separated and performed on different node groups. The nodes in the high-computing-power node group use a data parallel strategy, while the nodes in the low-computing-power node group use a distributed pipeline parallel strategy. Different types of requests are classified according to their SLO characteristics, and three request scheduling strategies are designed based on the request SLO characteristics to ensure full utilization of the device's computing power.

[0072] 1) Grouping of heterogeneous cluster nodes.

[0073] For a group of edge cluster devices The total number of devices is m. Assuming that all devices in the cluster are single-GPU devices, first, based on the total number of model parameters W to be deployed and the FLOPS (floating-point operations per second) computing power F of cluster node i, the process is as follows: i The cluster is divided into high-computing-power node groups. and low computing power node groups Where a+b=m. Specifically, for a large model to be deployed, the GPU memory required for a single inference (such as weight parameter usage, KV cache usage, etc.) is related to the input and output lengths. Let M be the minimum GPU memory requirement for deploying the complete model to a single device. min The video memory size should meet the M requirement. i ≥M min And computing power F i ≥F thres Nodes that meet the threshold are assigned to the high-computing-power node group, while other nodes that do not meet the criteria are assigned to the low-computing-power node group.

[0074] 2) Design of parallel and scheduling strategies.

[0075] In high-computing-power node groups In this system, each node can deploy a complete model and employs a data-parallel strategy for inference. This means that the inference computation for each request is executed by only one node, and different requests can be assigned to different nodes for independent inference computation, maximizing parallelism. In low-computing-power node groups... The system employs a distributed pipelined parallel inference strategy, dividing the model into S stages layer by layer, and satisfying the following conditions: Each stage is assigned to different nodes, with each node deploying only a portion of the model layers and corresponding weights. High-performance node groups... and low computing power node groups A waiting queue Q is set up in each of the following: H and Q L Used to store requests for computation to be performed, enabling batch processing of these requests. Node group and Due to differences in computational characteristics, the batch size is set to the hyperparameter B for each batch. H and B L .

[0076] For each time slot t∈{1,…,T}, the request R arriving... t The number of tokens in its input context is The number of output tokens is Different requests have different SLO attributes, which are used separately. This represents the maximum latency during the Prefill phase and the average maximum latency for each token generated during the Decode phase, corresponding to the two important SLO metrics: the latency of the first token and the average latency between tokens.

[0077] Based on the different characteristics of SLO for different types of requests, the request types are categorized according to SLO requirements, and three request scheduling strategies are designed as follows:

[0078] x tk ∈{0,1},k∈{HPHD,HPLD,LPLD}

[0079] Where x tk Represents the request R under time slot t. t Whether to select a certain scheduling strategy, where k represents one of the three specific strategies. The HPHD strategy indicates that in high-computing-power node groups... Prefill and Decode computations are completed in the middle, mainly for discriminative tasks or requests with very short outputs, which can effectively save KV cache transmission overhead to guarantee SLO; HPLD strategy means that in high-computing-power node groups Prefill calculations are completed in the low-computing-power node group. The Decode calculation is completed in the middle, and the Prefill stage is in The KV Cache calculated in the middle and the first token need to be transmitted to The subsequent decoding computation is performed in the middle, mainly targeting requests that are sensitive to TTFT and have long context inputs. It fully utilizes the two-stage computation characteristics of PD and the parallel strategy of node groups to maximize inference throughput by making full use of node computing power; the LPLD strategy means that in low-computing-power node groups The prefill and decode calculations are completed in the middle, which is mainly for TPOT-sensitive requests with short context inputs. This can effectively reduce energy consumption costs and avoid the transmission overhead of KVCache.

[0080] When a request is made to be scheduled to a high-computing-power node group During calculation, it will be added to the waiting queue Q. H In each node group, upon completion of inference, each node retrieves a batch of requests from the waiting queue for prefill computation to obtain the first token. If the number of remaining requests in the waiting queue is insufficient to form a batch, all remaining requests in the queue are combined into a batch for computation. Furthermore, to optimize KV Cache transmission under the HPLD strategy, during the computation process, after each model layer is completed, the calculated KV Cache is asynchronously transmitted to the corresponding model layer in the pipeline of the low-computing-power node group. This overlaps prefill transmission with computation, effectively reducing the transmission latency of the KV Cache under the HPLD strategy.

[0081] 3) Modeling of computation delay and waiting delay in each node group during the two-stage PD process.

[0082] In high-computing-power node groups Node H in i The computation delay for executing the Prefill batch is shown in the following formula:

[0083]

[0084] Where α is the computational cost per token (FLOPs / token) during full prefill computation, which is related to the model size but not to the token size. i Represents node H i The computing power (FLOP / s). And if the node group Node H in i Simultaneous processing of B H If there are multiple batch Decode requests, then each request is executed alternately based on the token. The computation time for each token is shown in the following formula:

[0085]

[0086] Where γ is the computational cost per token (FLOPs / token) during a single Decode operation. Because the decoding phase utilizes a KV Cache for auxiliary computation, its computational cost per token differs from α in the Prefill phase. Since batch processing is used, the total Decode batch processing time is related to the longest output sequence in the batch, as shown in the following formula:

[0087]

[0088] In low computing power node groups In this scenario, assuming pipelined inference is blocking, meaning the predecessor node must wait for the current node to become idle before it can transmit intermediate computation results and continue processing the next token, the computation latency for performing Prefill batch processing in the distributed pipeline is as follows:

[0089]

[0090] The batch size is B. H Furthermore, the large model is evenly divided into layers and distributed to each node in the pipeline. The latency calculation for pipeline batch prefill considers the worst-case scenario. Since the request context lengths scheduled to the pipeline for prefill have relatively small differences, analysis shows that the pipeline stage with the largest computational latency can cover the computational latency of other stages. In addition, due to the heterogeneity of pipeline nodes, the computation time of each node is uneven, which can easily cause some nodes to be idle and waiting, reducing pipeline efficiency, i.e., creating the so-called "bubble". Similarly, in the worst-case scenario, the decoding computation latency of a single token in the pipeline can be considered as the sum of the computational latency of each stage and the maximum bubble latency. Therefore, for the decoding stage of continuously generated tokens, the pipeline computation latency for each token is as follows:

[0091]

[0092] Similarly, the total time for computation using batch Decode is related to the longest output length sequence in the batch, as shown in the following formula:

[0093]

[0094] Queues are set up for batch processing, and due to the three designed request scheduling strategies, both the Prefill and Decode phases of a request may end up in one of the two waiting queues. Therefore, the queuing delay calculation formula based on the node group where the Prefill request is located is as follows:

[0095]

[0096] This is primarily determined by the number of requests currently in the queue and the expected computational latency, where This indicates the target queue to which the Prefill request is scheduled. Q under time slot t H The current number of prefill tasks in the queue does not include the current request. Similarly, Q under time slot t L The number of prefill tasks currently in the queue. Similarly, the formula for calculating the queuing delay for Decode requests is as follows:

[0097]

[0098] in This represents the target queue to which the Decode request is scheduled, where t′ represents the time slot when the Prefill is completed. Only after the calculation in the Prefill stage is completed can the request enter the Decode waiting queue to wait for calculation.

[0099] Based on the above calculation delay and queuing waiting delay, request R can be calculated. t The total end-to-end delay is shown in the following formula:

[0100]

[0101] in and These represent the total latency for the Prefill and Decode computations, respectively. This latency depends on the request's scheduling policy and batch processing characteristics, and is thus determined by... Sure.

[0102] 4) Energy consumption modeling.

[0103] Assume the idle power consumption of node i is The maximum power consumption during operation is The actual power consumption of a node in time slot t is proportional to the load. The formula for calculating the total energy consumption of all devices in the cluster during this time period is as follows:

[0104]

[0105] U ti The computing resource utilization rate of a node is defined as the ratio of the amount of computing power allocated to node i per unit time to the computing power of the device, satisfying U ti ∈[0,1], and This indicates the range of power consumption fluctuations at the node.

[0106] 5) Optimize the target definition.

[0107] In summary, the final optimization problem is modeled as follows:

[0108]

[0109] st (TTFT delay constraint)

[0110] (TPOT delay constraint)

[0111] (Total Delay Constraint)

[0112] Where z t ∈{0,1} represents R t Whether the task is completed within the time frame T is determined by λ1 and λ2, which represent the weighting coefficients for throughput and energy cost, respectively. The above optimization problem is a multi-objective optimization problem: maximizing throughput and minimizing energy cost while satisfying the request SLO constraint. This problem can be reduced to a heterogeneous parallel scheduling problem with multi-dimensional resource constraints and timing requirements; therefore, it belongs to the NP-hard category.

[0113] Step 2: Based on the heterogeneous computing power and energy consumption characteristics of the decoding nodes, implement a distributed pipeline non-uniform deployment strategy to further optimize model inference throughput. In low-computing-power node groups... To achieve a smooth pipeline, computation and communication should overlap as much as possible, and the computation time of each node should be similar to avoid frequent idle waiting by nodes due to pipeline inconsistencies, which would lead to wasted computing power and low resource utilization. At the same time, to achieve an efficient pipeline, nodes with higher computing power should be allocated more model layers for computation, but power consumption will be correspondingly higher. However, excessively pursuing low power consumption will result in wasted computing resources. Therefore, a trade-off must be struck between energy cost and performance. The specific steps are as follows:

[0114] 1) Considering the pipeline primarily handles the Decode stage computations, and each sequence in the input batch consists of a single token, the computational cost of each node can be assumed to be only related to the number of classification layers. Assume the deployed large model has M layers, and after uniform stratification in step 1, each low-computing-power node has the same number of layers, thus the FLOPs computation α of each layer is also approximately the same. The goal here is to determine the number of layers s allocated to each node. i To maximize the energy efficiency of pipelined inference through non-uniform layered deployment, it satisfies the following equation:

[0115]

[0116] In the Decode phase of the low-node group pipeline, the batch size to be processed in each step is B. L Meanwhile, the communication delay between nodes is also considered. Since the dimensions of the intermediate calculation results are fixed, and given stable batch size and network bandwidth, the communication latency also tends to stabilize. Therefore, by analyzing the computational load of node i, the corresponding computation time can be determined as follows:

[0117]

[0118] To ensure a uniform computational load across all nodes, we want node L to... i The computational load C is made as equal as possible across nodes to ensure similar computation times for each node. Energy consumption is also considered to achieve a trade-off between energy efficiency and performance. Therefore, the overall optimization objective for efficient pipelined inference in low-computing-power node groups is defined as follows:

[0119]

[0120] Wherein, the parameter λ∈[0,1] represents the trade-off between control performance and energy consumption, and T max =max j T j Calculate the delay time for the maximum stage in a stable pipeline.

[0121] 2) To achieve this optimization goal, a heuristic layering method based on computing power ratio allocation combined with energy efficiency correction is designed. Specifically, the model layers are first allocated according to the basic computing power ratio, then an energy consumption correction term is added to adjust the layer allocation, and finally iterative verification is performed.

[0122] The proportion is determined based on the computing power of each low-power node, let r i Let represent the proportion of node i's computing power to the total computing power of all nodes. The calculation formula is as follows:

[0123]

[0124] Define the energy efficiency factor η of the node i Let η be the computing power per watt, where the energy efficiency factor is positively correlated with the computing power of each node and negatively correlated with the peak power consumption of the node. We aim to reward high-energy-efficiency nodes (i.e., higher η values) and penalize low-energy-efficiency nodes during allocation. Therefore, the stratification ratio of node i obtained by adjusting the energy efficiency factor is:

[0125]

[0126] Where β>0 is the adjustment coefficient, and the larger the coefficient, the more it is biased towards nodes with higher energy efficiency. Using the basic computing power allocation ratio and energy efficiency factor correction term mentioned above, the layer allocation calculation formula for node i is as follows:

[0127]

[0128] In this system, parameter γ controls the allocation according to the computing power ratio, while β reflects the degree of influence of the energy efficiency correction term. If β = 0, it indicates a pure computing power ratio, while if γ = 0, it indicates a pure energy efficiency-driven approach. Taking γ = 1 and β = 0.5 is a relatively balanced scheme. In addition, δ represents the precision correction coefficient corresponding to node i. Since the number of layers corresponding to each node calculated proportionally may not be an integer, rounding down may lead to a difference between the overall corrected number of layers and the actual total number of layers M. Therefore, a precision correction coefficient needs to be added to ensure that the summation difference is eliminated.

[0129] Finally, verification and iterative adjustments are performed, and a threshold T is set. thres =0.05 indicates that the difference in computation delay between nodes should be less than this value, and the computation time T for each node should be less than this value. i And find the minimum time T min Then, if there exists a node T i >T min ·(1+T thres If a layer fails to migrate, it attempts to migrate one of the layers to the node with the shortest migration time. To avoid disrupting the sequential nature of the pipeline, the migration process is essentially a loading and unloading process of the first and last model layers at each intermediate node. This process is repeated until convergence is achieved, satisfying the constraint T. i <T min ·(1+T thres ), This allows the computational latency between nodes to be as close as possible.

[0130] Step 3: Design a request scheduler and collect request and cluster state trajectories and round-based rewards during exploration. Use a large model to assist in encoding potential rewards and train a decoder to decode the proxy rewards at each step. This assists in training the scheduler using the reinforcement learning PPO algorithm, enabling the scheduling direction designed in Step 1 to be determined based on the SLO characteristics of the request. This maximizes overall inference throughput and minimizes the total energy consumption of the cluster devices while satisfying the request SLO. The specific steps are as follows:

[0131] 1) Definition of state space, action space, and turn-based reward.

[0132] State space S t Defined as:

[0133]

[0134] in These respectively indicate that the Prefill and Decode tasks are included. and The number of nodes in each node group's waiting queue. R represents the resource utilization rate of all devices in the cluster from the previous time slot to the current time slot. t This indicates the description of the incoming request. The significance of this design is that the scheduler needs to comprehensively consider the queuing status of each node group, the device utilization rate, and the characteristics of the task description, and make a reasonable scheduling strategy by comprehensively analyzing the overall load of the cluster.

[0135] Action Space A t Defined as:

[0136] A t =k∈{HPHD,HPLD,LPLD}

[0137] The actions that the scheduler may take correspond to the three scheduling strategies designed in step 1. The HPHD strategy indicates that in high-computing-power node groups... Prefill and Decode computations are completed in the middle, and the HPLD strategy indicates that they are performed in the high-computing-power node group respectively. and low computing power node groups Prefill and Decode computations are completed in the middle, and the LPLD strategy indicates that in low-computing-power node groups The Prefill and Decode calculations are completed within the process. Based on the cluster status information in the environment and the potential SLO requirements in the task description, different scheduling strategies are adopted accordingly.

[0138] Round-based rewards are defined as follows: within a specified time period T, the overall cluster throughput, energy consumption cost, and SLO violation status are collected, and throughput rewards, energy consumption penalties, and SLO violation penalties are awarded. As shown in the following formula:

[0139]

[0140] Where, λ T , λ E , λ V These represent the weighting coefficients for throughput, energy consumption cost, and SLO violation rate, respectively. t ∈{0,1} represents the request R under time slot t. t Whether the corresponding SLO constraints are satisfied.

[0141] 2) Using an LLM-assisted reinforcement learning training scheduler, the overall training process is as follows: Figure 3 As shown in the diagram. Specifically, firstly, LLM is used to assist in encoding the potential reward and perform self-verification; then, the decoder is trained to decode the agent reward; and finally, the reinforcement learning PPO algorithm is used to train the scheduler.

[0142] To guide the LLM in understanding the task and outputting potential rewards, standard cue words are designed for the current task scenario. These cue words consist of a templated environment description and specific task instructions. The environment description primarily describes the current optimization problem and the task description for compressing and extracting state information to the LLM. The task instructions in the cue words require the LLM to generate a potential reward encoding function φ, and the input of function φ is the specific state s. t and action a t The output is a 3-dimensional potential reward vector z with explicit semantics. t =φ(s) t ,a t The vector is defined as [z1, z2, z3], where each dimension represents the performance of the input state and action at the corresponding level. z1 represents load balancing, negatively correlated with the standard deviation of node utilization; z2 represents energy efficiency ratio, positively correlated with the ratio of completed requests to energy consumption; and z3 represents SLO urgency, negatively correlated with the remaining time of the most urgent request in the queue. Feature engineering is performed using prior knowledge from LLM to filter out a large amount of redundant information irrelevant to the final goal.

[0143] The process of generating the latent reward encoding function needs to be self-verified to enhance stability and reliability. Specifically, LLM... First, generate n candidate response outputs. Each response output includes a Python code implementation of the potential reward encoding function. These candidate response outputs are then used as partial context for the cue words, prompting the LLM to summarize an improved response output. Extract the latent reward encoding function φ from the response ξ, and use pre-collected random state-action pairs. Verify φ and provide error feedback to the LLM until φ is executable. The process is as follows:

[0144]

[0145] Next, we define the reward decoding model f. ψ It receives potential reward z t As input, output a scalar value r t =f ψ (z t The specific reward value is represented by ), which uses multi-dimensional information about potential rewards to analyze and derive the agent reward for a specific state-action pair, thereby solving the problem of contribution allocation for round-based rewards and improving the training performance of reinforcement learning by alleviating the sparse feedback problem.

[0146] Finally, a policy network π is built into the scheduler. θTo make scheduling decisions, inference tests were conducted on the current system using a mixed request dataset with different SLO requirements. The scheduler was used as an agent to explore the experimental environment. A round was defined as T = 60s. After each round, the round containing multiple state-action pairs was stored as a trajectory τ in the experience pool. In the process, the reward R(τ) for that round is obtained according to the definition of round-based rewards, and then the experience pool is... The trajectories in the data are sampled to obtain a batch of trajectories. Estimate the loss of the potential reward-assisted analysis agent's reward contribution allocation using the following loss function, and update the reward decoder f. ψ Parameters:

[0147]

[0148] Where η is the reward decoder f ψ The learning rate of training, This represents the gradient of the loss function. Update the reward decoder f. ψ The parameters are then used to predict the agent reward r. t =f ψ (z t ), and used in the reinforcement learning PPO algorithm to apply scheduling strategy π θ During training, the algorithm incorporates a surrogate pruning objective function and a dominance function. The experience pool is also adjusted accordingly. Trajectories are sampled, and the advantage function is calculated using generalized advantage estimation (GAE).

[0149] A t =δ t +(γλ)δ t+1 +(γλ) 2 δ t+2 +...+(γλ) T-t-1 δ T-1

[0150] δ t =r t +γV(s t+1 )-V(s t )

[0151] Where, δ t The timing difference error is represented by the reward decoder f. ψ The estimated agent reward, γ is the discount factor, λ is the GAE smoothing parameter, and V(s) is a value network V ξ The estimated state-value function, for which the network is trained synchronously with the policy network, has the objective function of minimizing the mean squared error L(ξ) = (R... t -V ξ (st )) 2 R t This represents the cumulative agent reward for the current trajectory.

[0152] The PPO algorithm ensures stability by limiting the step size of policy updates. Based on the advantage estimation calculations described above, the scheduling policy network π... θ The training uses a pruned proxy objective function as follows:

[0153] L CLIP (θ)=E t [min(ρ t (θ)A t ,clip(ρ t (θ), 1-ε, 1+ε)A t )]

[0154] in, The importance sampling ratio, It is the old strategy before the update. ε is a hyperparameter used to prune the probability ratio to prevent the strategy from changing too much in a single update. The min operation ensures that the performance will not crash due to an excessively large update step size while the strategy is improved.

[0155] Repeat the above process until completion. Each round involves simultaneous training of the reward decoder f. ψ and the policy network π under the scheduler θ Value Network V ξ Finally, the trained scheduler is deployed to the cluster entry master node to schedule hybrid heterogeneous SLO requests to complete the inference service.

Claims

1. A large-scale model inference optimization method with SLO awareness in an edge heterogeneous computing network, characterized in that, Includes the following steps: Step 1: Group heterogeneous computing nodes according to the performance of edge network cluster nodes, and based on the large model Prefill-Decode separation strategy, adopt distributed data parallelism and pipeline parallel deployment and inference technology in the high computing power node group and the low computing power node group respectively, and design three different inference scheduling strategies for heterogeneous SLO requests. Step 2: Based on the cluster deployment and inference framework in Step 1, according to the heterogeneous computing power and energy consumption characteristics of nodes in the low computing power node group, implement the hierarchical non-uniform distribution adjustment of the large model in the distributed pipeline to ensure efficient pipeline inference. Step 3: Based on the pipeline model hierarchical allocation strategy in Step 2, design a request scheduler. During exploration, collect request and cluster state trajectories and round-based rewards. Use LLM (Large Language Model) to assist in encoding potential rewards and train the decoder to decode the proxy rewards at each step of the trajectory, which are then used to train the scheduler for the reinforcement learning PPO algorithm.

2. The SLO-aware large model inference optimization method in edge heterogeneous computing networks according to claim 1, characterized in that, The edge cluster heterogeneous devices Let the minimum video memory requirement for deploying the complete model to a single device be... Based on the total number of model parameters to be deployed and the FLOPS (floating-point operations per second) computing power of the cluster nodes. The video memory size should meet the video memory space requirements. And computing power Nodes that exceed the threshold are assigned to high-computing-power node groups. The remaining nodes that do not meet the criteria are assigned to the low-computing-power node group. ; In high-computing-power node groups The system employs a data-parallel inference strategy, assigning different requests to different nodes for independent inference to maximize parallelism; in low-computing-power node groups The model employs a distributed pipelined parallel inference strategy, dividing the model into layers. Each stage, and meets the following requirements. Each stage is assigned to different nodes, with each node deploying only a portion of the model layers and corresponding weights; high-computing-power node groups and low computing power node groups Waiting queues are set up separately. and Used to store requests for computation to be performed, enabling batch processing of the requests; node group and Due to differences in computational characteristics, the batch size is set as a hyperparameter for each. and ; For each time slot The request that was received Its attributes include the number of tokens in the input context and a description. Different requests have different SLO attributes, which are used accordingly. This represents the first token latency requirement (TimetoFirstToken, TTFT) and the average latency requirement between tokens (TimePerOutputToken, TPOT). and Let represent the computation latency of the request during the Prefill and Decode phases, respectively. The condition for satisfying the SLO is defined as follows: and Based on the different SLO characteristics of different request types, request types are categorized according to SLO requirements, and three request scheduling strategies are designed as follows: in express Requests under time slots Whether to select a certain scheduling strategy, and This indicates these three specific strategies. The strategy is represented in high-computing-power node groups The Prefill and Decode calculations are completed in the middle; The strategy is represented in high-computing-power node groups Prefill calculations are completed in the low-computing-power node group. The Decode calculation is completed in the middle, and the Prefill stage is in The KVCache calculated in the middle and the first token need to be transmitted to This is used for subsequent decoding calculations; The strategy is indicated in low-computing-power node groups The Prefill and Decode calculations are completed in the middle; When a request is scheduled to a high-computing-power node group During calculation, it will be added to the waiting queue. In this process, each node in the node group performs prefill calculations on a batch of requests from the waiting queue when each inference is completed. When the number of remaining requests in the waiting queue is insufficient to form a batch, all requests in the queue are combined into a batch for calculation. At the same time, during the calculation process, the KVCache is asynchronously transferred to the pipeline of the low-computing-power node group after each model layer calculation is completed, so as to achieve overlap between prefill calculation and transmission.

3. The SLO-aware large model inference optimization method in edge heterogeneous computing networks according to claim 1, characterized in that, The problem of non-uniform allocation of large model layers in the pipeline is solved using a heuristic layering method based on computing power ratio allocation combined with energy efficiency correction. The basic allocation ratio for model layering is determined according to the computing power of each low-computing-power node, letting... Represents a node The proportion of a node's computing power to the total computing power of all nodes is calculated using the following formula: Define the energy efficiency factor of a node The computing power per watt is positively correlated with the computing power of each node and negatively correlated with the peak power consumption of the node. The goal is to reward high-efficiency nodes and penalize low-efficiency nodes during allocation, and the resulting nodes are adjusted using an energy efficiency factor. The layering ratio is: in The larger the adjustment coefficient, the more it favors nodes with higher energy efficiency; using the aforementioned basic computing power allocation ratio and energy efficiency factor correction term, the nodes... The formula for calculating the layer allocation is: Among them, parameters Control the allocation based on computing power ratio. This reflects the extent of the impact of energy efficiency correction items. Represents a node The corresponding precision correction factor, because the number of layers corresponding to each node is not an integer when calculated proportionally, will result in a difference between the overall corrected number of layers and the actual total number of layers after rounding down. Since there is a difference, a precision correction factor is added to ensure that the summation difference is eliminated; Finally, verification and iterative adjustments are performed, and a threshold is set. This represents the maximum tolerable percentage of latency difference between nodes, calculated during the time interval of each node. And find the minimum time Afterwards, in accordance with The node migrates one layer to the node with the shortest time; repeat the above process until convergence is achieved and the constraints are satisfied. This achieves balanced computational latency among nodes.

4. The SLO-aware large model inference optimization method in edge heterogeneous computing networks according to claim 1, characterized in that, The method for training a scheduler using LLM-assisted reinforcement learning further includes at least the following steps: 1) Define the state space, action space, and turn-based rewards. State space... Defined as: in , , , These respectively indicate that the Prefill and Decode tasks are included. and The number of nodes in each node group's waiting queue. This represents the resource utilization rate of all devices in the cluster from the previous time slot to the current time slot. This indicates the description of the incoming request; Action space Defined as: The scheduler's action space corresponds to the three scheduling strategies designed in step 1. The strategy is represented in high-computing-power node groups The Prefill and Decode calculations are completed in the middle. The strategy is represented in high-computing-power node groups respectively. and low computing power node groups The Prefill and Decode calculations are completed in the middle. The strategy is indicated in low-computing-power node groups The Prefill and Decode calculations are completed in the process; based on the cluster status information in the environment and the potential SLO requirements in the task description, different scheduling strategies are adopted accordingly. A round-based reward is defined as a time period of a specified length. Internally, the system collects data on overall cluster throughput, energy consumption costs, and SLO violations, and assigns throughput rewards, energy consumption penalties, and SLO violation penalties, as shown in the following formula: in, , , These represent the weighting coefficients for throughput, energy consumption cost, and SLO violation rate, respectively. Indicates time slot The following request Are the corresponding SLO constraints satisfied? 2) Design standard prompts for the current task scenario, consisting of a templated environment description and specific task guidance. The environment description describes the current optimization problem to the LLM and the task description that requires the compression and extraction of state information, while the task guidance in the prompts describes the potential reward encoding function that the LLM needs to generate. The requirements, function The input is the state. and actions The output is a 3-dimensional potential reward vector with explicit semantics. Each dimension of the vector represents the performance level of the input state and action at the corresponding level. This indicates the degree of load balancing and is negatively correlated with the standard deviation of utilization among nodes. This represents the energy efficiency ratio, which is positively correlated with the ratio of completed requests to energy consumption. This indicates the SLO urgency, which is negatively correlated with the remaining time of the most urgent request in the queue; The process of generating the potential reward encoding function is self-verified to enhance stability and reliability. Specifically, LLM... First generate The response output of each candidate Each response output includes a Python code implementation of a potential reward encoding function. These candidate response outputs are then used as partial context for the cue words, prompting the LLM to summarize an improved response output. From the response Extracting the latent reward encoding function and using pre-collected random state-action pairs verify and provide error feedback to the LLM until The execution was successful and error-free. The process is as follows: Next, define the reward decoding model. It receives potential rewards As input, output a scalar value It represents the specific reward value, which uses multi-dimensional information of potential rewards to analyze and derive the agent reward of the specific state-action pair, so as to alleviate the sparse feedback problem and improve the training performance of reinforcement learning. 3) Built-in policy network in the scheduler Scheduling decisions are made by conducting inference tests on the current system using a mixed request dataset with different SLO requirements. The scheduler is used as an agent to explore the experimental environment, and a trajectory containing multiple state-action pairs is obtained after each round. Store in experience pool In the middle, the reward for that round is obtained according to the definition of round-based rewards. Next, the experience pool The trajectories in the data are sampled to obtain a batch of trajectories. The following objective function is used to estimate the loss of the potential reward-assisted analysis agent's reward contribution allocation and to update the reward decoder. Parameters: in, To reward decoders The learning rate of training, Represent the gradient of the loss function; update the reward decoder. The parameters are then used to predict the agent's reward. Scheduling strategy used in reinforcement learning PPO algorithm Optimize the training.

Images