Joint Model Inference Method Based on Model Inference Timeliness Optimization

Abstract

This invention discloses a joint model inference method based on model inference timeliness optimization, belonging to the field of deep learning technology. It includes defining a set of LLM inference requests, models, and edge servers; constructing a directed acyclic graph (DAG) for model calls for each request type; defining an inference timeliness metric, AoMI; modeling the request-model relationship as a hypergraph; using k-way hypergraph partitioning to optimize initial model deployment; designing a two-stage model redeployment mechanism of incremental repartitioning and global triggering redeployment to dynamically adapt to request changes; and constructing dynamic model optimization as a Markov decision process (MDP), combined with a curiosity-guided memory association learning method to optimize call decisions. This invention provides performance metrics for inference optimization through AoMI; dynamically addresses complex request-model relationships through hypergraph partitioning technology and a two-stage model deployment mechanism, optimizing communication and migration costs; and improves model call efficiency through a curiosity-based learning method.

Description

Technical Field

[0001] This invention relates to the field of deep learning technology, and in particular to a joint model inference method based on the optimization of model inference timeliness. Background Technology

[0002] In recent years, deep learning technologies, especially large language models (LLMs), have demonstrated outstanding performance in various generative tasks. However, LLM inference typically relies on centralized cloud computing, which suffers from high latency and low responsiveness. With the rise of edge computing, deploying the models required for LLM inference to distributed edge cloud systems has become a key approach to solving these problems. By deploying model inference services closer to users, distributed edge cloud systems can significantly reduce latency and improve response speed.

[0003] In distributed edge cloud systems, LLM inference requests typically involve multiple interdependent models. That is, the inference of a particular model requires the outputs of preceding models, and the relationships between these models are highly dynamic. Different types of models may be deployed on different edge servers. Summary of the Invention

[0004] The purpose of this invention is to provide a joint model inference method based on the optimization of model inference timeliness, so as to ensure that inference tasks can be completed efficiently and in a timely manner when multiple models are interdependent.

[0005] To achieve the above objectives, this invention provides a joint model inference method based on model inference timeliness optimization, comprising the following steps: S1. Constructing the edge cloud system model and AoMI quantification model: Define the set of LLM inference requests, models, and edge servers, construct a directed acyclic graph (DAG) for each request type, and integrate time penalty and inference latency penalty to define the inference timeliness index (AoMI). S2. Dynamic Model Deployment and Two-Phase Redeployment Based on Hypergraph Partitioning: The request-model relationship is modeled as a hypergraph, and k-way hypergraph partitioning is used to optimize the initial model deployment; a two-phase model redeployment mechanism of incremental repartitioning and global triggering redeployment is designed to dynamically adapt to request changes; S3. Model call decision based on curiosity-guided memory association learning: The model is dynamically optimized as a Markov decision process (MDP), and a burst request controller is set up to deal with request overload; high-value experience is propagated through memory association, and the curiosity mechanism of state novelty calculation is combined to optimize the call decision.

[0006] Preferably, in S1, the LLM inference request includes L different types, denoted as L(L, L, L) ,in A collection of request types; each request Each requires calling several models to generate a complete response; the collection of all available models. ,in Indicates the model type.

[0007] Preferably, in S1, the DAG relationship between each request type and the required model is denoted as... ;in, This represents the set of models involved in the request, with each vertex... Corresponding to a model deployed in an edge cloud system, for any two vertices and If execution dependency edges exist ,but exist Each model can only be invoked after execution is complete, and the execution of each model cannot be interrupted; these models are deployed on... On a set of edge servers, the set is denoted as .

[0008] Preferably, in S1, the AoMI metric measures the timeliness of the edge cloud system. The AoMI model construction specifically includes the following steps: S11, Define Time t The AoMI is: ;in, This represents the time penalty function, which measures the interval between the current time and the generation time of the most recently completed inference request; The LLM inference penalty function is used to capture whether the completion delay of the latest completed inference request is within its maximum acceptable delay range. Unlike traditional latency metrics that only measure request completion time, AoMI comprehensively evaluates the timeliness of the entire LLM inference process in the edge cloud system. S12, Time Penalty Function Reflecting the increase in time penalty, it is modeled as a linear time function to quantify the decline in timeliness, defined as: ; in, The generation time of the most recently completed inference request is represented by the equation. Get the index of the latest completed request. , Indicates a request The completion time; the time penalty function is designed to penalize the elapsed time since the last completed inference request, thereby encouraging the system to complete the next LLM inference request as quickly as possible; S13. Introduce an LLM inference penalty function to quantify the decrease in timeliness caused by inference expiration. The LLM inference penalty function is: ; in , This indicates a delay in the completion of the latest completed inference request. This is the maximum acceptable inference delay for the request.

[0009] Preferably, LLM inference requests involve joint inference across multiple edge servers. Due to the sequential dependencies between models, a model cannot be invoked until all preceding models have completed execution. Therefore, the model inference completion time is defined as: ; in, Representation Model The earliest time it becomes callable. This is the set of preceding models for this model. It transmits the output of the preceding model to the model. Time required; The completion time of the preceding model for this model; For model inference time; Model inference time From the model Computing requirements and edge servers The ratio between the computing power of the two is determined by the ratio of their respective computing capabilities. ,in It is a model The computational load, It is an edge server On the computing power; request The completion delay is defined as: ; in, It is a request A set of models that have no subsequent dependencies. and These are requests The generation time and completion time.

[0010] Preferably, the timeliness of inference is jointly affected by model deployment and invocation decisions. An optimization problem model is defined, with the following details: Model Deployment (MD): Determines which models will be hosted on edge servers, denoted as... Model deployment status A value of 1 indicates the model Deployed on the server Above, a value of 0 indicates that no deployment has occurred; Model Invocation (MI): Selects an already deployed model service LLM inference request, denoted as: Request model invocation scheme A value of 1 indicates the model Called to handle requests A value of 0 indicates that no call was made; Divide the timeline into non-overlapping time windows. And optimize independently within each window. The optimization problem can be expressed as: ; Constraints: Load balancing constraints limit the total size of the model on each edge server. This ensures that the imbalance remains within the tolerance range of the average value; the tolerance for this imbalance is determined by the parameter... control, Force a fully uniform deployment at that time; Ensure that the model can only be invoked after all preceding models have completed execution; Ensure that the model is invoked after the inference of its preceding requests has been completed; In each time window Within it, AoMI is a piecewise function that performs operations at each request completion time. There are discrete jump points, that is, in each interval Inside, It remains unchanged and equals Due to the LLM inference penalty function Depending on whether the completion delay of the most recently completed inference request is within the maximum acceptable delay, its value remains constant within the interval, thus dividing the time domain into unimportant intervals. And will optimize the problem Rewritten as: ; in, It is time reference mouth The collection of requests within, the request generation time Given a constant, minimizing AoMI is equivalent to minimizing and maintain a delay in request completion. Within its maximum acceptable delay.

[0011] Preferably, in S2, the hypergraph allows each hyperedge to connect any number of nodes, modeling the request as a hyperedge connecting all required models, thus defining the hypergraph. Contains a finite set of non-empty vertices and a set of super edges Each of the super edges It is a vertex set A non-empty subset of , the union of all hyperedges covers the entire vertex set, that is: ; In model deployment, a request-model hypergraph is built based on the model requirements of the request. Each vertex Corresponding to a deployment model Each super edge Represents an LLM inference request Connect all models that participated in executing the request; Because request inference involves multiple models deployed on different servers, binary variables are defined. To indicate a request Do we need to use a server? ,Right now: ; in, This determines where the model is hosted on an edge server, where Model deployment status A value of 1 indicates the model Deployed on the server Above, a value of 0 indicates that no deployment has occurred; This represents the deployment model for serving LLM inference requests. , A value of 1 indicates the model Called to handle requests A value of 0 indicates that no call was made; Indicates a request A server is required. Assuming there is no communication cost within the same server and the communication cost is the same between two different servers, it is expressed as follows: ; To reduce communication overhead and minimize the number of servers required, define the request. The cost of cross-server communication is: ; in, Indicates a request On the server and The amount of data transferred between them; , Each represents a request Do we need to use a server? and Since the model used for inference depends on the output of its predecessor model, the model... The time to receive results from its predecessor model is defined as: ; in, Indicates a request The amount of data transferred from the model; In the request-model hypergraph, requests are modeled as hyperedges connecting their required models. To reduce communication overhead, the k-way hypergraph partitioning method is applied to the constructed hypergraph to partition the model set. Divided into The plan is to divide the network into two unimportant blocks, namely servers, to minimize the connectivity metric. SuperMap k-way partitioning It is its vertex set Divided into A mutually exclusive subset, denoted as Each vertex in the hypergraph Only a subset of servers are allocated to each model, and each subset corresponds to a server. The goal of the model deployment (MD) problem is to place the model on the servers, minimizing cross-server communication for joint model inference. ; in, It is a load balancing constraint that limits the total size of the model on each edge server. This ensures that the imbalance remains within the tolerance range of the average value; the tolerance for this imbalance is determined by the parameter... control, Force a fully uniform deployment at that time; Indicates the popularity of the request, reflecting the frequency of its occurrence. C4 avoids double-counting communication costs.

[0012] Preferably, in S2, the specific method for redeploying the two-stage model is as follows: Incremental Repartitioning: To adapt to the evolving distribution of LLM inference request types while avoiding excessive overhead from global redeployment, an incremental repartitioning is introduced. This repartitioning incrementally adjusts model deployment based on request patterns and dependency structures: First, high-frequency request types are identified, and incremental model migration scores are calculated; second, after identifying models with high migration values, suitable servers for these models to be migrated are determined, with the principle of incremental model deployment to minimize inter-server communication costs during federated inference; before migration, the models are checked to ensure they meet the requirements of reducing access span and satisfying server balance constraints. Trigger global redeployment: When the migration of models exceeds a predefined threshold, the model deployment on all servers is re-optimized. This phase is based on a k-way hypergraph partition and aims to generate a new deployment scenario X′ that jointly minimizes inter-server communication and model migration costs from previous deployments.

[0013] Preferably, in S3, the Markov Decision Process (MDP) specifically includes the following steps: S31, at time step At that time, the edge cloud system observed the state. include: The model calls to the DAG for all requests are denoted as follows: ,in Model call relationships for each request type; model deployment status. ; most recent request completion time The remaining time required for the model to complete the current inference. and the model execution status of the currently ongoing request. ,in It is a request The model execution state set is defined. ,in Indicates the execution status of the model. This indicates that the reasoning has been completed. This indicates that the reasoning is not yet complete; S32, Motion Space Includes model invocation decisions, The representative request-model invocation scheme is defined as follows: , indicating a request Should the model be called? To reason; S33. The reward function is based on the AoMI minimization design, and the calculation formula is as follows: ; S34. The burst request controller uses an AoMI-based heuristic strategy to select a request that is expected to minimize the AoMI increment; the MDP uses Deep Reinforcement Learning (DRL) for model invocation, and when the request arrival rate exceeds a safety threshold... At that time, an emergency request was made for the controller to take over the call process; S35, State-Action Value Function Estimate based on strategy The cumulative reward is generated through a deep neural network. To approximate, during training, the predicted values ​​are aligned with the target values ​​derived from observed rewards and estimated future rewards; during training, experience... Sampled from the playback buffer and updated via the following targets: ; in, These are parameters of the target network that are updated periodically. It is a discount factor; the DRL path is trained solely based on its own exploration, and the burst request controller serves as a backup mechanism, which is only activated under burst conditions and does not interfere with the learning process.

[0014] Preferably, in S3, memory association learning demonstrates associative reasoning by linking relevant experiences to adapt to new but similar situations, introducing a relational state structure to capture associations from observed trajectories; Specifically, state relation structure as follows: ; relation It means state It is a state Given a state relation structure, the successor in the sampled trajectory under a certain action is... Value propagation mechanism is defined as This propagation relationship allows highly rewarding experiences to spread between relevant states; Curiosity-driven associative learning of memories includes: Memory association: Using high-value estimation to supervise value functions Updates will be provided, and exploration of unexplored areas will be encouraged; Curiosity mechanism: calculating the novelty score of state embeddings ,in It is a state The nearest neighbor in the embedding space; the intrinsic reward calculation formula is: ,in The weighting of curiosity is controlled; the total reward, combining extrinsic and intrinsic rewards, is expressed as: ; Setting the loss function: The value function is updated under the learning objective of time difference, guided by memory association and curiosity. Given an empirical tuple sampled from a replay buffer, the loss function is defined as: ; in, It is a balance coefficient that controls the influence of memory guidance.

[0015] Therefore, this invention employs the aforementioned joint model inference method based on model inference timeliness optimization. By modeling the request-model relationship as a hypergraph and applying hypergraph partitioning techniques, it solves the problem of optimizing model deployment under dynamic and coupled request-model relationships. Through the k-way hypergraph partitioning method, it minimizes the communication cost between edge servers while ensuring balanced model deployment. A two-stage model redeployment mechanism is proposed to address the adaptability of model deployment in the face of dynamically changing requests and model relationships. By combining incremental redistribution and global redeployment, the model deployment location is dynamically adjusted to cope with various challenges. The changing requirements not only reduce communication overhead but also adapt to the dynamic changes in requests and model needs, improving the timeliness of the edge cloud system when handling large-scale LLM inference tasks. A curiosity-guided memory association learning method is introduced for model invocation decisions, significantly improving the accuracy of model invocation by enhancing sample efficiency and the ability to explore unexplored states. Within the framework of deep reinforcement learning, a high-value experience propagation mechanism is incorporated, enabling the system to make more accurate invocation decisions when facing complex model dependencies. This not only accelerates the learning process but also improves the system's generalization ability, thereby enhancing the real-time response capability of the edge cloud system.

[0016] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0017] Figure 1 This is a framework diagram of an edge cloud system according to an embodiment of the present invention; Figure 2 This is a graphical illustration of the AoMI according to an embodiment of the present invention; Figure 3 This is a framework diagram of the model call optimization scheme according to an embodiment of the present invention; Figure 4 This is a performance comparison of the embodiments of the present invention with other model deployment schemes; Figure 5 This is a performance comparison of the embodiments of the present invention with other model invocation schemes; Figure 6 This is a comparison of the inference timeliness of AoMI and latency-optimized in this invention embodiment; Figure 7 This refers to the performance of the heuristic strategy guided by AoMI under burst conditions in this embodiment of the invention. Detailed Implementation

[0018] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0019] Unless otherwise defined, the technical or scientific terms used in this invention shall have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0020] Example 1 This invention provides a joint model inference method based on model inference timeliness optimization, comprising the following steps: S1. Constructing the Edge Cloud System Model and AoMI Quantization Model: Define the set of LLM inference requests, models, and edge servers; construct a Directed Acyclic Graph (DAG) for model invocation for each request type; and integrate time penalty and inference latency penalty to define the inference timeliness metric, AoMI. The edge cloud system framework is as follows: Figure 1 As shown.

[0021] LLM inference requests contain L different types, denoted as L(L, L) ,in A collection of request types; each request Each requires calling several models to generate a complete response; the collection of all available models. ,in Indicates the model type.

[0022] Each request type has a DAG (Directed Acyclic Graph) relationship with the required model, denoted as […]. ;in, This represents the set of models involved in the request, with each vertex... Corresponding to a model deployed in an edge cloud system, for any two vertices and If execution dependency edges exist ,but exist Each model can only be invoked after execution is complete, and the execution of each model cannot be interrupted; these models are deployed on... On a set of edge servers, the set is denoted as .

[0023] The AoMI metric measures the timeliness of edge cloud systems. The construction of the AoMI model includes the following steps: S11, Define Time t The AoMI is: ;in, This represents the time penalty function, which measures the interval between the current time and the generation time of the most recently completed inference request; The LLM inference penalty function is used to capture whether the completion delay of the latest completed inference request is within its maximum acceptable delay range. Unlike traditional latency metrics that only measure request completion time, AoMI comprehensively evaluates the timeliness of the entire LLM inference process in the edge cloud system. S12, Time Penalty Function Reflecting the increase in time penalty, it is modeled as a linear time function to quantify the decline in timeliness, defined as: ; in, The generation time of the most recently completed inference request is represented by the equation. Get the index of the latest completed request. , Indicates a request The completion time; the time penalty function is designed to penalize the elapsed time since the last completed inference request, thereby encouraging the system to complete the next LLM inference request as quickly as possible; S13. Introduce an LLM inference penalty function to quantify the decrease in timeliness caused by inference expiration. The LLM inference penalty function is: ; in , This indicates a delay in the completion of the latest completed inference request. This is the maximum acceptable inference delay for the request.

[0024] Figure 2 This demonstrates how AoMI varies over time in an edge cloud system, with each request having a specific generation time and a maximum acceptable inference latency. The completion time of a request is determined by the system's decisions regarding model deployment and invocation. For example, suppose the latest completed inference request A is generated in 1 second and completed in 2.5 seconds, which is within the acceptable inference latency range. After a few seconds, before any other inference request completes, the LLM inference penalty function is defined as follows: The corresponding AoMl is calculated as follows: This represents the elapsed time since the system last completed the inference request. When the edge cloud system completes the request in 3.5 seconds... However, if the delay exceeds its maximum acceptable delay of 0.5 seconds, it indicates that the system has failed to complete the inference in time, and therefore will trigger a factor. Applying an LLM inference penalty accelerates the increase in AoMI, denoted as: When the system completes the inference for request C within 4 seconds, and this is within its acceptable inference latency, the inference is considered timely, and the LLM inference penalty function returns 0. AoMI is .

[0025] In this way, AoMI quantifies the inference timeliness of the system by merging time and LLM inference penalties, encouraging the system to adjust its optimization strategy to maintain the timeliness of distributed joint model inference. LLM inference requests involve joint inference across multiple edge servers. Due to the sequential dependencies between models, a model cannot be invoked until all preceding models have completed execution. Therefore, the model inference completion time is defined as: ; in, Representation Model The earliest time it becomes callable. This is the set of preceding models for this model. It transmits the output of the preceding model to the model. Time required; The completion time of the preceding model for this model; For model inference time; Model inference time From the model Computing requirements and edge servers The ratio between the computing power of the two is determined by the ratio of their respective computing capabilities. ,in It is a model The computational load, It is an edge server On the computing power; request The completion delay is defined as: ; in, It is a request A set of models that have no subsequent dependencies. and These are requests The generation time and completion time.

[0026] The specific definition of the optimization problem in this embodiment is as follows: A lower AoMI value indicates that the federated model inference can be executed in a timely manner, reflecting the timeliness when serving LLM inference requests. Inference timeliness is jointly influenced by model deployment and invocation decisions, which determine the output transmission time between models and the model inference time.

[0027] Model Deployment (MD): Determines which models will be hosted on edge servers, denoted as... Model deployment status A value of 1 indicates the model Deployed on the server A value of 0 indicates that no deployment has been made.

[0028] Model Invocation (MI): Selects an already deployed model service LLM inference request, denoted as: Request model invocation scheme A value of 1 indicates the model Called to handle requests A value of 0 indicates that no call was made.

[0029] Therefore, to ensure timely and effective inference in edge cloud systems, the goal is to minimize AoMI. Considering that optimizing across the entire timeframe might diminish the impact of recent system decisions, the timeline is divided into non-overlapping time windows. And optimize independently within each window. The optimization problem can be expressed as: ; Constraints: Load balancing constraints limit the total size of the model on each edge server. This ensures that the imbalance remains within the tolerance range of the average value. This imbalance tolerance is determined by the parameter... control, Force a fully uniform deployment at that time.

[0030] : Ensure that the model can only be invoked after all preceding models have completed execution.

[0031] Ensure that the model is invoked after the inference of its preceding requests has been completed.

[0032] In each time window Within it, AoMI is a piecewise function that performs operations at each request completion time. There are discrete jump points. That is, in each interval... Inside, It remains unchanged and equals Due to the LLM inference penalty function Depending on whether the completion delay of the most recently completed inference request is within the maximum acceptable delay, its value remains constant within the interval. Therefore, the time domain is divided into unimportant intervals. And will optimize the problem Rewritten as: ; in, It is time reference mouth The set of requests within. Since LLM infers that request arrival time depends on user behavior, the request generation time... These are known constants. Minimizing AoMI is equivalent to minimizing and maintain a delay in request completion. Within its maximum acceptable delay.

[0033] S2. Dynamic model deployment and two-stage redeployment based on hypergraph partitioning: The request-model relationship is modeled as a hypergraph, and the initial model deployment is optimized by k-way hypergraph partitioning; a two-stage model redeployment mechanism of incremental repartitioning and global triggering redeployment is designed to dynamically adapt to request changes.

[0034] Hypergraphs allow each hyperedge to connect any number of nodes. By modeling a request as a hyperedge connecting all the required nodes, a hypergraph is defined. Contains a finite set of non-empty vertices and a set of super edges Each of the super edges It is a vertex set A non-empty subset of , the union of all hyperedges covers the entire vertex set, that is: ; In model deployment, a request-model hypergraph is built based on the model requirements of the request. Each vertex Corresponding to a deployment model Each super edge Represents an LLM inference request Connect all models that participated in executing the request; Because request inference involves multiple models, which are typically deployed on different servers, defining binary variables... To indicate a request Do we need to use a server? ,Right now: ; in, This determines where the model is hosted on an edge server, where Model deployment status A value of 1 indicates the model Deployed on the server Above, a value of 0 indicates that no deployment has occurred; This represents the deployment model for serving LLM inference requests. , A value of 1 indicates the model Called to handle requests A value of 0 indicates that no call was made; Indicates a request A server is required. Assuming there is no communication cost within the same server and the communication cost is the same between two different servers, it is expressed as follows: ; To reduce communication overhead, minimize the number of servers required, as inter-server data transfer typically accounts for a major portion of communication costs. Define the request. The cost of cross-server communication is: ; in, Indicates a request On the server and The amount of data transferred between them; , Each represents a request Do we need to use a server? and Since the model used for inference depends on the output of its predecessor model, the model... The time to receive results from its predecessor model is defined as: ; in, Indicates a request The amount of data transferred from the model.

[0035] In the request-model hypergraph, requests are modeled as hyperedges connecting their required models. To reduce communication overhead, the k-way hypergraph partitioning method is applied to the constructed hypergraph to partition the model set. Divided into The plan is to divide the network into two unimportant blocks, namely servers, to minimize the connectivity metric. SuperMap k-way partitioning It is its vertex set Divided into A mutually exclusive subset, denoted as Each vertex in the hypergraph Only a subset of servers are allocated to each model, and each subset corresponds to a server. The goal of the model deployment (MD) problem is to place the model on the servers, minimizing cross-server communication for joint model inference. ; in, It is a load balancing constraint that limits the total size of the model on each edge server. This ensures that the imbalance remains within the tolerance range of the average value; the tolerance for this imbalance is determined by the parameter... control, Force a fully uniform deployment at that time; Indicates the popularity of the request, reflecting the frequency of its occurrence. C4 avoids double-counting communication costs.

[0036] The specific steps for redeploying the two-stage model are as follows: Incremental Repartitioning: To adapt to the evolving distribution of LLM inference request types while avoiding excessive overhead from global redeployment, an incremental repartitioning is introduced. This repartitioning incrementally adjusts model deployment based on request patterns and dependency structures: First, high-frequency request types are identified, and incremental model migration scores are calculated; second, after identifying models with high migration values, suitable servers for these models to be migrated are determined, with the principle of incremental model deployment to minimize inter-server communication costs during federated inference; before migration, the models are checked to ensure they meet the requirements of reducing access span and satisfying server balance constraints. Trigger global redeployment: When the migration of models exceeds a predefined threshold, the model deployment on all servers is re-optimized. This phase is based on a k-way hypergraph partition and aims to generate a new deployment scenario X′ that jointly minimizes inter-server communication and model migration costs from previous deployments.

[0037] S3. Model Recall Decision Based on Curiosity-Driven Memory Association Learning: The model is dynamically optimized as a Markov Decision Process (MDP), and a burst request controller is set up to handle request overload. High-value experiences are propagated through memory association, combined with the curiosity mechanism of state novelty calculation, to optimize the recall decision, such as... Figure 3 As shown.

[0038] The Markov Decision Process (MDP) specifically includes the following steps: S31, at time step At that time, the edge cloud system observed the state. include: The model calls to the DAG for all requests are denoted as follows: ,in Model call relationships for each request type; model deployment status. ; most recent request completion time The remaining time required for the model to complete the current inference. and the model execution status of the currently ongoing request. ,in It is a request The model execution state set is defined. ,in Indicates the execution status of the model. This indicates that the reasoning has been completed. This indicates that the reasoning is not yet complete; S32, Motion Space Includes model invocation decisions, The representative request-model invocation scheme is defined as follows: , indicating a request Should the model be called? To reason; S33. The reward function is based on the AoMI minimization design, and the calculation formula is as follows: ; S34. The burst request controller uses an AoMI-based heuristic strategy to select a request that is expected to minimize the AoMI increment; the MDP uses Deep Reinforcement Learning (DRL) for model invocation, and when the request arrival rate exceeds a safety threshold... At that time, an emergency request was made for the controller to take over the call process; S35, State-Action Value Function Estimate based on strategy The cumulative reward is generated through a deep neural network. To approximate, during training, the predicted values ​​are aligned with the target values ​​derived from observed rewards and estimated future rewards; during training, experience... Sampled from the playback buffer and updated via the following targets: ; in, These are parameters of the target network that are updated periodically. It is a discount factor; the DRL path is trained solely based on its own exploration, and the burst request controller serves as a backup mechanism, which is only activated under burst conditions and does not interfere with the learning process.

[0039] Memory association learning demonstrates associative reasoning by linking relevant experiences to adapt to new but similar situations, introducing a relational state structure to capture associations from observed trajectories; Specifically, state relation structure as follows: ; relation It means state It is a state Given a state relation structure, the successor in the sampled trajectory under a certain action is... Value propagation mechanism is defined as This propagation relationship allows highly rewarding experiences to spread between relevant states.

[0040] Curiosity-driven associative learning of memories includes: Memory association: Using high-value estimation to supervise value functions Updates will be provided, and exploration of unexplored areas will be encouraged; Curiosity mechanism: calculating the novelty score of state embeddings ,in It is a state The nearest neighbor in the embedding space; the intrinsic reward calculation formula is: ,in The weighting of curiosity is controlled; the total reward, combining extrinsic and intrinsic rewards, is expressed as: ; Setting the loss function: The value function is updated under the learning objective of time difference, guided by memory association and curiosity. Given an empirical tuple sampled from a replay buffer, the loss function is defined as: ; in, It is a balance coefficient that controls the influence of memory guidance.

[0041] The technical advantages of this embodiment are illustrated below with specific data.

[0042] Real-world LLM inference request data is used, with each request involving 2 to 8 models for joint inference. The arrival patterns of LLM inference requests are derived from real-world data from an edge cloud inference service provider, capturing fluctuations and sudden increases in request demand. Each optimization window contains 100 requests. The distributed inference environment consists of 10 edge servers with heterogeneous computing capabilities, and the communication costs between the servers vary.

[0043] like Figure 4 As shown in (a), this embodiment introduces fast convergence and maintains the highest reward compared to the baseline method. This improvement is attributed to the continuous adaptation of model deployment through hypergraph partitioning to address dynamically changing LLM inference requests. In contrast, the InitHP method deploys only based on the initial request-model relationship, lacking further updates and failing to keep pace with the dynamic changes in real-world requests. In the comparison of Makespan and total Flowtime (…), Figure 4(b) This method reduces task completion time by 19.29%, 24.49%, 42.97%, and 21.48% and traffic time by 16.26%, 18.17%, 41.82%, and 14.16% respectively compared to the baseline methods i) to iv). Through hypergraph-based partitioning, this method reduces communication between edge servers during joint model inference, especially under complex request-model relationships.

[0044] In terms of performance comparison in model invocation, this method is compared with four benchmark methods, such as... Figure 5 As shown in (a), our method converges faster than GE-DQN and improves rewards by 15.9%. This is attributed to the ability to propagate high-reward experiences across relevant states, thereby enhancing sample efficiency and accelerating the learning process. Figure 5 In (b), the present invention consistently maintains a high request completion rate. Furthermore, in Figure 5 In (c) and (d), the present invention reduces completion time by an average of 68.98% and process time by 69.66% compared to the baseline. Even compared to HEFT, a time-efficiency-focused approach, the present invention achieves a 40.80% and 38.29% reduction in maximum completion time and process time, respectively, due to simultaneously considering request inference latency and inter-server communication overhead.

[0045] To evaluate inference timeliness, AoMI and request completion latency were used as optimization targets. The latter measures the total inference time for service requests and is a widely used metric in edge cloud systems. Figure 6 AoMI and latency optimization were compared in terms of inference timeliness. Figure 6 (a) presents the AoMI curve, which reflects the timeliness of model inference over time. Delayed optimization strategies often lead to delayed request inference, resulting in an increased slope. Furthermore, the AoMI curve shows a higher jump point value when the system completes a long-pending request. Figure 6 (b) shows that, under the AoMI target, a larger proportion of requests are completed within shorter time intervals. Figure 6 (c) Further illustrates the difference in arrival request completion rate between AoMI and latency optimization, showing that AoMI as an optimization metric enables the system to serve a higher percentage of arrival requests. Figure 6 (d) shows that, under the AoMI objective, more requests are completed in a timely manner within the maximum acceptable delay, representing a 17.1% improvement in timely inference. These results collectively highlight the effectiveness of AoMI as an optimization objective for improving the timeliness of joint model inference.

[0046] Further evaluate AoMI-guided heuristic strategies under contingency conditions. Figure 7In (a), arrival requests fluctuate during the burst perception controller's time period, prioritizing requests that cause the minimum increase in AoMI. The baseline policy selects requests with the shortest immediate inference latency. Figure 7 (b) shows that the AoMI-guided heuristic strategy achieved a high on-time completion rate of 14.3%, demonstrating its benefits in terms of reasoning timeliness under contingent conditions.

[0047] Therefore, this invention employs the aforementioned joint model inference method based on model inference timeliness optimization. By modeling the request-model relationship as a hypergraph and applying hypergraph partitioning techniques, it solves the problem of optimizing model deployment under dynamic and coupled request-model relationships. Through the k-way hypergraph partitioning method, it minimizes the communication cost between edge servers while ensuring balanced model deployment. A two-stage model redeployment mechanism is proposed to address the adaptability of model deployment in the face of dynamically changing requests and model relationships. By combining incremental redistribution and global redeployment, the model deployment location is dynamically adjusted to cope with various challenges. The changing requirements not only reduce communication overhead but also adapt to the dynamic changes in requests and model needs, improving the timeliness of the edge cloud system when handling large-scale LLM inference tasks. A curiosity-guided memory association learning method is introduced for model invocation decisions, significantly improving the accuracy of model invocation by enhancing sample efficiency and the ability to explore unexplored states. Within the framework of deep reinforcement learning, a high-value experience propagation mechanism is incorporated, enabling the system to make more accurate invocation decisions when facing complex model dependencies. This not only accelerates the learning process but also improves the system's generalization ability, thereby enhancing the real-time response capability of the edge cloud system.

[0048] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A joint model inference method based on optimization of model inference timeliness, characterized in that, Includes the following steps: S1. Constructing the edge cloud system model and AoMI quantification model: Define the set of LLM inference requests, models, and edge servers, construct a directed acyclic graph (DAG) for each request type, and integrate time penalty and inference latency penalty to define the inference timeliness index (AoMI). The AoMI metric measures the timeliness of edge cloud systems. The construction of the AoMI model includes the following steps: S11, Define Time t The AoMI is: ;in, This represents the time penalty function, which measures the interval between the current time and the generation time of the most recently completed inference request; The LLM inference penalty function is used to capture whether the completion delay of the latest completed inference request is within its maximum acceptable delay range. Unlike traditional latency metrics that only measure request completion time, AoMI comprehensively evaluates the timeliness of the entire LLM inference process in the edge cloud system. S12, Time Penalty Function Reflecting the increase in time penalty, it is modeled as a linear time function to quantify the decline in timeliness, defined as: ； in, The generation time of the most recently completed inference request is represented by the equation. Get the index of the latest completed request. , Indicates a request The completion time; the time penalty function is designed to penalize the elapsed time since the last completed inference request, thereby encouraging the system to complete the next LLM inference request as quickly as possible; S13. Introduce an LLM inference penalty function to quantify the decrease in timeliness caused by inference expiration. The LLM inference penalty function is: ； in , This indicates a delay in the completion of the latest completed inference request. This is the maximum acceptable inference delay for the request; LLM inference requests involve joint inference across multiple edge servers. Due to the sequential dependencies between models, a model cannot be invoked until all preceding models have completed execution. Therefore, the model inference completion time is defined as: ； in, Representation Model The earliest time it becomes callable. This is the set of preceding models for this model. It transmits the output of the preceding model to the model. Time required; The completion time of the preceding model for this model; For model inference time; Model inference time From the model Computing requirements and edge servers The ratio between the computing power of the two is determined by, that is ,in It is a model The computational load, It is an edge server On the computing power; request The completion delay is defined as: ； in, It is a request A set of models that have no subsequent dependencies. and These are requests The generation time and completion time; The timeliness of inference is jointly affected by model deployment and invocation decisions. An optimization problem model is defined, as follows: Model Deployment (MD): Determines which models will be hosted on edge servers, denoted as... Model deployment status A value of 1 indicates the model Deployed on the server Above, a value of 0 indicates that no deployment has occurred; Model Invocation (MI): Selects an already deployed model service LLM inference request, denoted as: Request model invocation scheme A value of 1 indicates the model Called to handle requests A value of 0 indicates that no call was made; Divide the timeline into non-overlapping time windows. And optimize independently within each window. The optimization problem can be expressed as: ； Constraints: Load balancing constraints limit the total size of the model on each edge server. This ensures that the imbalance remains within the tolerance range of the average value; the tolerance is determined by the parameter... control, Forced full uniform deployment; : Ensure that the model can only be invoked after all preceding models have completed execution; Ensure that the model is invoked after the inference of its preceding requests has been completed; In each time window Internally, AoMI is a piecewise function that performs operations at each request completion time. There are discrete jump points, that is, in each interval Inside, It remains unchanged and equals Due to the LLM inference penalty function Depending on whether the completion delay of the most recently completed inference request is within the maximum acceptable delay, its value remains constant within the interval, thus dividing the time domain into unimportant intervals. And will optimize the problem Rewritten as: ； in, It is time reference mouth The request set within, request generation time Given a constant, minimizing AoMI is equivalent to minimizing and maintain a delay in request completion. Within its maximum acceptable delay; S2. Dynamic Model Deployment and Two-Phase Redeployment Based on Hypergraph Partitioning: The request-model relationship is modeled as a hypergraph, and k-way hypergraph partitioning is used to optimize the initial model deployment; a two-phase model redeployment mechanism of incremental repartitioning and global triggering redeployment is designed to dynamically adapt to request changes; Hypergraphs allow each hyperedge to connect any number of nodes. By modeling a request as a hyperedge connecting all the required nodes, a hypergraph is defined. Contains a finite set of non-empty vertices and a set of super edges Each of the super edges It is a vertex set A non-empty subset of , the union of all hyperedges covers the entire vertex set, that is: ； In model deployment, a request-model hypergraph is built based on the model requirements of the request. Each vertex Corresponding to a deployment model Each super edge Represents an LLM inference request Connect all models that participated in executing the request; Because request inference involves multiple models deployed on different servers, binary variables are defined. To indicate a request Do we need to use a server? ,Right now: ； in, This determines where the model is hosted on an edge server, where Model deployment status A value of 1 indicates the model Deployed on the server Above, a value of 0 indicates that no deployment has occurred; This represents the deployment model for serving LLM inference requests. , A value of 1 indicates the model Called to handle requests A value of 0 indicates that no call was made; Indicates a request A server is required. Assuming there is no communication cost within the same server and the communication cost is the same between two different servers, it is expressed as follows: ; To reduce communication overhead and minimize the number of servers required, define the request. The cost of cross-server communication is: ； in, Indicates a request On the server and The amount of data transferred between them; , Each represents a request Do we need to use a server? and Since the model used for inference depends on the output of its predecessor model, the model... The time to receive results from its predecessor model is defined as: ； in, Indicates a request The amount of data transferred from the model; In the request-model hypergraph, requests are modeled as hyperedges connecting their required models. To reduce communication overhead, the k-way hypergraph partitioning method is applied to the constructed hypergraph to partition the model set. Divided into The plan is to divide the network into unimportant blocks, i.e., servers, to minimize the connectivity metric. SuperMap k-way partitioning It is its vertex set Divided into A mutually exclusive subset, denoted as Each vertex in the hypergraph Only a subset of servers are allocated to each model, and each subset corresponds to a server. The goal of the model deployment (MD) problem is to place the model on the servers, minimizing cross-server communication for joint model inference. in, It is a load balancing constraint that limits the total size of the model on each edge server. This ensures that the imbalance remains within the tolerance range of the average value; the tolerance is determined by the parameter... control, Forced full uniform deployment; Indicates the popularity of the request, reflecting the frequency of its occurrence; C4 avoids double-counting communication costs. The specific steps for redeploying the two-stage model are as follows: Incremental Repartitioning: To adapt to the evolving distribution of LLM inference request types while avoiding excessive overhead from global redeployment, an incremental repartitioning is introduced. This repartitioning incrementally adjusts model deployment based on request patterns and dependency structures: First, high-frequency request types are identified, and incremental model migration scores are calculated; second, after identifying models with high migration values, suitable servers for these models to be migrated are determined, with the principle of incremental model deployment to minimize inter-server communication costs during federated inference; before migration, the models are checked to ensure they meet the requirements of reducing access span and satisfying server balance constraints. Trigger global redeployment: When the migration of models exceeds a predefined threshold, the model deployment on all servers is re-optimized. This phase is based on the k-way hypergraph partition and aims to generate a new deployment scenario X′ that jointly minimizes inter-server communication and model migration costs from previous deployments. S3. Model call decision based on curiosity-guided memory association learning: The model is dynamically optimized as a Markov decision process (MDP), and a burst request controller is set up to deal with request overload; high-value experience is propagated through memory association, and the call decision is optimized by combining the curiosity mechanism of state novelty calculation. Memory association learning demonstrates associative reasoning by linking relevant experiences to adapt to new but similar situations, introducing a relational state structure to capture associations from observed trajectories; Specifically, state relation structure as follows: ； relation Representing state It is a state Given a state relation structure, the successor in the sampled trajectory under a certain action is... Value propagation mechanism is defined as This propagation relationship allows highly rewarding experiences to spread between relevant states; Curiosity-driven associative learning of memories includes: Memory association: Using high-value estimation to supervise value functions Updates will be provided, and exploration of unexplored areas will be encouraged; Curiosity mechanism: calculating the novelty score of state embeddings ,in It is a state The nearest neighbor in the embedding space; the intrinsic reward calculation formula is: ,in The weighting of curiosity is controlled; the total reward, combining extrinsic and intrinsic rewards, is expressed as: ; Setting the loss function: The value function is updated under the learning objective of time difference, guided by memory association and curiosity. Given an empirical tuple sampled from a replay buffer, the loss function is defined as: ； in, It is a balance coefficient that controls the influence of memory guidance.

2. The joint model inference method based on model inference timeliness optimization according to claim 1, characterized in that, In S1, LLM inference requests contain L different types, denoted as L(L1, L2, L3, L4, L5, L6, L7, L8, L9, L10, L11, L12, L13, L14, L15, L16, L17, L18, L19, L10, L11, L12, L13, L14, L16, L17, L18, L19, L12, ,in A collection of request types; each request Each requires calling several models to generate a complete response; the collection of all available models. ,in Indicates the model type.

3. The joint model inference method based on model inference timeliness optimization according to claim 2, characterized in that, In S1, the DAG relationship between each request type and the required model is denoted as... ;in, This represents the set of models involved in the request, with each vertex... Corresponding to a model deployed in an edge cloud system, for any two vertices and If execution dependency edges exist ,but exist Each model can only be invoked after execution is complete, and the execution of each model cannot be interrupted; these models are deployed on... On a set of edge servers, the set is denoted as .

4. The joint model inference method based on model inference timeliness optimization according to claim 3, characterized in that, In S3, the Markov Decision Process (MDP) specifically includes the following steps: S31, at time step At that time, the edge cloud system observed the state. include: The model calls to the DAG for all requests are denoted as follows: ,in Model call relationships for each request type; model deployment status. ; most recent request completion time The remaining time required for the model to complete the current inference. and the model execution status of the currently ongoing request. ,in It is a request The model execution state set is defined. ,in Indicates the execution status of the model. This indicates that the reasoning has been completed. This indicates that the reasoning is not yet complete; S32, Motion Space Includes model invocation decisions, The representative request-model invocation scheme is defined as follows: , indicating a request Should the model be called? To reason; S33. The reward function is based on the AoMI minimization design, and the calculation formula is as follows: ； S34. The burst request controller uses an AoMI-based heuristic strategy to select a request that is expected to minimize the AoMI increment; the MDP uses Deep Reinforcement Learning (DRL) for model invocation, and when the request arrival rate exceeds a safety threshold... At that time, an emergency request was made for the controller to take over the call process; S35, State-Action Value Function Estimate based on strategy The cumulative reward is generated through a deep neural network. To approximate, during training, the predicted values ​​are aligned with the target values ​​derived from observed rewards and estimated future rewards; during training, experience... Sampled from the playback buffer and updated via the following targets: ； in, These are parameters of the target network that are updated periodically. It is a discount factor; the DRL path is trained solely based on its own exploration, and the burst request controller serves as a backup mechanism, only activating under burst conditions and not interfering with the learning process.

Images