Joint model reasoning method based on model reasoning timeliness optimization

Abstract

The invention discloses a joint model reasoning method based on model reasoning timeliness optimization, and belongs to the technical field of deep learning, and the method comprises the steps: defining a set of LLM reasoning requests, models and edge servers, constructing a model for each request type, and calling a directed acyclic graph (DAG); a reasoning timeliness index AoMI is defined; the request-model relation is modeled into a hypergraph, and k-way hypergraph division is adopted to optimize initial model deployment; designing a two-stage model redeployment mechanism of progressive increment repartitioning and global triggering redeployment, and dynamically adapting to request changes; and dynamically optimizing the model to construct a Markov decision process MDP, and optimizing a call decision by combining a curiosity-guided memory association learning method. According to the method, performance indexes are provided for reasoning optimization through AoMI; a complex request model relation is dynamically coped with through a hypergraph segmentation technology and a two-stage model deployment mechanism, and the communication and migration cost is optimized; and the model calling efficiency is improved through a curiosity-based learning method.

Description

Technical Field

[0001] This invention relates to the field of deep learning technology, and in particular to a joint model inference method based on the optimization of model inference timeliness. Background Technology

[0002] In recent years, deep learning technologies, especially large language models (LLMs), have demonstrated outstanding performance in various generative tasks. However, LLM inference typically relies on centralized cloud computing, which suffers from high latency and low responsiveness. With the rise of edge computing, deploying the models required for LLM inference to distributed edge cloud systems has become a key approach to solving these problems. By deploying model inference services closer to users, distributed edge cloud systems can significantly reduce latency and improve response speed.

[0003] In distributed edge cloud systems, LLM inference requests typically involve multiple interdependent models. That is, the inference of a particular model requires the outputs of preceding models, and the relationships between these models are highly dynamic. Different types of models may be deployed on different edge servers. Summary of the Invention

[0004] The purpose of this invention is to provide a joint model inference method based on the optimization of model inference timeliness, so as to ensure that inference tasks can be completed efficiently and in a timely manner when multiple models are interdependent.

[0005] To achieve the above objectives, this invention provides a joint model inference method based on model inference timeliness optimization, comprising the following steps: S1. Constructing the edge cloud system model and AoMI quantification model: Define the set of LLM inference requests, models, and edge servers, construct a directed acyclic graph (DAG) for each request type, and integrate time penalty and inference latency penalty to define the inference timeliness index (AoMI). S2. Dynamic Model Deployment and Two-Phase Redeployment Based on Hypergraph Partitioning: The request-model relationship is modeled as a hypergraph, and k-way hypergraph partitioning is used to optimize the initial model deployment; a two-phase model redeployment mechanism of incremental repartitioning and global triggering redeployment is designed to dynamically adapt to request changes; S3. Model call decision based on curiosity-guided memory association learning: The model is dynamically optimized as a Markov decision process (MDP), and a burst request controller is set up to deal with request overload; high-value experience is propagated through memory association, and the curiosity mechanism of state novelty calculation is combined to optimize the call decision.

[0006] Preferably, in S1, the LLM inference request includes L different types, denoted as L(L, L, L) ,in A collection of request types; each request Each requires calling several models to generate a complete response; the collection of all available models. ,in Indicates the model type.

[0007] Preferably, in S1, the DAG relationship between each request type and the required model is denoted as... ;in, This represents the set of models involved in the request, with each vertex... Corresponding to a model deployed in an edge cloud system, for any two vertices and If execution dependency edges exist ,but exist Each model can only be invoked after execution is complete, and the execution of each model cannot be interrupted; these models are deployed on... On a set of edge servers, the set is denoted as .

[0008] Preferably, in S1, the AoMI metric measures the timeliness of the edge cloud system. The AoMI model construction specifically includes the following steps: S11, Define Time t The AoMI is: ;in, This represents the time penalty function, which measures the interval between the current time and the generation time of the most recently completed inference request; The LLM inference penalty function is used to capture whether the completion delay of the latest completed inference request is within its maximum acceptable delay range. Unlike traditional latency metrics that only measure request completion time, AoMI comprehensively evaluates the timeliness of the entire LLM inference process in the edge cloud system. S12, Time Penalty Function Reflecting the increase in time penalty, it is modeled as a linear time function to quantify the decline in timeliness, defined as: ; in, The generation time of the most recently completed inference request is represented by the equation. Get the index of the latest completed request. , Indicates a request The completion time; the time penalty function is designed to penalize the elapsed time since the last completed inference request, thereby encouraging the system to complete the next LLM inference request as quickly as possible; S13. Introduce an LLM inference penalty function to quantify the decrease in timeliness caused by inference expiration. The LLM inference penalty function is: ; in , This indicates a delay in the completion of the latest completed inference request. This is the maximum acceptable inference delay for the request.

[0009] Preferably, LLM inference requests involve joint inference across multiple edge servers. Due to the sequential dependencies between models, a model cannot be invoked until all preceding models have completed execution. Therefore, the model inference completion time is defined as: ; in, Representation Model The earliest time it becomes callable. This is the set of preceding models for this model. It transmits the output of the preceding model to the model. Time required; The completion time of the preceding model for this model; For model inference time; Model inference time From the model Computing requirements and edge servers The ratio between the computing power of the two is determined by the ratio of their respective computing capabilities. ,in It is a model The computational load, It is an edge server On the computing power; request The completion delay is defined as: ; in, It is a request A set of models that have no subsequent dependencies. and These are requests The generation time and completion time.

[0010] Preferably, the timeliness of inference is jointly affected by model deployment and invocation decisions. An optimization problem model is defined, with the following details: Model Deployment (MD): Determines which models will be hosted on edge servers, denoted as... Model deployment status A value of 1 indicates that the model... Deployed on the server Above, a value of 0 indicates that no deployment has occurred; Model Invocation (MI): Selects an already deployed model service LLM inference request, denoted as: Request model invocation scheme A value of 1 indicates the model Called to handle requests A value of 0 indicates that no call was made; Divide the timeline into non-overlapping time windows. And optimize independently within each window. The optimization problem can be expressed as: ; Constraints: Load balancing constraints limit the total size of the model on each edge server. This ensures that the imbalance remains within the tolerance range of the average value; the tolerance for this imbalance is determined by the parameter... control, Force a fully uniform deployment at that time; : Ensure that the model can only be invoked after all preceding models have completed execution; Ensure that the model is invoked after the inference of its preceding requests has been completed; In each time window Internally, AoMI is a piecewise function that performs operations at each request completion time. There are discrete jump points, that is, in each interval Inside, It remains unchanged and equals Due to the LLM inference penalty function Depending on whether the completion delay of the most recently completed inference request is within the maximum acceptable delay, its value remains constant within the interval, thus dividing the time domain into unimportant intervals. And will optimize the problem Rewritten as: ; in, It is time reference mouth The collection of requests within, the request generation time Given a constant, minimizing AoMI is equivalent to minimizing and maintain a delay in request completion. Within its maximum acceptable delay.

[0011] Preferably, in S2, the hypergraph allows each hyperedge to connect any number of nodes, modeling the request as a hyperedge connecting all required models, thus defining the hypergraph. Contains a finite set of non-empty vertices and a set of super edges Each of the super edges It is a vertex set A non-empty subset of , the union of all hyperedges covers the entire vertex set, that is: ; In model deployment, a request-model hypergraph is built based on the model requirements of the request. Each vertex Corresponding to a deployment model Each super edge Represents an LLM inference request Connect all models that participated in executing the request; Because request inference involves multiple models, which are typically deployed on different servers, defining binary variables... To indicate a request Do we need to use a server? ,Right now: ; in, This determines where the model is hosted on an edge server, where Model deployment status A value of 1 indicates that the model... Deployed on the server Above, a value of 0 indicates that no deployment has occurred; This represents the deployment model for serving LLM inference requests. , A value of 1 indicates the model Called to handle requests A value of 0 indicates that no call was made; Indicates a request A server is required. Assuming there is no communication cost within the same server and the communication cost is the same between two different servers, it is expressed as follows: ; To reduce communication overhead and minimize the number of servers required, define the request. The cost of cross-server communication is: ; in, Indicates a request On the server and The amount of data transferred between them; , Each represents a request Do we need to use a server? and Since the model used for inference depends on the output of its predecessor model, the model... The time to receive results from its predecessor model is defined as: ; in, Indicates a request The amount of data transferred from the model; In the request-model hypergraph, requests are modeled as hyperedges connecting their required models. To reduce communication overhead, the k-way hypergraph partitioning method is applied to the constructed hypergraph to partition the model set. Divided into The plan is to divide the network into two unimportant blocks, namely servers, to minimize the connectivity metric. SuperMap k-way partitioning It is its vertex set Divided into A mutually exclusive subset, denoted as Each vertex in the hypergraph Only a subset of servers are allocated to each model, and each subset corresponds to a server. The goal of the model deployment (MD) problem is to place the model on the servers, minimizing cross-server communication for joint model inference. ; in, It is a load balancing constraint that limits the total size of the model on each edge server. This ensures that the imbalance remains within the tolerance range of the average value; the tolerance for this imbalance is determined by the parameter... control, Force a fully uniform deployment at that time; Indicates the popularity of the request, reflecting the frequency of its occurrence. C4 avoids double-counting communication costs.

[0012] Preferably, in S2, the specific method for redeploying the two-stage model is as follows: Incremental Repartitioning: To adapt to the evolving distribution of LLM inference request types while avoiding excessive overhead from global redeployment, an incremental repartitioning is introduced. This repartitioning incrementally adjusts model deployment based on request patterns and dependency structures: First, high-frequency request types are identified, and incremental model migration scores are calculated; second, after identifying models with high migration values, suitable servers for these models to be migrated are determined, with the principle of incremental model deployment to minimize inter-server communication costs during federated inference; before migration, the models are checked to ensure they meet the requirements of reducing access span and satisfying server balance constraints. Trigger global redeployment: When the migration of models exceeds a predefined threshold, the model deployment on all servers is re-optimized. This phase is based on a k-way hypergraph partition and aims to generate a new deployment scenario X′ that jointly minimizes inter-server communication and model migration costs from previous deployments.

[0013] Preferably, in S3, the Markov Decision Process (MDP) specifically includes the following steps: S31, at time step At that time, the edge cloud system observed the state. include: The model calls to the DAG for all requests are denoted as follows: ,in Model call relationships for each request type; model deployment status. ; most recent request completion time The remaining time required for the model to complete the current inference. and the model execution status of the currently ongoing request. ,in It is a request The model execution state set is defined. ,in Indicates the execution status of the model. This indicates that the reasoning has been completed. This indicates that the reasoning is not yet complete; S32, Motion Space Includes model invocation decisions, The representative request-model invocation scheme is defined as follows: , indicating a request Should the model be called? To reason; S33. The reward function is based on the AoMI minimization design, and the calculation formula is as follows: ; S34. The burst request controller uses an AoMI-based heuristic strategy to select a request that is expected to minimize the AoMI increment; the MDP uses Deep Reinforcement Learning (DRL) for model invocation, and when the request arrival rate exceeds a safety threshold... At that time, an emergency request was made for the controller to take over the call process; S35, State-Action Value Function Estimate based on strategy The cumulative reward is generated through a deep neural network. To approximate, during training, the predicted values ​​are aligned with the target values ​​derived from observed rewards and estimated future rewards; during training, experience... Sampled from the playback buffer and updated via the following targets: ; in, These are parameters of the target network that are updated periodically. It is a discount factor; the DRL strategy is trained solely based on its own exploration, and the burst request controller serves as a backup mechanism, which is only activated under burst conditions and does not interfere with the learning process.

[0014] Preferably, in S3, memory association learning demonstrates associative reasoning by linking relevant experiences to adapt to new but similar situations, introducing a relational state structure to capture associations from observed trajectories; Specifically, state relation structure as follows: ; relation It means state It is a state Given a state relation structure, the successor in the sampled trajectory under a certain action is... Value propagation mechanism is defined as This propagation relationship allows highly rewarding experiences to spread between relevant states; Curiosity-driven associative learning of memories includes: Memory association: Using high-value estimation to supervise value functions Updates will be provided, and exploration of unexplored areas will be encouraged; Curiosity mechanism: calculating the novelty score of state embeddings ,in It is a state The nearest neighbor in the embedding space; the intrinsic reward calculation formula is: ,in The weighting of curiosity is controlled; the total reward, combining extrinsic and intrinsic rewards, is expressed as: ; Setting the loss function: The value function is updated under the learning objective of time difference, guided by memory association and curiosity. Given an empirical tuple sampled from a replay buffer, the loss function is defined as: ; in, It is a balance coefficient that controls the influence of memory guidance.

[0015] Therefore, this invention employs the aforementioned joint model inference method based on model inference timeliness optimization. By modeling the request-model relationship as a hypergraph and applying hypergraph partitioning techniques, it solves the problem of optimizing model deployment under dynamic and coupled request-model relationships. Through the k-way hypergraph partitioning method, it minimizes the communication cost between edge servers while ensuring balanced model deployment. A two-stage model redeployment mechanism is proposed to address the adaptability of model deployment in the face of dynamically changing requests and model relationships. By combining incremental redistribution and global redeployment, the model deployment location is dynamically adjusted to cope with various challenges. The changing requirements not only reduce communication overhead but also adapt to the dynamic changes in requests and model needs, improving the timeliness of the edge cloud system when handling large-scale LLM inference tasks. A curiosity-guided memory association learning method is introduced for model invocation decisions, significantly improving the accuracy of model invocation by enhancing sample efficiency and the ability to explore unexplored states. Within the framework of deep reinforcement learning, a high-value experience propagation mechanism is incorporated, enabling the system to make more accurate invocation decisions when facing complex model dependencies. This not only accelerates the learning process but also improves the system's generalization ability, thereby enhancing the real-time response capability of the edge cloud system.

[0016] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0017] Figure 1 This is a framework diagram of an edge cloud system according to an embodiment of the present invention; Figure 2 This is a graphical illustration of the AoMI according to an embodiment of the present invention; Figure 3 This is a framework diagram of the model call optimization scheme according to an embodiment of the present invention; Figure 4 This is a performance comparison of the embodiments of the present invention with other model deployment schemes; Figure 5 This is a performance comparison of the embodiments of the present invention with other model invocation schemes; Figure 6 This is a comparison of the inference timeliness of AoMI and latency-optimized in this invention embodiment; Figure 7 This refers to the performance of the heuristic strategy guided by AoMI under burst conditions in this embodiment of the invention. Detailed Implementation

[0018] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0019] Unless otherwise defined, the technical or scientific terms used in this invention shall have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0020] Example 1 This invention provides a joint model inference method based on model inference timeliness optimization, comprising the following steps: S1. Constructing the Edge Cloud System Model and AoMI Quantization Model: Define the set of LLM inference requests, models, and edge servers; construct a Directed Acyclic Graph (DAG) for model invocation for each request type; and integrate time penalty and inference latency penalty to define the inference timeliness metric, AoMI. The edge cloud system framework is as follows: Figure 1 As shown.

[0021] LLM inference requests contain L different types, denoted as L(L, L) ,in A collection of request types; each request Each requires calling several models to generate a complete response; the collection of all available models. ,in Indicates the model type.

[0022] Each request type has a DAG (Directed Acyclic Graph) relationship with the required model, denoted as […]. ;in, This represents the set of models involved in the request, with each vertex... Corresponding to a model deployed in an edge cloud system, for any two vertices and If execution dependency edges exist ,but exist Each model can only be invoked after execution is complete, and the execution of each model cannot be interrupted; these models are deployed on... On a set of edge servers, the set is denoted as .

[0023] The AoMI metric measures the timeliness of edge cloud systems. The construction of the AoMI model includes the following steps: S11, Define Time t The AoMI is: ;in, This represents the time penalty function, which measures the interval between the current time and the generation time of the most recently completed inference request; The LLM inference penalty function is used to capture whether the completion delay of the latest completed inference request is within its maximum acceptable delay range. Unlike traditional latency metrics that only measure request completion time, AoMI comprehensively evaluates the timeliness of the entire LLM inference process in the edge cloud system. S12, Time Penalty Function Reflecting the increase in time penalty, it is modeled as a linear time function to quantify the decline in timeliness, defined as: ; in, The generation time of the most recently completed inference request is represented by the equation. Get the index of the latest completed request. , Indicates a request The completion time; the time penalty function is designed to penalize the elapsed time since the last completed inference request, thereby encouraging the system to complete the next LLM inference request as quickly as possible; S13. Introduce an LLM inference penalty function to quantify the decrease in timeliness caused by inference expiration. The LLM inference penalty function is: ; in , This indicates a delay in the completion of the latest completed inference request. This is the maximum acceptable inference delay for the request.

[0024] Figure 2 This demonstrates how AoMI varies over time in an edge cloud system, with each request having a specific generation time and a maximum acceptable inference latency. The completion time of a request is determined by the system's decisions regarding model deployment and invocation. For example, suppose the latest completed inference request A is generated in 1 second and completed in 2.5 seconds, which is within the acceptable inference latency range. After a few seconds, before any other inference request completes, the LLM inference penalty function is defined as follows: The corresponding AoMl is calculated as follows: This represents the elapsed time since the system last completed the inference request. When the edge cloud system completes the request in 3.5 seconds... However, if the delay exceeds its maximum acceptable delay of 0.5 seconds, it indicates that the system has failed to complete the inference in time, and therefore will trigger a factor. Applying an LLM inference penalty accelerates the increase in AoMI, denoted as: When the system completes the inference for request C within 4 seconds, and this is within its acceptable inference latency, the inference is considered timely, and the LLM inference penalty function returns 0. AoMI is .

[0025] In this way, AoMI quantifies the inference timeliness of the system by merging time and LLM inference penalties, encouraging the system to adjust its optimization strategy to maintain the timeliness of distributed joint model inference. LLM inference requests involve joint inference across multiple edge servers. Due to the sequential dependencies between models, a model cannot be invoked until all preceding models have completed execution. Therefore, the model inference completion time is defined as: ; in, Representation Model The earliest time it becomes callable. This is the set of preceding models for this model. It transmits the output of the preceding model to the model. Time required; The completion time of the preceding model for this model; For model inference time; Model inference time From the model Computing requirements and edge servers The ratio between the computing power of the two is determined by the ratio of their respective computing capabilities. ,in It is a model The computational load, It is an edge server On the computing power; request The completion delay is defined as: ; in, It is a request A set of models that have no subsequent dependencies. and These are requests The generation time and completion time.

[0026] The specific definition of the optimization problem in this embodiment is as follows: A lower AoMI value indicates that the federated model inference can be executed in a timely manner, reflecting the timeliness when serving LLM inference requests. Inference timeliness is jointly influenced by model deployment and invocation decisions, which determine the output transmission time between models and the model inference time.

[0027] Model Deployment (MD): Determines which models will be hosted on edge servers, denoted as... Model deployment status A value of 1 indicates that the model... Deployed on the server A value of 0 indicates that no deployment has been made.

[0028] Model Invocation (MI): Selects an already deployed model service LLM inference request, denoted as: Request model invocation scheme A value of 1 indicates the model Called to handle requests A value of 0 indicates that no call was made.

[0029] Therefore, to ensure timely and effective inference in edge cloud systems, the goal is to minimize AoMI. Considering that optimizing across the entire timeframe might diminish the impact of recent system decisions, the timeline is divided into non-overlapping time windows. And optimize independently within each window. The optimization problem can be expressed as: ; Constraints: Load balancing constraints limit the total size of the model on each edge server. This ensures that the imbalance remains within the tolerance range of the average value. This imbalance tolerance is determined by the parameter... control, Force a fully uniform deployment at that time.

[0030] : Ensure that the model can only be invoked after all preceding models have completed execution.

[0031] Ensure that the model is invoked after the inference of its preceding requests has been completed.

[0032] In each time window Internally, AoMI is a piecewise function that performs operations at each request completion time. There are discrete jump points. That is, in each interval... Inside, It remains unchanged and equals Due to the LLM inference penalty function Depending on whether the completion delay of the most recently completed inference request is within the maximum acceptable delay, its value remains constant within the interval. Therefore, the time domain is divided into unimportant intervals. And will optimize the problem Rewritten as: ; in, It is time reference mouth The set of requests within. Since LLM infers that request arrival time depends on user behavior, the request generation time... These are known constants. Minimizing AoMI is equivalent to minimizing and maintain a delay in request completion. Within its maximum acceptable delay.

[0033] S2. Dynamic model deployment and two-stage redeployment based on hypergraph partitioning: The request-model relationship is modeled as a hypergraph, and the initial model deployment is optimized by k-way hypergraph partitioning; a two-stage model redeployment mechanism of incremental repartitioning and global triggering redeployment is designed to dynamically adapt to request changes.

[0034] Hypergraphs allow each hyperedge to connect any number of nodes. By modeling a request as a hyperedge connecting all the required nodes, a hypergraph is defined. Contains a finite set of non-empty vertices and a set of super edges Each of the super edges It is a vertex set A non-empty subset of , the union of all hyperedges covers the entire vertex set, that is: ; In model deployment, a request-model hypergraph is built based on the model requirements of the request. Each vertex Corresponding to a deployment model Each super edge Represents an LLM inference request Connect all models that participated in executing the request; Because request inference involves multiple models, which are typically deployed on different servers, defining binary variables... To indicate a request Do we need to use a server? ,Right now: ; in, This determines where the model is hosted on an edge server, where Model deployment status A value of 1 indicates that the model... Deployed on the server Above, a value of 0 indicates that no deployment has occurred; This represents the deployment model for serving LLM inference requests. , A value of 1 indicates the model Called to handle requests A value of 0 indicates that no call was made; Indicates a request A server is required. Assuming there is no communication cost within the same server and the communication cost is the same between two different servers, it is expressed as follows: ; To reduce communication overhead, minimize the number of servers required, as inter-server data transfer typically accounts for a major portion of communication costs. Define the request. The cost of cross-server communication is: ; in, Indicates a request On the server and The amount of data transferred between them; , Each represents a request Do we need to use a server? and Since the model used for inference depends on the output of its predecessor model, the model... The time to receive results from its predecessor model is defined as: ; in, Indicates a request The amount of data transferred from the model.

[0035] In the request-model hypergraph, requests are modeled as hyperedges connecting their required models. To reduce communication overhead, the k-way hypergraph partitioning method is applied to the constructed hypergraph to partition the model set. Divided into The plan is to divide the network into two unimportant blocks, namely servers, to minimize the connectivity metric. SuperMap k-way partitioning It is its vertex set Divided into A mutually exclusive subset, denoted as Each vertex in the hypergraph Only a subset of servers are allocated to each model, and each subset corresponds to a server. The goal of the model deployment (MD) problem is to place the model on the servers, minimizing cross-server communication for joint model inference. ; in, It is a load balancing constraint that limits the total size of the model on each edge server. This ensures that the imbalance remains within the tolerance range of the average value; the tolerance for this imbalance is determined by the parameter... control, Force a fully uniform deployment at that time; Indicates the popularity of the request, reflecting the frequency of its occurrence. C4 avoids double-counting communication costs.

[0036] The specific steps for redeploying the two-stage model are as follows: Incremental Repartitioning: To adapt to the evolving distribution of LLM inference request types while avoiding excessive overhead from global redeployment, an incremental repartitioning is introduced. This repartitioning incrementally adjusts model deployment based on request patterns and dependency structures: First, high-frequency request types are identified, and incremental model migration scores are calculated; second, after identifying models with high migration values, suitable servers for these models to be migrated are determined, with the principle of incremental model deployment to minimize inter-server communication costs during federated inference; before migration, the models are checked to ensure they meet the requirements of reducing access span and satisfying server balance constraints. Trigger global redeployment: When the migration of models exceeds a predefined threshold, the model deployment on all servers is re-optimized. This phase is based on a k-way hypergraph partition and aims to generate a new deployment scenario X′ that jointly minimizes inter-server communication and model migration costs from previous deployments.

[0037] S3. Model Recall Decision Based on Curiosity-Driven Memory Association Learning: The model is dynamically optimized as a Markov Decision Process (MDP), and a burst request controller is set up to handle request overload. High-value experiences are propagated through memory association, combined with the curiosity mechanism of state novelty calculation, to optimize the recall decision, such as... Figure 3 As shown.

[0038] The Markov Decision Process (MDP) specifically includes the following steps: S31, at time step At that time, the edge cloud system observed the state. include: The model calls to the DAG for all requests are denoted as follows: ,in Model call relationships for each request type; model deployment status. ; most recent request completion time The remaining time required for the model to complete the current inference. and the model execution status of the currently ongoing request. ,in It is a request The model execution state set is defined. ,in Indicates the execution status of the model. This indicates that the reasoning has been completed. This indicates that the reasoning is not yet complete; S32, Motion Space Includes model invocation decisions, The representative request-model invocation scheme is defined as follows: , indicating a request Should the model be called? To reason; S33. The reward function is based on the AoMI minimization design, and the calculation formula is as follows: ; S34. The burst request controller uses an AoMI-based heuristic strategy to select a request that is expected to minimize the AoMI increment; the MDP uses Deep Reinforcement Learning (DRL) for model invocation, and when the request arrival rate exceeds a safety threshold... At that time, an emergency request was made for the controller to take over the call process; S35, State-Action Value Function Estimate based on strategy The cumulative reward is generated through a deep neural network. To approximate, during training, the predicted values ​​are aligned with the target values ​​derived from observed rewards and estimated future rewards; during training, experience... Sampled from the playback buffer and updated via the following targets: ; in, These are parameters of the target network that are updated periodically. It is a discount factor; the DRL strategy is trained solely based on its own exploration, and the burst request controller serves as a backup mechanism, which is only activated under burst conditions and does not interfere with the learning process.

[0039] Memory association learning demonstrates associative reasoning by linking relevant experiences to adapt to new but similar situations, introducing a relational state structure to capture associations from observed trajectories; Specifically, state relation structure as follows: ; relation It means state It is a state Given a state relation structure, the successor in the sampled trajectory under a certain action is... Value propagation mechanism is defined as This propagation relationship allows highly rewarding experiences to spread between relevant states.

[0040] Curiosity-driven associative learning of memories includes: Memory association: Using high-value estimation to supervise value functions Updates will be provided, and exploration of unexplored areas will be encouraged; Curiosity mechanism: calculating the novelty score of state embeddings ,in It is a state The nearest neighbor in the embedding space; the intrinsic reward calculation formula is: ,in The weighting of curiosity is controlled; the total reward, combining extrinsic and intrinsic rewards, is expressed as: ; Setting the loss function: The value function is updated under the learning objective of time difference, guided by memory association and curiosity. Given an empirical tuple sampled from a replay buffer, the loss function is defined as: ; in, It is a balance coefficient that controls the influence of memory guidance.

[0041] The technical advantages of this embodiment are illustrated below with specific data.

[0042] Real-world LLM inference request data is used, with each request involving 2 to 8 models for joint inference. The arrival patterns of LLM inference requests are derived from real-world data from an edge cloud inference service provider, capturing fluctuations and sudden increases in request demand. Each optimization window contains 100 requests. The distributed inference environment consists of 10 edge servers with heterogeneous computing capabilities, and the communication costs between the servers vary.

[0043] like Figure 4 As shown in (a), this embodiment introduces fast convergence and maintains the highest reward compared to the baseline method. This improvement is attributed to the continuous adaptation of model deployment through hypergraph partitioning to address dynamically changing LLM inference requests. In contrast, the InitHP method deploys only based on the initial request-model relationship, lacking further updates and failing to keep pace with the dynamic changes in real-world requests. In the comparison of Makespan and total Flowtime (…), Figure 4(b) This method reduces task completion time by 19.29%, 24.49%, 42.97%, and 21.48% and traffic time by 16.26%, 18.17%, 41.82%, and 14.16% respectively compared to the baseline methods i) to iv). Through hypergraph-based partitioning, this method reduces communication between edge servers during joint model inference, especially under complex request-model relationships.

[0044] In terms of performance comparison in model invocation, this method is compared with four benchmark methods, such as... Figure 5 As shown in (a), our method converges faster than GE-DQN and improves rewards by 15.9%. This is attributed to the ability to propagate high-reward experiences across relevant states, thereby enhancing sample efficiency and accelerating the learning process. Figure 5 In (b), the present invention consistently maintains a high request completion rate. Furthermore, in Figure 5 In (c) and (d), the present invention reduces completion time by an average of 68.98% and process time by 69.66% compared to the baseline. Even compared to HEFT, a time-efficiency-focused approach, the present invention achieves a 40.80% and 38.29% reduction in maximum completion time and process time, respectively, due to simultaneously considering request inference latency and inter-server communication overhead.

[0045] To evaluate inference timeliness, AoMI and request completion latency were used as optimization targets. The latter measures the total inference time for service requests and is a widely used metric in edge cloud systems. Figure 6 AoMI and latency optimization were compared in terms of inference timeliness. Figure 6 (a) presents the AoMI curve, which reflects the timeliness of model inference over time. Delayed optimization strategies often lead to delayed request inference, resulting in an increased slope. Furthermore, the AoMI curve shows a higher jump point value when the system completes a long-pending request. Figure 6 (b) shows that, under the AoMI target, a larger proportion of requests are completed within shorter time intervals. Figure 6 (c) Further illustrates the difference in arrival request completion rate between AoMI and latency optimization, showing that AoMI as an optimization metric enables the system to serve a higher percentage of arrival requests. Figure 6 (d) shows that, under the AoMI objective, more requests are completed in a timely manner within the maximum acceptable delay, representing a 17.1% improvement in timely inference. These results collectively highlight the effectiveness of AoMI as an optimization objective for improving the timeliness of joint model inference.

[0046] Further evaluate AoMI-guided heuristic strategies under contingency conditions. Figure 7In (a), arrival requests fluctuate during the burst perception controller's time period, prioritizing requests that cause the minimum increase in AoMI. The baseline policy selects requests with the shortest immediate inference latency. Figure 7 (b) shows that the AoMI-guided heuristic strategy achieved a high on-time completion rate of 14.3%, demonstrating its benefits in terms of reasoning timeliness under contingent conditions.

[0047] Therefore, this invention employs the aforementioned joint model inference method based on model inference timeliness optimization. By modeling the request-model relationship as a hypergraph and applying hypergraph partitioning techniques, it solves the problem of optimizing model deployment under dynamic and coupled request-model relationships. Through the k-way hypergraph partitioning method, it minimizes the communication cost between edge servers while ensuring balanced model deployment. A two-stage model redeployment mechanism is proposed to address the adaptability of model deployment in the face of dynamically changing requests and model relationships. By combining incremental redistribution and global redeployment, the model deployment location is dynamically adjusted to cope with various challenges. The changing requirements not only reduce communication overhead but also adapt to the dynamic changes in requests and model needs, improving the timeliness of the edge cloud system when handling large-scale LLM inference tasks. A curiosity-guided memory association learning method is introduced for model invocation decisions, significantly improving the accuracy of model invocation by enhancing sample efficiency and the ability to explore unexplored states. Within the framework of deep reinforcement learning, a high-value experience propagation mechanism is incorporated, enabling the system to make more accurate invocation decisions when facing complex model dependencies. This not only accelerates the learning process but also improves the system's generalization ability, thereby enhancing the real-time response capability of the edge cloud system.

[0048] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for joint model inference based on model inference timeliness optimization, characterized in that, The method comprises the following steps: S1, constructing an edge cloud system model and an AoMI quantization model: defining a LLM inference request, a model, and a set of edge servers, constructing a model invocation directed acyclic graph (DAG) for each request type, and fusing a time penalty and an inference delay penalty to define an inference timeliness index AoMI; S2, dynamic model deployment based on hypergraph partitioning and two-stage redeployment: modeling the request-model relationship as a hypergraph, optimizing the initial model deployment by using k-way hypergraph partitioning, designing a two-stage model redeployment mechanism of gradual incremental re-partitioning and globally triggered redeployment to dynamically adapt to request changes; S3, model invocation decision based on curiosity-guided memory associative learning: constructing a Markov decision process (MDP) for dynamic model optimization, and setting a burst request controller to cope with request overload; propagating high-value experience through memory association, and combining a curiosity mechanism calculated based on state novelty to optimize the invocation decision.

2. The joint model inference method based on model inference timeliness optimization according to claim 1, characterized in that, In S1, the LLM inference request contains L different types, denoted as where is a set of request types; each request requires invocation of several models to generate a complete response; the set of all available models where denotes a model type.

3. The joint model inference method for model inference timeliness optimization according to claim 2, characterized in that, In S1, the association of each request type with the DAG of models needed is denoted as ; where, denotes the set of models involved in the request, each vertex corresponds to a model deployed in the edge cloud system, and for any two vertices and , if there exists an execution dependency edge , then cannot be invoked until has completed execution, and each model cannot be interrupted during execution; these models are deployed on edge servers, denoted as .

4. The joint model inference method for model inference timeliness optimization according to claim 3, characterized in that, In S1, the AoMI index measures the timeliness of the edge cloud system, and the AoMI model construction specifically comprises the following steps: S11, define time t AoMI is: ; wherein, represents a time penalty function, which measures the interval between the current time and the generation time of the latest completed inference request; represents an LLM inference penalty function, which is used to capture whether the completion delay of the latest completed inference request is within its maximum acceptable delay range; unlike the traditional delay metric which only measures the completion time of the request, the AoMI comprehensively evaluates the timeliness of the entire LLM inference process in the edge cloud system; S12, Time penalty function Reflecting the growth of time penalty, modeled as a linear time function, used to quantify the decline in timeliness, defined as: ； where, denotes the time of generation of the most recently completed inference request, by the equation obtain the index of the most recently completed request , denotes the completion time of the request ; the time penalty function aims to penalize the time elapsed since the last completed inference request, encouraging the system to complete the next LLM inference request as soon as possible; S13, a LLM inference penalty function is introduced to quantify the timeliness decline caused by inference expiration, and the LLM inference penalty function is: ； wherein , denotes the inference completion delay of the latest completed request, is the maximum acceptable inference delay for the request.

5. The joint model inference method for model inference timeliness optimization according to claim 4, characterized in that, The LLM inference request involves joint inference across multiple edge servers, and due to the sequential dependency between models, a model cannot be invoked until all preceding models have completed execution, therefore, the model inference completion time is defined as: ； wherein, representing the model the earliest time at which it becomes available, is a set of predecessor models for the model, is the time required to transfer the output of the predecessor models to the model ; is the completion time of the predecessor models for this model; is the model inference time; Model inference time By model The ratio between the computational demand of the model and the computational capacity on the edge server , i.e. where is the computational amount of the model , and is the computational capacity on the edge server; the completion delay of the request is defined as: ； wherein, is a request for a model collection with no subsequent dependencies, and is the generation time and completion time, respectively, of the request 6. The joint model inference method for model inference timeliness optimization according to claim 5, characterized in that, The inference timeliness is jointly affected by model deployment and invocation decision, and the optimization problem model is defined as follows: Model Deployment MD: decides which models are hosted on the edge server, denoted as ; Model Deployment Status , which takes the value 1 if the model is deployed on the server and 0 if it is not deployed; Model invocation MI: selecting a deployed model service for a LLM inference request, denoted as: ; request model invocation scheme a value of 1 indicates that the model was invoked to process the request a value of 0 indicates that it was not invoked; Divide the timeline into non-overlapping time windows and optimize independently within each window, the optimization problem can be represented as: ； Constraint condition: : load balancing constraints, limiting the total size of models on each edge server , such that it is within a tolerance of the average, the imbalance tolerance being controlled by a parameter , enforcing a perfectly uniform deployment when : Ensure that a model can only be invoked once all of its prerequisites have completed execution; : Ensures that the invocation of the model happens after the inference of its preceding request is completed; At each time window , AoMI is a piecewise function with discrete jump points at each request completion time , i.e. within each interval , remains constant and equal to ; since the LLM inference penalty function depends on whether the completion delay of the latest completed inference request is within the maximum acceptable delay, its value remains constant within an interval, so the time domain is divided into unimportant intervals , and the optimization problem is rewritten as: ； wherein, is the time parameter is the set of requests within the time parameter is a known constant, minimizing the AoMI is equivalent to minimizing and keeping the request completion delay within its maximum acceptable delay.

7. The joint model inference method of claim 6, wherein, In S2, hypergraphs allow each hyperedge to connect any number of nodes, modeling requests as hyperedges connecting all required models, defining a hypergraph containing a finite non-empty set of vertices and a set of hyperedges where each hyperedge is a non-empty subset of the set of vertices and the union of all hyperedges covers the entire set of vertices, i.e.: ； In model deployment, a request-model hypergraph is constructed based on the request-based model requirements Each vertex Corresponds to a deployed model Each hyperedge Represents an LLM inference request Connects all models involved in executing the request; Since the request reasoning involves multiple models, which are usually deployed on different servers, define binary variables to represent the request whether the server needs to be used i.e.: ； wherein, determines the location where the model is hosted on the edge server, wherein ; model deployment status is 1, indicating that the model is deployed on the server , and is 0, indicating that it is not deployed; indicates the deployed model serving LLM inference requests, , is 1, indicating that the model is invoked to process the request , and is 0, indicating that it is not invoked; indicates the request needs to use the server , assuming that there is no communication cost within the same server, and the communication cost between two different servers is the same, denoted as ; To reduce the communication overhead, to reduce the number of servers required as much as possible, define the cross-server communication cost of a request as: ； wherein, represents a request to the server to transmit a data amount; between the server , represent a request whether the server and , since the model for inference depends on the output of its predecessor model, the time at which the model receives the result from its predecessor model is defined as: ； wherein, indicates a request amount of data transferred from the model; In the request-model hypergraph, a request is modeled as a hyperedge connecting the models it requires. To reduce the communication overhead, the model set is partitioned into non-interdependent blocks, i.e., servers, by applying a k-way hypergraph partitioning method to the constructed hypergraph, to minimize the connectivity metric;​ hypergraph k-way partitioning is to partition its vertex set into mutually exclusive subsets, denoted as , where each vertex is assigned to only one subset in the hypergraph , and each subset corresponds to one server; the goal of model deployment MD problem is to place models onto servers such that the cross-server communication for joint model inference is minimized: wherein, is a load balancing constraint, limiting the total size of models on each edge server such that it is within a tolerance of the average, the imbalance tolerance being controlled by the parameter , enforcing a perfectly uniform deployment when represents the popularity of the request, reflecting the frequency of occurrence of the request, C4 avoids repeated computation of communication costs.

8. The joint model inference method of claim 7, wherein, In S2, the specific way of two-stage model redeployment is as follows: Gradual incremental re-partitioning: to adapt to the distribution of the continuously developing LLM inference request types, while avoiding excessive overhead of global redeployment, a gradual incremental re-partitioning is introduced, which incrementally adjusts the model deployment based on request patterns and dependency structure: firstly, high-frequency request types are identified, and an incremental model migration value score is calculated; Secondly, after identifying the models with high migration values, the appropriate servers to which these models can be migrated are determined, and the principle is to incrementally deploy the models to minimize the inter-server communication cost during joint inference; before migration, it is checked that the models need to meet the access span reduction and server balance constraints; Triggering global redeployment: when the migrated models exceed a predefined threshold, the model deployment on all servers is re-optimized, and based on the k-way hypergraph partitioning, the new deployment X' is generated, which jointly minimizes the inter-server communication and model migration cost from the previous deployment.

9. The joint model inference method for model inference timeliness optimization according to claim 8, characterized in that, In S3, the Markov decision process (MDP) specifically comprises the following steps: S31, at a time step The edge cloud system observes a state comprising: All requested model invocations DAG, denoted as where is the model invocation relationship for each request type; model deployment case ; the latest request completion time ; the remaining time required by the model to complete the current inference and the model execution state of the current ongoing request where is the model execution state set of request , defined as where represents the execution state of the model, represents that the inference has been completed, represents that the inference has not been completed; S32, action space contains model invocation decisions, representative request-model invocation scheme, defined as represents a request whether to invoke a model to perform inference; S33, the reward function is designed based on AoMI minimization, and the calculation formula is: ； S34, the burst request controller selects a request that is expected to minimize the AoMI increment based on the heuristic policy of the AoMI; the MDP invokes the model through deep reinforcement learning (DRL) when the request arrival rate exceeds a safety threshold the burst request controller takes over the invocation process S35, state-action value function estimating cumulative rewards according to a policy approximated by a deep neural network aligning predicted values with target values derived from observed rewards and estimated future rewards; during training, experiences are sampled from a replay buffer and updated by the following objective: ； wherein, are parameters of the target network updated periodically, is a discount factor; DRL policy is trained only based on its own exploration, the burst request controller serves as a backup mechanism and will only be activated under burst conditions and does not interfere with the learning process.

10. The joint model inference method for model inference timeliness optimization according to claim 9, characterized in that, In S3, memory associative learning exhibits associative reasoning by relating relevant experiences to adapt to new but similar situations, and a relationship state structure is introduced to capture associations from observed trajectories; In particular, the state relationship structure is as follows: ； relationship representing a state is a state successor in a sampled trajectory under some action, given state relationship structure , the value propagation mechanism is defined as This propagation relationship allows high-reward experiences to propagate between related states; Curiosity-guided memory associative learning includes: Memory associations: use high-value estimates to supervise the update of the value function, and encourage exploration of unexplored regions ; Curiosity mechanism: novelty score of state embedding is computed where is the state the nearest neighbor in the embedding space; the intrinsic reward is computed as: where the weight controlling curiosity; the total reward combines the extrinsic and intrinsic rewards and is denoted as: ; Setting up the loss function: update the value function under the time difference learning goal under the guidance of memory association and curiosity Given an experience tuple sampled from the replay buffer, define the loss function as: ； wherein is a balancing coefficient, controlling the influence of the memory guidance.

Images