Service arrangement and scheduling method and system based on large language model

Abstract

This invention discloses a service-oriented orchestration and scheduling method and system based on a large language model, comprising: encapsulating the model and its working model into an MSU and generating a general interface specification file based on the MCP protocol; generating a unique Context ID for each inference task and registering the context information in external shared storage; automatically generating a model service orchestration graph by an LLM based on the UIS file of each MSU; according to the topology and context dependencies of the orchestration graph, a custom scheduler comprehensively scores candidate nodes in a Kubernetes cluster based on context affinity and resource score and selects the optimal node to deploy the MSU; and adaptively optimizing scheduling weights and resource allocation through a fast / slow feedback loop of the runtime monitoring module. This invention significantly reduces cross-node communication latency and improves task throughput while ensuring system scalability.

Description

Technical Field

[0001] This invention relates to the fields of model service architecture and cloud-native deployment technology, and in particular to a service architecture orchestration and scheduling method and system based on large language models, for efficiently scheduling multi-model collaborative tasks in heterogeneous computing clusters. Background Technology

[0002] Deep learning models have been widely applied across various fields. However, traditional AI systems, with their independent models designed for single tasks and fixed input / output formats, struggle to adapt to complex, multi-stage business processes. As application scenarios become more complex, single-model architectures reveal limitations when handling cross-task, cross-modal, or dynamic decision-making needs. Therefore, multi-model task collaboration has become a development trend. It automates the processing of complex tasks through model combination and coordination, but this places higher demands on invocation mechanisms, data pathways, and unified scheduling management.

[0003] The rise of Large Language Models (LLMs) has driven multi-model collaboration. LLMs, with their powerful natural language understanding and reasoning capabilities, can act as "cognitive centers" or "scheduling agents" among different models, enabling dynamic task decomposition, result aggregation, and autonomous decision-making at the semantic level. However, existing multi-model collaboration schemes typically lack standardized context interaction protocols, making collaborative invocation and context sharing between different models difficult, thus affecting the system's scalability and composability. Therefore, emerging technical standards such as Model Context Protocols (MCPs) have emerged.

[0004] This protocol defines a unified model context interface and communication protocol, enabling different types of models to share task context, execution state, and data dependencies in a structured manner, thereby achieving interoperability and composability between models. However, the MCP protocol itself does not involve underlying resource management and scheduling mechanisms, so it still needs to be combined with containerization and cloud-native architectures in practical applications. In cloud-native environments, Kubernetes (K8s), as the mainstream container orchestration platform, provides automatic scheduling, scaling, and fault tolerance mechanisms for computing resources, providing a good foundation for model service-oriented architecture. However, existing solutions still face challenges when combining LLM services with the MCP protocol and achieving efficient orchestration through K8s.

[0005] The current default scheduling strategy in Kubernetes is resource-driven, not context-driven. The scheduler cannot perceive the business logic dependencies between models, and may schedule strongly dependent service instances to different nodes with high network latency, resulting in increased overall inference latency and decreased resource utilization. In addition, how to achieve "plug-in" dynamic management of working models (workloads) in multi-model systems and seamlessly integrate them with service orchestration and context protocols is also a current technical challenge.

[0006] Therefore, the industry urgently needs a new technology solution that can standardize and encapsulate large models and their working models, manage context dependencies using a unified protocol, and combine Kubernetes orchestration capabilities to achieve context-aware intelligent scheduling and dynamic service management. Summary of the Invention

[0007] To overcome the shortcomings of existing technologies, the purpose of this invention is to propose a service-oriented orchestration and scheduling method and system based on a large language model.

[0008] To achieve the above objectives, the technical solution adopted by this invention is: a service-oriented orchestration and scheduling method based on a large language model, comprising the following steps:

[0009] S1. System Startup and Service Encapsulation. The system first scans registered models, including the Large Language Model (LLM) and multiple working models, and generates a Universal Interface Specification (UIS) file. Simultaneously, the system activates a slow path to periodically update UIS annotations.

[0010] S2. When a user submits a task request through the task orchestration module, the system first sends a request to the model service module to read the UIS. After receiving the request, the model service module will generate a system-level prompt message based on the registered UIS file and return the prompt message.

[0011] S3. After receiving the information, the task orchestration module will execute the LLM inference task. Based on system-level and user-defined prompts, the LLM will attempt to generate a service orchestration graph (a type of Directed Acyclic Graph, describing the order of calls and input / output dependencies between models, and supporting dynamic updates at runtime). If the orchestration graph cannot be generated (e.g., missing functional modules, unclear task descriptions), the task orchestration module will return an error message to the user and terminate the task.

[0012] S4. Once the orchestration diagram is determined, the task orchestration module will send a task registration request to the context protocol management module. The context protocol management module will execute the task registration process and return a global context identifier (Context ID).

[0013] S5. After receiving the global context identifier (Context ID), the task orchestration module starts the task, sends the orchestration diagram and task scheduling request to the scheduling and deployment module, and waits for the task to be executed.

[0014] S6, the scheduling and deployment module, creates Kubernetes Pods according to the MSUs identified in the orchestration diagram in topological order. A custom controller (Operator) and a custom scheduler (Scheduler) then perform node selection and instance management based on context ID affinity and node resource status. During this process, a fast feedback path enables elastic scaling through custom metrics (such as resource load).

[0015] S7. After the scheduling and deployment module completes the task execution, it will return the result to the model orchestration module, which will then return the final execution result to the user, and the task will end.

[0016] Accordingly, a service-based orchestration and scheduling system based on a large language model includes:

[0017] Model Service Module: This module encapsulates the LLM and working model into a containerized MSU with an MCP interface, and provides management and storage functions for model images.

[0018] Context Protocol Management Module: Assigns a unique Context ID to each task to identify context information and task status.

[0019] Service Orchestration Module: With LLM as the decision-making core, it is used to build and dynamically update the model service orchestration graph and define dependencies between models.

[0020] Cluster scheduling and deployment module: Runs on a K8s cluster, including a custom controller for automated lifecycle management and a custom scheduler for context-aware Pod scheduling, dynamically scheduling the deployment of each MSU instance in the K8s cluster.

[0021] Runtime monitoring and optimization module: Used to collect system performance indicators and adjust scheduling strategies and resource configurations through feedback loop.

[0022] Beneficial Effects: This invention achieves efficient, scalable, and fault-tolerant model service deployment in multi-model collaboration scenarios by employing a standardized context protocol based on MCP (Model Context Protocol), combining automatic orchestration capabilities generated from large language models, and a context-aware cluster scheduling mechanism. Compared with existing technologies, this invention has the following significant technical effects:

[0023] 1. Standardization and scalability: Through the encapsulation of MCP and MSU, unified access to heterogeneous models and "plug-in" dynamic management are achieved, improving system flexibility.

[0024] 2. Low latency and bandwidth consumption: The system adopts a lightweight interaction method of command messages plus Context ID. The actual context is accessed on demand through an external shared key-value store or passed by reference, thereby avoiding the copying of a large amount of context data (such as large vectors or images) during scheduling and execution, significantly reducing network bandwidth consumption and reducing cross-node transmission latency.

[0025] 3. Context-aware scheduling: Custom schedulers can be aware of the context dependencies in the orchestration graph, optimize Pod layout (such as node affinity of strongly dependent instances), and significantly reduce cross-node transmission latency.

[0026] 4. Intelligent closed loop: Using LLM as the decision center to generate orchestration diagrams, combined with the automated execution capabilities of custom controllers, an intelligent closed loop is achieved from task understanding to service deployment.

[0027] 5. Adaptive optimization: Through runtime monitoring and adaptive feedback loop, resource allocation is dynamically adjusted to ensure low latency and high resource utilization under different loads. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of the overall architecture of the service-oriented orchestration and scheduling system based on a large language model, as shown in this embodiment.

[0029] Figure 2 This is a flowchart illustrating the model service orchestration and scheduling method based on big language as shown in this embodiment.

[0030] Figure 3 This is a schematic diagram of the context passing timing based on the context protocol management module in this embodiment. Detailed Implementation

[0031] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Any modifications made to the technical solutions based on the technical concept proposed in this invention shall fall within the scope of protection of this invention.

[0032] Example 1:

[0033] like Figure 1 As shown, this embodiment of the invention provides a service-oriented orchestration and scheduling system based on a large language model, including: a model service module 101, a context protocol management module 102, a service orchestration module 103, a scheduling and deployment module 104, and a runtime monitoring and optimization module 105. The system is built and runs on a K8s cluster 106 containing nodes 107a, 107b, and 107c.

[0034] The model service module 101 is used to encapsulate the LLM and working model into a containerized MSU with an MCP interface, and to provide storage and management functions for the model image.

[0035] This module includes a built-in container image repository and a UIS file storage database for storing and managing various heterogeneous models, including large language models (101a) and working models (101b). Through containerization technology, each model and its dependencies (such as Python runtime, CUDA libraries, dependency packages, etc.) and service communication interfaces are packaged into portable container images, ensuring model consistency and repeatability across different nodes. The UIS file storage database is a dedicated database for storing UIS files, associated with the model image repository, and provides interfaces for model registration, model management, and model querying.

[0036] A UIS file is a JSON-formatted unified interface specification file that contains a natural language description of the model (for LLM understanding), its MCP interface definition (including explicit input and output declarations), communication protocols, resource requirements, and model annotations (usually performance-related). This file enables the system to achieve consistency in communication semantics and standardize invocation across different models. By storing the UIS file in a database, new models can be dynamically loaded into the system simply by providing their image and interface files, ensuring system scalability.

[0037] The context protocol management module 102 provides a unified communication protocol and context management. This module decouples control flow from data flow; models only exchange instruction information and context identification information, and all models access data through a shared external key-value store, significantly reducing communication overhead between services. Centralized context management improves consistency and ease of use, while shared external storage effectively enhances high availability and partition tolerance.

[0038] The service orchestration module 103 generates decisions based on LLM, which are then used as the model service orchestration graph in subsequent tasks. Upon receiving a user request, the module queries the registered MSUs in the model service module 101 and, using the query interface of the UIS file storage database, dynamically constructs system-level prompt words from information such as MSU natural language descriptions, MCP interface definitions, and model annotations. The module uses the user request and system-level prompt words to construct a task request. The LLM responds to the task request by performing inference and generating a structured (e.g., JSON or YAML) model service orchestration graph (DAG). The orchestration graph defines runtime information such as basic task information, model usage sequence, execution order, dependent modules, and resource requirements. The module has a built-in validator that verifies the topology, dependencies, and interface matching of the orchestration graph before execution.

[0039] The scheduling and deployment module 104 runs on the Kubernetes cluster and includes a custom controller 104a for automated lifecycle management and a custom scheduler 104b for context-aware Pod scheduling, dynamically scheduling the deployment of each MSU instance in the Kubernetes cluster. The custom controller 104a receives and parses the orchestration graph, determines the creation order of model instances based on topology sorting, and submits it to the scheduler. The custom scheduler 104b executes a context-aware scheduling algorithm for Pods, enabling manual scheduling of Pods through the Kubernetes API. This algorithm includes filtering logic (selecting nodes based on available resources) and scoring logic (selecting the optimal node based on a weighted scoring formula).

[0040] The runtime monitoring and optimization module 105 is used to collect runtime metrics in real time and perform adaptive optimization to ensure the overall performance and stability of the system. The module integrates monitoring tools such as Prometheus, collects system runtime status periodically, and establishes two types of feedback loops: fast feedback is used to detect and respond immediately to sudden load or latency anomalies, and uses Horizontal PodAutoscaler (HPA) to automatically scale up and down Pod replicas based on real-time metrics to cope with load fluctuations; slow feedback is used to periodically update performance annotations and adjust scheduling weights, thereby achieving a long-term balance between stability and performance and improving the availability and throughput of the system under sudden load changes.

[0041] Example 2:

[0042] Figure 2 The method flow of an embodiment of the present invention is illustrated.

[0043] S201. Initialization Phase. Upon system startup, the model service module scans the list of registered models and generates a corresponding UIS file for each model. The UIS file includes a model functional description, MCP interface description (input / output declaration), communication protocol, resource requirements, and performance annotations. The UIS file uses a structured JSON format, which can be automatically parsed by LLM.

[0044] S202. Slow feedback loop starts. The runtime monitoring and optimization module 105 starts the periodic slow feedback loop, performs performance sampling and statistics on all model service units in the system, and writes these performance data into the corresponding UIS file to provide performance reference for subsequent orchestration decisions, thereby achieving continuous optimization of global performance.

[0045] S203, Task Reception Phase. After receiving the user request, the Service Orchestration Module 103 sends a gRPC request command to the Model Service Module 101, requesting the retrieval of the UIS files of all currently registered MSUs. Upon receiving the request, the Model Service Module 101 constructs a system-level prompt message and returns it. The Service Orchestration Module combines the user request and the system-level request to construct a task request. After performing inference, the LLM generates one or more model orchestration diagrams. If the LLM determines that the request cannot be fulfilled, it returns an error message to the user.

[0046] S204, Context Registration Phase. Once the scheme is determined, the service orchestration module 103 sends a task request to the context protocol management module 102. The context protocol module will assign a unique Context ID to the task and write the initial context information and model orchestration diagram into the shared key-value store.

[0047] S205, Context-Aware Scheduling and Deployment Phase. Custom controller 104a continuously listens for new orchestration task events. Custom scheduler 104b executes Pod scheduling based on context affinity and resource score. Context affinity represents the tightness of shared Context IDs between dependency models; resource score represents the degree to which the target node meets the requirements for CPU, memory, GPU memory, etc. Custom scheduler 104b uses a comprehensive scoring function:

[0048]

[0049] The optimal node is selected to deploy the MSU, and Pods (108a, 108b, etc.) are started via the Kubernetes API according to the topology order. After the Pod is deployed and activated, the input context is read from the shared storage based on the Context ID in the message instruction. After the task is completed, the result is written back to the shared storage using the Context ID as the key. The system triggers subsequent models sequentially according to the orchestration diagram definition until the task is completed.

[0050] S206, Result Return and Feedback Optimization Phase. The final model writes the results to shared storage. Service orchestration module 103 detects task completion. Simultaneously, runtime monitoring and optimization module 105 records the performance metrics of this task (such as latency and load), updates the performance database (for the slow feedback loop in S201), and forms an adaptive closed loop.

[0051] Example 3:

[0052] Combination Figure 3 This embodiment provides a context organization method based on the context protocol management module 102, which is used to decouple the control flow and data flow, thereby improving the scalability and consistency of the system in the process of multi-model task collaboration.

[0053] In this embodiment, context passing follows the MCP (Model Context Protocol), and model units interact through lightweight instruction messages. Each message contains only a Context ID and control instructions, while the actual context content (including input data, intermediate results, and status information) is stored in an external shared key-value store (such as Redis, etcd, etc.). In this way, the system achieves "instruction and data separation," reducing communication overhead and improving parallel processing capabilities.

[0054] The specific process is as follows:

[0055] S301. When a task starts, the system assigns a globally unique Context ID (e.g., Task-wytDhYeP).

[0056] S302, The system enters an instruction waiting loop (corresponding to...) Figure 3 (Regarding "whether the instruction has arrived"). Communication between model units is achieved through lightweight instruction messages (such as gRPC), which only contain the instruction type and Context ID (control flow). All actual context data is associated with this Context ID and stored in an external shared key-value store (such as Redis) (data flow).

[0057] S303. Upon receiving an instruction, the system performs a branching decision based on the instruction type:

[0058] a) If it is a read instruction: The model unit uses the Context ID in the message to read the corresponding context data from the shared storage. After processing, it returns to S302 to wait for the next instruction.

[0059] b) If it is a write instruction: The model unit will append or update the processing result to the existing context using the Context ID as the key, and write it back to the shared storage. After processing is complete, it returns to S302 to wait for the next instruction.

[0060] c) If it is a termination instruction: The system determines that the task should end normally and jumps to S304.

[0061] d) If it is another illegal instruction: The system throws an exception error message and jumps to S304 to prepare to terminate the task.

[0062] S304, Mission complete.

[0063] Example 4:

[0064] This embodiment illustrates the implementation details of a scheduling algorithm in this invention.

[0065] Problem Modeling: This invention relates to a service-oriented orchestration and scheduling method based on a large language model. The process first involves problem modeling, then transforming the problem into a computable optimization form for solution. The technical problem to be solved is to implement an efficient and low-latency scheduling strategy for an orchestration graph dynamically generated by an LLM in a heterogeneous computing environment managed by a Kubernetes cluster. This problem is modeled as an online multi-objective optimization problem with multi-dimensional resource constraints (especially GPU memory) and context dependencies. Its core objective is to rationally place model service units in the orchestration graph as Pods on cluster nodes while ensuring the correctness of context dependency constraints, thereby minimizing end-to-end inference latency and improving resource utilization efficiency. The Kubernetes node set is defined as follows:

[0066]

[0067] Let K be the number of available physical nodes; the set of online requests at any time t is:

[0068]

[0069] Each inference request A corresponding layout diagram is denoted as:

[0070]

[0071] in This represents the I model nodes (Pods to be scheduled) that make up the orchestration graph. This indicates the context dependencies between Pods. express The startup depends on The context output.

[0072] For each node Define its total resource vector as:

[0073]

[0074] Represents total resources, of which Represents CPU capability. Indicates memory capacity. This represents the GPU memory capacity for each model Pod. Define its resource requirement vector Indicates execution The system resources required at that time. Each Pod should also have its own context identifier. For the same arrangement diagram All Pods in the set share all Context IDs, and their predecessor set is defined as follows:

[0075]

[0076] and the context data generated by its predecessor. With the deadline of the graph .

[0077] Define scheduling decision variables It is represented as:

[0078]

[0079] Also define the Pod's start execution time. and completion time And the total completion time of the layout diagram:

[0080]

[0081] Define end-to-end experimental delay ,in for Arrival time. When When, the timeout is defined as:

[0082]

[0083] The system's context transmission delay is defined as follows:

[0084]

[0085] in, Used to indicate time This represents the time cost of using IPC communication within the same node; It refers to network transmission delay, and its functional relationship depends on the context. and bandwidth .

[0086] The start time of a Pod is constrained by the end time of its predecessor and the context transfer latency, and must satisfy the following constraints:

[0087]

[0088] Regarding the objective function, this invention is based on multi-objective optimization, taking minimizing end-to-end delay and minimizing transmission overhead as joint optimization objectives. Its objective function is defined as follows:

[0089]

[0090] in These are weighting coefficients used to balance latency and resource efficiency. Objective The timeout cost used to minimize end-to-end latency is defined as follows:

[0091]

[0092] Target To minimize context transmission overhead, the sum of transmission delays corresponding to dependent edges is defined:

[0093]

[0094] This naturally encourages deploying Pods with strong dependencies on the same or nearby nodes to reduce latency in transferring context from distant nodes.

[0095] To ensure the feasibility of the solution, a unique allocation constraint requires that each Pod can only be assigned to one node:

[0096]

[0097] At the same time, resource constraints are imposed:

[0098]

[0099] Solving the global optimum for this NP-hard problem within a millisecond-level scheduling window is infeasible. Therefore, this invention transforms it into a two-stage heuristic scheduling algorithm of filtering and scoring: constraints serve as filtering logic, and the optimization objective is transformed into a scoring standard.

[0100] During the filtering phase, when a Pod At that time, the scheduler traverses all nodes. Filter to meet the requirements Nodes of resource demand ( This forms the candidate node set ValidNodes.

[0101] During the scoring phase, this invention will use a multi-objective function. Transform into a weighted heuristic function And select the node with the highest score. :

[0102]

[0103] The scoring function is defined as follows:

[0104]

[0105] in and Preset weights and limits ,and .

[0106] for Defined as:

[0107]

[0108] That is, when the current node is running on this node, it is expected that subsequent nodes will be scheduled to this node.

[0109] The definition of is:

[0110]

[0111] in, (t) refers to the value of a node. Currently available GPU memory capacity;

[0112] Refers to Pod The amount of GPU memory required to perform this task;

[0113] This embodiment uses a minimum allocation strategy, selecting nodes with the most available resources (GPU memory resources are the most important factor here).

[0114] By transforming the problem as described above, the NP-hard global optimization problem is transformed into a heuristic algorithm that enables awareness of MCP context dependencies during scheduling.

[0115] It should be understood that although this specification describes the system according to embodiments, not every embodiment contains only one independent technical solution. This narrative style is only for clarity, and those skilled in the art should consider this specification as a whole. The technical features in each embodiment can also be reasonably combined to form other implementation methods that can be understood by those skilled in the art.

Claims

1. A service-oriented orchestration and scheduling system based on a large language model, characterized in that, include: Model service module: used to encapsulate LLM and working models into containerized MSU with MCP interface, and provide management and storage functions for model images; Context protocol management module: Assigns a unique Context ID to each task to identify context information and task status; Service Orchestration Module: With LLM as the decision-making core, it is used to build and dynamically update the model service orchestration graph and define dependencies between models; Cluster scheduling and deployment module: running on a K8s cluster, including a custom controller for automated lifecycle management and a custom scheduler for context-aware Pod scheduling, dynamically scheduling the deployment of each MSU instance in the K8s cluster; Runtime monitoring and optimization module: Used to collect system performance indicators and adjust scheduling strategies and resource configurations through feedback loop.

2. The system according to claim 1, characterized in that: The model service module includes a container image repository and a UIS file storage database. The container image repository is used to store and manage various heterogeneous models. Through containerization technology, each model and its dependent environment and service communication interface are packaged into a portable container image, thereby ensuring the consistency and repeatability of the model on different nodes. The UIS file storage database is a dedicated database for storing UIS files. It is associated with the model image repository and provides interfaces for model registration, model management, and model querying.

3. The system according to claim 2, characterized in that: The UIS file is a unified interface specification file in JSON format, which contains a natural language description of the model, its MCP interface definition communication protocol, resource requirements, and model annotations.

4. The system according to claim 1, characterized in that: The context protocol management module provides a unified communication protocol and context management; models only transmit instruction information and context identification information, and all models access data through a shared external key-value store.

5. The system according to claim 1, characterized in that: The service orchestration module generates decisions based on LLM, and these decisions are used as the model service orchestration graph for subsequent tasks. Upon receiving a user request, the service orchestration module queries the registered MSUs in the model service module and uses the query interface of the UIS file storage database to dynamically construct system-level prompt words from the MSU's natural language description, MCP interface definition, and model annotation information. The service orchestration module constructs task requests using user requests and system-level prompts. The LLM responds to the task requests, performs inference, and generates a structured model service orchestration graph. The service orchestration module has a built-in validator that verifies the topology, dependencies, and interface matching of the orchestration graph before execution.

6. The system according to claim 1, characterized in that: In the cluster scheduling and deployment module, the custom controller receives and parses the orchestration diagram, determines the creation order of model instances based on the topology sorting, and submits it to the custom scheduler; the custom scheduler executes a context-aware scheduling algorithm for Pod execution, and implements manual scheduling of Pods through the K8s API. The scheduling algorithm includes filtering logic based on available resources to select nodes and scoring logic based on a weighted scoring formula to select the optimal node.

7. The system according to claim 1, characterized in that: The runtime monitoring and optimization module is used to collect runtime metrics in real time and perform adaptive optimization. The module integrates monitoring tools, collects system operating status regularly, and establishes two types of feedback loops: fast feedback and slow feedback. Fast feedback is used to detect and respond to sudden load or latency anomalies in real time, and uses Horizontal PodAutoscaler to automatically scale up and down Pod replicas based on real-time metrics to cope with load fluctuations. Slow feedback is used to periodically update performance annotations and adjust scheduling weights.

8. A method based on the system of any one of claims 1-7, characterized in that, Includes the following steps: S1. Initialization Phase: When the system starts, the model service module scans the list of registered models and generates a corresponding UIS file for each model. S2. Slow feedback loop starts: The runtime monitoring and optimization module starts the cycle slow feedback loop to sample and statistically analyze the performance of all model service units in the system, and writes these performance data into the corresponding UIS file to provide performance reference for subsequent orchestration decisions. S3, Task Reception Phase: After receiving a user request, the service orchestration module sends a gRPC request command to the model service module, requesting to obtain the UIS files of all currently registered MSUs. After receiving the request, the model service module constructs a system-level prompt message and returns it. The service orchestration module combines the user request and the system-level request to construct a task request. After performing inference, the LLM generates one or more model orchestration graphs. If the LLM believes that the request cannot be satisfied, it returns an error message to the user. S4. Context Registration Phase: Once the scheme is determined, the service orchestration module sends a task request to the context protocol management module. The context protocol management module will determine a unique Context ID for the task and write the initial context information and model orchestration graph into the shared key-value store. S5, Context-Aware Scheduling and Deployment Phase: The custom controller continuously listens for new orchestration task events; the custom scheduler executes Pod scheduling based on context affinity and resource score; context affinity represents the tightness of shared Context IDs between dependent models; resource score represents the degree to which the target node meets the CPU, memory, and GPU memory requirements; the custom scheduler selects nodes to deploy MSUs based on a comprehensive scoring function and starts Pods via the K8s API according to the topology order; after Pod deployment is activated, the input context is read from shared storage based on the Context ID in the message instruction; after task execution is completed, the result is written back to shared storage using the Context ID as the key; the system triggers subsequent models sequentially according to the orchestration graph definition until the task is completed; The comprehensive scoring function is: ； in, To orchestrate the i-th Pod to be deployed in the diagram, For the first 1 node Indicates the first The Pod is deployed in the... The score for each node; this score is determined by the Pod's affinity score for the node. and node resource availability score The weighted calculation is obtained, where the weight coefficients are... and Meet the conditions and This indicates that both node affinity and resource availability conditions are considered simultaneously. Affinity Score for: ； It is a Pod waiting to be scheduled; It is a candidate node that can be scheduled; yes The set of Pods that it depends on; yes The dependent nodes have been deployed on the node. A set; It evaluates the quantity of the set; Node resource availability score for: ； (t) refers to the value of a node. Currently available GPU memory capacity; Refers to Pod The amount of GPU memory required to perform this task; It is a node The total capacity of the video memory; S6. Result Return and Feedback Optimization Phase: The final model writes the results to shared storage; the service orchestration module detects that the task is completed; at the same time, the runtime monitoring and optimization module records the performance indicators of this task, updates the performance database, and forms an adaptive closed loop.

9. The method according to claim 8, characterized in that, S4 specifically includes: S41. When a task starts, the system assigns a globally unique Context ID; S42, The system enters an instruction waiting loop; communication between model units is achieved through lightweight instruction messages, which contain only the instruction type and Context ID, while all actual context data is associated through the Context ID and stored in an external shared key-value store; S43. Upon receiving an instruction, the system performs a branching decision based on the instruction type: If it is a read instruction: the model unit uses the Context ID in the message to read the corresponding context data from the shared storage; after processing, it returns to S42 to wait for the next instruction; If it is a write instruction: the model unit will append or update the processing result based on the existing context using the Context ID as the key, and write it back to the shared storage; after processing is complete, it returns to S42 to wait for the next instruction; If it is a termination instruction: the system determines that the task should end normally and jumps to S44; If the command is illegal: the system throws an exception error message and jumps to S44 to prepare to terminate the task; S44, Mission complete.

Images