A large model service intelligent operation and maintenance system and method based on graph engineering

Abstract

The application discloses a large model service intelligent operation and maintenance system and method based on graph engineering, and belongs to the technical field of operation and maintenance. The system comprises a causal perception multi-agent routing module which completes intelligent dispatching of tasks and outputs a feasible action mask based on an operation and maintenance causal graph. A dynamic graph construction engine constructs a three-layer nested graph task graph topology based on a standardized node template library. An execution engine completes task node scheduling and execution in combination with real-time resource load. A causal constraint self-adaptive convergence controller locates a fault root cause based on causal tracing, completes Bayesian optimization optimization in a causal alignment subspace, and outputs optimal repair parameters. A memory deposition and knowledge update module differentially deposits operation and maintenance experience, and converts positioning errors into regular penalty reverse iteration routing network weights. The application realizes autonomous closed-loop operation and maintenance of large model services from routing dispatching, computing power scheduling to fault self-healing and strategy self-iteration through a three-layer nested graph topology bearing full-link operation and maintenance and a three-depth coupling mechanism.

Description

Technical Field

[0001] This invention relates to the field of intelligent operation and maintenance technology for large model services, and in particular to an intelligent operation and maintenance system and method for large model services based on graph engineering. Background Technology

[0002] Currently, large-scale model cluster operation and maintenance (O&M) and agent automation technologies are increasingly widely used in various fields. A complete large-scale model service O&M system needs to include model deployment, computing resource scheduling, gateway routing O&M, fault diagnosis, disaster recovery backup, monitoring and early warning, and model evaluation and optimization to ensure the stability and iterability of the system. Currently, large-scale model service O&M systems mainly have the following three architectural patterns: The first type is a static operation and maintenance model with strong human involvement. It relies entirely on manual operation and operation and maintenance scripts. Each link is independent but inefficient. Data and experience cannot be accumulated and reused, making it difficult to achieve autonomous closed-loop operation and maintenance throughout the entire process.

[0003] The second type is the agent architecture driven by a large model. This approach replaces manual decision-making by orchestrating fixed processes or introducing a large model into various stages of operation and maintenance, commonly using the ReAct pattern or multi-agent architecture. This model reduces the cost of manual monitoring, but the ReAct paradigm suffers from linear blocking in heterogeneous resource parallel scenarios, and the final decision of the multi-agent architecture still heavily depends on the skill level of the operation and maintenance personnel. Furthermore, data from each stage is passed through a single model without filtering to make the final decision, which, in the absence of supervision and constraints, is prone to data interference leading to model illusions.

[0004] The third type is the graph orchestration-based operation and maintenance system mode, which uses a directed acyclic graph to orchestrate the process of each operation and maintenance link and supports parallel task triggering. However, its scheduling decision is based on static dependencies and time triggering, lacks the ability to perceive the dynamic load of the underlying GPU / NPU heterogeneous computing power, and does not have a closed-loop optimization mechanism for specific operation and maintenance indicators such as AI inference latency and model accuracy decay.

[0005] In summary, existing technologies suffer from the following shortcomings: They are limited to a single operational scenario, lacking unified and comprehensive capabilities across the entire chain, including gateway scheduling, computing power maintenance, model evaluation, monitoring and early warning, disaster recovery backup, and fault self-healing; the agent architecture lacks global causal cognition, with single agents prone to decision-making biases, and multi-agent processes having fixed processes and failing to perceive dynamic resource constraints; routing allocation relies on pure semantic recognition, resulting in randomness, lack of interpretability, and difficulty in production deployment; graph orchestration is based on static dependencies, lacking dynamic perception of heterogeneous computing power and adaptation specifically for large-scale model maintenance; fault repair relies on fixed retry strategies, lacking adaptive parameter optimization capabilities based on causal constraints; and operational experience is merely documented in text, lacking a closed-loop mechanism for downstream repair results to optimize upstream routing decisions. Therefore, there is an urgent need for an intelligent operation and maintenance system for large-scale model services that can achieve end-to-end autonomous closed-loop, interpretable decision-making, dynamic resource perception, and adaptive fault convergence.

[0006] The disclosure of the above background technical content is only for the purpose of assisting in understanding the concept and technical solution of this application, and does not necessarily provide technical instruction. Summary of the Invention

[0007] The purpose of this invention is to provide an intelligent operation and maintenance system and method for large model services based on graph engineering. It carries out full-link operation and maintenance through a three-layer nested graph topology and realizes the full-process interpretable autonomous closed-loop operation and maintenance of large model services from routing assignment and computing power scheduling to fault self-healing and policy self-iteration through a triple deep coupling mechanism of constraint domain generation, optimization and feedback.

[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A large-scale model service intelligent operation and maintenance system based on graph engineering is provided. The system adopts a three-layer nested topology architecture, which includes a multi-expert hybrid decision-making top layer, a business execution middle layer, and an atomic execution bottom layer from top to bottom. The system as a whole relies on graph engineering modeling to realize the structured arrangement of operation and maintenance tasks, fault causal reasoning, and knowledge iteration closed loop. The system includes functional modules mounted on the three-layer nested topology architecture and collaboratively completing the entire process of operation and maintenance closed loop. The functional modules include: a causal perception multi-agent routing module, a dynamic graph construction engine, an execution engine, a causal constraint adaptive convergence controller, and a memory accumulation and knowledge update module. Deployed at the top layer of the multi-expert hybrid decision-making system, it is used to complete the causal relationship matching between tasks and expert agents based on the operation and maintenance causal graph, realize the intelligent allocation of operation and maintenance tasks, and output the feasibility action mask of the corresponding tasks. The dynamic graph building engine is used to dynamically generate a nested task graph topology structure that adapts to the three-layer nested topology architecture based on the atomic task dependencies of operation and maintenance tasks, thereby connecting the hierarchical linkage relationships of the three-layer architecture. The execution engine is configured with reinforcement learning dynamic scheduling capabilities, which are used to complete the scheduling and automated execution of mid-level business subgraphs and bottom-level atomic units under the constraints of feasible action masks and real-time resource load. The causal constraint adaptive convergence controller is used to complete fault and anomaly tracing and root cause location based on the operation and maintenance causal graph, and output the optimal repair parameters through the optimization search of causal subspace constraints to achieve fault adaptive convergence repair at the entire topology level. The memory accumulation and knowledge update module is used to accumulate operation and maintenance repair experience, update the top-level RAG retrieval knowledge base, and iteratively optimize the routing strategy to achieve continuous learning and closed-loop knowledge update for large-scale model operation and maintenance scenarios.

[0009] Furthermore, following any one or a combination of the aforementioned technical solutions, the multi-expert hybrid decision-making top-level configuration includes an operation and maintenance causal knowledge graph and a RAG retrieval enhancement module, used to realize causal distance calculation, feasible action mask generation, and operation and maintenance knowledge retrieval iteration, undertaking the global intelligent decision-making of the system.

[0010] Furthermore, based on any or a combination of the aforementioned technical solutions, the middle layer of business execution includes a gateway intelligent scheduling subgraph, a heterogeneous hardware operation and maintenance subgraph, a fully automated model evaluation subgraph, a monitoring, perception and early warning subgraph, a disaster recovery and backup operation and maintenance subgraph, and a fault self-healing iteration subgraph, providing a business execution carrier for intelligent operation and maintenance of large models across all scenarios.

[0011] Furthermore, following any one or a combination of the aforementioned technical solutions, the atomic execution layer is composed of various atomic execution units generated by a unified standardized node template, providing standardized and reusable basic execution capabilities for each business subgraph in the middle layer of the business execution.

[0012] Furthermore, based on any or a combination of the aforementioned technical solutions, the dynamic graph construction engine is configured with a standardized node template library. The standardized node template library defines six types of core standardized nodes. Each type of standardized node structure uniformly sets seven fields: node type, input parameters, output indicators, execution permissions, dependency conditions, convergence threshold, and status label. After the target expert intelligent agent decomposes the atomic operation and maintenance task, it matches the corresponding standardized node template, automatically completes the initialization of node parameters, binding of execution permissions and configuration of convergence threshold, and generates the available task graph nodes of the atomic execution underlying layer.

[0013] Furthermore, following any one or a combination of the aforementioned technical solutions, the dynamic graph construction engine dynamically constructs directed edges based on the data dependencies between nodes in the task graph. The construction rules include: Directed data flow edges are automatically generated only when the output data of the preceding task graph node can cover the input requirements of the following task graph node and the preceding task graph node is executed successfully. Task graph nodes without data dependencies do not generate link edges and are executed in parallel by default; at the same time, conditional edge links can be dynamically activated or deactivated based on real-time system monitoring indicators and fault status, so as to achieve adaptive adjustment of the three-layer nested task graph topology.

[0014] Furthermore, based on any or a combination of the aforementioned technical solutions, the six business subgraphs in the middle layer of the business execution operate independently, communicate with each other, and do not interfere with each other. Within each subgraph, the atomic execution units at the bottom layer of the atomic execution are connected by dynamic directed edges, and the business subgraphs without dependencies are executed in parallel.

[0015] Furthermore, following any one or a combination of the aforementioned technical solutions, the causal perception multi-agent routing module relies on the multi-expert hybrid decision-making top-level operation and maintenance causal graph to calculate causal correlation and allocate task routes, specifically including: In the operation and maintenance causal graph, the variable set corresponding to each domain expert agent is traced back from the current task alarm node along the causal edge. The shortest directed path length is used as the causal distance. When there are multiple paths, the reciprocal of the sum of the causal effect weights of the paths is taken as the final distance metric. The causal perception multi-agent routing module uses a routing agent gating network to calculate the routing probability of each domain expert agent based on the calculated causal distance, completes the intelligent assignment of operation and maintenance tasks, and outputs the feasible action mask.

[0016] Furthermore, following any one or a combination of the aforementioned technical solutions, the resource-aware dynamic scheduler in the execution engine models the graph task scheduling process as a Markov decision process. The state space of the Markov decision process includes the graph topology and node states, a real-time resource view, and a ready task queue. The action space of the Markov decision process is constrained by a feasible action mask, allowing only feasible task actions to be selected. The reward function weights of the Markov decision process are dynamically configured by the causal-aware multi-agent routing module according to the task type and target expert agent, enabling different expert agents to correspond to differentiated scheduling optimization objectives. The dynamic weights are fitted using offline simulation to the three-dimensional Pareto front of resource utilization, task completion time, and task hunger rate, and the weight combination corresponding to the inflection point of the front is selected as the default configuration.

[0017] Furthermore, following any or a combination of the aforementioned technical solutions, the parameter space mapping unit of the causal constraint adaptive convergence controller limits the optimization search range through manifold constraints. The parameter space mapping unit transforms the Gaussian process covariance function of the Bayesian optimization algorithm into the product of a standard kernel function and a causal alignment indicator function. The causal alignment indicator function takes a value of 1 in the causal alignment parameter subspace of the matching minimum intervention set and a value of 0 in other regions, so that the prior confidence of the Bayesian optimization algorithm in the non-causal alignment region is set to zero, and the search for the optimal repair parameters is completed only in the causal alignment repair parameter subspace.

[0018] Furthermore, following any or a combination of the aforementioned technical solutions, the causal intervention positioning unit of the causal constraint adaptive convergence controller constructs the top-level operation and maintenance causal graph of the multi-expert hybrid decision-making system using an offline construction and online iteration approach. In the offline phase, based on historical operation and maintenance logs, an initial causal skeleton is generated through a time-series causal discovery algorithm, and the initial causal skeleton is trimmed and corrected by combining prior knowledge of physical topology in the operation and maintenance domain as a hard constraint. In the online phase, small-scale causal intervention verification is carried out only through shadow traffic or gray-scale environments. After each fault repair is completed, the actual intervention effect is used as the basis for Bayesian updating of the edge weights of the operation and maintenance causal graph to continuously optimize the causal positioning accuracy.

[0019] According to another aspect of the present invention, the present invention provides an intelligent operation and maintenance method for large model services based on graph engineering, applied to the system described in any of the preceding claims, the method comprising the following steps: S1. Receive operation and maintenance task instructions, calculate the causal correlation degree between expert agents in each domain and the current task alarm node based on the pre-built operation and maintenance causal graph, assign the task to the target expert agent according to the causal correlation degree, and output the feasible action mask under the current task type simultaneously. S2. Receive the list of atomic tasks output by the target expert agent, call the standardized node template library to generate task graph nodes, dynamically construct directed edges based on the data dependencies between the task graph nodes, and generate a nested task graph topology that adapts to the three-layer nested topology architecture. S3. Based on the dependencies of the task graph topology, within the sub-action space of the feasible action mask constraint, each of the task graph nodes is scheduled and executed according to the real-time computing resource load using a reinforcement learning scheduler. S4. When the task graph node fails to execute, trace the anomaly propagation path on the operation and maintenance causal graph, locate the root dependent variable and calculate the minimum intervention set; map the minimum intervention set to the manifold constraint of the repair parameter space, perform Bayesian optimization search in the causally aligned repair parameter subspace, and output the optimal repair parameter combination; S5. Based on the location confidence of the root cause of the fault, the repair experience is stored differentially in the retrieval enhancement knowledge base or shared counterexample library of the corresponding domain expert agent; and the location error is converted into a regularization penalty term of the routing agent gating network to update the network weight of the routing agent gating network.

[0020] The beneficial effects of the technical solution provided by this invention are as follows: a. A three-layer nested graph topology is adopted, consisting of a top-level causal graph, a middle-level six-type business sub-graphs, and a bottom-level standardized execution nodes. This architecture unifies gateway scheduling, computing power operation and maintenance, model evaluation, monitoring and early warning, disaster recovery backup, and fault self-healing under the same graph architecture. Combined with data-driven dynamic directed edges and parallel execution of dependent nodes, this eliminates the linear blocking of the traditional ReAct paradigm and achieves autonomous closed-loop operation and maintenance throughout the entire process. b. The action mask output by the routing agent directly constrains the action space of the reinforcement learning scheduler, avoiding invalid decisions from the source; the causal minimum intervention set rewrites the Bayesian optimization kernel function, which mathematically prunes the search space to a causal aligned subspace, greatly improving the convergence speed of fault repair; the triple coupling synergy produces a superlinear effect, rather than a simple superposition of the algorithms; c. The fault location error is transformed into a regularization penalty term of the routing gating network. The failure expert route probability decays exponentially over time. The downstream execution result dynamically reconstructs the upstream optimization target, forming a closed-loop control loop. This allows the system to autonomously iterate the operation and maintenance strategy without manual parameter tuning, continuously reducing the recurrence rate of faults. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 A schematic diagram of the overall architecture of a three-layer nested graph topology is provided as an exemplary embodiment of the present invention; Figure 2 A schematic diagram of the overall process of the operation and maintenance system provided as an exemplary embodiment of the present invention; Figure 3 A schematic diagram of the operation and maintenance cause-effect graph topology provided as an exemplary embodiment of the present invention; Figure 4 A schematic diagram of a causal effect adjacency matrix is ​​provided as an exemplary embodiment of the present invention; Figure 5A schematic diagram of the operation and maintenance causal graph topology (sampling) during the construction and training of the operation and maintenance causal graph, provided as an exemplary embodiment of the present invention; Figure 6 A schematic diagram of the causal effect weight distribution and online intervention verification results in the construction and training of the operation and maintenance causal graph, as provided in an exemplary embodiment of the present invention; Figure 7 A schematic diagram of experimental parameters provided for an exemplary embodiment of the present invention; Figure 8 A schematic diagram of adjustment efficiency is provided for an exemplary embodiment of the present invention; Figure 9 A schematic diagram of average repair time provided for an exemplary embodiment of the present invention; Figure 10 A schematic diagram illustrating the root sound localization accuracy provided for an exemplary embodiment of the present invention; Figure 11 This is a schematic diagram of an intelligent operation and maintenance method for large model services based on graph engineering, provided as an exemplary embodiment of the present invention. Detailed Implementation

[0023] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0024] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, apparatus, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0025] In one embodiment of the present invention, a large-scale model service intelligent operation and maintenance system based on graph engineering is provided, see appendix. Figure 1The system adopts a three-layer nested topology architecture, which includes a multi-expert hybrid decision-making top layer, a business execution middle layer containing business execution subgraphs, and an atomic execution bottom layer from top to bottom. The system as a whole relies on graph engineering modeling to realize the structured orchestration of operation and maintenance tasks, fault causal reasoning, and knowledge iteration closed loop. The system includes functional modules mounted on the three-layer nested topology architecture and collaboratively completing the entire process of operation and maintenance closed loop. The functional modules include: a causal perception multi-agent routing module, a dynamic graph construction engine, an execution engine, a causal constraint adaptive convergence controller, and a memory accumulation and knowledge update module. The causal perception multi-agent routing module is deployed at the top layer of the multi-expert hybrid decision-making system. It is used to complete the causal association matching between tasks and expert agents, realize the intelligent assignment of operation and maintenance tasks, and output the feasible action mask of the corresponding operation and maintenance tasks. The dynamic graph construction engine is used to dynamically construct the topological connection relationship between task graph nodes based on the atomic task dependency relationship and node data dependency relationship of the operation and maintenance task, generate a nested task graph topology structure that adapts to the three-layer nested topology architecture, and realize the hierarchical linkage relationship of the three-layer nested topology architecture. The execution engine is configured with reinforcement learning dynamic scheduling capability, which is used to complete the scheduling and automated execution of the business execution subgraph and the atomic execution unit corresponding to the atomic execution underlying layer under the constraints of the feasible action mask, combined with real-time resource load; The causal constraint adaptive convergence controller is used to complete fault and anomaly tracing and root cause localization based on the top-level causal reasoning mechanism. It outputs the optimal repair parameters through the optimized search of causal subspace constraints, and realizes fault adaptive convergence repair at the full topology level. The memory accumulation and knowledge update module is used to accumulate operation and maintenance repair experience, update the top-level retrieval knowledge base, and iteratively optimize the routing strategy to achieve continuous learning and closed-loop knowledge update for large-scale model operation and maintenance scenarios.

[0026] This invention achieves intelligent assignment of operation and maintenance tasks based on causal correlation and outputs feasible action masks through a causal-aware multi-agent routing module. A three-layer nested task graph topology is generated using a dynamic graph construction engine. An execution engine, combined with a reinforcement learning scheduler, executes task graph nodes based on resource load under mask constraints. A causal-constrained adaptive convergence controller traces anomalies and locates root causes when task graph nodes fail, performing Bayesian optimization within the causal alignment subspace to obtain optimal repair parameters. A memory accumulation and knowledge update module then accumulates repair experience and converts the location error into a regularization penalty term for the routing agent gating network, iteratively updating routing weights. This enables interpretable routing assignment for large-scale model operation and maintenance tasks, dynamic graph topology construction, resource-aware scheduling, causal-driven adaptive fault repair, and closed-loop self-iteration of operation and maintenance experience. It overcomes the shortcomings of traditional multi-agent allocation in operation and maintenance, static graph orchestration, blind retries in fault repair, and the inability of operation and maintenance experience to optimize decisions in reverse, achieving autonomous closed-loop intelligent operation and maintenance across the entire large-scale model service chain.

[0027] In one embodiment of the present invention, a large-scale model service intelligent operation and maintenance system based on graph engineering is provided, see appendix. Figure 1 , 2 The system employs a three-layer nested topology architecture. The top layer consists of a multi-expert hybrid decision-making layer based on an operational causal knowledge graph to calculate causal distance and generate action masks, as well as a RAG retrieval enhancement module. The middle layer comprises six business execution subgraphs: gateway intelligent scheduling, heterogeneous hardware operation and maintenance, fully automated model evaluation, monitoring and early warning, disaster recovery backup operation and maintenance, and fault self-healing iteration. The bottom layer consists of atomic execution units generated from standardized node templates. The system also includes a dynamic graph construction engine, an execution engine, a causal constraint adaptive convergence controller, and a memory accumulation and knowledge update module to achieve causal perception routing, dynamic construction of task graphs, fault root cause location and repair, and closed-loop iteration of operation and maintenance knowledge.

[0028] The specific structure, algorithm, and parameter configuration of each module are as follows: This embodiment is deployed on a multi-machine heterogeneous computing cluster, including various GPU computing nodes such as NVIDIA A100, H100, H20, 4090, and Ascend910. A unified large-model gateway service is set up at the front end, covering general-purpose large language models and multimodal models, and providing complete service operation, resource scheduling, and fault self-healing capabilities. The system adopts a three-layer nested graph topology architecture, enabling dynamic node construction, automatic edge generation, RAG knowledge retrieval, dynamic context updates, multi-expert hybrid intelligent agent collaborative scheduling (including causal-aware routing), causal inference-enhanced fault diagnosis, an RL-RDS intelligent scheduler (including action mask constraints), an ACC-BO adaptive convergence controller (including causal pruning kernel functions), and a Graph-Loop bidirectional convergence iterative framework. A standardized operation and maintenance node schema is configured, and structured management of the entire data log is enabled.

[0029] The dynamic graph building engine has a built-in standardized node template library. Based on the abstract definition of the full-link operation and maintenance business of the large model gateway, it defines six types of core standardized nodes. These six core standardized nodes are related to the six types of middle-layer business sub-nodes. Figure 1 With a one-to-one correspondence, all nodes share a unified structured input / output, data format, execution logic, and status identifier. Each node's schema uniformly includes seven fields: node type, input parameters, output metrics, execution permissions, dependency conditions, convergence threshold, and status label.

[0030] The node creation logic is as follows: After the system receives the operation and maintenance target, the multi-expert intelligent agent automatically parses the target semantics and assigns it to the most matching domain expert intelligent agent. The domain expert intelligent agent then breaks down the atomic operation and maintenance tasks, matches them with the corresponding standardized node templates, automatically initializes the node parameters, binds execution permissions, sets the convergence threshold, and completes the dynamic node construction.

[0031] The dynamic graph construction engine uses a self-developed dynamic dependency resolution algorithm to automatically identify data dependencies between nodes and generate directed topological edges. The directed edge identifier is defined as:

[0032] in, For nodes i Pointing to node j The directed edge identifier (1 indicates that an edge exists, 0 indicates that no edge exists). For the preceding node i The output dataset; The input dataset for the subsequent node j; For the preceding node i The execution status.

[0033] The edge construction rule is: only the current node outputs data. It can cover the input requirements of subsequent nodes. And the execution status of the preceding node. Upon success, directed data flow edges are automatically generated; edges are not generated for nodes without data dependencies, and execution is performed in parallel by default, completely eliminating invalid serial waiting. It also supports conditional dynamic edges, dynamically activating or deactivating edge links based on real-time monitoring metrics and fault status, achieving adaptive topology adjustment.

[0034] The system architecture consists of a three-layer nested dedicated graph architecture, comprising a top-level decision master graph, a middle-level business sub-graph, and a bottom-level execution node graph. It integrates context engineering, RAG knowledge retrieval, and data log management to form a complete closed-loop operation and maintenance link.

[0035] Top-level multi-expert hybrid decision-making layer: The top layer is responsible for global scheduling, semantic parsing, task distribution, and multi-expert collaborative decision-making capabilities. It deeply integrates context engineering and RAG retrieval enhancement technology, and introduces a hybrid expert architecture to replace the traditional monolithic intelligent agent mode. The system deploys a lightweight routing intelligent agent and multiple domain expert intelligent agents. After receiving the user's operation and maintenance goals, the routing intelligent agent first performs semantic parsing and intent recognition, and then calculates the causal correlation between each domain expert and the current fault based on the alarm type of the current task and the operation and maintenance cause-effect graph, routing the task to the most matching domain expert intelligent agent. When the complexity of the task exceeds the capabilities of a single expert, multiple expert intelligent agents can collaborate to handle the same task.

[0036] The causal awareness routing logic is as follows: the calculation of the routing gated network introduces operational causal constraints, and its formula is:

[0037] The final route probability is determined by Softmax Output:

[0038] in, Experts The causal distance between the covered area and the current alarm node in the causal graph, where λ is the causal constraint weight coefficient. For maintenance tasks T Encoding vector For the first i The gating network weight vectors corresponding to each domain expert agent. The causal distance is defined as: in the operation and maintenance causal graph... In the process, tracing back along the causal edge from the Alert node to the expert... The shortest directed path length of the covered variable set; if there are multiple paths, the reciprocal of the sum of the causal effect weights on the paths is taken as the distance metric.

[0039] The action space constraint output is as follows: After the routing agent completes the expert assignment, it synchronously outputs the feasible action mask under the current task type. This mask dynamically masks behaviors in the scheduler that are not applicable to the current expert domain. The scheduler's deep Q network only selects policies within the sub-action space scaled by the mask.

[0040] Context engineering adaptation involves the system automatically collecting all context information, including historical operation and maintenance dialogues, cluster running status, hardware configuration information, gateway rule parameters, and historical fault records. It also involves structured organization of context constraints to provide complete background support for task splitting, node parameter initialization, and edge relationship construction.

[0041] RAG knowledge empowerment involves: building a dedicated RAG knowledge base for operations and maintenance, accumulating historical operations and maintenance solutions, fault handling experience, optimal resource scheduling strategies, and model optimization rules; after receiving the operations and maintenance objectives, the domain expert intelligent agent automatically searches the knowledge base to match similar scenarios, reuses mature operations and maintenance strategies, and dynamically adjusts the topology structure and execution parameters in combination with real-time context.

[0042] Mid-level business execution subgraphs: Based on top-level multi-expert collaborative decision-making, six types of dedicated business subgraphs are generated: gateway intelligent scheduling subgraph, heterogeneous hardware operation and maintenance subgraph, fully automated model evaluation subgraph, monitoring, perception, and early warning subgraph, disaster recovery and backup operation and maintenance subgraph, and fault self-healing iteration subgraph. Each subgraph runs independently, shares data, and does not interfere with each other. Internally, the underlying nodes are connected by dynamic directed edges. Dependency-free subgraphs execute in parallel, relying on a diamond-shaped structure. Fan-out-Fan-in Topology enables efficient processing of batch tasks.

[0043] The underlying execution node unit consists of six types of standardized nodes, forming the smallest execution unit. All nodes operate independently and in isolation; a single node failure only affects that node and does not impact the overall topology. During node execution, runtime data, operation logs, and exception information are collected in real time and synchronously uploaded to the memory node, achieving full-process structured management of data and logs.

[0044] The system constructs a graph-driven full lifecycle data management system. All operation and maintenance data is stored in a structured manner in the form of graph node attributes and edge data streams. Each piece of data is bound to a corresponding execution node, timestamp, context scenario, and execution log, forming a traceable, searchable, and reusable operation and maintenance data graph.

[0045] A dedicated log collection and governance mechanism is established, enabling real-time collection of all process node execution logs, topology change logs, policy adjustment logs, and fault repair logs. These logs are automatically structured, anonymized, and categorized for storage. Logs are bidirectionally bound to graph topology and operations tasks, supporting precise retrieval based on nodes, tasks, time, and fault type. Simultaneously, log data is synchronized in real-time to the RAG knowledge base, serving as a core data source for accumulating operational experience.

[0046] Breaking away from the traditional fixed context loading mode, it achieves dynamic incremental context updates. In each iteration of the graph, it automatically updates context data such as cluster status, task progress, fault information, and resource reserves, and corrects node execution parameters and edge link relationships in real time. For complex operation and maintenance scenarios, it automatically filters highly correlated context information and filters invalid and redundant data to ensure the accuracy and efficiency of the agent's decision-making.

[0047] Based on structured operation and maintenance data and logs, an incrementally updated RAG knowledge base is built, comprising four major modules: scenario library, fault library, policy library, and indicator library. Before each decision, the agent uses semantic retrieval to match the current operation and maintenance scenario, fault type, and resource status, and retrieves the optimal historical policy; at the same time, it incrementally archives the results, logs, and data of newly executed tasks, continuously expanding the knowledge base and enabling autonomous iterative upgrades of operation and maintenance capabilities.

[0048] In this embodiment of the invention, the "constraint domain" refers to the feasible subspace for downstream module optimization search or strategy selection, determined by the output of the upstream module. The boundary of this subspace is defined by causal relationships or expert domain knowledge, and the search algorithm defined thereon is mathematically constrained to not explore outside the boundary. Formally, let the constraint of the upstream module's output on the downstream decision space be a feasible domain C, where Ω is the original decision space of the downstream module, and the constraint domain C is defined by the set of constraint functions. Definition:

[0049] in, m The total number of constraint functions. The threshold parameter is determined by the output of the upstream module. No. j The constraint functions mean that the search algorithm of the downstream module is mathematically constrained to operate within C, which... \C (Set difference, universal set) The outputs subtracted from the feasible set C are defined as invalid or unexecutable.

[0050] The algorithm forms an inseparable technical whole through the generation, propagation, and feedback of the constraint domain, specifically manifested in the following triple coupling mechanism: The first coupling (MoERL-RDS): When the routing agent completes expert assignment, it synchronously outputs a feasible action mask. This mask directly serves as the action space constraint for the reinforcement learning scheduler, so that the output of MoE is no longer the task label, but the feasible domain boundary of the RL policy, thus avoiding ineffective scheduling decisions from the source.

[0051] The second coupling (Causal-ACCACC-BO): The minimum set of interventions output by the causal inference engine is transformed into manifold constraints for repairing the parameter space. The causal graph information is encoded as the inherent prior of the Bayesian optimization algorithm and solidified into the kernel function property, so that the optimization algorithm mathematically loses its ability to explore non-causal paths, rather than just serving as a starting point for heuristic search.

[0052] The third coupling (Causal-ACCMoE-RAGMoE): After the repair is completed, the positioning error calculated by the causal inference engine does not directly update the knowledge base text, but is transformed into a regularization penalty term of the routing agent gating network. The downstream execution result dynamically reconstructs the mathematical optimization objective of the upstream route, forming a closed-loop control loop in a strict sense.

[0053] The specific mathematical implementation of the above triple coupling is as follows: Resource-Aware Dynamic Scheduler Based on Reinforcement Learning (RL-RDS) Regarding MDP modeling: The graph task scheduling problem is modeled as a Markov decision process – state space: the system's state at time t is a high-dimensional vector, containing the graph topology and node states, real-time resource view, and ready queue states.

[0054] in, For the task diagram, For resource view, This is the ready task queue.

[0055] Action space (constrained by action mask): At each decision time step, the scheduler selects a task from the ready queue and executes the action:

[0056] in, For the scheduler at time t The chosen action As an action type, The selected task number.

[0057] Action selection is constrained by the mask output by the MoE module, and the strategy is constrained to the sub-action space marked as feasible by the mask:

[0058] in, For a moment t state, For a single action, For the action space, For a moment t Feasibility action mask function.

[0059] Task-aware reward function: The reward function weights are designed to be dynamically adjustable, and are dynamically generated by the routing agent based on the current task type and the assignment expert.

[0060] The dynamic weight vector is generated as follows:

[0061] in, For a moment t The immediate reward value of the lower Markov decision process; the higher the reward, the better the overall performance of the current task scheduling scheme. 、 、 、 Let be the four dynamic weights of the reward function at time t. The cluster calculates resource utilization at time t. For a moment t Total observed task completion time (span). For a moment t The scheduling penalty cost, For a moment t Average waiting time of tasks in the ready queue (representing the degree of task starvation). For a moment t The dynamic weight vector of the reward function, For normalization function, It is a multilayer perceptron network. For a moment t Operation and maintenance tasks semantic encoding vector, To assign the processing of the current task k Feature vectors of domain expert agents.

[0062] That is, the scheduling optimization focus of different experts on the same task varies, with different computing power scheduling experts prioritizing resource utilization. Higher; repair tasks triggered by fault diagnosis experts have a higher waiting time weight. Reduce the priority to ensure repair speed. The weight coefficients are determined by Pareto front scanning: In the offline simulation environment, multiple rounds of scheduling simulation are performed on each weight combination to draw a three-dimensional Pareto front of resource utilization, completion time, and task starvation rate. The weight combination corresponding to the inflection point on the front is selected as the default configuration.

[0063] The objective is to maximize the cumulative discount reward: the algorithm implementation employs depth... Q The Distributed Qualification Network (DQN), as a learning framework for scheduling strategies, has the following update formula:

[0064] in, For experience replay collection D The mathematical expectation of the action selection is affected by the mask. constraint, Only The maximum value is taken within the action subspace where =1. For the target Q-network output, This is the current output of the master Q network, representing the execution of a scheduling action in state s. Action value, network parameters are , To perform the action The instant reward received afterwards To reinforce the learning discount factor, the value is (0 < <1), For the next state Traverse all available actions Take the largest Q value.

[0065] For example, the specific parameter configuration of the RL-RDS scheduler is as follows: The Q-network structure is a three-layer fully connected neural network with hidden layer dimensions of [512, 256, 128]. ReLU Activation function; experience replay pool capacity 100,000 records; discount factor =0.95; ε-greedy exploration strategy with initial ε=1.0, decaying to a minimum of 0.01 every 1000 steps; batch size 64; learning rate 1×10⁻⁶. -4 The target network synchronization interval is 1000 steps.

[0066] The adaptive convergent controller with Bayesian optimization (ACC-BO) models the fault repair parameter optimization problem as a black-box optimization problem:

[0067] in X To repair the parameter space, The cost function for evaluating the repair effect.

[0068] The core principle of Bayesian optimization is to construct a Gaussian process. f The surrogate model uses a sampling function to balance exploration and exploitation, and sequential sampling efficiently approximates the global optimum. The Gaussian process prior is:

[0069] in, It is a mean function. Let covariance function be used. It is a Gaussian process.

[0070] The modified implementation uses a causal pruning kernel function (i.e., the specific mathematical implementation of the second layer of coupling):

[0071] in, For standard kernel functions, For parameter points x With minimal intervention set Causal alignment This is the causal alignment threshold. That is, within the standard... Multiply the kernel by the causal alignment indicator function, when the parameter x With minimal intervention set Causal alignment score CausalAlign When the value is greater than the threshold δ, the indicator function is set to 1; otherwise, it is set to 0. The prior confidence of the Bayesian optimization is directly set to zero in the region outside the causal alignment direction, mathematically eliminating the possibility of the optimization algorithm exploring non-causal paths. The desired improvement in the acquisition function is:

[0072] in, The optimal value observed so far. For mathematical expectation operators, For parameter points x The objective function value at that point, ,like If it is better than the current best value (the difference is positive), take the difference; otherwise, take 0 (if there is no improvement, the improvement amount is 0).

[0073] For example, the ACC-BO controller parameters are configured as follows: Gaussian process kernel function for causal pruning 5 / 2 cores, initial length scale l=0.1, causal alignment threshold δ=0.3; acquisition function is desired improvement (EI); maximum number of iterations 50; minimum improvement threshold 1×10 -4 Weights of the causal perception term in the convergence criteria parameters α, β, η The convergence threshold ε = 0.01.

[0074] Multi-Expert Hybrid Agent Cooperative Scheduling (MoE-Agent): This involves a gated network design that deploys multiple domain expert agents. The routing agent assigns tasks based on task type and a causal graph. The gated network computation introduces causal constraints, i.e., the aforementioned... Final route probability The gated network employs hard routing and load balancing constraints.

[0075] After the differential knowledge is accumulated and repaired, the causal inference engine evaluates the accuracy of root cause localization: if accurate, the complete causal chain and repair strategy are stored in the RAG knowledge base of the fault diagnosis expert; if the deviation is large, the failed path is stored in the shared counterexample library of all experts for subsequent adjustment of the gating weights of the routing agent.

[0076] The mathematical implementation of the third coupling is as follows: Let the first i The mean causal localization bias of the experts within the time window was: The corresponding forgetting factor is defined as:

[0077] Where σ is Sigmoid function, β This is the forgetting rate coefficient. Let be the tolerance threshold for positioning deviation. The loss function of the routing gating network is expanded to:

[0078] in Classify the losses for the task. For load balancing losses, and η These are weighting coefficients. Let be the gating network weight of the i-th expert. Experts with high failure rates are subject to greater weight decay penalties in the gating network, and their routing probability decays exponentially over time. When an expert's frequent misjudgments result in a large number of failed experiences being written into the negative example library, the gating network increases the expert's forgetting factor during backpropagation, causing the probability of that expert being routed subsequently to decay exponentially over time until new successful experiences restore their confidence.

[0079] For example, the parameter configuration of a multi-expert intelligent agent system is as follows: The routing agent is fine-tuned based on the LLaMA-3 open-source large language model, and the gating network is trained independently of the base model. The initial deployment consists of 5 domain experts (computing power scheduling, model evaluation, fault diagnosis, gateway policy, and security audit), which can be dynamically expanded to a maximum of 20 during runtime. The gating network is a two-layer fully connected neural network (512256N). ReLU The activation function, softmax, outputs the route probability, and the load constraint parameters; the causal constraint weight coefficient λ=0.6 in the causal-aware routing parameters, and the causal distance threshold. =5.

[0080] A causal inference-enhanced fault diagnosis and adaptive repair algorithm (Causal-ACC) The feasible strategy for constructing the operation and maintenance cause-effect graph is a four-step process that primarily uses offline learning and secondarily uses online verification: 1. Offline phase: Utilize historical operation and maintenance logs (time-series data + fault events) to construct a preliminary causal framework through time-series causal discovery (PCMCI algorithm); 2. Knowledge Graph Injection: Using prior physical topology in the operations and maintenance domain (such as the physical dependency between GPU and inference service) as hard constraints, the skeleton of offline learning is trimmed and modified; 3. Online verification: Small-scale intervention verification is only conducted in low-risk scenarios (shadow traffic, gray-scale environment), rather than full-scale online A / B testing; 4. Incremental update: After each successful fault repair, the actual intervention effect is fed back as the basis for Bayesian update of the edge weights of the causal graph.

[0081] Root cause localization and minimum intervention calculations include: Step 1: Anomaly Propagation Path Tracing. Load the operation and maintenance cause-effect graph, start from the fault node and backtrack along the causal edges to calculate the causal effect of each antecedent variable. The causal effect is defined as:

[0082] in, This means that intervention is applied to variable X to make it take the value x. The baseline value is used. For enforcement order Then, the probability distribution of Y, Let Y be the probability distribution under the baseline parameters.

[0083] Step 2: Minimal Intervention Set Calculation. Based on the backdoor and frontdoor criteria, the minimum intervention set is defined as the smallest subset of variables that satisfies the intervention effect:

[0084] in, Let I be the candidate set consisting of all modifiable variables in the operation and maintenance causal graph. The selected candidate intervention subset; |I| is the number of elements in the candidate intervention subset, which serves as the optimization objective to minimize the intervention scale; This indicates that causal intervention is applied to all variables in set I; Let Y be the probability distribution of the outcome variable Y under the current fault state; The target probability distribution of the outcome variable Y is defined as follows: the constraint condition requires that the probability distribution of Y after intervention causally approach the target distribution.

[0085] Step 3: Causal-driven search space pruning. Transforming into manifold constraints that repair the parameter space, the reduced parameter space is:

[0086] in, This is the original, complete repair parameter space; This is a candidate parameter vector in the parameter space; For causal alignment function; Preset causal alignment threshold; Let be the reduced parameter space obtained after pruning.

[0087] Only in the repair parameter subspace aligned with the minimal set of interventions for causality Internal search significantly reduces the number of search dimensions.

[0088] Corrected convergence criteria for causal perception Add a causal perception term to the convergence coefficients:

[0089] in, For state changes, It is the change in energy. α represents the change in causal structure, and β and η are weighting coefficients. The calculation method is as follows:

[0090] When the repair operation successfully intervenes in the root cause path Approaching , ≈0, Approaching zero; conversely, if the repair only improves the indicator at the observational level without changing the causal structure, Keep a large value. If the value remains relatively large, it can be used to determine whether the repair has truly converged.

[0091] For example, the parameters of the causal inference system are configured as follows: In the offline phase, a preliminary causal framework is constructed using 30 days of historical operation and maintenance logs based on the PCMCI algorithm; a knowledge graph is injected into the GPU-Gateway-Service three-level dependency graph as a hard constraint; the initial node size of the causal graph is about 300 operation and maintenance variables, which can be increased during runtime; the causal effect threshold τ=0.15, and root cause relocation is triggered only when the causal effect deviation exceeds τ; the causal weight coefficient β=0.7.

[0092] The simulation environment was configured as follows: simulating 128 heterogeneous computing nodes (A100×32, H100×32, H20×16, RTX4090×32, Ascend910B×16), interconnected via a simulated 10GbE network; the task types included 40% model evaluation tasks, 25% computing power inspection tasks, 20% fault repair tasks, and 15% gateway policy update tasks; five common faults (GPU / NPU memory overflow, inference latency spikes, model accuracy degradation, network partitioning, and service process deadlock) were randomly injected according to a Poisson distribution, with an average fault interval of 300 seconds; each round of the simulation ran continuously for 72 hours, repeated for 10 rounds and the average was taken.

[0093] The results of the operation and maintenance cause-effect graph construction and causal effect analysis are attached. Figure 3 , 4 As shown in the attached diagram, the process of constructing and training causal relationships in operation and maintenance is as follows. Figure 5 , 6 As shown in the attached figure, the various indicators during training are as follows. Figure 7 As shown in the attached diagram, the ablation experiment results are as follows: Figure 8 , 9 As shown in Figure 10, the experimental results show that the performance improvement of the complete system is not the arithmetic sum of the gains of each variant, but rather a result of the decoupling of the scheduling strategy of RL from the spatiotemporal complexity of causal root cause localization through the constraint domain generation-optimization mechanism, which produces a better synergistic effect.

[0094] The feasible action mask is a task-level static mask that remains unchanged throughout the lifecycle of a single maintenance task. For tasks that run for an extremely long time, the system supports incremental recalculation of the mask, whereby the routing agent regenerates the mask based on the latest causal graph state and synchronizes it to the scheduler.

[0095] Beneficial technical effects of this embodiment Architectural innovation significantly improves operational efficiency and computing power utilization: By using Graph Engineering parallel topology to achieve non-blocking parallel execution of multiple tasks, the problem of single-fault cascading interruption is completely solved, and the operational efficiency at scale is significantly improved compared with traditional solutions, and the utilization rate of heterogeneous computing power resources is greatly increased.

[0096] Intelligent scheduling effectively alleviates resource contention and task hunger: By using the RL-RDS reinforcement learning scheduler (constrained by action mask), it achieves globally optimal scheduling decisions in resource-constrained heterogeneous computing scenarios. Compared with traditional fixed-rule scheduling, the average waiting time of tasks is significantly reduced, the response speed of high-priority tasks is greatly improved, and the resource fragmentation rate is effectively reduced.

[0097] Adaptive repair significantly improves fault recovery efficiency and quality: By using the ACC-BO adaptive convergence controller (the search space is constrained by the causal pruning of the kernel function), fault repair is upgraded from fixed strategy retry to intelligent sequential optimization. Compared with the traditional fixed retry scheme, the fault repair success rate is significantly improved, the average number of repair iterations is greatly reduced, and the MTTR is shortened to less than one-third of the traditional scheme.

[0098] Refined division of labor across domains significantly improves the accuracy of long-tail task processing: Through a multi-expert hybrid intelligent agent collaborative scheduling mechanism (including causal perception routing logic), operation and maintenance tasks in different domains are assigned to expert intelligent agents in their respective domains for processing. Compared with single intelligent agent solutions, the decision-making accuracy of long-tail operation and maintenance problems is significantly improved, and the degree of knowledge specialization and the breadth of domain coverage are significantly enhanced.

[0099] Causal diagnosis improves both root cause localization accuracy and repair efficiency: Through a fault diagnosis algorithm enhanced by causal inference (adopting an engineering feasible strategy of offline learning as the main method and online verification as a supplement), a paradigm upgrade from correlation diagnosis to causal diagnosis is achieved. In complex fault scenarios, the root cause localization accuracy is significantly improved, the false repair rate is greatly reduced, and the fault recurrence rate is effectively reduced.

[0100] Data empowerment significantly improves the intelligence of operation and maintenance and the accuracy of decision-making: By integrating context engineering, enhanced RAG retrieval, and full-link data log management, the structured accumulation, reuse, and iteration of operation and maintenance experience are realized, which completely solves the problems of traditional intelligent agent decision-making that lacks data support and is highly random.

[0101] Deeply coupled algorithms form a superlinear synergistic effect: The four core algorithms form a deeply integrated technical whole through constraint domain generation and constraint domain optimization mechanisms. The preceding algorithm generates the constraint domain, and the subsequent algorithm optimizes within the constraint domain. Moreover, the optimization result corrects the constraint generation logic of the preceding algorithm. Ablation experiments prove that the performance improvement of the complete system far exceeds the arithmetic sum of the gains of each module.

[0102] End-to-end closed-loop operation, significantly reducing manual operation and maintenance costs: It achieves autonomous closed-loop operation of the entire process from task triggering, topology construction, parallel execution, fault repair, data accumulation, and strategy optimization, without the need for manual step-by-step operation, manual decision-making, and manual review. 24 / 7 unattended operation and maintenance is fully implemented, significantly reducing manual operation and maintenance costs and greatly improving fault self-healing response speed.

[0103] The architecture is highly versatile, scalable, and adaptable: the customized graph architecture, multi-expert collaboration mechanism, and causal inference framework serve as a general operation and maintenance foundation, which can be quickly adapted to various large models, multimodal models, heterogeneous computing power clusters and gateway architectures, and can be extended to various AI operation and maintenance scenarios, with extremely high engineering implementation value and scenario adaptability.

[0104] This paper presents an intelligent operation and maintenance method for large model services based on graph engineering. See [link to relevant documentation]. Figure 11 This method is applied to the intelligent operation and maintenance system for large model services based on graph engineering, as described above. The execution process of this method is as follows: S1. Receive external input operation and maintenance task instructions to trigger the system operation and maintenance process; the causal perception multi-agent routing module retrieves the pre-built operation and maintenance causal graph, takes the alarm node corresponding to the current task as the starting point, traverses the operation and maintenance causal graph in reverse, and calculates the causal correlation degree between each domain expert agent and the alarm node; the routing agent gating network calculates the routing probability of each domain expert agent based on the calculated causal distance, and distributes the operation and maintenance task to the target expert agent with the highest matching degree according to the routing probability; at the same time, the routing agent gating network outputs the feasible action mask corresponding to the current task type, limits and filters the action space that the subsequent scheduler can execute, and blocks invalid scheduling actions that are irrelevant to this task.

[0105] S2. After receiving the assigned operation and maintenance task, the target expert agent decomposes the overall operation and maintenance task and outputs a list of atomic tasks that can be executed independently. The dynamic graph construction engine obtains this list of atomic tasks, calls the internal standardized node template library, matches the corresponding standardized node template for each atomic task, instantiates and generates each task graph node, and automatically completes the initialization of node input parameters, execution permission binding, and convergence threshold configuration. The dynamic graph construction engine parses the input and output data dependencies between all task graph nodes. Only when the output dataset of the current task graph node can cover the input requirements of the subsequent task graph node, and the execution status of the current task graph node is successful, will a directed data flow edge be automatically generated between the nodes. Task graph nodes that do not have data dependencies with each other will not generate directed edges and will be configured to execute in parallel by default. At the same time, combined with real-time monitoring indicators and fault status, the conditional edge links are dynamically activated or deactivated to adaptively adjust the topology structure, and finally generate a three-layer nested task graph topology containing a top-level decision graph, a middle-level business subgraph, and a bottom-level execution node.

[0106] S3, the execution engine's internal resource-aware dynamic scheduler, based on reinforcement learning, reads the complete three-layer nested task graph topology and simultaneously collects real-time cluster computing resource load information and ready task queue status. The scheduler strictly adheres to the data dependency constraints defined in the task graph topology and performs task selection within the sub-action space of the feasible action mask constraints output by S1, allowing scheduling only for tasks marked as feasible. Combining the dynamically generated reward function weights during the Markov decision process, it comprehensively considers indicators such as resource utilization, task completion time, and task starvation rate to complete the resource allocation and scheduling execution of each task graph node. During node operation, it continuously reports node running status, output indicators, and execution logs.

[0107] S4. Continuously monitor the running return status of each task graph node. When it is determined that the task graph node has failed to execute, trigger the fault handling process of the causal constraint adaptive convergence controller. The causal intervention positioning unit loads the operation and maintenance causal graph, traces the complete abnormal propagation path along the causal edge from the fault alarm node, calculates the root cause variable corresponding to the fault in combination with the causal effect, and solves to obtain the minimum intervention set that can eliminate the impact of the fault. The parameter space mapping unit maps the minimum intervention set to the manifold constraint of the repair parameter space, modifies the Gaussian process covariance function of the Bayesian optimization algorithm, and limits the optimization search to the causal alignment indicator function within the causal alignment repair parameter subspace. Iterative optimization search is performed in this subspace to find the optimal repair parameter combination, and the fault repair operation is carried out using this set of repair parameters.

[0108] S5. After the fault repair operation is completed, the causal intervention localization unit outputs the localization reliability of this root cause analysis; the memory accumulation and knowledge update module performs differentiated experience storage based on the localization reliability: if the localization reliability reaches the preset threshold, the causal link, fault phenomenon, and repair strategy are stored in the retrieval enhancement knowledge base of the corresponding domain expert agent; if the localization reliability is lower than the preset threshold, the failed localization path of this fault is stored in the system's shared counterexample library; the average localization error of each domain expert agent within the set time window is calculated, and a data set with localization reliability is generated. Sigmoid The forgetting factor with exponential decay characteristics transforms the localization error into a regularization penalty term for the routing agent gating network, which is then weighted and expanded to update the weights of the routing agent gating network. This causes the probability of subsequent route assignment for a domain expert agent that has failed to locate multiple times to decay exponentially over time, until new successful operation and maintenance experience re-enhances the confidence of the expert agent, thus completing a full operation and maintenance loop and realizing the autonomous iterative optimization of the system's operation and maintenance strategy.

[0109] In this embodiment, a complete closed-loop process is used to solve the problems of randomness in multi-agent allocation, process serial blocking, blind retrying of faults, and inability to iterate backwards on experience in traditional large-scale model operation and maintenance. This is achieved by using a complete closed-loop process of causal perception task assignment, dynamic three-layer nested task graph topology construction, reinforcement learning resource scheduling constrained by mask, fault parameter optimization and repair under causal constraints, and reverse updating of the routing network based on fault experience. This enables interpretable, autonomous closed-loop, and adaptive iteration of large-scale model cluster operation and maintenance, reducing the need for manual operation and maintenance intervention.

[0110] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0111] The above description is only a specific embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A large-scale model service intelligent operation and maintenance system based on graph engineering, characterized in that, The system adopts a three-layer nested topology architecture, which includes a multi-expert hybrid decision-making top layer, a business execution middle layer containing business execution subgraphs, and an atomic execution bottom layer from top to bottom. The system as a whole relies on graph engineering modeling to realize the structured orchestration of operation and maintenance tasks, fault causal reasoning, and knowledge iteration closed loop. The system includes functional modules mounted on the three-layer nested topology architecture and collaboratively completing the entire process of operation and maintenance closed loop. The functional modules include: a causal perception multi-agent routing module, a dynamic graph construction engine, an execution engine, a causal constraint adaptive convergence controller, and a memory accumulation and knowledge update module. The causal perception multi-agent routing module is deployed at the top layer of the multi-expert hybrid decision-making system. It is used to complete the causal association matching between tasks and expert agents, realize the intelligent assignment of operation and maintenance tasks, and output the feasible action mask of the corresponding operation and maintenance tasks. The dynamic graph construction engine is used to dynamically construct the topological connection relationship between task graph nodes based on the atomic task dependency relationship and node data dependency relationship of the operation and maintenance task, generate a nested task graph topology structure that adapts to the three-layer nested topology architecture, and realize the hierarchical linkage relationship of the three-layer nested topology architecture. The execution engine is configured with reinforcement learning dynamic scheduling capability, which is used to complete the scheduling and automated execution of the business execution subgraph and the atomic execution unit corresponding to the atomic execution underlying layer under the constraints of the feasible action mask, combined with real-time resource load; The causal constraint adaptive convergence controller is used to complete fault and anomaly tracing and root cause localization based on the top-level causal reasoning mechanism. It outputs the optimal repair parameters through the optimized search of causal subspace constraints, and realizes fault adaptive convergence repair at the full topology level. The memory accumulation and knowledge update module is used to accumulate operation and maintenance repair experience, update the top-level retrieval knowledge base, and iteratively optimize the routing strategy adopted by the causal perception multi-agent routing module to achieve continuous learning and closed-loop knowledge update in large model operation and maintenance scenarios.

2. The intelligent operation and maintenance system for large model services based on graph engineering according to claim 1, characterized in that, The multi-expert hybrid decision-making top-level configuration includes an operation and maintenance causal knowledge graph and a RAG retrieval enhancement module, which are used to realize causal distance calculation, generation of the feasible action mask, and operation and maintenance knowledge retrieval iteration, thereby achieving global intelligent decision-making for the system.

3. The intelligent operation and maintenance system for large model services based on graph engineering according to claim 1, characterized in that, The middle layer of business execution includes a gateway intelligent scheduling subgraph, a heterogeneous hardware operation and maintenance subgraph, a fully automatic model evaluation subgraph, a monitoring, perception and early warning subgraph, a disaster recovery and backup operation and maintenance subgraph, and a fault self-healing iteration subgraph, providing a business execution carrier for intelligent operation and maintenance of large models in all scenarios.

4. The intelligent operation and maintenance system for large model services based on graph engineering according to claim 1, characterized in that, The atomic execution layer consists of various atomic execution units generated from a unified standardized node template, providing standardized and reusable basic execution capabilities for each business subgraph in the middle layer of business execution.

5. The intelligent operation and maintenance system for large model services based on graph engineering according to claim 1, characterized in that, The dynamic graph construction engine is configured with a standardized node template library, which includes standardized nodes. The standardized node structure is uniformly set with fields including node type, input parameters, output metrics, execution permissions, dependency conditions, convergence threshold, and status label. The target expert agent, assigned by the causal perception multi-agent routing module, splits the received operation and maintenance task into several atomic tasks, matches the corresponding standardized node template, and automatically completes node parameter initialization, execution permission binding, and convergence threshold configuration to generate the available task graph nodes at the atomic execution layer.

6. The intelligent operation and maintenance system for large model services based on graph engineering according to claim 5, characterized in that, The specific execution rules for constructing topological connections between nodes in the task graph by the dynamic graph building engine include: When the output data of the current available task graph node can cover the input requirements of the subsequent available task graph node, and the current available task graph node is executed successfully, a directed data flow edge is automatically generated. The available task graph nodes without data dependencies do not generate link edges and are executed in parallel by default. Based on real-time system monitoring indicators and fault status, conditional edge links are dynamically activated or deactivated to achieve adaptive adjustment of the task graph topology of the three-layer nested topology architecture.

7. The intelligent operation and maintenance system for large model services based on graph engineering according to claim 3 or 6, characterized in that, The business subgraphs in the middle layer of the business execution operate independently, communicate with each other, and do not interfere with each other. Within each task subgraph, the atomic execution units at the bottom layer of the atomic execution are connected by dynamic directed edges. Business subgraphs without dependencies are executed in parallel.

8. The intelligent operation and maintenance system for large model services based on graph engineering according to claim 2, characterized in that, The causal perception multi-agent routing module relies on the top-level operation and maintenance causal knowledge graph of the multi-expert hybrid decision-making system to calculate causal correlation and allocate task routes, specifically including: In the operation and maintenance causal graph, the variable set corresponding to each domain expert agent is traced back from the current task alarm node along the causal edge. The shortest directed path length is used as the causal distance. When there are multiple paths, the reciprocal of the sum of the causal effect weights of the paths is taken as the final distance metric. The causal perception multi-agent routing module uses a routing agent gating network to calculate the routing probability of each domain expert agent based on the calculated causal distance, completes the intelligent assignment of operation and maintenance tasks, and outputs the feasible action mask.

9. The intelligent operation and maintenance system for large model services based on graph engineering according to claim 1, characterized in that, The resource-aware dynamic scheduler in the execution engine models the graph task scheduling process as a Markov decision process. The state space of the Markov decision process includes a graph topology, node states, a real-time resource view, and a ready task queue. The action space of the Markov decision process is constrained by the feasible action mask, allowing only feasible tasks to be selected and actions to be performed. The reward function weights of the Markov decision process are dynamically configured by the causal perception multi-agent routing module according to the task type and target expert agent, so that different expert agents correspond to differentiated scheduling optimization targets. The default configuration of the reward function weights is obtained by fitting the three-dimensional Pareto front of resource utilization, task completion time, and task hunger rate through offline simulation, and the weight combination corresponding to the inflection point of the front is selected as the default configuration of the reward function weights.

10. A method for intelligent operation and maintenance of large model services based on graph engineering, characterized in that, Applied to the system of any one of claims 1-9, the method comprises the following steps: S1. Receive operation and maintenance task instructions, calculate the causal correlation degree between expert agents in each domain and the current task alarm node based on the pre-built operation and maintenance causal knowledge graph, assign the task to the target expert agent according to the causal correlation degree, and output the feasible action mask under the current task type simultaneously. S2. Receive the list of atomic tasks output by the target expert agent, call the standardized node template library to generate task graph nodes, dynamically construct directed edges based on the data dependencies between the task graph nodes, and generate a nested task graph topology that adapts to the three-layer nested topology architecture. S3. Based on the dependencies of the task graph topology, within the sub-action space of the feasible action mask constraint, each of the task graph nodes is scheduled and executed according to the real-time computing resource load using a reinforcement learning scheduler. S4. When the task graph node fails to execute, trace the anomaly propagation path on the operation and maintenance causal graph, locate the root dependent variable and calculate the minimum intervention set; map the minimum intervention set to the manifold constraint of the repair parameter space, perform Bayesian optimization search in the causally aligned repair parameter subspace, and output the optimal repair parameter combination; S5. Based on the confidence level of fault root cause localization, the repair experience is differentially stored in the retrieval enhancement knowledge base or shared counterexample library of the corresponding domain expert agent; and the localization error is converted into a regularization penalty term of the routing agent gating network, and the network weight of the routing agent gating network is updated.

Images