An agent engine-based micro-service operation and maintenance fault repair method and system

Abstract

The present disclosure relates to an agent engine-based microservice operation and maintenance fault repair method and system. The method can determine the node fault probability based on the current microservice data exception, combined with the service topology characteristics and the log semantic characteristics corresponding to the application log data, to guarantee the accuracy and explainability of root cause positioning through the graph neural network and semantic model; based on the repair action sequence generated by the strategy network and the dynamic risk budget, the safety and efficiency of the operation and maintenance operation are effectively guaranteed; the agent engine performs repair operations based on the estimated execution time and the repair action sequence, and by extending the operation and maintenance special execution engine based on the general agent framework, not only the semantic loss between modules is eliminated, but also the stability of the execution success rate is guaranteed; the whole method forms a complete closed-loop process from fault perception, root cause positioning, strategy generation and risk assessment to standardized execution, significantly improves the end-to-end fault repair efficiency and success rate, and greatly shortens the repair time.

Description

Technical Field

[0001] The embodiments in this specification belong to the field of cloud computing and large model technology, and specifically relate to a microservice operation and maintenance fault repair method and system based on an intelligent agent engine. Background Technology

[0002] With the widespread adoption of microservice architecture, large-scale internet systems typically consist of dozens or even hundreds of interdependent microservices. A single business request may often involve a chain of calls across multiple service nodes, increasing system complexity and posing significant challenges to operations and maintenance. To ensure high availability and stability, intelligent operations and maintenance (AIOps) technology is widely used to leverage artificial intelligence algorithms for the detection, location, and repair of operational faults.

[0003] Existing mainstream microservice fault handling solutions are mainly divided into rule engine solutions, isolated AI model solutions, and manual orchestration solutions. They mainly suffer from defects such as link fragmentation, risk blind spots, static decision-making, and chaotic execution, thus failing to guarantee the effectiveness and accuracy of fault repair. Summary of the Invention

[0004] The embodiments of this disclosure propose a microservice operation and maintenance fault repair method and system based on an intelligent agent engine.

[0005] In a first aspect of this disclosure, a microservice operation and maintenance fault repair method based on an intelligent agent engine is provided. The method includes determining abnormal data characteristics based on the current microservice data, and determining service topology characteristics based on the abnormal data characteristics and a service topology graph; the current microservice data includes at least application log data. The method further includes determining node failure probabilities based on the service topology characteristics and log semantic features corresponding to the application log data, and determining a current state vector based on a first result of the node failure probability and a preset probability interval. The method also includes generating a repair action sequence based on the current state vector and a policy network, and determining a comprehensive risk value based on a second result of the current risk value corresponding to the repair action sequence and a dynamic risk budget; the dynamic risk budget is determined based on the node failure probability, the service topology graph, and a historical database. Furthermore, the method includes determining an estimated execution time based on the repair action sequence using an intelligent agent engine, based on a third result of the comprehensive risk value and a preset risk interval, and performing repair operations based on the estimated execution time and the repair action sequence.

[0006] In a second aspect of this disclosure, a microservice operation and maintenance fault repair system based on an intelligent agent engine is provided. The system includes an anomaly detection module configured to determine anomaly data characteristics based on the current microservice data, and to determine service topology characteristics based on the anomaly data characteristics and a service topology graph; the current microservice data includes at least application log data. The system also includes a root cause localization module configured to determine the node failure probability based on the service topology characteristics and log semantic features corresponding to the application log data, and to determine the current state vector based on a first result of the node failure probability and a preset probability interval. The system further includes a policy generation module configured to generate a repair action sequence based on the current state vector and a policy network, and to determine a comprehensive risk value based on a second result of the current risk value corresponding to the repair action sequence and a dynamic risk budget; the dynamic risk budget is determined based on the node failure probability, the service topology graph, and a historical database. Furthermore, the system includes an engine execution module configured to determine the estimated execution time based on the comprehensive risk value and a third result of a preset risk interval, using the intelligent agent engine to determine the estimated execution time based on the repair action sequence, and to perform repair operations based on the estimated execution time and the repair action sequence.

[0007] In a third aspect of this disclosure, a computer program product is provided, comprising a computer program that is executed by a processor to implement the method according to the first aspect.

[0008] In a fourth aspect of this disclosure, a machine-readable storage medium is provided. The machine-readable storage medium stores machine-executable instructions, which are executed by a processor to implement the method provided according to a first aspect of this disclosure.

[0009] It should be understood that the description in the Summary of the Invention section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0010] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0011] Figure 1 A flowchart illustrating a microservice operation and maintenance fault repair method based on an intelligent agent engine, according to some embodiments of this disclosure, is shown.

[0012] Figure 2 A schematic diagram of the overall architecture of an operation and maintenance terminal according to some embodiments of this disclosure is shown;

[0013] Figure 3A schematic diagram of the overall architecture of an intelligent agent engine according to some embodiments of this disclosure is shown;

[0014] Figure 4 A block diagram of a microservice operation and maintenance fault repair system based on an intelligent agent engine, according to some embodiments of this disclosure, is shown. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0016] The terms “comprising” and “having”, and any variations thereof, in this specification, claims, and the foregoing drawings are intended to cover a non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the steps or units listed, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus. Depending on the context, the word “if” as it applies herein may be interpreted as “when”, “when”, “in response to determination”, or “in response to detection”.

[0017] As mentioned above, existing mainstream microservice fault handling solutions are mainly divided into rule engine solutions, isolated AI model solutions, and manual orchestration solutions. Rule engine solutions rely on fixed thresholds and preset alarm rules (such as Prometheus alarm rules) pre-set by operations personnel for fault detection and handling. For example, an alarm is triggered when CPU utilization exceeds 90%. The drawback of this solution is that it cannot cover unknown fault modes, has weak generalization ability, and a high false alarm rate (over 45%). Isolated AI model solutions apply artificial intelligence models to fault handling, such as using neural networks for fault detection or decision trees for root cause localization. The drawback of this solution is that the detection, localization, and repair modules are isolated from each other, meaning they run independently. This leads to semantic loss in information transmission between modules, a low end-to-end repair success rate (less than 40%), and a lack of interpretability, making it difficult for operations personnel to immediately understand the AI's decision-making basis. Manual orchestration solutions rely on experts writing fault handling scripts (such as Ansible). Playbook (Operations Orchestration Scripts) solidifies the troubleshooting and repair steps of common faults into executable scripts, which can be manually or automatically triggered to execute the corresponding scripts when a fault occurs. The drawback of this approach is that script writing and maintenance rely too much on expert experience, the average repair time is long (e.g., more than 30 minutes), and it cannot cope with complex cascading fault scenarios.

[0018] Based on this, existing mainstream microservice fault handling solutions mainly suffer from the following technical shortcomings:

[0019] (1) Link fragmentation: The three stages of fault detection, root cause localization and repair decision-making lack an effective coordination mechanism. The independent operation of each stage leads to high information loss during transmission (e.g., up to 60%), making the repair effect far lower than the individual performance of each link.

[0020] (2) Risk blind spots: Existing automated repair tools lack quantitative assessment of operational risks before performing maintenance operations, and cannot predict the service impact that the operation may bring. For example, the error execution rate of high-risk operations (such as database parameter changes or service circuit breakers) is relatively high (e.g., more than 5%).

[0021] (3) Static decision-making: Repair strategies are usually based on static rules or fixed scripts, which cannot be dynamically adjusted according to the real-time running status of the environment and the context of the fault, and have weak generalization ability.

[0022] (4) Execution disorder: The lack of a standardized execution framework and the inconsistent calling formats of different operation and maintenance tools (such as Kubernetes API, database client and monitoring system interface) lead to large fluctuations in execution success rate.

[0023] In addition, there is a lack of effective closed-loop optimization mechanisms, which prevents continuous learning from historical fault experience, thus failing to guarantee the effectiveness and accuracy of fault repair.

[0024] To address this, embodiments of this disclosure propose a microservice operation and maintenance fault repair method based on an intelligent agent engine. The method includes determining abnormal data characteristics based on the current microservice data, and determining service topology characteristics based on the abnormal data characteristics and the service topology graph; the current microservice data includes at least application log data. The method further includes determining node failure probabilities based on the service topology characteristics and log semantic features corresponding to the application log data, and determining a current state vector based on a first result of the node failure probability and a preset probability interval. The method also includes generating a repair action sequence based on the current state vector and a policy network, and determining a comprehensive risk value based on a second result of the current risk value corresponding to the repair action sequence and a dynamic risk budget; the dynamic risk budget is determined based on the node failure probability, the service topology graph, and a historical database. Furthermore, the method includes determining an estimated execution time based on the comprehensive risk value and a preset risk interval, using the intelligent agent engine to determine the estimated execution time based on the repair action sequence, and performing repair operations based on the estimated execution time and the repair action sequence.

[0025] This approach enables the determination of node failure probabilities based on anomalies in current microservice data, combined with service topology features and log semantic features corresponding to application log data. Graph neural networks and semantic models ensure the accuracy and interpretability of root cause localization. Furthermore, the repair action sequences and dynamic risk budgets generated by the policy network effectively guarantee the security and efficiency of operational processes. The intelligent agent engine performs repair operations based on estimated execution time and repair action sequences. By extending the general intelligent agent framework with a dedicated operational execution engine, it not only eliminates semantic losses between modules but also ensures the stability of execution success rates. In addition, the entire approach forms a complete closed-loop process from fault perception, root cause localization, policy generation and risk assessment to standardized execution, significantly improving end-to-end fault repair efficiency and success rate while drastically shortening repair time.

[0026] Please see Figure 1The flowchart shown illustrates a microservice operation and maintenance fault repair method based on an intelligent agent engine, representing some embodiments of this disclosure. Method 100 is executed, for example, by an operation and maintenance terminal, which can be deployed on the control plane of a Kubernetes cluster to act as an intelligent operation and maintenance middleware to achieve automatic detection, analysis, repair, and learning of microservice faults. Furthermore, it supports seamless integration with monitoring systems (such as Prometheus, ELK, or Jaeger), providing operation and maintenance personnel with visual decision support and risk-controlled automated execution capabilities. Here, the overall architecture of the operations and maintenance terminal has at least a data acquisition layer, an intelligent decision-making layer, and an execution engine layer. The data acquisition layer collects real-time data from the current microservices, including multi-dimensional data, and sends this data to the intelligent decision-making layer via bidirectional gRPC. The intelligent decision-making layer performs anomaly detection, root cause localization, and repair strategy generation based on the current microservice data, thereby obtaining a repair action sequence, which is then sent to the execution engine layer via the REST API protocol. The execution engine layer can be understood as a dedicated operations and maintenance execution engine extended from the general intelligent agent framework, used to perform standardized repair operations (such as parameter validation, permission checks, action scheduling, risk control, and result parsing) based on the repair action sequence and the estimated execution time corresponding to that sequence. In addition, the overall architecture of the operation and maintenance terminal can also have a feedback learning layer, which is used to receive the repair feedback data collected by the execution engine layer after the repair operation is completed through the message bus, and perform reward calculation and policy update based on the repair feedback data (such as feeding the updated policy network back to the intelligent decision layer through gRPC bidirectional stream); and is also used to determine policy characteristics and root cause characteristics to update the historical database (such as the FAISS vector database) that stores multiple microservice fault repair cases. This not only continuously optimizes the decision quality of the policy network, but also shortens the fault handling time by using the continuously updated historical database, and realizes the continuous accumulation and efficient reuse of operation and maintenance knowledge.

[0027] like Figure 1As shown in box 102, method 100 can determine abnormal data characteristics based on the existence of anomalies in the current microservice data, and determine service topology characteristics based on the abnormal data characteristics and the service topology graph. Here, the data acquisition layer can send the real-time collected current microservice data to the intelligent decision layer. This current microservice data has multi-dimensional data, such as at least application log data (and may also include call event data and resource indicator data). Then, the intelligent decision layer performs anomaly detection on the current microservice data. After determining that the current microservice is abnormal, it determines the abnormal data characteristics based on the current microservice data, and determines the service topology characteristics based on the abnormal data characteristics and the service topology graph constructed from the service registry data. It is understandable that abnormal data features may include abnormal service nodes and the abnormal scores corresponding to those abnormal service nodes. These features are determined based on the reconstruction time-series features determined when performing anomaly detection on the current microservice data, and the current time-series features corresponding to the current microservice data. Of course, they may also include anomaly dimensions and anomaly patterns, but are not limited to these. Service topology features may be the topology anomaly probability of abnormal service nodes, which is determined through the service topology graph and graph neural networks (such as multi-hop GraphSAGE feature aggregation).

[0028] In some implementations, the data acquisition layer is also specifically used for:

[0029] Based on the current microservice data, the data format is converted to obtain the current time-series characteristics. Here, the data acquisition layer can be deployed on the DaemonSets of each node in the Kubernetes cluster. Each node in the Kubernetes cluster can be understood as a worker node (i.e., a service carrier) running Pods, microservices, and monitoring and collection programs. This data acquisition layer is specifically subdivided into eBPF probe units, cgroup collection units, and log collection units. The eBPF probe unit is used to capture network syscalls and container events at a preset sampling frequency (e.g., 100Hz) using kprobe and tracepoints. It also extracts abnormal service events from these network syscalls and container events (e.g., by building a portable eBPF program using a CO-RE approach, capturing 23 key events covering socket, TCP, UDP, and file I / O, with memory usage <50MB / node, and comparing the TCP retransmission rate calculated using a sliding window with a 5% threshold to determine...). The system identifies anomalies and assigns service Pod identifiers to abnormal service events, enabling microservice-level anomaly identification by associating them with cgroup collection units. The cgroup collection unit collects 42 container resource metrics, including CPU utilization, memory usage, disk I / O, and network traffic, at preset time intervals (e.g., 5 seconds). It then identifies anomalous metrics extracted from all container resource metrics (e.g., identifying all metrics exceeding preset thresholds as anomalous) as resource metric data. The log collection unit captures the container's standard output and standard error log streams in real time. It then merges and extracts anomalous logs from this stream (e.g., merging subsequent unmarked stack lines by identifying the start line of the stack, and extracting logs containing log level, anomaly type, error keywords, or business stack traces through keyword matching and semantic parsing) as application log data. This method of retaining anomalous data (while filtering out a large amount of normal data) effectively reduces the central transmission and computation load.

[0030] After collecting current microservice data containing call event data, resource metric data, and application log data, time alignment can be performed on the current microservice data (including dividing the sliding alignment window and aggregating or completing the data within the same window, such as calculating the mean, peak, and retransmission rate of call event data within the window, using adjacent data for resource metric data if there is no data in the window, and calculating the number of ERRORs and exceptions within the window for application log data). Then, according to the service tag and / or pod tag (which can also include container tags and namespace tags), the data of each service node is concatenated on the time-aligned current microservice data (that is, the data of each service node is associated and bound based on the tags of the time-aligned call event data, resource metric data, and application log data) to obtain the microservice data corresponding to each service node, thereby ensuring that the call event data, resource metric data, and application log data are consistent in the time dimension (such as the same nanosecond-level timestamp), smoothing out the time unit differences between eBPF microseconds, log milliseconds, and cgroup seconds.

[0031] In the embodiments of this disclosure, the data acquisition layer may also have a protocol adaptation unit. This unit performs format conversion on the microservice data corresponding to each service node to uniformly convert it into OpenMetrics format (i.e., a standardized metric data transmission and storage format). All microservice data after format conversion is used as the current time-series feature and sent to the intelligent decision-making layer via gRPC bidirectional stream (protocol P99 latency less than 5 milliseconds). Here, the dimensions of the current time-series feature can be, for example, 60 dimensions, including 12-dimensional network time-series features such as TCP retransmission rate, connection rejection number, RTT round-trip time, port occupancy, packet loss, handshake timeout, and syscall call time corresponding to call event data; 42-dimensional resource metric features such as CPU utilization, throttling count, memory RSS / cache, disk IOPS, read / write latency, number of threads, number of handles, and container runtime corresponding to resource metric data; and 6-dimensional abnormal log features such as the number of ERROR logs, number of WARN logs, frequency of exception stacks, interface error count, average exception interval, and error code distribution statistics corresponding to application log data.

[0032] Next, based on the current temporal features and the deep learning model, the reconstructed temporal features are determined, and it is determined whether the error between the reconstructed temporal features and the current temporal features exceeds a preset error threshold. Here, the deep learning model can be understood as a lightweight variational autoencoder (VAE) model (less than 10MB in size, with an inference latency of less than 15 milliseconds). It adopts a hierarchical structure (e.g., including a low-level encoder-decoder and a high-level encoder-decoder). The low-level encoder-decoder is used to detect resource-level anomalies (such as CPU spikes and memory leaks), and the high-level encoder-decoder is used to detect business-level anomalies (such as sudden drops in QPS and sudden increases in error rate). It can output reconstructed temporal features of the same dimension (i.e., restoring the temporal features under normal conditions) based on the microservice data corresponding to each service node in the input current temporal features. In addition, when training the deep learning model based on temporal feature samples under normal conditions in the past (e.g., the past 30 days), the optimization objective can be to minimize the reconstruction error and to calculate the mean and standard deviation of all reconstruction errors in the temporal feature samples.

[0033] It is understood that the intelligent decision-making layer can be specifically subdivided into an anomaly detection unit, a topology reasoning unit, a log coding unit, a gating fusion unit, and a policy generation unit. The anomaly detection unit outputs reconstruction time-series features corresponding to each service node based on a deep learning model. It then calculates the error between these reconstruction time-series features and the current time-series features (i.e., calculating the error dimension-by-dimensionally and averaging the sum of all errors) to determine whether the error exceeds a preset error threshold. In embodiments of this disclosure, the preset error threshold can be, for example, the sum of the mean of all reconstruction errors in the time-series feature samples and three times the standard deviation. When the error corresponding to a certain service node exceeds the preset error threshold (and the duration of the anomaly can also be simultaneously determined to exceed a preset error time threshold, such as 15 seconds), it indicates that an anomaly has been detected in the current microservice data (i.e., the corresponding service node is abnormal). Otherwise, it is considered that no anomaly has been detected in the current microservice data, and the system continues to monitor and acquire the current microservice data for the next time period collected by the data acquisition layer.

[0034] Subsequently, in response to the determination that the error between the reconstructed time-series characteristics and the current time-series characteristics exceeds a preset error threshold, it is determined that the current microservice data is abnormal. Here, after determining that the current microservice data is abnormal, the abnormal data characteristics can be determined based on the reconstructed time-series characteristics and the current time-series characteristics. For example, based on the errors corresponding to each service node mentioned above and the preset error threshold, abnormal service nodes (i.e., all service nodes whose errors exceed the preset error threshold) can be identified; the reconstruction errors of each dimension can be determined based on the reconstruction time-series features corresponding to the abnormal service nodes and the current time-series features; then, based on the mean and standard deviation of all reconstruction errors in the time-series feature samples mentioned above, each reconstruction error is standardized, and the standardized reconstruction errors can be pruned based on a preset pruning interval to avoid the information of other dimensions being buried due to excessively high abnormal scores in some dimensions; then, based on the ratio between each standardized reconstruction error and the maximum value of the preset pruning interval, the abnormal contribution of each dimension can be determined; then, the product between the preset weight corresponding to each dimension and the corresponding abnormal contribution is calculated, and the mean of all product results is determined as the abnormal score corresponding to the abnormal service node (between 0 and 1, the larger the value, the more serious the fault of the service node, and if there are multiple abnormal service nodes, each abnormal service node has a corresponding abnormal score). Understandably, after determining the reconstruction errors of each dimension based on the reconstruction time sequence characteristics and the current time sequence characteristics, the total reconstruction error corresponding to the call event data, the total reconstruction error corresponding to the resource indicator data, and the total reconstruction error corresponding to the application log data can be calculated according to all dimensions corresponding to the call event data, resource indicator data, and application log data, as mentioned above. All dimensions corresponding to the maximum total reconstruction error are then identified as abnormal dimensions. Furthermore, a trend comparison can be made between the time-series curve of the indicator corresponding to the anomaly dimension (determined based on the current microservice data after time alignment mentioned above) and the historical stable baseline of the corresponding dimension. For example, if the trend comparison result is a sudden increase (e.g., the indicator in the time-series curve is greater than 1.5 times the indicator in the historical stable baseline), the anomaly pattern is determined to be a sudden increase; or, if the trend comparison result is a sudden decrease (e.g., the indicator in the time-series curve is less than 1.5 times the indicator in the historical stable baseline), the anomaly pattern is determined to be a sudden decrease; or, if the trend comparison result is that the time-series curve of the indicator continues to oscillate up and down, and the variance is far greater than the variance range of the historical stable baseline, the anomaly pattern is determined to be fluctuation. The anomaly dimension and the corresponding anomaly pattern, together with the aforementioned anomaly service node and the anomaly score corresponding to the anomaly service node, are jointly determined as anomaly data features. Of course, embodiments of this disclosure can also introduce a time attention mechanism, which prioritizes anomaly dimensions with higher mutation time step weights by superimposing time attention weights to mark mutation-type patterns, and is not limited to this.

[0035] In this way, based on unified format conversion and reconstruction error detection, standardized processing of multi-source heterogeneous monitoring data can be achieved, ensuring that the anomaly detection output (anomaly data features) has a unified format and semantics, thereby providing reliable input for the root cause localization module and eliminating information loss at the front end of the link caused by inconsistent data formats.

[0036] In some implementations, when the intelligent decision layer determines service topology features based on abnormal data characteristics and the service topology graph, it specifically determines the anomaly propagation weight of each abnormal service node in the service topology graph based on the abnormal data characteristics and preset service scores. Then, based on the anomaly propagation weight of each abnormal service node and its metadata vector, it determines the initial feature vector. Here, the service topology graph can be constructed from service registry data provided by a service registry. This service registry can be understood as a microservice component (commonly such as Nacos, Consul, Eureka, or Kubernetes Service). After all microservices start, they actively report their addresses and instance information to this service registry. Understandably, the topology inference unit of the aforementioned intelligent decision-making layer calls the service registry data (or the service registry data corresponding to the aforementioned abnormal service nodes) from the service registry data. From this data, it extracts all registered service names to identify each noticed service name as a service node. Next, based on the call chains (such as Jaeger tracing / eBPF network call records) in the service registry data, it determines the call flow between every two service nodes and determines the connection edges (i.e., directed connections) between the corresponding two service nodes based on all call flows. This allows it to construct a service topology graph based on all service nodes and all connection edges. Of course, the topology inference unit can also call the service registry data in real time to update the service topology graph in real time.

[0037] After constructing the service topology graph, the topology inference unit can mark abnormal service nodes in the service topology graph based on the abnormal service nodes in the abnormal data features. Based on the abnormal score corresponding to the abnormal service node in the abnormal data features and the preset service score, it determines the abnormal propagation weight (i.e., the initial propagation weight, used to characterize the basic strength of the outward spread of the fault) of each abnormal service node in the service topology graph. The process of determining the abnormal propagation weight of each abnormal service node can be seen, for example, in the expression shown below:

[0038]

[0039] In the above formula, Let V be the anomaly propagation weight for the v-th abnormal service node. This represents the anomaly score corresponding to the v-th abnormal service node. This is a preset service score corresponding to the vth abnormal service node. The preset service score (between 0 and 1) can be determined based on the service importance of the abnormal service node. For example, the preset service score corresponding to a core service (such as orders and payments) can be 0.9, and the preset service score corresponding to an important service (such as product recommendations and inventory) can be 0.7.

[0040] In addition to determining the abnormal propagation weight of each abnormal service node in the service topology graph, it is also necessary to determine the abnormal propagation weight of the remaining normal service nodes in the service topology graph. The abnormal propagation weight of the normal service node can be a fixed value, or it can be determined by referring to the above, and is not limited to this.

[0041] Next, the initial feature vector can be determined based on the anomaly propagation weight and node metadata vector of each abnormal service node. Here, the node metadata vector can be understood as a vector obtained by transforming attributes such as service type, deployment specifications, and number of instances corresponding to the abnormal service node. The initial feature vector of the corresponding abnormal service node is obtained by concatenating the node metadata vector and the anomaly propagation weight. Of course, the initial feature vectors of the other normal service nodes in the service topology graph also need to be determined in the same way, but we will not go into details here.

[0042] Next, based on the abnormal data features, the edge propagation weights of each abnormal connection edge in the service topology graph are determined. Then, feature aggregation is performed based on all initial feature vectors and all edge propagation weights to obtain the target feature vectors of each abnormal service node. Here, the edge propagation weights of the corresponding abnormal connection edges (i.e., connection edges pointing from an abnormal service node to other service nodes) can be determined based on the abnormal score corresponding to the abnormal service node in the abnormal data features and the preset basic edge weights. The determination process can be seen in the expression shown below:

[0043]

[0044] In the above formula, The edge propagation weights for abnormal connections. Preset the basic edge weights (all connecting edges have fixed values, such as 1). The abnormal score corresponds to the abnormal service node, and 3 is the abnormal amplification coefficient (used to amplify the edge propagation weight of abnormal connection edges). Of course, the edge propagation weight of other normal connection edges in the service topology graph also needs to be determined in the same way, but we will not go into details here.

[0045] Next, based on the initial feature vectors of all abnormal service nodes, the edge propagation weights of all abnormal connections, the initial feature vectors of all normal service nodes, and the edge propagation weights of all normal connections, a multi-hop GraphSAGE feature aggregation can be performed using a graph neural network to obtain the target feature vectors of each abnormal service node. Here, multi-hop GraphSAGE feature aggregation can be, for example, performing 3-hop GraphSAGE feature aggregation, and its aggregation iteration process can be seen in the expression shown below:

[0046]

[0047] In the above formula, The target feature vector of the vth abnormal service node after feature aggregation of the kth hop (when k is 0, it corresponds to the initial feature vector mentioned above, and the value of k is iterated as 1, 2 and 3). The activation function (such as ReLU or Sigmoid function, to introduce a nonlinear transformation) This is a vector concatenation operation (that is, concatenating the feature vector of the abnormal service node with the mean feature vector of the adjacent service nodes). Let v be the target feature vector of the vth abnormal service node after feature aggregation of the (k-1)th hop. Let V be the feature vector of the neighboring service nodes of the v-th abnormal service node after feature aggregation of the k-1-th hop. It is the mean aggregation function (that is, the element-wise average of the feature vectors of the neighboring service nodes of the v-th abnormal service node). This is the set of neighboring service nodes of the v-th abnormal service node (including all upstream calling service nodes and all downstream calling service nodes). A weight matrix is ​​pre-set for the k-th layer (which can be obtained through pre-training).

[0048] Next, service topology features are determined based on all target feature vectors. Here, after determining the target feature vectors of all abnormal service nodes, each target feature vector can be converted into a topology anomaly probability (between 0 and 1) through a fully connected network and activation function, and the concatenation result of the topology anomaly probabilities of all abnormal service nodes is determined as the service topology features.

[0049] In this way, based on anomaly propagation weight calculation and multi-hop feature aggregation, single-point anomaly signals can be effectively propagated along the service call chain, enabling the target feature vector of each node to be fused with full-link multi-hop topology information. Compared with the existing technology of independently analyzing each service, this effectively achieves lossless propagation of anomaly information in the topology graph, ensuring that root cause localization can utilize the global topology context rather than isolated judgment.

[0050] In box 104, method 100 can determine the node failure probability based on service topology features and log semantic features corresponding to application log data, and determine the current state vector based on the first result of the node failure probability and the preset probability interval. Here, the log encoding unit of the intelligent decision layer mentioned above preprocesses the application log data in the current microservice data (such as removing variables such as timestamps and addresses) and extracts features from the preprocessed application log data through a language processing model to obtain the corresponding log semantic features (such as a 768-dimensional log semantic vector for output adaptation to fault identification). The language processing model can be, for example, a fine-tuned BERT-based model (such as one containing 12 encoder layers, a hidden layer dimension of 768, and 12 attention heads). The fine-tuning method includes adding a fully connected layer for operation and maintenance log semantic adaptation at the output position, using massive microservice fault logs (OOM, connection exhaustion, TCP timeout, cascading errors, etc.) for supervised fine-tuning, so that the model can identify operation and maintenance specific abnormal terms (such as Connection pool, retransmit, and OOM). (killed, etc.), update the weight parameters of the upper-layer Encoder and the newly added classification adaptation layer (freeze the underlying general semantic weights to reduce training costs and adapt to container operation and maintenance log scenarios), and adjust the loss function (use the generation of 768-dimensional vectors with close distances from similar fault logs as the optimization objective), and is not limited to this.

[0051] In some implementations, when determining the node failure probability based on service topology features and log semantic features corresponding to application log data, the intelligent decision layer specifically performs a fusion calculation on the service topology features and the log semantic features corresponding to application log data to obtain a fusion weight. Here, the gating fusion unit of the aforementioned intelligent decision layer calculates the fusion weight based on the service topology features and the log semantic features corresponding to application log data through a gating fusion neural network. The determination process can be seen in the expression shown below:

[0052]

[0053] In the above formula, G is the fusion weight (also known as the gating fusion weight; the larger the value, the more the current sample relies on log semantics to determine the root cause; the smaller the value, the more it relies on the service topology anomaly propagation probability). For activation functions (such as the Sigmoid activation function, which maps the output to between 0 and 1). The preset weight matrix (which can be obtained through pre-training) is used. , represents the service topology feature, , represents the log semantic feature, [;] represents the vector concatenation operation, and , represents the gating bias parameter (which can be obtained through pre-training).

[0054] Next, the log semantic features are mapped to log fault data, and the node fault probability is determined based on the log fault data, fusion weights, and service topology features. Here, the gated fusion unit can map the log semantic vector to the log fault score corresponding to the abnormal service node (i.e., log fault data, between 0 and 1; if there are multiple abnormal service nodes, each abnormal service node has a corresponding log fault score) through a fully connected network layer, and determine the node fault probability based on the log fault score, fusion weights, and service topology features. The determination process can be seen in the expression shown below:

[0055]

[0056] In the above formula, Let G be the node failure probability of the v-th abnormal service node, and G be the fusion weight. Let v be the log fault score of the v-th abnormal service node. Let be the topology anomaly probability corresponding to the v-th abnormal service node in the service topology features.

[0057] In this way, the confidence weights of topological features and log semantic features can be adaptively balanced based on the gating fusion mechanism, realizing deep collaboration of cross-modal information. Compared with the existing technology where topological analysis and log analysis output results independently and then are manually synthesized, this method effectively achieves the organic fusion of the two types of information at the model level. It not only improves the accuracy of root cause localization, but also uses the value of the fusion weight to intuitively reflect the credibility of the two types of information in the current scenario, and has good interpretability.

[0058] In some implementations, when the intelligent decision layer determines the current state vector based on a first result of the node failure probability and a preset probability interval, it specifically determines whether the first result, based on the node failure probability and the preset probability interval, indicates that the maximum probability of the node failure falls within the preset probability interval. Here, taking a node failure probability that includes multiple abnormal service nodes as an example, the maximum probability can be identified from all node failure probabilities, and it can be determined whether this maximum probability falls within the preset probability interval. This preset probability interval can, for example, be set to between 0.7 and 1 (inclusive). It is understandable that when the maximum probability is within the preset probability range, it indicates that the root cause can be directly identified based on the maximum probability of node failure. When the maximum probability is between 0.5 and 0.7 (inclusive), it indicates that the confidence of the root cause is insufficient (if it is necessary to expand the topology awareness range), for example, the maximum probability of node failure can be re-determined as described above, and then judged again with the preset probability range. In addition, multi-hop GraphSAGE feature aggregation needs to be changed to 5-hop GraphSAGE feature aggregation, the process of which will not be elaborated here. When the maximum probability is between 0 and 0.5, it indicates that there is no clearly high-confidence faulty service node, and manual intervention alarm can be directly triggered.

[0059] Subsequently, in response to the determination that the maximum probability of the node failure probability falls within a preset probability range, the current state vector is determined based on service topology features, current microservice data, and historical database. Here, the current state vector can be understood as a 184-dimensional state vector concatenated from a topology feature vector (64-dimensional), a resource indicator vector (42-dimensional), a service indicator vector (32-dimensional), and a historical action vector (42-dimensional). The topology feature vector can be obtained, for example, based on service topology features and topology-related values ​​extracted during multi-hop GraphSAGE feature aggregation, such as the anomaly weights of upstream and downstream adjacent service nodes, connection edge failure propagation weights, service levels, business importance, and link hop counts, through compression and normalization. The resource indicator vector can be, for example, based on resource indicator data in the current microservice data (such as failed Pods). The service metrics vector can be obtained by standardizing the mean, peak, and variance of the current sliding window, including CPU, memory, disk I / O, handles, threads, and network connections corresponding to the service. Service metric vectors can be obtained by summarizing and normalizing service statistics within the current microservice data, such as QPS, error rate, P95 / P99 response time, transaction success rate, order volume, and payment success rate for each interface. Historical action vectors can be obtained by extracting the most recent N historical repair action sequences (such as expanding the connection pool, circuit breaking, restarting instances, or master-slave switching), comprehensive risk scores, execution time, repair success rate, and business impact from a historical database, and then encoding and padding these data. This process is not limited to these methods.

[0060] Furthermore, in response to determining that the maximum probability of the first result not being a node failure probability falls within a preset probability range, the service topology features are updated based on the abnormal data features and the service topology graph. Here, after determining that the maximum probability of a node failure probability falls between 0.5 and 0.7 (inclusive) as mentioned above, the maximum probability of a node failure probability is re-determined based on the abnormal data features and the service topology graph, referring to the above, and then compared with the preset probability range again. Additionally, multi-hop GraphSAGE feature aggregation needs to be changed to 5-hop GraphSAGE feature aggregation; the process will not be elaborated further here.

[0061] Furthermore, in response to determining that the maximum probability of the first result not being a node failure probability falls within a preset probability range, fault alarm information is generated based on abnormal data characteristics and current microservice data. Here, after determining that the maximum probability of a node failure probability falls between 0 and 0.5 as mentioned above, fault alarm information can be analyzed from abnormal data characteristics and current microservice data. This fault alarm information may include, for example, a fault timestamp window, fault phenomenon data (such as an 82% error rate for an order interface, a sudden increase of 2000ms in interface RTT, or a 10-fold increase in the number of TCP connection failures), and an abnormal log summary. It may also include the end-to-end inference results of the intelligent decision-making layer (such as the service topology characteristics, fusion weights, and node failure probabilities mentioned above). It is understandable that after generating fault alarm information, manual troubleshooting guidance can be provided. For example, taking a payment business cluster failure as an example, the network connectivity and interface availability of the third-party payment channel can be checked, and / or the payment gateway connection pool and firewall rate limiting policies can be investigated, and / or the slow query and lock wait status of the database can be verified, and this is not limited to these.

[0062] In this way, a three-level probability interval branch decision mechanism can be used to adaptively select subsequent processing paths based on the confidence level of root cause localization. This not only solves the problem of mechanically executing fixed processes regardless of the confidence level in existing technologies, but also enables dynamic adjustment of decision paths, significantly enhancing generalization ability.

[0063] In box 106, method 100 can generate a repair action sequence based on the current state vector and the policy network, and determine the comprehensive risk value based on the current risk value corresponding to the repair action sequence and the second result of the dynamic risk budget. Here, the overall architecture of the policy network adopts an Actor-Critic dual-branch shared bottom-level feature extraction layer, which may consist of a shared layer (such as a multi-layer fully connected layer), a policy head, and a value head. The shared layer is used to extract the operational fault features from the current state vector; the policy head is used to output the probability distribution of one or more operational actions based on the multi-layer fully connected layer and the normalization function (i.e., to output the repair action sequence); the value head is used to determine the corresponding state value scalar based on the multi-layer fully connected layer (representing the overall benefit value of performing repair under the current fault scenario, which may not be output but is mainly used in the training process of the policy network).

[0064] Understandably, the policy generation unit of the aforementioned intelligent decision-making layer inputs the current state vector into the policy network to generate a sequence of repair actions containing probability distributions of one or more operational actions. This sequence is then sent to the execution engine layer (i.e., a dedicated operational execution engine extended from the general intelligent agent framework OpenClaw) via a REST API protocol. This execution engine layer can be further subdivided into, for example, a risk management unit, a parameter verification unit, a permission check unit, an action scheduling unit, and a result parsing unit. Here, the risk management unit determines the comprehensive risk value based on the current risk value corresponding to the repair action sequence and the second result of the dynamic risk budget. The dynamic risk budget is determined based on node failure probabilities, the service topology graph, and the historical database. Its determination process can be seen in the expression shown below:

[0065]

[0066] In the above formula, For dynamic risk budgeting, To serve the importance score, To change the range of scores, To apply historical scores, , and These are preset weight parameters. Here, the service importance score can be determined based on the service importance of the abnormal service node corresponding to the highest probability of node failure, and preset importance score levels. For example, a service importance score between 0.9 and 1.0 can correspond to service nodes with core service importance (such as payment, orders, and MySQL master database), a service importance score between 0.6 and 0.8 can correspond to service nodes with important service importance (such as inventory, recommendation, and Redis cache), and a service importance score between 0.1 and 0.5 can correspond to service nodes with ordinary service importance (such as backend reports, log analysis, and internal management). The change scope score can be based on the repair action sequence, by statistically analyzing the number of affected services or instances in the service topology diagram (statistical indicators include the number of modified Pod replicas, database changes, number of downstream circuit breakers, and number of cross-namespace instances, etc.), and obtained through normalization (i.e., by calculating the ratio with the total number of instances in the cluster); the execution history score can be based on the historical database, by statistically analyzing the historical success rate of the same type of fault and the same type of repair action within the past 30 days, and obtained through mapping (i.e., by calculating the ratio with the total number of all repair actions within 30 days), and is defaulted to 0.5 if there is no corresponding historical repair action.

[0067] In the embodiments disclosed herein, the execution engine layer extends vertical domains based on the general intelligent agent framework OpenClaw, and is equipped with a variety of operation and maintenance adapters such as a unified K8s API adapter, a PromQL query adapter, and a SQL change adapter. This not only enables unified adaptation to operation and maintenance-specific parameter validation (such as JSON Schema parameter validation), permission control (such as RBAC permission authentication), asynchronous scheduling, and risk control capabilities (such as uniformly encapsulating execution results into status, data, error, or metadata formats), but also effectively shields the differences in interfaces, input parameters, and return formats of various operation and maintenance tools, thereby ensuring data standardization.

[0068] In some implementations, when the execution engine layer determines the comprehensive risk value based on the second result of the current risk value and the dynamic risk budget corresponding to the repair action sequence, it specifically determines whether the second result indicates that the current risk value is less than the dynamic risk budget. Here, the current risk value corresponding to the repair action sequence is understood as the sum of the risk scores corresponding to all maintenance actions in the repair action sequence. The risk score corresponding to each maintenance action can be, for example, a preset value, or it can be obtained by weighted summation based on a preset base value, the operation parameters corresponding to each maintenance action, and the service importance score mentioned above, and is not limited to this.

[0069] Understandably, if the current risk value is less than the dynamic risk budget, it indicates that the repair action sequence is executable and the subsequent process can be executed; if the current risk value is greater than or equal to the dynamic risk budget, it indicates that the repair action sequence is not executable (i.e., invalid), and the above should be referred to to regenerate the repair action sequence, and it should be determined again whether the corresponding current risk value is less than the dynamic risk budget. If the constraints still cannot be met after multiple attempts, a risk over-limit alarm can be generated to promptly switch to manual intervention.

[0070] Subsequently, in response to the determination that the second result is that the current risk value is less than the dynamic risk budget, the service impact score is determined based on the node failure probability, the preset service score, and the service monitoring data. Here, the service importance can be determined based on the service importance of the service node corresponding to the highest probability in the node failure probability and the preset service score. Then, the service monitoring data during the service anomaly can be obtained through the data acquisition layer mentioned above (such as the eBPF probe unit) to calculate the total transaction volume and the lost transaction volume from the service monitoring data, and the ratio of the lost transaction volume to the total transaction volume is determined as the lost transaction percentage. Then, the product of 100, the service importance, and the lost transaction percentage (i.e., 100 * service importance * lost transaction percentage) can be determined as the service impact score.

[0071] Next, based on the repair action sequence and service topology diagram, the operation impact score is determined, and the historical success score is determined based on the historical database. Here, the operation impact score can be determined based on the services (such as one or more) corresponding to the maintenance actions in the repair action sequence. The total number of upstream and downstream dependent service nodes of the corresponding service node is counted in the service topology diagram, and the score is determined based on the number of services and the corresponding total number of upstream and downstream dependent service nodes. For example, when there is one service and the total number of upstream and downstream dependent service nodes only involves local related operations, the corresponding operation impact score can be set to 20; when there is one service and the total number of upstream and downstream dependent service nodes involves all related operations, the corresponding operation impact score can be set to 50; when there are multiple service nodes, or when the total number of all upstream and downstream dependent service nodes of all service nodes is close to the total number of service nodes in the service topology diagram, the corresponding operation impact score can be set to a score between 80 and 100.

[0072] The historical success score can be determined based on the historical database by statistically analyzing the historical success rate of the same type of fault and the same type of repair action sequence. It is determined by calculating the difference between 1 and the historical success rate, and the product of the difference and 100, i.e., 100*(1 - historical success rate).

[0073] Next, the operational complexity score is determined based on the number of actions in the repair action sequence. The operational complexity score, historical success score, operational impact score, and service impact score are then weighted and summed to obtain a comprehensive risk value. Here, the operational complexity score can be determined based on the product of the number of actions in the repair action sequence and a preset score (e.g., a preset score of 10 for a single maintenance action, with an upper limit of 100 for the product). Alternatively, it can be determined based on the action type level corresponding to each maintenance action in the repair action sequence. For example, simple (e.g., scaling up Pods and adjusting rate limiting thresholds), with a corresponding operational complexity score of 20; medium (e.g., restarting services and clearing caches), with a corresponding operational complexity score of 50; and complex (e.g., SQL changes, read / write splitting switching, and kernel parameter modifications), with a corresponding operational complexity score between 80 and 100.

[0074] Understandably, after determining the operational complexity score, historical success score, operational impact score, and service impact score, the weighted summation process can be illustrated by the expression shown below:

[0075]

[0076] In the above formula, For the comprehensive risk value, To influence the score, The operation affects the score. As a historical success score, The score is complex to operate.

[0077] In this way, a multi-dimensional risk assessment can be conducted on the sequence of repair actions before execution based on a weighted quantification of four dimensions: service impact, operational impact, historical success rate, and operational complexity. This transforms the previously invisible operational risks into quantifiable comprehensive risk values, providing an objective basis for risk classification and access control, and effectively addressing the technical deficiency of existing technologies that lack prior risk assessment for high-risk operations.

[0078] In box 108, method 100 can determine the estimated execution time based on a third result of the comprehensive risk value and a preset risk range, using the intelligent agent engine and a repair action sequence, and then perform repair operations based on the estimated execution time and the repair action sequence. Here, the risk control unit of the aforementioned execution engine layer determines the third result based on whether the comprehensive risk value is within the preset risk range. When the third result indicates that the comprehensive risk value is within the preset risk range, the action scheduling unit of the aforementioned execution engine layer determines the estimated execution time based on the repair action sequence, and performs standardized repair operations based on the estimated execution time and the repair action sequence by combining the parameter verification unit and the permission check unit. It is understandable that the intelligent agent engine is the execution engine layer mentioned above (that is, an operation and maintenance-specific execution engine that is extended on the basis of the general intelligent agent framework OpenClaw). It is deployed on the control plane of the K8s cluster and runs as an independent Deployment container. Its underlying layer is equipped with the original OpenClaw general intelligent agent framework, and on this basis, operation and maintenance-specific extension plugins (that is, the risk management unit, parameter verification unit, permission check unit, action scheduling unit, and result parsing unit mentioned above) are developed. It can call, remember, and plan the native OpenClaw tools without changing them.

[0079] In some implementations, when the intelligent agent engine determines the estimated execution time based on the repair action sequence using a third result that is a comprehensive risk value and a preset risk range, it specifically determines whether the third result indicates that the comprehensive risk value falls within the preset risk range. Here, the preset risk range can be set, for example, between 0 and 50. When the comprehensive risk value is within this preset risk range, it indicates that the repair action sequence is low-risk and can be automatically executed. When the comprehensive risk value is higher than this preset risk range, it indicates that the repair action sequence is medium-high risk, and manual approval is required to determine whether to execute the subsequent process.

[0080] Subsequently, in response to the determination that the third result indicates the comprehensive risk value falls within a preset risk range, the intelligent agent engine determines the estimated execution time based on the repair action sequence. Here, the estimated execution time can be determined based on the type of maintenance action in the repair action sequence, identifying the standard baseline duration corresponding to the maintenance adapter's execution of the corresponding maintenance action (e.g., the standard baseline duration for Pod scaling or rate limiting adjustment is 10 seconds, the standard baseline duration for service rolling restart is 25 seconds, and the standard baseline duration for MySQL parameter changes or read / write splitting switch is 60 seconds), and calculating the sum of the standard baseline durations for all maintenance action types. Then, based on the aforementioned operation complexity score, a corresponding complexity correction parameter can be determined (the higher the operation complexity score, the larger the complexity correction parameter; the operation complexity score and the complexity correction parameter have a preset correspondence), and the product of the sum of the standard baseline durations and the complexity correction parameter is determined as the estimated execution time. Of course, historical action sequences belonging to the same type of fault and the same type of repair action sequence can also be identified through historical databases. The average execution time of the historical action sequence can be used as a benchmark, and the product of the benchmark and the complexity correction parameter can be used as the estimated execution time. This is not limited to this method.

[0081] Subsequently, in response to the determination that the third result is not within the preset risk range and the comprehensive risk value is not within the overall risk range, an approval work order is generated based on the current microservice data, service topology characteristics, repair action sequence, historical database, and comprehensive risk value. The approval work order includes a root cause report, repair instructions, estimated impact, and rollback plan. Here, if the third result is not within the preset risk range and the comprehensive risk value is between 50 and 69 (inclusive), it indicates that the repair action sequence is of medium risk. In this case, an approval work order can be generated and fed back to the operations team leader for manual approval. The subsequent process can only be executed if the approval is granted within a specified time period (e.g., 15 minutes) (otherwise, the repair action will not be executed), or the subsequent process can be executed according to the modified repair action sequence. If the comprehensive risk value is greater than or equal to 70, it indicates that the repair action sequence is of high risk. In this case, an approval work order can be generated and fed back to the operations director for manual approval. The subsequent process can only be executed if the approval is granted within a specified time period (e.g., 15 minutes) (otherwise, the repair action will not be executed), or the subsequent process can be executed according to the modified repair action sequence.

[0082] It is understandable that approval work orders can be generated based on current microservice data, service topology characteristics, repair action sequences, historical databases, and comprehensive risk values, and include root cause reports, repair descriptions, estimated impacts, and rollback plans. The root cause report may include, for example, multi-source collected data analyzed from current microservice data (such as at least one of eBPF network metrics, cgroup resource time series, application log data, and log semantic features), service topology data extracted when determining service topology characteristics (such as at least one of service topology graphs, edge propagation weights of abnormal connections, and topology anomaly probabilities), root cause fusion data applied when determining node failure probabilities (such as at least one of fusion weights, node failure probabilities, the maximum probability of node failure, and whether 5-hop GraphSAGE feature aggregation was performed), and service foundation data (such as at least one of the aforementioned service importance scores and lost transaction volumes). Its content structure may include, for example, a failure time window, a summary of abnormal monitoring metrics, a log semantic anomaly summary, a NN topology propagation link, a gated fusion root cause reasoning process, and a final root cause determination conclusion. In one example, the root cause report could be “Fault window: 2026-06-23, 14:10-14:20; Order service memory usage 96%, TCP retransmission rate 8.2%; Logs contain a large number of Connection pool exhausted errors; The topology anomaly probability of the 3-hop GraphSAGE output node is 0.92, the fusion weight is 0.78, and the maximum probability of node failure is 0.89≥0.7. The root cause is determined to be the exhaustion of the database connection pool in the order service”, and is not limited to this.

[0083] The repair description may include, for example, strategy output data determined based on the repair action sequence (such as at least one of the operation parameters of the repair action sequence and maintenance actions, such as the maximum value of the connection pool, the number of replicas to be expanded, or the circuit breaker threshold), historical case data determined based on the historical database (such as at least one of the historical action sequences and corresponding execution effects belonging to the same type of fault and repair action as the repair action sequence, such as success or failure), and risk assessment data determined based on the comprehensive risk value (which may also include the operational capabilities supported by the maintenance adapter). Its content structure may include, for example, the source of the strategy, detailed step-by-step operation, expected effect of each step, and success rate of similar historical strategies. In one example, the repair description may be "The repair strategy is derived from a historical successful case with a similarity of 0.91 in the historical database. The execution order is: 1. Expand the order DB connection pool max_size from 200 to 400; 2. Expand the order Pod replicas from 3 to 6; 3. Observe for 10 seconds to verify QPS and error rate; the success rate of similar historical action sequences is 70%, the comprehensive risk value is 57.8, and it is a medium risk," and is not limited to this.

[0084] The estimated impact may include, for example, service topology data determined based on the service topology diagram (such as at least one of the upstream and downstream dependent service nodes and corresponding call chains of the root cause service node), strategy action data determined based on the repair action sequence (such as maintenance actions in the repair action sequence), service indicator data (such as at least one of the affected transaction chains associated with the root cause service node, user traffic share, and service importance), and risk item data (such as the quantitative value of the operation coverage). Its content structure may include, for example, direct changes to resources, upstream and downstream related services, affected business traffic share, and potential scope of fault propagation. In one example, the estimated impact may be "modifying the connection pool parameters of all 6 Pods of the order service; downstream dependent services are product recommendation and inventory, with 65% of online transaction traffic flowing through this link; if the operation is abnormal, it will cause a large-scale timeout of the order and payment interfaces, affecting payment transactions across the entire platform," and is not limited to this.

[0085] Rollback plans may include, for example, the original configuration parameters (such as at least one of the original connection pool size, original number of replicas, and original SQL parameters) before the execution of the repair action sequence, the capabilities of the operation and maintenance adapter (such as at least one of the K8s configuration rollback interface, database transaction rollback and cache recovery interface), topology and monitoring data (such as verification metrics after rollback, including at least one of CPU, QPS, error rate and TCP retransmission), and risk assessment data (such as operation complexity score and at least one of the historical failure cases in the historical database that belong to the same type of fault and the same type of repair action sequence). Its content structure may include, for example, trigger rollback conditions, step-by-step rollback operations, verification metrics, and rollback timeout handling. In one example, a rollback plan could be defined as follows: "Rollback trigger condition: Error rate > 30% within a 10-second window after the repair is performed; Rollback steps: 1. Restore the DB connection pool max_size to 200; 2. Shrink the Pod replicas back to 3; 3. Continuously observe for 30 seconds, and the rollback is considered complete if the error rate falls below 1%; If the rollback timeout exceeds 60 seconds without recovery, an emergency manual alert will be automatically pushed," and it is not limited to this.

[0086] In this way, a graded processing mechanism based on comprehensive risk values ​​can be used to achieve synergy between risk graded control and standardized execution processes. For high-risk operations, a complete approval work order containing root cause reports, remediation instructions, estimated impact and rollback plans can be automatically generated to ensure that high-risk operations can only be executed after manual confirmation. At the same time, the introduction of estimated execution time can also select synchronous or asynchronous execution mode according to the operation time, avoiding confusion and resource blockage at the execution level.

[0087] When performing repair operations based on the estimated execution time and repair action sequence, the action scheduling unit at the execution engine layer, as mentioned above, determines the execution mode based on the estimated execution time. For example, when the estimated execution time is less than or equal to a preset time threshold (e.g., 30 seconds), the execution mode is determined to be synchronous execution, and the process is blocked and waits for the result to avoid timeout and deadlock. That is, after issuing the operation and maintenance command, the main thread stops all subsequent processes (it is stuck, such as not performing root cause secondary verification in parallel, not pushing task receipts to the front end, and all downstream processes are suspended), and continuously polls or waits for the operation and maintenance adapter to return a complete execution result (e.g., success, failure, error, or rollback trigger). Only after confirming the final execution receipt will the repair process continue. Subsequent processes include result verification, reward calculation, and case entry into the database. When the estimated execution time exceeds a preset time threshold, the execution mode is set to asynchronous mode, and a Webhook callback is registered. This means that after the operation and maintenance instructions are issued, the main thread will not block and wait but will immediately return the task ID, allowing the process to proceed directly to the subsequent processes (without blocking the main thread, and can simultaneously handle the repair actions of other faulty services). In addition, a dedicated callback interface address is pre-registered with the operation and maintenance adapter so that after all time-consuming operation and maintenance operations are completed (such as success, failure, or rollback), the operation and maintenance adapter will proactively push the complete execution result to the Webhook address, and the background child thread will receive the callback data and then perform subsequent processes such as repair verification, reward calculation, and case entry into the database.

[0088] After determining the execution mode, the parameter verification unit of the execution engine layer mentioned above performs parameter verification on the repair action sequence (or the repair action sequence modified by humans, which may also include service input parameters of operation and maintenance actions, such as service name, namespace, resource threshold, connection pool parameters, circuit breaker ratio, change duration or rollback configuration, etc.). For example, using JSON Schema parameter verification, a preset operation and maintenance action JSON constraint template is loaded to verify whether the field format, upper and lower limits of values, and enumeration range of the repair action sequence and service input parameters conform to the specifications. If they do not conform to the specifications, the parameter verification is determined to be unsuccessful, the abnormal parameters need to be intercepted and reported, and the process should be stopped from proceeding to the next step. If they conform to the specifications, the parameter verification is determined to be successful, and the process can proceed to the next step.

[0089] After the parameters pass the verification, the permission checking unit at the execution engine layer mentioned above performs a three-level permission check on the operation and maintenance action type of the repair action sequence (such as at least one of the following: scaling, circuit breaking, restart, database parameter modification, instance switching, or resource deletion), the target resource corresponding to the operation and maintenance action type (such as at least one of the following: Pod, Service, MySQL, Redis, namespace, or configuration file), and the initiator's identity (such as the execution engine layer account or the operation and maintenance personnel's approval account). For example, using a preset RBAC model (i.e., a preset access permission configuration relationship), it checks whether the initiator has the permission to execute the corresponding operation and maintenance action type and the application target resource. If the initiator has the permission, the permission check is deemed to have passed, and the process proceeds to the next step. If the initiator does not have the permission, the permission check is deemed to have failed, and a permission exception must be reported and the process to the next step must be stopped.

[0090] After the permission verification is passed, the risk control unit of the aforementioned execution engine layer performs a pre-execution snapshot (i.e., the prerequisite for abnormal rollback) before executing the repair action sequence. For example, it performs a full backup of the current configuration parameters and running status of the target resource corresponding to the operation and maintenance action type of the repair action sequence, so as to serve as the data source if automatic rollback is required in the future. The current configuration parameters and running status of the target resource can be obtained by calling the query interface of the corresponding operation and maintenance adapter, and is not limited to this.

[0091] After taking a pre-execution snapshot, the action scheduling unit of the aforementioned execution engine layer matches the corresponding operation and maintenance adapter (such as a Kubernetes API adapter, database operation and maintenance adapter, or middleware adapter) based on the repair action, and issues a call instruction according to the determined execution mode to execute the repair action. In addition, the result parsing unit of the aforementioned execution engine layer captures the return result data after the repair action is executed in real time, and parses the return result data to obtain structured repair feedback data. Here, the return result data includes, for example, the output data returned by the operation and maintenance adapter after executing the repair action (such as core service metrics pulled by the PromQL adapter, Kubernetes...). The parsing process includes API adapter return resource status, database operation and maintenance adapter output, or intermediate adapter circuit breaker or restart operation log results, execution process metadata (such as estimated execution time, actual execution time, synchronous / asynchronous identifier, initiator identity, comprehensive risk value, or operation timestamp), and exception information data (such as error code, permission denial, operation timeout, database error, or automatic rollback record). It can also include pre-existing auxiliary data (such as fusion weight, node failure probability, service Pod label, or failure window indicator), and is not limited to these. The parsing process includes unified cleaning and identification of core information (such as return output data, execution process metadata, exception information data, and pre-existing auxiliary data), and encapsulation into standard structured (such as status, data, error, and metadata, where status can be success, failed, or rollback, data can be resource status or core service indicator, error can be an error code or error description filled in when execution fails or an empty string filled in when execution succeeds, and metadata can be execution time, comprehensive risk value, or service label) repair feedback data.

[0092] In the embodiments of this disclosure, the risk management unit at the aforementioned execution engine layer can also automatically roll back based on the repair feedback data. If it determines that there is an execution anomaly in the repair feedback data (such as returned error codes, deterioration of resource indicators, or execution timeout with no response), it can do so through the aforementioned pre-execution snapshot. For example, it can retrieve the current configuration parameters and running status of the target resource from the pre-execution snapshot and reversely call the operation and maintenance adapter to restore the target resource to its configuration parameters and running status before the repair, thereby achieving anomaly repair. Furthermore, the risk management unit can also verify the indicator recovery status after automatic rollback and generate a complete rollback audit report for push notifications of anomaly upgrades.

[0093] Please see Figure 2 The diagram illustrates the overall architecture of an intelligent agent engine according to some embodiments of the present disclosure. Figure 2As shown, the overall architecture 200 of the intelligent agent engine may include a parameter verification unit, a permission check unit, a risk control unit, an action scheduling unit, and a result parsing unit. The risk control unit is equipped with engine support components (such as an operation and maintenance adapter). The processes executed by each unit can be referred to the above, and will not be elaborated here.

[0094] In some implementations, after performing repair operations based on the estimated execution time and repair action sequence, the operation and maintenance terminal can also obtain repair feedback data through the intelligent agent engine. The effectiveness of the repair actions can be verified by determining whether the repair feedback data is within a preset baseline range and whether its duration exceeds a preset duration threshold. Here, after determining the repair feedback data, the risk management unit of the aforementioned intelligent decision layer extracts core service metrics (such as service QPS, interface error rate, interface RTT, TCP connection failures, service CPU or memory usage, or database connection pool utilization) from the repair feedback data and sends them to the feedback learning layer via the message bus. It is understandable that the feedback learning layer can be further subdivided into a reward calculation unit and an update and accumulation unit. The reward calculation unit can determine whether the core service indicator is within a preset baseline range and whether the duration exceeds a preset duration threshold (e.g., 60 seconds) based on the core service indicator. The preset baseline range can be, for example, between the product of the service indicator baseline (i.e., the average value of the service indicator during long-term stable operation) and 95% and the service indicator baseline and 105%. If the repair feedback data is within the preset baseline range and the duration exceeds the preset duration threshold, it indicates that the repair action was successfully executed and can proceed to the subsequent process. Otherwise, it indicates that the repair action failed and can be automatically rolled back by the risk control unit mentioned above. The process can be referred to above, and the corresponding repair action sequence can also be marked as failed and updated to the historical database.

[0095] Subsequently, in response to the determination that the repair feedback data is within the preset baseline range and the duration exceeds the preset duration threshold, the current multidimensional reward is determined based on the repair feedback data, and the current training sample is determined based on the current state vector, the repair action sequence, the current multidimensional reward, and the repair state vector corresponding to the repair action sequence.Here, after confirming the successful execution of the repair action, the reward calculation unit of the aforementioned feedback learning layer determines the current multi-dimensional reward corresponding to the repair action sequence based on the repair feedback data. This current multi-dimensional reward can be, for example, a weighted sum of the repair effect reward (40% weight), execution time reward (25% weight), business impact reward (15% weight), resource consumption reward (10% weight), and risk control reward (10% weight). The repair effect reward can be, for example, the ratio between the difference between the repaired metric and the abnormal metric, and the difference between the baseline metric and the abnormal metric, i.e., (repaired metric - abnormal metric) / (baseline metric - abnormal metric). Post-recovery metrics can be understood as core service metrics extracted from repair feedback data. Anomaly metrics can be understood as the peak or average value of the fault data (which can be determined based on the aforementioned deep learning model) of the microservice data corresponding to the root cause service node after time alignment and data concatenation. This includes metrics such as CPU, memory, or disk I / O usage during the fault (corresponding to resource metrics data), TCP retransmission rate, connection failures, or RTT during the fault period (corresponding to call event data), and the percentage of erroneous requests within the fault window (corresponding to application log data). Baseline metrics can be understood as the average value of service metrics during long-term stable operation. Execution time rewards can be, for example, 1 times the actual time and SL. The ratio of the A threshold to the difference between the two is 1 - (actual time / SLA threshold). The actual time can be understood as the difference between the timestamp when the result parsing unit receives the returned result data and the timestamp when the action scheduling unit issues the maintenance action instruction. The SLA threshold can be understood as the preset standard duration corresponding to the executed maintenance action. The business impact reward can be, for example, the ratio between the lost transaction volume during the abnormal period and the total transaction volume, i.e., lost transaction volume during the abnormal period / total transaction volume. These figures can be calculated from the service monitoring data during the aforementioned service abnormality period. The resource consumption reward can be, for example, 1 - (actual resources / preset resources). The ratio result, the difference between the two, is 1 - (actual resources / budgeted resources). The actual resources can be understood as the resources currently occupied by the service read from the resource status returned above after the operation and maintenance action is completed. The budgeted resources can be understood as the resource limit preset before the operation and maintenance action is executed (such as the maximum number of Pod replicas allowed for expansion, the maximum number of database connections, or temporary CPU and memory quotas, etc.). The risk control reward can be, for example, the difference between 1 and the ratio of actual risk and preset risk, that is, 1 - (actual risk / budgeted risk). The actual risk can be understood as the current risk value corresponding to the repair action sequence mentioned above. The budgeted risk can be understood as the dynamic risk budget mentioned above.

[0096] After determining the current multidimensional reward, the update and accumulation unit of the aforementioned feedback learning layer forms a four-tuple of experience based on the current multidimensional reward (e.g., Rtotal), the current state vector (e.g., st), the repair action sequence (e.g., at), and the repair state vector corresponding to the repair action sequence (e.g., st+1). This four-tuple of experience is then used as the current training sample. Here, the repair state vector corresponding to the repair action sequence can be understood as the current state vector determined based on the current microservice data collected again by the aforementioned data acquisition layer after the operation and maintenance action is completed. The determination process can be found above and will not be elaborated further here.

[0097] Next, the sampling probability of the current training sample is determined based on the current multidimensional reward, and the training set of the policy network is updated based on the current training sample and the corresponding sampling probability. Here, the update and saturation unit of the feedback learning layer mentioned above determines the sampling probability of the current training sample based on the current multidimensional reward and the state value scalar output by the value head when the policy network outputs the repair action sequence based on the current state vector. The determination process can be seen in the following expression:

[0098]

[0099]

[0100] In the above formula, (or) Let be the temporal difference error (TD-error) of the i-th training sample (i.e., the current training sample). For the current multi-dimensional rewards, The preset discount factor, The value head of the policy network is the state value scalar output based on the current state vector determined again after the repair action is performed. The value head of the policy network outputs a state value scalar based on the current state vector. For a preset very small positive number (such as 10) -6 (magnitude) Let be the sampling probability of the i-th training sample. Let j be the temporal difference error of the j-th training sample in the training set. These are preset priority control hyperparameters.

[0101] After determining the sampling probability of the current training sample, the update and accumulation unit of the feedback learning layer mentioned above updates the current training sample and the corresponding sampling probability to the training set of the policy network. The training set includes multiple historical training samples and the sampling probability corresponding to each historical training sample (and after the training set is updated, the sampling probability corresponding to each historical training sample needs to be updated).

[0102] Next, it is determined that the policy network has met the update conditions, and the policy network is updated based on the updated training set. Here, after updating the training set of the policy network, the update and stagnation unit of the aforementioned feedback learning layer determines whether the policy network has met the update conditions. These update conditions may be, for example, that the number of training samples in the updated training set exceeds 1000, or that the policy network has not been updated for 24 hours. Once it is determined that the policy network has met the update conditions, a priority experience replay mechanism is used to randomly sample a batch of training samples from the updated training set according to the sampling probability, and the parameters of the policy network are updated based on this batch of training samples.

[0103] It is understandable that when updating the parameters of the policy network, for example, the advantage function can be calculated based on all multidimensional rewards and all state value scalars corresponding to the batch training samples, and the parameters of the policy network (composed of a shared layer and a policy head) and the value network (composed of a shared layer and a value head) can be updated in K (e.g., K=10) rounds mini-batch mode using the batch training samples. Furthermore, in the embodiments of this disclosure, the optimization objective function (adding a risk penalty term) applied to the policy network during the update can be seen, for example, in the expression shown below:

[0104]

[0105] In the above formula, To optimize the objective function, The objective function is the original objective function of a policy network well-known in this field (e.g., determined based on the dominance function, the probability ratio of new and old policies, and the pruning threshold), and the risk penalty weight hyperparameter is a preset parameter. The current risk value (each training sample in the batch training samples has a corresponding current risk value, and the determination process can be found above). A dynamic risk budget is used (each training sample in the batch of training samples has a corresponding dynamic risk budget, the determination process of which can be found above). In this way, when the actual risk of a repair action exceeds the budget, the loss can be increased additionally, thereby suppressing high-risk repair actions.

[0106] In this way, a comprehensive quantitative evaluation of each repair operation can be carried out based on the five-dimensional reward function. Combined with the priority experience replay mechanism, high-information samples can obtain a higher sampling probability, realizing continuous iterative optimization of the policy network. In addition, experience can be summarized from each operation and maintenance practice to improve decision-making, fundamentally solving the defects of static and solidified policies in existing technologies, thus possessing self-evolution capabilities.

[0107] In some implementations, after performing repair operations based on the estimated execution time and repair action sequence, the operation and maintenance terminal can further determine root cause features based on abnormal data characteristics and node failure probabilities by the feedback learning layer, and determine policy features based on the repair action sequence and estimated execution time. Here, the update and sedimentation unit of the aforementioned feedback learning layer (or the case sedimentation unit alone, not limited thereto) determines the root cause features based on abnormal data characteristics and node failure probabilities. These root cause features can be, for example, a 64-dimensional feature vector composed of abnormal data characteristics (such as abnormal service nodes, abnormal scores corresponding to abnormal service nodes, abnormal dimensions, and abnormal patterns), node failure probabilities, and the aforementioned compressed fragments of log semantic features, topological abnormality probabilities, edge propagation weights, etc., and are not limited thereto; and, based on the repair action sequence and estimated execution time, policy features are also determined. These policy features can be, for example, a 64-dimensional feature vector composed of the repair action sequence, estimated execution time, and the aforementioned historical action vector (42-dimensional), execution mode, approval level, etc., in the current state vector, and are not limited thereto.

[0108] Subsequently, the historical database is updated based on root cause features, strategy features, comprehensive risk value, repair feedback data, and current multidimensional rewards. Here, root cause features and strategy features can be concatenated to obtain a case feature vector (e.g., a 128-dimensional feature vector). Then, a repair strategy sequence can be concatenated based on the aforementioned repair action sequence (or a manually modified repair action sequence), the actual repair actions performed by the execution engine layer during repair (or possibly including parameter-validated repair actions), and additional parameters (e.g., connection pool limit, circuit breaker ratio, or timeout). Next, the repair effect verification data determined by whether the repair feedback data is within the preset baseline range and the stable duration after the completion of the maintenance actions, and the current multidimensional rewards (or...) can be used. The structured result data, determined based on the corresponding repair effect rewards, execution time rewards, business impact rewards, resource consumption rewards, and risk control rewards, and the execution status, whether rollback was triggered, or the comparison of indicators before and after the execution of maintenance actions in the repair feedback data, are concatenated with the actual time (i.e., the duration of the fault) mentioned above to obtain the execution effect indicator data. Then, the case feature vector, repair strategy sequence, repair effect verification data, current multi-dimensional rewards, execution effect indicator data, comprehensive risk value, and success / failure labels (and other data may also be included, not limited to this) mentioned above are updated to the historical database in a structured format.

[0109] In embodiments of this disclosure, the historical database can be configured, for example, as a FAISS (Facebook AI SimilaritySearch) vector database, to update the vector index of the aforementioned case feature vectors in the FAISS vector database. It is understood that the vector index of the FAISS vector database adopts an IVF_PQ (Inverted File with Product Quantization) composite index structure. The IVF (Inverted File Index) divides the search space into multiple inverted lists through clustering, and only searches the clusters closest to the query vector during retrieval, thereby reducing the search scope. The PQ (Product Quantization) divides the high-dimensional vector into multiple sub-vectors and quantizes them separately, significantly compressing storage space and accelerating distance calculation. With the IVF1024_PQ16 index configuration, the retrieval latency for 1 million cases can be guaranteed to be less than 15 milliseconds, supporting millisecond-level similar case retrieval.

[0110] Furthermore, after determining the 64-dimensional feature vector composed of abnormal data features (such as abnormal service nodes, abnormal scores corresponding to abnormal service nodes, abnormal dimensions, and abnormal patterns), node failure probability, and the aforementioned compressed fragments of log semantic features, topological anomaly probability, and edge propagation weights, high-matching historical cases with a similarity greater than 85% can be selected based on the aforementioned FAISS vector database. Then, successful historical action sequences are quickly determined from the high-matching historical cases to replace the repair action sequences generated by the policy network, thereby improving the generation efficiency of the repair policy. The historical action sequence is then executed with reference to the above. After the repair operation is performed, the case feature vector corresponding to the historical action sequence can be updated to the vector index in the FAISS vector database.

[0111] In this way, the root cause characteristics and strategy characteristics of each failure can be structurally stored in the historical database, forming a searchable and reusable operation and maintenance knowledge base. When similar failures occur in the future, historical successful cases can be retrieved first and their repair strategies can be recommended, realizing the continuous accumulation and efficient reuse of operation and maintenance experience, further shortening the failure handling time and improving the repair success rate.

[0112] Please see Figure 3 The diagram illustrates the overall architecture of an operation and maintenance terminal according to some embodiments of this disclosure. Figure 3 As shown, the overall architecture 300 of the operation and maintenance terminal can include a data acquisition layer, an intelligent decision-making layer, an execution engine layer, and a feedback learning layer. The data acquisition layer can be further subdivided into an eBPF probe unit, a cgroup acquisition unit, a log acquisition unit, and a protocol adaptation unit. The intelligent decision-making layer can be further subdivided into an anomaly detection unit, a topology reasoning unit, a log encoding unit, a gating fusion unit, and a policy generation unit. The execution engine layer can have an operation and maintenance-specific execution engine extended on the general intelligent agent framework OpenClaw. Its specific subdivision structure can be referred to above. The feedback learning layer can be further subdivided into a reward calculation unit and an update and accumulation unit. The update and accumulation unit can be further divided into a case accumulation unit, for example. The execution process of each unit can be referred to above, and will not be elaborated further here.

[0113] In some implementations, taking the scenario of an e-commerce promotional event where the order service connection pool is exhausted and the error rate suddenly increases from 0.1% to 25% as an example, referring to one or more of the above embodiments, the process can be simplified as follows: the eBPF probe unit detects a sudden increase in the MySQL connection rejection rate to 18%, the cgroup collection unit shows that the number of blocked order service threads has increased by 300%, and the log collection unit collects "Connection pool" data. An "exhausted" exception occurred; the error between the reconstructed time-series features and the current time-series features was 0.38, which is greater than the preset error threshold (e.g., 0.25) and lasted for 20 seconds, indicating that fault repair was required; the edge propagation weight of the abnormal connection edge in the service topology graph was determined to be 0.92, log semantic feature analysis confirmed the connection pool exhaustion semantics, and based on the maximum probability of node failure, the root cause service node failure was determined to be insufficient MySQL connection count; the repair action sequence generated by the policy network included [querying the current configuration, calculating the optimal number of connections, expanding the connection pool, and verifying the load]; the comprehensive risk value was determined to be 74 points, triggering manual approval, which was approved by the operations director within 10 minutes, with the additional condition of monitoring connection utilization within 2 hours after expansion; the estimated execution time was determined to be 45 seconds, asynchronous mode was enabled, and parameter verification was performed sequentially: max_connections=2000 (originally 1500), permission check: authorization through DBA role, and execution change: ALTERGLOBAL VARIABLES SET `max_connections=2000`, Result Capture: Operation successful, new connection count effective; After the repair action was completed, connection waiting time decreased from 1200 milliseconds to 20 milliseconds, service error rate returned to 0.2%, and remained stable for 120 seconds, confirming successful repair; The current multidimensional reward was determined to be 0.91, and the policy network parameters were updated to enhance the handling capability of connection pool-type faults; The corresponding case feature vectors were updated to the historical database, improving the hit rate of subsequent similar fault retrieval by 40%. The overall processing time was 4 minutes and 15 seconds, and the service interruption time was only 68 seconds, which is 6.6 times more efficient than manual processing (average 28 minutes).

[0114] In some implementations, taking the failure of a recommendation service cache node, which triggers cascading anomalies in order, payment, and inventory services, reducing overall availability to 65%, as an example, the process can be simplified as follows, referring to one or more of the above embodiments: Simultaneously, four service nodes are detected as abnormal, with error rates all greater than 20%. Based on service topology feature analysis, a call chain relationship is confirmed, indicating a cascading failure, requiring the activation of a fast circuit breaker mechanism. Log semantic feature analysis confirms an OOM (Out of Memory) error, and based on the highest probability of node failure, the root cause service node failure is determined to be the Redis cache cluster. The repair action sequence generated by the policy network includes: immediate execution: order to recommendation service circuit breaker (80% degradation ratio); immediate execution: payment to recommendation service circuit breaker (100% degradation ratio); asynchronous execution: Redis master-slave switch; asynchronous execution: cache preheating; recovery phase: gradually increasing the circuit breaker rate. The circuit breaker thresholds were determined as follows: A comprehensive risk value of 65 points was determined for the circuit breaker operation, triggering manual approval, which was granted by the operations team leader within 2 minutes. Similarly, a comprehensive risk value of 82 points was determined for the Redis switch, triggering manual approval, which was granted by the operations director within 5 minutes. The operation was implemented in stages, prioritizing core service links. Circuit breaker degradation was executed sequentially, restoring core service availability to 92% within 30 seconds, asynchronously executing the Redis switch, registering Webhook callbacks, automatically triggering cache preheating after the switch, and gradually canceling the circuit breaker after 15 minutes for complete recovery. After the repair actions were completed, the core transaction link interruption time was less than 90 seconds, the cascading impact was reduced by 76%, and the complete recovery time was 102 seconds. The repair was confirmed as successful, and for subsequent similar faults, the policy network would prioritize a fast circuit breaker strategy to ensure a 40% optimization of response time.

[0115] Figure 4 A block diagram of a microservice operation and maintenance fault repair system based on an intelligent agent engine according to some embodiments of this disclosure is shown. The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for the apparatus embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and relevant parts can be referred to the description of the method embodiments. Figure 4As shown, the microservice operation and maintenance fault repair system 400 based on the intelligent agent engine may include at least an anomaly detection module 402, configured to determine anomaly data characteristics based on the current microservice data, and determine service topology characteristics based on the anomaly data characteristics and the service topology graph; the current microservice data includes at least application log data. The microservice operation and maintenance fault repair system 400 also includes a root cause localization module 404, configured to determine the node failure probability based on the service topology characteristics and the log semantic features corresponding to the application log data, and determine the current state vector based on a first result of the node failure probability and a preset probability interval. The microservice operation and maintenance fault repair system 400 also includes a policy generation module 406, configured to generate a repair action sequence based on the current state vector and a policy network, and determine a comprehensive risk value based on a second result of the current risk value corresponding to the repair action sequence and a dynamic risk budget; the dynamic risk budget is determined based on the node failure probability, the service topology graph, and the historical database. In addition, the microservice operation and maintenance fault repair system 400 based on the intelligent agent engine also includes an engine execution module 408, which is configured to determine the estimated execution time based on the third result of the comprehensive risk value and the preset risk range, and perform repair operations based on the estimated execution time and the repair action sequence.

[0116] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this specification are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in or transmitted through a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., Digital Versatile Discs (DVDs)), or semiconductor media (e.g., Solid State Disks (SSDs)).

[0117] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0118] This disclosure can be a method, apparatus, system, and / or program product. The program product may include a machine-readable storage medium on which machine-readable program instructions for performing various aspects of this disclosure are loaded. The machine-readable program instructions described herein can be downloaded from the machine-readable storage medium to various computing / processing devices, or downloaded via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the machine-readable program instructions from the network and forwards them to the machine-readable storage medium in the respective computing / processing device.

[0119] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing. Furthermore, although operations are depicted in a specific order, this should be understood as requiring that such operations be performed in the specific order shown or in sequential order, or requiring that all illustrated operations be performed to achieve the desired result. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the foregoing discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented individually or in any suitable sub-combination in multiple implementations.

[0120] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A microservice operation and maintenance fault repair method based on an intelligent agent engine, characterized in that, include: Based on the anomalies in the current microservice data, the abnormal data characteristics are determined based on the current microservice data, and the service topology characteristics are determined based on the abnormal data characteristics and the service topology graph. The current microservice data includes at least application log data; Based on the service topology features and the log semantic features corresponding to the application log data, the node failure probability is determined, and based on the first result of the node failure probability and the preset probability interval, the current state vector is determined. A repair action sequence is generated based on the current state vector and the policy network, and a comprehensive risk value is determined based on the current risk value corresponding to the repair action sequence and the second result of the dynamic risk budget. The dynamic risk budget is determined based on the node failure probability, the service topology graph, and the historical database. as well as Based on the third result of the comprehensive risk value and the preset risk range, the intelligent agent engine determines the estimated execution time based on the repair action sequence, and performs the repair operation based on the estimated execution time and the repair action sequence.

2. The method according to claim 1, characterized in that, The method further includes: Based on the current microservice data, the current microservice data is converted into a new format to obtain the current time-series features; Based on the current temporal features and the deep learning model, reconstructed temporal features are determined, and it is determined whether the error between the reconstructed temporal features and the current temporal features exceeds a preset error threshold; and In response to determining that the error between the reconstructed time-series feature and the current time-series feature exceeds the preset error threshold, it is determined that the current microservice data is abnormal.

3. The method according to claim 1, characterized in that, The process of determining service topology features based on the abnormal data features and service topology graph includes: Based on the abnormal data characteristics and the preset service score, the abnormal propagation weight of each abnormal service node in the service topology graph is determined, and the initial feature vector is determined based on the abnormal propagation weight of each abnormal service node and the node metadata vector. Based on the abnormal data characteristics, the edge propagation weights of each abnormal connection edge in the service topology graph are determined, and feature aggregation is performed based on all the initial feature vectors and all the edge propagation weights to obtain the target feature vectors of each abnormal service node; and The service topology features are determined based on all the target feature vectors.

4. The method according to claim 1, characterized in that, The step of determining the node failure probability based on the service topology features and the log semantic features corresponding to the application log data includes: The service topology features and the log semantic features corresponding to the application log data are fused and calculated to obtain the fusion weight; and The log semantic features are mapped to log fault data, and the node fault probability is determined based on the log fault data, the fusion weight, and the service topology features.

5. The method according to claim 1 or 3, characterized in that, The determination of the current state vector based on the first result of the node failure probability and the preset probability interval includes: Based on the first result of the node failure probability and the preset probability interval, determine whether the first result is the maximum probability that the node failure probability is within the preset probability interval; In response to determining that the first result indicates that the maximum probability of the node failure probability is within the preset probability range, a current state vector is determined based on the service topology features, the current microservice data, and the historical database. In response to determining that the highest probability that the first result is not the node failure probability falls within the preset probability range, the service topology features are updated based on the abnormal data features and the service topology graph; or In response to determining that the highest probability that the first result is not the node failure probability falls within the preset probability range, a fault alarm message is generated based on the abnormal data characteristics and the current microservice data.

6. The method according to claim 1 or 3, characterized in that, The determination of the comprehensive risk value based on the second result of the current risk value and dynamic risk budget corresponding to the repair action sequence includes: Based on the second result of the current risk value and dynamic risk budget corresponding to the repair action sequence, determine whether the second result is that the current risk value is less than the dynamic risk budget; In response to determining that the second result is that the current risk value is less than the dynamic risk budget, a service impact score is determined based on the node failure probability, the preset service score, and the service monitoring data. Based on the repair action sequence and the service topology map, an operation impact score is determined, and a historical success score is determined based on the historical database; and The operation complexity score is determined based on the number of actions in the repair action sequence. The operation complexity score, the historical success score, the operation impact score, and the service impact score are then weighted and summed to obtain a comprehensive risk value.

7. The method according to claim 1, characterized in that, The third result, based on the comprehensive risk value and the preset risk range, is used by the intelligent agent engine to determine the estimated execution time based on the repair action sequence, including: Based on the third result of the comprehensive risk value and the preset risk range, determine whether the third result indicates that the comprehensive risk value is within the preset risk range; In response to determining that the third result indicates the comprehensive risk value falls within the preset risk range, the agent engine determines the estimated execution time based on the repair action sequence; or In response to determining that the third result is not within the preset risk range of the comprehensive risk value, an approval work order is generated based on the current microservice data, the service topology characteristics, the repair action sequence, the historical database, and the comprehensive risk value; the approval work order includes a root cause report, repair instructions, estimated impact, and rollback plan.

8. The method according to claim 1, characterized in that, The method further includes: The intelligent agent engine obtains repair feedback data and determines whether the repair feedback data is within a preset baseline range and whether the duration exceeds a preset duration threshold. In response to determining that the repair feedback data is within the preset baseline range and the duration exceeds the preset duration threshold, the current multidimensional reward is determined based on the repair feedback data, and the current training sample is determined based on the current state vector, the repair action sequence, the current multidimensional reward, and the repair state vector corresponding to the repair action sequence. The sampling probability of the current training sample is determined based on the current multidimensional reward, and the training set of the policy network is updated based on the current training sample and the corresponding sampling probability; and Once the policy network is determined to meet the update conditions, the policy network is updated based on the updated training set.

9. The method according to claim 8, characterized in that, The method further includes: Based on the abnormal data characteristics and the node failure probability, root cause characteristics are determined, and based on the repair action sequence and the estimated execution time, strategy characteristics are determined; and The historical database is updated based on the root cause features, the strategy features, the comprehensive risk value, the repair feedback data, and the current multidimensional reward.

10. A microservice operation and maintenance fault repair system based on an intelligent agent engine, characterized in that, include: The anomaly detection module is configured to determine the abnormal data characteristics based on the current microservice data, and determine the service topology characteristics based on the abnormal data characteristics and the service topology graph. The current microservice data includes at least application log data; The root cause localization module is configured to determine the node failure probability based on the service topology features and the log semantic features corresponding to the application log data, and to determine the current state vector based on the first result of the node failure probability and the preset probability interval. The strategy generation module is configured to generate a repair action sequence based on the current state vector and the policy network, and determine a comprehensive risk value based on the current risk value corresponding to the repair action sequence and a second result of dynamic risk budget. The dynamic risk budget is determined based on the node failure probability, the service topology graph, and the historical database. as well as The engine execution module is configured to determine the estimated execution time based on the third result of the comprehensive risk value and the preset risk range, and to perform the repair operation based on the estimated execution time and the repair action sequence.

Images