Fault determination methods, devices, products and electronic equipment for microservice systems

Abstract

This invention discloses a method, apparatus, product, and electronic device for fault determination in microservice systems, relating to the field of artificial intelligence technology. The method involves determining anomaly event flows based on anomaly monitoring information; determining an anomaly event causal graph based on the anomaly event flows; and then using a pre-trained model-driven agent, combined with the event causal graph and fault mode criteria, determining the fault point, fault chain, and fault classification. This method achieves anomaly detection, fault classification, and root cause localization in microservice systems, realizing a closed-loop diagnosis from anomaly discovery to root cause localization. Furthermore, from the perspective of events and causal relationships, deep feature engineering and semantic abstraction are performed on the refined data. Through graph modeling technology, system behaviors and dependencies scattered across different modalities are integrated into a unified event causal graph, providing comprehensive and high-dimensional input for the agent's fault reasoning, thus improving the accuracy of microservice fault determination.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a fault determination method and device for a micro-service system, a product and an electronic device. BACKGROUND

[0002] Micro-service architecture is a software architecture style that splits monolithic applications into small service units deployed independently. With the wide application of micro-service architecture in cloud computing systems, the complexity and dynamics of the system have increased significantly, leading to frequent system failures.

[0003] In related technologies, the method of artificial intelligence for IT operations (AIOps) is used for fault diagnosis. By extracting shallow statistical features such as the number of occurrences of specific error logs, the average length of service call chains, and the fluctuation range of resource usage, these features are compared with historical normal baselines or input into a simple model for anomaly recognition. If the features deviate from the normal range, it is determined that there is a potential fault, and the associated nodes are preliminarily located in combination with service dependency relationships. Since these shallow statistical features only reflect the quantitative rules on the surface of the data, they cannot capture deep information, making it difficult to accurately trace the root cause when facing complex faults.

[0004] Therefore, how to improve the accuracy of micro-service fault determination is a technical problem that needs to be solved by those skilled in the art. SUMMARY

[0005] The purpose of the present application is to provide a fault determination method, device, product and electronic device for a micro-service system, to solve the problem of low accuracy in analyzing the fault condition of a micro-service system based on shallow features of the micro-service system.

[0006] To solve the above technical problems, the present application provides a fault determination method for a micro-service system, comprising:

[0007] Obtaining abnormal monitoring information of the micro-service system, and determining an abnormal event stream according to the abnormal monitoring information; wherein the monitoring information at least includes index information, log information and call chain information;

[0008] Obtaining entity information in the abnormal event stream and the relationship between entities; wherein the entity information in the abnormal event stream is determined by event description information of the abnormal event stream;

[0009] Determining an event causal graph according to the entity information and the relationship between entities;

[0010] The pre-trained model driven agent is combined with an event causal graph and a pre-set fault mode to determine an abnormal service in the event causal graph, a propagation path related to the abnormal service, and a fault classification of the abnormal service.

[0011] In one aspect, determining an abnormal event flow according to the abnormal monitoring information includes:

[0012] Determining an index abnormal event according to the abnormal index information; wherein the index at least includes a utilization rate index of a processor and a delay index of a system;

[0013] Determining a log abnormal event according to the abnormal log information;

[0014] Determining an abnormal tracking event according to the abnormal call chain information;

[0015] Determining an abnormal event flow according to the index abnormal event, the log abnormal event, and the abnormal tracking event.

[0016] In another aspect, the monitoring information is index information, and obtaining abnormal index information of the microservice system includes:

[0017] Obtaining an observation value and a preset value of a target index;

[0018] In a case where a deviation between the observation value and the preset value is greater than a preset deviation, determining that the target index is an abnormal index;

[0019] Obtaining abnormal index information; wherein the abnormal index information at least includes a time at which an abnormal index occurs, a service affected by the abnormal index, a service instance affected by the abnormal index, an abnormal index name, an abnormal type, and an abnormal score of the abnormal index;

[0020] The information of the index abnormal event at least includes a unique identifier of the event and the abnormal index information.

[0021] In another aspect, determining a log abnormal event according to the abnormal log information includes:

[0022] Extracting entity information and events in the abnormal log information through a pre-trained model; wherein the entity information in the abnormal log information at least includes a service, a resource, an error code, and a request identifier;

[0023] Determining a type of the extracted event according to a pre-set fault mode or event type; wherein the type of the event at least includes a connection failure event type and a business logic error event type.

[0024] In another aspect, the monitoring information is call chain information; obtaining abnormal call chain information of the microservice system includes:

[0025] According to the call chain data, a service topology graph is constructed, and information of the service topology graph is obtained; wherein the information of the service topology graph at least includes inter-service dependency direction, service call frequency and call delay;

[0026] According to the information of the service topology graph, abnormal call chain information is determined;

[0027] The information of the abnormal tracking event at least includes event type, tracking identifier, root service, bottleneck service and error type.

[0028] In another aspect, it also includes:

[0029] The service topology graph and the information of the service topology graph are stored in a database, and inter-service call events are determined according to the information of the service topology graph;

[0030] The information of the inter-service call event at least includes event type, target caller service, source caller service and delay.

[0031] In another aspect, obtaining entity information in the abnormal event stream and relationships between entities includes:

[0032] The abnormal event stream and the service topology graph stored in the database are input into a pre-trained model;

[0033] Entity information in the abnormal event stream and relationships between entities are extracted by the pre-trained model;

[0034] The entity information in the abnormal event stream at least includes service name, instance identifier, associated performance indicators, abnormal type and resources extracted from event description of the abnormal event stream;

[0035] The relationships between entities at least include dependency relationships determined based on the abnormal tracking event, causal relationships determined based on error stacks in log abnormal events, propagation relationships determined based on error transmission on the tracking path, and associated relationships determined based on request identifiers and thread identifiers associated with different modes of events.

[0036] In another aspect, determining an event causal graph according to the entity information and the relationships between entities includes:

[0037] The entity information is taken as a node of the event causal graph;

[0038] The relationships between entities are taken as edges between nodes of the event causal graph;

[0039] After determining the event causal graph according to the entity information and the relationships between entities, it further includes:

[0040] at least acquiring an index abnormal value and a log error summary;

[0041] determining attribute information of nodes and attribute information of edges between nodes in the event causal graph based on the index abnormal value and the log error summary.

[0042] On the other hand, after determining the event causal graph according to the entity information and the relationship between the entities, further comprising:

[0043] storing the event causal graph, the attribute information of nodes in the event causal graph and the attribute information of edges between nodes in the event causal graph in the database.

[0044] On the other hand, before the agent driven by the pre-trained model determines the abnormal service in the event causal graph, the propagation path related to the abnormal service and the fault classification of the abnormal service in combination with the event causal graph and the pre-set fault mode, further comprising:

[0045] detecting the fault condition of the microservice system through the agent coordinator; wherein the fault condition includes a fault occurrence condition or a fault non-occurrence condition;

[0046] when the agent coordinator detects that the fault condition of the microservice system is the fault occurrence condition, sending prompt information for representing fault detection to the target agent;

[0047] the agent driven by the pre-trained model determines the abnormal service in the event causal graph, the propagation path related to the abnormal service and the fault classification of the abnormal service in combination with the event causal graph and the pre-set fault mode, comprising:

[0048] the target agent driven by the pre-trained model determines the abnormal service in the event causal graph, the propagation path related to the abnormal service and the fault classification of the abnormal service in combination with the event causal graph and the pre-set fault mode.

[0049] On the other hand, the target agent driven by the pre-trained model determines the abnormal service in the event causal graph, the propagation path related to the abnormal service and the fault classification of the abnormal service in combination with the event causal graph and the pre-set fault mode, comprising:

[0050] in the target agent, the state of the event causal graph is obtained through the pre-trained model; the abnormal signal is recognized according to the state of the event causal graph, and the graph query tool is called to extract the abnormal service related to the abnormal signal from the event causal graph;

[0051] the graph query tool and the knowledge base retrieval tool are called to query the propagation path of the abnormal signal in the microservice system, and the causal chain leading to the abnormal service is determined according to the propagation path.

[0052] determine the fault classification of the abnormal service according to the fault classification stored in the database.

[0053] In another aspect, the target agent at least includes a first type target agent, a second type target agent and a third type target agent;

[0054] In the case where the agent coordinator detects that the fault condition of the microservice system is a fault, sending prompt information for representing fault detection to the target agent includes:

[0055] In the case where the agent coordinator detects that the fault condition of the microservice system is a fault, sending prompt information for representing starting fault detection to the first type target agent by the agent coordinator;

[0056] The target agent driven by the pre-trained model, in combination with the event causal graph and the pre-set fault mode, determines the abnormal service in the event causal graph, the propagation path related to the abnormal service and the fault classification of the abnormal service includes:

[0057] In the first type target agent, the state of the event causal graph is obtained through the pre-trained model; the abnormal signal is identified according to the state of the event causal graph, and the graph query tool is called to extract the abnormal service related to the abnormal signal from the event causal graph; the abnormal service related to the abnormal signal is sent to the agent coordinator; so that the agent coordinator determines the candidate causal chain causing the abnormality of the abnormal service according to the event causal graph, and sends prompt information for representing fault detection of the candidate causal chain to the second type target agent;

[0058] In the second type target agent, the graph query tool and the knowledge base retrieval tool are called to query the propagation path of the abnormal signal in the microservice system, and it is determined that the causal chain causing the abnormality of the abnormal service is the candidate causal chain according to the propagation path; information containing the candidate causal chain is sent to the agent coordinator; so that the agent coordinator sends prompt information for representing fault classification to the third type target agent;

[0059] In the third type target agent, the fault classification of the abnormal service is determined according to the fault classification stored in the database.

[0060] In another aspect, calling the graph query tool to extract the abnormal service related to the abnormal signal from the event causal graph includes:

[0061] In the first type target agent, it is judged whether the result returned by the graph query tool meets the first preset requirement;

[0062] If no, the thought chain reasoning is performed according to the result returned by the graph query tool and the first preset requirement to generate a new thinking path and determine the query parameter of the graph query tool based on the new thinking path; the result returned by the graph query tool after querying according to the query parameter is acquired; and it is determined whether the result returned by the graph query tool meets the first preset requirement.

[0063] If yes, the abnormal service related to the abnormal signal is determined according to the result returned by the graph query tool.

[0064] The calling of the graph query tool and the knowledge base retrieval tool to query the propagation path of the abnormal signal in the microservice system comprises:

[0065] In the second type of target agent, it is determined whether the result returned by the graph query tool and the knowledge base retrieval tool meets the second preset requirement.

[0066] If no, the thought chain reasoning is performed according to the result returned by the graph query tool and the second preset requirement to generate a new thinking path and determine the query parameter of the result returned by the graph query tool and the knowledge base retrieval tool based on the new thinking path; the result returned by the graph query tool and the knowledge base retrieval tool after querying according to the query parameter is acquired; and it is determined whether the result returned by the graph query tool and the knowledge base retrieval tool meets the second preset requirement.

[0067] If yes, the propagation path of the abnormal signal in the microservice system is determined according to the result returned by the graph query tool and the knowledge base retrieval tool.

[0068] On the other hand, before the agent coordinator sends the prompt information for representing the fault detection of the candidate causal chain to the second type of target agent, the agent coordinator further comprises:

[0069] The actual fault determination progress and the load condition of the agent are acquired.

[0070] The second type of target agent is determined according to the actual fault determination progress and the load condition of the agent.

[0071] Before the agent coordinator sends the prompt information for representing the fault classification to the third type of target agent, the agent coordinator further comprises:

[0072] The third type of target agent is determined according to the actual fault determination progress and the load condition of the agent.

[0073] On the other hand, the graph query tool is at least used to perform the query on the nodes, edges and paths in the event causal graph.

[0074] The knowledge base retrieval tool is used at least to query historical fault cases, service metadata, operation and maintenance manuals, fault modes, fault solutions in a database, and to provide context and prior knowledge for reasoning.

[0075] In another aspect, it also includes:

[0076] Obtaining the thinking path of each agent, the called tool, the query parameters of the tool, and the result returned by the tool;

[0077] Output information containing the thinking path of each agent, the called tool, the query parameters of the tool, and the result returned by the tool.

[0078] To solve the above technical problems, the application further provides a fault determination device of a micro-service system, comprising:

[0079] A first acquisition module is configured to acquire abnormal monitoring information of the micro-service system and determine an abnormal event stream according to the abnormal monitoring information, wherein the monitoring information at least includes index information, log information, and call chain information;

[0080] A second acquisition module is configured to acquire entity information in the abnormal event stream and the relationship between entities, wherein the entity information in the abnormal event stream is determined by event description information of the abnormal event stream;

[0081] A first determination module is configured to determine an event causal diagram according to the entity information and the relationship between entities;

[0082] A second determination module is configured to determine an abnormal service in the event causal diagram, a propagation path related to the abnormal service, and a fault classification of the abnormal service by using a pre-trained model-driven agent and combining the event causal diagram and a pre-set fault mode.

[0083] To solve the above technical problems, the application further provides a computer program product, comprising computer programs / instructions, which, when executed by a processor, implement the steps of the above-mentioned fault determination method of the micro-service system.

[0084] To solve the above technical problems, the application further provides an electronic device, comprising:

[0085] A memory is configured to store computer programs;

[0086] A processor is configured to implement the steps of the above-mentioned fault determination method of the micro-service system when executing the computer programs.

[0087] To solve the above technical problems, the application further provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the fault determination method of the micro-service system.

[0088] The application has the advantages that: firstly, in the method, after obtaining the abnormal monitoring information of the micro-service system, the abnormal event stream is determined according to the abnormal monitoring information; the abnormal event causal diagram is determined based on the abnormal event stream; then, the pre-trained model driven agent is used, and the fault point, fault chain and fault classification are determined in combination with the event causal diagram and the fault mode standard. That is, the abnormal detection, fault classification and root cause positioning of the micro-service system are realized through the method, and the closed-loop diagnosis from abnormal discovery to root cause positioning is realized; secondly, compared with the method of determining the micro-service fault based on the shallow statistical features, in the method provided by the application, the deep feature engineering and semantic abstraction are performed on the refined data from the perspective of events and causal relationships, the system behaviors and dependency relationships scattered in different modal data are fused into a unified event causal diagram through the graph modeling technology, comprehensive and high-dimensional inputs are provided for the fault reasoning of the agent, and the accuracy of the micro-service fault determination is improved; thirdly, when determining the micro-service fault, the pre-trained model driven agent is used for analysis, so that the general knowledge and semantic understanding ability learned by the pre-trained model can be used for fault analysis, and the efficiency and accuracy of the micro-service system fault determination are improved.

[0089] In addition, the abnormal event stream is determined according to the index abnormal event, the log abnormal event and the abnormal tracking event, the information limitation of a single event dimension is broken, the abnormal root cause is more accurately and comprehensively positioned, and the accuracy of the fault determination is improved.

[0090] When the abnormal call chain information of the micro-service system is obtained, the service topology graph is constructed according to the call chain data, then the information of the service topology graph is obtained, and finally the abnormal call chain information is determined according to the information of the service topology graph. Through the construction of the service topology graph, the user can intuitively understand the call relationship between the services in the micro-service system; and based on the information of the service topology graph such as the dependency direction between the services, the call frequency and the call delay of the services, the abnormal call chain information can be accurately determined.

[0091] The entity information and the relationship between the entities in the abnormal event stream are extracted through the pre-trained model, the dependence on the domain prior knowledge and the artificial rule construction can be effectively reduced, the cross-domain adaptability and the complex text understanding ability are stronger, the efficiency and the accuracy of the entity and relationship extraction can be improved while the artificial cost investment is reduced, and the method is especially suitable for processing the unstructured and semantic complex text data scene.

[0092] By taking the entity information and the relationship between entities as the nodes and edges of the event causal graph respectively, and combining the index abnormal value and the log error summary to supplement the attribute information of the nodes and edges, the event causal graph constructed can not only have a clear entity association structure, but also contain accurate abnormal related attribute data, thereby significantly improving the description depth and information integrity of the event causal graph on the event logic.

[0093] The agent coordinator detects the fault condition of the microservice system, sends prompt information of fault detection to the target agent after detecting the fault, and then uses the target agent driven by the pre-trained model to analyze the fault. That is, the agent coordinator realizes the macroscopic detection of the microservice system fault and realizes the unified management of the target agent.

[0094] Through the agent coordinator, different types of agents (such as the first type of target agent for abnormal signal recognition, the second type of target agent for causal chain determination, and the third type of target agent for fault classification processing) are triggered, and based on different types of agents, abnormal detection, fault classification and root cause positioning are completed. This collaborative mechanism effectively overcomes the limitations of traditional single models and fixed expert architectures, improves the accuracy, robustness and explainability of diagnosis, and better adapts to the dynamic changes of the microservice environment.

[0095] A dynamic and adaptive multi-agent collaborative architecture is adopted instead of a fixed expert approach. The agent has the ability of autonomous planning and tool invocation, and can dynamically select the most suitable tool and reasoning path according to the real-time state of the event causal graph and the diagnosis target, and iteratively converge to the root cause. This flexibility enables the system to better adapt to the dynamic changes of the microservice environment and cope with unknown or newly emerging fault modes without the need for frequent retraining of the entire model, significantly improving the robustness and generalization ability of the system.

[0096] The information output includes the thinking path of each agent, the called tool, the query parameters of the tool, and the result returned by the tool. That is, the system not only gives the diagnosis conclusion (fault point, fault chain and fault classification), but also records the observation, thinking and action of the agent at each step (including each tool invocation and its result) in detail. The operation and maintenance personnel can accurately track each step of logic from the original data to the final decision of the model, greatly improving the explainability, credibility and auditability of the diagnosis result. This enables the operation and maintenance personnel to trust the judgment of the model, conduct in-depth verification, and learn from it, making up for the shortcomings of traditional Artificial Intelligence (AI) black box models, which is particularly important for production environments with high safety and compliance requirements.

[0097] In addition, the application further provides a fault determination device, a computer program product, an electronic device and a computer readable storage medium of a micro-service system, which have the same or corresponding technical features and effects as the above-mentioned fault determination method of a micro-service system. BRIEF DESCRIPTION OF DRAWINGS

[0098] In order to more clearly illustrate the embodiments of the present application, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0099] Figure 1 A flowchart of a fault determination method of a micro-service system provided for the embodiments of the present application is shown in

[0100] Figure 2 A structural diagram of a large model driven micro-service fault diagnosis system based on adaptive multi-modal intelligent agent collaborative reasoning provided for the embodiments of the present application is shown in

[0101] Figure 3 A structural diagram of an electronic device provided for the embodiments of the present application is shown in DETAILED DESCRIPTION

[0102] The technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the present application.

[0103] The core of the present application is to provide a fault determination method, device, product and electronic device of a micro-service system, so as to solve the problem of low accuracy in analyzing the fault condition of a micro-service system based on the shallow features of the micro-service system.

[0104] In order to make those skilled in the art better understand the present application, the present application will be further described in detail in combination with the drawings and specific embodiments. Figure 1 A flowchart of a fault determination method of a micro-service system provided for the embodiments of the present application is shown in Figure 1 As shown in the figure, the method comprises:

[0105] S10: acquiring abnormal monitoring information of a micro-service system, and determining an abnormal event stream according to the abnormal monitoring information; wherein the monitoring information at least comprises index information, log information and calling chain information;

[0106] S11: Obtain entity information in the abnormal event stream and the relationship between entities; wherein the entity information in the abnormal event stream is determined by the event description information of the abnormal event stream;

[0107] S12: Determine the event causal graph according to the entity information and the relationship between entities;

[0108] S13: Determine the abnormal service in the event causal graph, the propagation path related to the abnormal service, and the fault classification of the abnormal service by using a pre-trained model driven agent in combination with the event causal graph and the pre-set fault mode.

[0109] In order to improve the accuracy of microservice system fault determination, the abnormal event stream is determined according to the abnormal monitoring information, which comprises:

[0110] Determine the index abnormal event according to the abnormal index information; wherein the index at least includes the utilization rate index of the processor and the delay index of the system;

[0111] Determine the log abnormal event according to the abnormal log information;

[0112] Determine the abnormal tracking event according to the abnormal call chain information;

[0113] Determine the abnormal event stream according to the index abnormal event, the log abnormal event and the abnormal tracking event.

[0114] Specifically, the monitoring information is index information, and the abnormal index information of the microservice system is obtained, which comprises:

[0115] Obtain the observation value and the preset value of the target index;

[0116] In the case where the deviation between the observation value and the preset value is greater than the preset deviation, determine that the target index is an abnormal index;

[0117] Obtain the abnormal index information; wherein the abnormal index information at least includes the time when the abnormal index occurs, the service affected by the abnormal index, the service instance affected by the abnormal index, the abnormal index name, the abnormal type and the abnormal score of the abnormal index;

[0118] The information of the index abnormal event at least includes: the unique identifier of the event and the abnormal index information.

[0119] Determine the log abnormal event according to the abnormal log information, which comprises:

[0120] Extract the entity information and the event in the abnormal log information by the pre-trained model; wherein the entity information in the abnormal log information at least includes the service, the resource, the error code and the request identifier;

[0121] Determine the type of the extracted event according to a preset fault mode or event type; wherein the type of the event at least includes a connection failure event type and a business logic error event type.

[0122] The monitoring information is call chain information; the abnormal call chain information of the micro-service system is obtained by:

[0123] According to the call chain data, a service topology graph is constructed, and information of the service topology graph is obtained; wherein the information of the service topology graph at least includes the inter-service dependency direction, the service call frequency and the call delay;

[0124] The abnormal call chain information is determined according to the information of the service topology graph;

[0125] The information of the abnormal tracking event at least includes: event type, tracking identifier, root service, bottleneck service and error type.

[0126] In addition, the inter-service call event can also be determined according to the information of the service topology graph; wherein the information of the inter-service call event at least includes: event type, target caller service, source caller service and delay.

[0127] In order to obtain the service topology graph, in the implementation, the fault determination method of the micro-service system further comprises: storing the service topology graph and the information of the service topology graph in the database.

[0128] The above describes the process of obtaining the abnormal monitoring information of the micro-service system and determining the abnormal event flow according to the abnormal monitoring information; wherein the monitoring information at least includes the index information, the log information and the call chain information. The module that realizes this function is called a multi-modal semantic extraction module. In order to make the person skilled in the art better understand the function realized by the multi-modal semantic extraction module, the heterogeneous data multi-level semantic extraction mechanism will be described again in combination with specific embodiments.

[0129] The present application introduces a more detailed multi-level semantic extraction strategy, which aims to extract structured and semanticized "events" from the original massive heterogeneous monitoring data, and provides high-quality input for subsequent event causal graph construction and agent reasoning.

[0130] 1) Index data processing: index eventization and anomaly detection.

[0131] Time series prediction and anomaly detection: for each index, the deviation between the observed value and the predicted value (based on the Z-score method) is calculated to identify abnormal points or abnormal segments.

[0132] Event Conversion: Once an anomaly is detected, it is encapsulated into an "Anomaly Event". Each event contains: a unique identifier (event_id), a timestamp, the affected service (service_name), the affected service instance (instance_id), the anomaly metric name (metric_name, e.g., CPU_Usage, P99_Latency), the anomaly type (anomaly_type, e.g., spike, drop, sustained_high, sudden_change), the anomaly score (anomaly_score, representing the quantified anomaly degree), and other relevant metadata (tags, e.g., deployment_version).

[0133] This process enables the conversion from continuous time-series data to discrete, semanticized "Anomaly Events", greatly simplifying subsequent processing.

[0134] 2) Log Data Processing: Log Structuring and Anomaly Event Extraction.

[0135] Keyword and Regular Expression Matching: Quickly filter out a large number of normal logs.

[0136] LLM-driven Anomaly Event Extraction and Classification: Combined with filtered text, LLM performs information extraction (Information Extraction) to identify key entities (services, resources, error codes, request IDs, etc.) and event types (connection failure, OOM, deadlock, business logic error, etc.). LLM is explicitly prompted to map log content to predefined fault patterns or event types (from general knowledge base). For example, the prompt may include "Based on the following log content, extract key entities and events, and classify them into known fault patterns, if no match, identify as a new anomaly event".

[0137] Output: Structured "Log Event Stream". Each event contains:

[0138] event_id, timestamp, service_name, instance_id.

[0139] semantic_event_type: LLM-identified semantic event type (e.g., DB_Connection_Refused, Application_Crash, API_Auth_Failure).

[0140] original_log_summary: A concise semantic summary of the original log.

[0141] 3) Trace data processing: Trace context and service behavior analysis, extract "inter-service call events" and "anomaly trace events" from the original call chain.

[0142] Dynamic topology construction and update: continuously construct and maintain service topology graph G=(V,E) from call chain data, including inter-service dependency direction, call frequency, average delay, etc. This graph will be stored in the general knowledge base for agent query.

[0143] P95 threshold anomaly detection: combined with dynamic threshold, identify response time abnormal Span (call relationship) or entire Trace (call chain).

[0144] Span attribute analysis: extract key attributes of each Span, such as service.name, operation.name, duration, status.code, error label, and associated log event ID.

[0145] Event transformation: encapsulate analysis results as "trace event stream".

[0146] Inter-service call event: {“event_type”:“ServiceCall”,“caller_service”:“UserService”,“callee_service”:“OrderService”,“latency_ms”:50,“success”:true}.

[0147] Anomaly trace event: {“event_type”:“TraceAnomaly”,“trace_id”:“abc”,“root_service”:“Gateway”:“bottleneck_service”:“PaymentService”,“error_type”:“Timeout”}.

[0148] After determining the abnormal event stream, obtain the entity information in the abnormal event stream and the relationship between the entities, including:

[0149] Input the abnormal event stream and the service topology graph stored in the database into the pre-trained model;

[0150] Extract entity information and relationships in the abnormal event stream through the pre-trained model;

[0151] Among them, the entity information in the abnormal event stream at least includes the service name, instance identifier, associated performance indicators, anomaly type and resources extracted from the event description of the abnormal event stream.

[0152] The relationships between entities include at least: dependencies determined based on anomaly tracing events, causal relationships determined based on error stacks in log anomaly events, propagation relationships determined based on error propagation on the tracing path, and association relationships determined based on events that associate different patterns with request identifiers and thread identifiers.

[0153] Determining an event cause-effect graph based on entity information and the relationships between entities includes:

[0154] Use entity information as nodes in the event cause-effect graph;

[0155] The relationships between entities are represented as edges between nodes in an event cause-effect graph;

[0156] After determining the event cause-effect graph based on entity information and the relationships between entities, the process also includes:

[0157] At least obtain outlier metrics and log error summaries;

[0158] As for determining the attribute information of nodes and the attribute information of edges between nodes in the event causal graph based on outlier indicators and log error summaries.

[0159] After determining the event cause-effect graph based on entity information and the relationships between entities, the process also includes:

[0160] Store the event cause-effect graph, the attribute information of the nodes in the event cause-effect graph, and the attribute information of the edges between the nodes in the graph in the database.

[0161] The process of determining the event cause-effect graph has been described above. The module that implements this function is called the event cause-effect graph modeling and system state representation module. To enable those skilled in the art to better understand the function implemented by the event cause-effect graph modeling and system state representation module, specific embodiments will be used to illustrate it again below.

[0162] The core of the event causal graph modeling and system state representation module is the dynamic construction and maintenance of an event causal graph (ECG_t), which represents the behavior, anomalies, and causal relationships of a microservice system within a specific time window. This graph will be the primary "observation object" for agents to perform high-level reasoning.

[0163] 1) Event Aggregation and Time Window Management: Metric event streams, log event streams, and trace event streams from the previous stage are aggregated within a specific time window (e.g., the past 5 minutes). These events constitute the "raw evidence" of ECG_t. A sliding time window mechanism ensures that ECG_t always reflects the latest system state.

[0164] 2) LLM-driven entity-relation extraction and graph construction: This is the core of this module. This invention utilizes a dedicated LLM as the entity-relation extractor.

[0165] Input: The aggregated event stream (in JSON array form), and the latest service topology and metadata from the general knowledge base.

[0166] Prompt Example: "You are a professional graph knowledge builder. Given the following list of microservice events and system topology, identify the key entities (services, instances, resources, fault types, request IDs, etc.) and their relationships (call relationships, exception propagation, causal relationships, attribute relationships). Build a graph structure representing these entities and relationships, outputting it in JSON or GraphML format. For example: 'entity': 'OrderService', 'type': 'Service', 'properties': {'cpu_util': 'high'}, 'relations': [{'target': 'DB', 'type': 'depends_on', 'reason': 'DB_Deadlock'}]. Pay special attention to entity ID and type consistency."

[0167] The core logic is as follows:

[0168] (1) Entity identification: Extract service name, instance ID, KPI (key performance indicator), error code, exception type, resources (such as database, cache, message queue) from the event description as nodes in the graph.

[0169] (2) Relation extraction: Identify the edges between nodes.

[0170] (3) Dependency: Based on service calls in the traced events.

[0171] (4) Causal relationship: based on the error stack in the log event (e.g., causing a service to fail).

[0172] (5) Propagation relationship: based on error propagation on the tracing path.

[0173] (6) Association: Associate events of different modalities based on metadata such as request ID and thread ID (e.g., abnormal metrics, log errors, and call chain timeouts under a certain request ID).

[0174] (7) Attribute association: Treat outlier values ​​of indicators, log error summaries, etc. as attributes of nodes or edges.

[0175] Graph Structure Representation: The generated ECG_t is a directed weighted graph. Nodes and edges have rich attributes. Node attributes: Service nodes can have CPU_anomaly_score and Memory_anomaly_score; error event nodes can have error_type and count; resource nodes can have status. Edge attributes: Dependency edges can have avg_latency, error_rate, and frequency; causal edges can have causal_strength.

[0176] 3) Graph Storage and Query Optimization: After being dynamically constructed, ECG_t is stored in a graph database, providing agents with graph query tools that enable them to efficiently retrieve specific patterns, paths, or associations within the graph. For example, querying "all services with CPU anomalies and related to the database".

[0177] After determining the event causal graph, an agent driven by a pre-trained model is used, combined with the event causal graph and pre-defined fault modes, to determine the abnormal services in the event causal graph, the propagation paths related to the abnormal services, and the fault classification of the abnormal services.

[0178] Furthermore, before using a pre-trained model-driven agent, combined with an event causal graph and pre-defined fault modes, to determine anomalous services in the event causal graph, propagation paths associated with anomalous services, and fault classifications of anomalous services, the process also includes:

[0179] The intelligent agent coordinator detects fault conditions in the microservice system; these fault conditions include both cases where faults have occurred and cases where no faults have occurred.

[0180] When the agent coordinator detects a fault in the microservice system, it sends a prompt message to the target agent to characterize the fault detection.

[0181] Using a pre-trained model-driven agent, and combining an event cause-effect graph with pre-defined fault modes, we determine anomalous services in the event cause-effect graph, the propagation paths associated with anomalous services, and the fault classifications of anomalous services, including:

[0182] By using a pre-trained model-driven target agent and combining it with an event causal graph and pre-defined fault modes, we can determine the abnormal services, the propagation paths associated with the abnormal services, and the fault classifications of the abnormal services in the event causal graph.

[0183] In some embodiments, using a target agent driven by a pre-trained model, and combining an event causal graph and pre-defined fault modes, determining anomalous services, propagation paths associated with anomalous services, and fault classifications of anomalous services in the event causal graph includes:

[0184] In the target agent, the state of the event causal graph is obtained through a pre-trained model; abnormal signals are identified based on the state of the event causal graph, and a graph query tool is invoked to extract abnormal services related to the abnormal signals from the event causal graph;

[0185] Use graph query tools and knowledge base retrieval tools to query the propagation path of abnormal signals in the microservice system, and determine the causal chain that caused the abnormal service to fail based on the propagation path;

[0186] The fault classification of the abnormal service is determined based on the fault classification stored in the database.

[0187] Among them, the graph query tool is at least used to perform queries on nodes, edges, and paths in the event causal graph;

[0188] Knowledge base retrieval tools are used at least to query historical failure cases, service metadata, operation and maintenance manuals, failure modes, failure solutions in the database, and to provide context and prior knowledge for reasoning.

[0189] Traditional single-model architectures, even those based on large pre-trained models, face challenges such as context length limitations, loss of key information, and inconsistent inference paths. When dealing with massive amounts of heterogeneous data, a single model struggles to effectively integrate all information and maintain consistent inference. For example, directly inputting all metrics, logs, and tracking data into a single large language model may exceed its context window limit, leading to information truncation. Therefore, this invention employs multiple agents for fault analysis. In some embodiments, the target agents include at least a first type of target agent, a second type of target agent, and a third type of target agent.

[0190] When the agent coordinator detects a failure in the microservice system, it sends a notification message to the target agent to characterize the failure detection, including:

[0191] When the agent coordinator detects a fault in the microservice system, it sends a prompt message to the first type of target agent to indicate that fault detection has begun.

[0192] Using a pre-trained model-driven target agent, and combining an event causal graph with pre-defined fault modes, we determine anomalous services, propagation paths associated with anomalous services, and fault classifications of anomalous services within the event causal graph, including:

[0193] In the first type of target agent, the state of the event causal graph is obtained through a pre-trained model; abnormal signals are identified based on the state of the event causal graph, and a graph query tool is called to extract abnormal services related to the abnormal signals from the event causal graph; the abnormal services related to the abnormal signals are sent to the agent coordinator; so that the agent coordinator can determine the candidate causal chain that causes the abnormal service to be abnormal based on the event causal graph, and send a prompt message to the second type of target agent to characterize the fault detection of the candidate causal chain;

[0194] In the second type of target agent, the graph query tool and knowledge base retrieval tool are invoked to query the propagation path of the abnormal signal in the microservice system, and the causal chain that caused the abnormal service to be abnormal is determined as a candidate causal chain based on the propagation path; the information containing the candidate causal chain is sent to the agent coordinator; so that the agent coordinator can send the prompt information used to characterize the fault classification to the third type of target agent;

[0195] In the third type of target agent, the fault classification of abnormal services is determined based on the fault classification stored in the database.

[0196] In some embodiments, invoking a graph query tool to extract anomalous services related to anomalous signals from an event cause-effect graph includes:

[0197] In the first type of target agent, determine whether the results returned by the graph query tool meet the first preset requirements;

[0198] If not, then perform thought chain reasoning based on the results returned by the graph query tool and the first preset requirement to generate a new thinking path and determine the query parameters of the graph query tool based on the new thinking path; obtain the results returned by the graph query tool after querying according to the query parameters; and return a judgment on whether the results returned by the graph query tool meet the first preset requirement.

[0199] If so, determine the abnormal service related to the abnormal signal based on the results returned by the graph query tool;

[0200] The propagation paths of abnormal signals in a microservice system can be queried using graph query tools and knowledge base retrieval tools, including:

[0201] In the second type of target intelligent agent, determine whether the results returned by the graph query tool and the knowledge base retrieval tool meet the second preset requirements;

[0202] If not, then based on the results returned by the graph query tool and the knowledge base retrieval tool, and the second preset requirement, perform thought chain reasoning to generate a new thinking path and determine the query parameters of the results returned by the graph query tool and the knowledge base retrieval tool based on the new thinking path; obtain the results returned by the graph query tool and the knowledge base retrieval tool after querying according to the query parameters; return to the step of judging whether the results returned by the graph query tool and the knowledge base retrieval tool meet the second preset requirement;

[0203] If so, the propagation path of the abnormal signal in the microservice system is determined based on the results returned by the graph query tool and the knowledge base retrieval tool.

[0204] The first and second preset requirements are to achieve the diagnostic goal or meet the stopping conditions.

[0205] Furthermore, before the agent coordinator sends a prompt message to the second type of target agent to characterize the fault detection of the candidate causal chain, it also includes:

[0206] Obtain the actual fault determination progress and the agent's load status;

[0207] The second type of target agent is determined based on the actual fault determination schedule and the agent's load status.

[0208] Before the agent coordinator sends the prompt information characterizing the fault classification to the third type of target agent, it also includes:

[0209] The third type of target agent is determined based on the actual fault determination schedule and the agent's load status.

[0210] To enable operations and maintenance personnel to accurately track every step of the model's logic from raw data to final decision, fault determination methods for microservice systems also include:

[0211] Obtain the thought process of each agent, the tools it calls, the query parameters of the tools, and the results returned by the tools;

[0212] The output includes information such as the thought process of each agent, the tools called, the query parameters of the tools, and the results returned by the tools.

[0213] The process described above, which utilizes a pre-trained model-driven agent and combines an event causal graph with pre-defined fault modes to determine anomalous services, propagation paths associated with anomalous services, and fault classifications of anomalous services, can be implemented through an adaptive agent collaborative reasoning module. To enable those skilled in the art to better understand this function implemented by the adaptive agent collaborative reasoning module, the process is described below with reference to specific embodiments.

[0214] Adaptive multi-agent cooperative reasoning module:

[0215] The aim is to drive intelligent agents with a set of large language models that have autonomous planning, decision-making and tool invocation capabilities, so as to achieve dynamic, iterative and interpretable diagnosis of complex faults.

[0216] 1) Agent Orchestrator: Acts as the "brain" of the entire agent system, responsible for: initial perception and task initialization: receiving the event-cause graph (ECG_t) constructed in the previous stage as initial observation. Based on the abnormal patterns in the graph and the preset diagnostic goals, selects and activates the initial perceptual agents.

[0217] Dynamic programming and scheduling: Coordinates the interactions between multiple agents. Based on the current diagnostic progress and agent feedback, it dynamically determines the next agent to be activated, as well as how to allocate tasks and share information.

[0218] Iteration and convergence control: Monitor the diagnostic process, including inference depth, time limits, or confidence thresholds, until convergence conditions are met (e.g., finding a high-confidence root cause or being unable to infer further).

[0219] Context management: Maintain the "shared memory" or "global context" of the entire diagnostic session to ensure information consistency and historical traceability among agents.

[0220] 2) LLM-driven Agent Swarm and Dynamic Collaboration: This invention constructs a group of agents driven by a large language model. These agents are no longer fixed "experts" with specific roles, but possess autonomous planning, decision-making, and tool-calling capabilities. They can dynamically play different roles and collaborate according to task requirements. During the diagnostic process, these agents dynamically play roles and collaborate based on the system state and task requirements. Perception Agent (as described in the first type of target agent above): Primarily responsible for identifying and confirming initial anomalous signals, extracting the most prominent and relevant anomalous events and affected entities from ECG_t. Reasoning Agent (as described in the second type of target agent above): Primarily responsible for constructing causal chains and propagation paths, analyzing anomalous information, iteratively exploring causal relationships, and eliminating irrelevant factors. Decision Agent: Primarily responsible for integrating the evidence and inferences provided by all perception and reasoning agents, combining them with fault patterns in a general knowledge base, ultimately providing the most likely root cause, fault classification, and generating a structured diagnostic report and detailed reasoning process.

[0221] Each intelligent agent possesses the following core capabilities:

[0222] Perception: Through LLM, the perception agent understands the state of the observed event causal graph (or its subgraphs), identifies preliminary anomalous signals, and extracts the most prominent and relevant anomalous events and affected entities from ECG_t. For example, the perception agent might invoke a graph query tool to find the node with the highest CPU utilization.

[0223] Thinking / Planning: The LLM within the agent performs deep reasoning and planning based on current observations (using the ReAct pattern combined with advanced hinting techniques such as Chain-of-Thought), determining the next analytical direction, the tools to be called, and their parameters. This internal thinking enables the agent to autonomously construct causal chains and propagation paths, and eliminate irrelevant factors. For example, after analyzing the anomaly information provided by the perception agent, the reasoning agent will iteratively call graph query tools and knowledge base retrieval tools to explore the propagation path of the anomaly in the system, and may discover a causal chain such as "high CPU usage in service A causes slow response in service B".

[0224] Action (Tool Calling): The agent invokes various external tools through predefined interfaces to obtain more information or perform specific analyses. The results of the tool invocation will be fed back to the agent as new observations, driving the next round of "thinking" and "action".

[0225] Common Toolset: Tools that an agent can invoke include, but are not limited to:

[0226] Graph Query Tool: Executes queries on nodes, edges, paths, and patterns in ECG_t, for example, query_graph(query_type='get_neighbors',node_id='OrderService',relation_type='depends_on').

[0227] Knowledge Base Retrieval Tool: Queries historical failure cases, service metadata, operation and maintenance manuals, failure modes, common solutions, etc. in a general knowledge base, providing context and prior knowledge for reasoning, such as retrieve_kb_entry(query='deadlock common causes').

[0228] Raw Data Query Tool: When necessary, directly query raw logs, metrics, or tracking storage (such as Elasticsearch, Prometheus, Jaeger) to obtain more detailed context and assist in verification, such as query_raw_log(request_id='xyz').

[0229] 3) Adaptive Iterative Reasoning Process. The diagnostic process is an iterative "observation-thinking-action-loop":

[0230] Observe: The agent coordinator gives part or all of ECG_t as the current observation input to the agent.

[0231] Think / Plan: The LLM inside the intelligent body performs chain-of-thought reasoning based on the observation results and the current goal, generating a thinking path and the next action plan (i.e., which tool to call and the parameters of the tool).

[0232] Action (Act): The agent executes the plan, calls the corresponding external tools, and uses the output of the tools as new observations.

[0233] Loop: New observations are fed back to the agent, prompting it to perform the next round of "thinking" and "acting" until the diagnostic goal is achieved or the stopping condition is met. This iterative process allows the agent to dynamically adjust its reasoning strategy based on real-time feedback.

[0234] Generation and structuring of inference chains:

[0235] To maximize interpretability, this invention enhances the generation and structuring of the reasoning chain, making it include the agent's complete "thought trail." The enhanced reasoning chain structure meticulously records every "observation," "thinking," and "action" of the agent throughout the diagnostic process, forming a traceable and auditable trajectory. The reasoning chain not only includes the final inference but also explicitly records each agent's "thinking" (internal planning), "action" (tools invoked and their parameters), and "observation" (results returned by the tools) at each step. Operations personnel can clearly see when and why the agent invoked which tools, and what information the tools returned to drive the reasoning progress, greatly enhancing transparency and verifiability. The entire reasoning chain demonstrates the complete decision-making process from initial anomaly detection to final root cause localization and recommendations, moving beyond a simple accumulation of conclusions.

[0236] The above describes the entire process of fault determination in microservice systems, namely, the large model-driven microservice fault diagnosis process based on adaptive multimodal intelligent agent collaborative reasoning. In order to enable those skilled in the art to better understand the overall technical solution of the present invention, the following description will continue with reference to the accompanying drawings and specific embodiments. Figure 2 A structural diagram of a large model-driven microservice fault diagnosis system based on adaptive multimodal intelligent agent collaborative reasoning, provided in an embodiment of the present invention, is shown below. Figure 2 As shown, the system includes a raw data layer 1, a multimodal semantic extraction module 2, a general knowledge base 3, an event-cause graph modeling and state representation module 4, a multi-agent collaborative reasoning module 5, and a module 6 for generating structured diagnostic reports. The raw data layer 1 includes metrics, logs, and traces. The multimodal semantic extraction module 2 performs metric event identification, structured log extraction, and trace behavior analysis. The general knowledge base 3 stores service topology, fault cases, and domain knowledge. The event-cause graph modeling and state representation module 4 constructs event-cause graphs, extracts entity-relationships, and dynamically updates service states. The multi-agent collaborative reasoning module 5 includes perception / reasoning / decision agents capable of tool calls and adaptive iteration. The module 6 for generating structured diagnostic reports performs anomaly detection (AD), fault classification (FT), and root cause localization (RCL).

[0237] Specifically, the large model-driven microservice fault diagnosis system based on adaptive multimodal intelligent agent collaborative reasoning provided by this invention includes the following modules:

[0238] 1) General knowledge base: As an auxiliary capability throughout the process, it provides structured knowledge such as system history failures and operation and maintenance experience for each module and agent to query, thereby enhancing the accuracy and efficiency of reasoning.

[0239] 2) Multimodal semantic extraction module: Processes raw metrics, logs, and call chain data into a higher-level "event stream".

[0240] Metrics: Transform raw time-series data into “metric anomaly events”, combined with anomaly detection algorithms.

[0241] Logs: Emphasis is placed on semantic event extraction after log templates are created, and LLM is used to classify and refine events.

[0242] Call chain: Emphasizing the extraction of "inter-service call events" and "abnormal call events," as well as behavioral patterns, while preserving the context.

[0243] The core objective is to provide rich, semantically labeled events for the next stage of graph modeling.

[0244] The multimodal semantic extraction module is responsible for multi-level structuring, cleaning, and preliminary semantic extraction of massive heterogeneous raw monitoring data. This module has designed customized hierarchical processing mechanisms for different data types (metrics, logs, traces), aiming to retain the semantic information and fault-related features of the raw data to the greatest extent possible, while eliminating redundancy and noise, providing high-quality, semantically rich input for subsequent modules.

[0245] 3) Event Causal Graph Modeling and System State Representation Module: The core is to construct a dynamically updated event causal graph (ECG_t). Through the entity-relationship extraction capability driven by LLM, it identifies node entities such as services, instances, resources, and error types from the event flow in the previous stage, and extracts the causal, dependency, propagation, and association relationships between them as edges.

[0246] The Event-Cause-Effect Graph Modeling and System State Representation module focuses on deep feature engineering and semantic abstraction of refined data from the perspective of events and causal relationships. This module uses graph modeling techniques to integrate system behaviors and dependencies scattered across different modalities into a unified event-cause-effect graph, providing comprehensive, high-dimensional input for intelligent agents to perform reasoning.

[0247] 4) Adaptive Multi-Agent Collaborative Reasoning Module: An Agent Orchestrator is introduced, responsible for perception, planning, and scheduling. A more general and flexible LLM-driven agent swarm is employed, including perception agents, reasoning agents, and decision-making agents. These agents possess tool calling capabilities, dynamically invoking various external tools to query knowledge bases, analyze graphs, retrieve raw data, and execute specific algorithms. The reasoning process follows an autonomous planning paradigm of "Observe-Think-Act-Loop." Based on observations of the event causal graph, agents autonomously determine their next analytical direction and tool invocation, iteratively converging to the root cause. This mechanism significantly improves the system's adaptability, robustness, and ability to handle unknown faults.

[0248] This adaptive multi-agent collaborative reasoning module consists of agents (rather than fixed experts) driven by a large language model, possessing autonomous planning and tool invocation capabilities. These agents dynamically select, schedule, and iteratively execute tasks based on system state and task requirements, collaboratively performing anomaly detection, fault classification, and root cause localization. This collaborative mechanism effectively overcomes the limitations of traditional single-model and fixed-expert architectures, improving the accuracy, robustness, and interpretability of diagnostics, and better adapting to dynamically changing microservice environments.

[0249] 5) Module for generating structured diagnostic reports: The diagnostic reports will include more detailed reasoning chains, clearly recording the agent's thought process and each tool call and its result, further enhancing transparency and verifiability. Remedial measures will also be recommended.

[0250] The fault determination method for microservice systems provided by this invention implements:

[0251] 1. Unified End-to-End Intelligent Diagnostic Flow: This invention integrates the traditionally fragmented core tasks of anomaly detection (AD), fault classification (FT), and root cause localization (RCL) into a highly automated end-to-end diagnostic process through a carefully designed intelligent agent collaborative reasoning mechanism. This avoids the cumbersome process of switching between different tools and manually integrating results, significantly reducing the integration complexity and operational overhead of the operation and maintenance process. Operation and maintenance engineers can now obtain complete diagnostic reports from a unified system, greatly shortening the Mean Time To Recovery (MTTR) from anomaly discovery to root cause localization. Statistics show that MTTR reduction is a key indicator for measuring the effectiveness of AIOps systems, and this method is expected to reduce it by more than 30%.

[0252] 2. Deep Causal Relationship Discovery and Understanding: Unlike existing methods that only perform shallow data fusion, this invention constructs an event causal graph (ECG_t) and utilizes intelligent agents for deep reasoning. This enables the automatic discovery, modeling, and understanding of complex causal relationships and fault propagation paths between entities from massive heterogeneous events. This capability allows the system to go beyond superficial fault symptoms and directly address the root cause. For example, it can not only identify high CPU usage but also trace back to the specific operation or internal deadlock that caused the high CPU usage, thereby discovering complex fault modes involving cross-service and cross-modal operations that are difficult to identify using traditional methods.

[0253] 3. High Transparency and Auditable Reasoning Process: This invention introduces a structured, detailed, and traceable agent reasoning trace into the diagnostic results. This means the system not only provides diagnostic conclusions but also meticulously records the agent's "observations," "thoughts," and "actions" at every step (including each tool call and its result). Operations personnel can precisely track every step of the model's logic from raw data to final decision, greatly improving the interpretability, credibility, and auditability of the diagnostic results. This allows operations personnel to trust the model's judgments, conduct in-depth verification, and learn from them, compensating for the shortcomings of traditional AI "black box" models, which is particularly important for production environments with high security and compliance requirements.

[0254] 4. Adaptability and Robustness: This invention employs a dynamic, adaptive multi-agent collaborative architecture, rather than fixed experts. Agents possess autonomous planning and tool invocation capabilities, dynamically selecting the most suitable tools and inference paths based on the real-time state of the event causal graph and diagnostic objectives, iteratively converging to the root cause. This flexibility allows the system to better adapt to the dynamic changes in the microservice environment and cope with unknown or emerging failure modes without frequent retraining of the entire model, significantly improving the system's robustness and generalization ability.

[0255] 5. Low False Alarm Rate and High-Accuracy Diagnosis: This method deeply correlates and cross-validates abnormal events, log semantic events, and tracked behavioral events in an event causal graph. Multiple agents collaboratively reason and verify each other from different perspectives, effectively identifying and eliminating false anomalies or irrelevant information. This multimodal, multi-agent interactive verification mechanism significantly reduces the false alarm rate and improves the accuracy of fault diagnosis. In real-world production environments, it is expected to achieve higher accuracy than single-modal or shallow fusion schemes, while reducing MTTD (Mean Time To Detect) to minutes or even seconds.

[0256] 6. Scalability and Domain Adaptability: This system adopts an architecture that separates the agent and tool layers, possessing excellent scalability. In the future, new LLM agent roles (e.g., fault repair agents, security audit agents) can be easily introduced, or new specialized tools (e.g., code analysis tools, configuration management tools) can be integrated to address evolving operational needs. This architecture allows the invention to extend beyond fault diagnosis to a broader AIOps domain, demonstrating strong foresight and cutting-edge technology.

[0257] 7. Significantly Reduced Operation and Maintenance Costs and Increased Efficiency: Through highly intelligent automated fault diagnosis, this invention can greatly reduce the time and effort that operation and maintenance engineers spend on fault discovery, localization, and analysis. This allows the operation and maintenance team to devote more energy to system optimization, architecture improvement, and innovative business support, thereby improving overall operation and maintenance efficiency and team value. In large-scale complex systems, this can save significant human resources, significantly reduce operating costs, and reduce service interruptions caused by human error.

[0258] In the above embodiments, the fault determination method for microservice systems has been described in detail. This invention also provides embodiments of fault determination devices and electronic devices for microservice systems. It should be noted that this invention describes the embodiments of the device portion from two perspectives: one based on functional modules, and the other based on hardware.

[0259] The fault determination device for a microservice system provided in the embodiments of the present invention, based on the perspective of functional modules, includes:

[0260] The first acquisition module is used to acquire abnormal monitoring information of the microservice system and determine the abnormal event flow based on the abnormal monitoring information; wherein, the monitoring information includes at least indicator information, log information and call chain information;

[0261] The second acquisition module is used to acquire entity information and relationships between entities in the abnormal event stream; wherein, the entity information in the abnormal event stream is determined by the event description information of the abnormal event stream;

[0262] The first determination module is used to determine the event cause-effect graph based on entity information and the relationships between entities;

[0263] The second determination module is used to determine the abnormal services, the propagation paths associated with the abnormal services, and the fault classifications of the abnormal services in the event cause-effect graph by using an agent driven by a pre-trained model and combining the event cause-effect graph with pre-defined fault modes.

[0264] Since the embodiments of the apparatus and the embodiments of the method correspond to each other, please refer to the description of the embodiments of the method for the embodiments of the apparatus, which will not be repeated here.

[0265] Figure 3 A structural diagram of an electronic device provided in an embodiment of the present invention. This embodiment is based on a hardware perspective, such as... Figure 3 As shown, the electronic device includes:

[0266] Memory 20 is used to store computer programs;

[0267] The processor 21 is configured to implement the steps of the fault determination method for a microservice system as described in the above embodiments when executing a computer program.

[0268] The processor 21 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 21 may be implemented using at least one of the following hardware forms: Digital Signal Processor (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array. The processor 21 may also include a main processor and a coprocessor. The main processor, also known as the Central Processing Unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 21 may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 21 may also include an Artificial Intelligence (AI) processor, which handles computational operations related to machine learning.

[0269] The memory 20 may include one or more computer-readable storage media, which may be non-transitory. The memory 20 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In this embodiment, the memory 20 is used to store at least the following computer program 201, which, after being loaded and executed by the processor 21, is capable of implementing the relevant steps of the fault determination method for the microservice system disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 20 may also include an operating system 202 and data 203, and the storage method may be temporary or permanent storage. The operating system 202 may include Windows, Unix, Linux, etc. The data 203 may include, but is not limited to, the data involved in the aforementioned fault determination method for the microservice system.

[0270] In some embodiments, the electronic device may further include a display screen 22, an input / output interface 23, a communication interface 24, a power supply 25, and a communication bus 26.

[0271] Those skilled in the art will understand that Figure 3 The structures shown do not constitute a limitation on electronic devices and may include more or fewer components than those shown.

[0272] The electronic device provided in this embodiment of the invention includes a memory and a processor. When the processor executes a program stored in the memory, it can implement the following method: a fault determination method for a microservice system, with the same effect as above.

[0273] This invention also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the above-described fault determination method for a microservice system.

[0274] Finally, the present invention also provides an embodiment corresponding to a computer-readable storage medium. The computer-readable storage medium stores a computer program, which, when executed by a processor, performs the steps described in the above method embodiments.

[0275] It is understood that if the methods in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and executes all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0276] The computer-readable storage medium provided by this invention includes the aforementioned fault determination method for microservice systems, and has the same effect.

[0277] The foregoing has provided a detailed description of the fault determination method, apparatus, product, and electronic device for the microservice system provided by this invention. The various embodiments in the specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section. It should be noted that those skilled in the art can make several improvements and modifications to this invention without departing from the principles of this invention, and these improvements and modifications also fall within the protection scope of this invention.

[0278] It should also be noted that, in this specification, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A method for failure determination of a microservice system, characterized in that, The method comprises: obtaining abnormal monitoring information of a microservice system, and determining an abnormal event stream according to the abnormal monitoring information; wherein the monitoring information at least comprises index information, log information and call chain information; obtaining entity information in the abnormal event stream and relationships between entities; wherein the entity information in the abnormal event stream is determined by event description information of the abnormal event stream; determining an event causal graph according to the entity information and the relationships between entities; determining an abnormal service in the event causal graph, a propagation path related to the abnormal service and a fault classification of the abnormal service by using a pre-trained model driven intelligent agent in combination with the event causal graph and a pre-set fault mode; obtaining entity information in the abnormal event stream and relationships between entities comprises: inputting the abnormal event stream and a service topology graph stored in a database into a pre-trained model; extracting the entity information in the abnormal event stream and the relationships between entities by the pre-trained model; wherein the entity information in the abnormal event stream at least comprises a service name, an instance identifier, an associated performance index, an abnormal type and a resource extracted from event description of the abnormal event stream; the relationships between entities at least comprise a dependency relationship determined based on an abnormal tracking event, a causal relationship determined based on an error stack in a log abnormal event, a propagation relationship determined based on an error transmission on a tracking path, and an association relationship determined based on a request identifier and a thread identifier to associate events of different modes. 2.The method for failure determination of a microservice system according to claim 1, characterized in that, determining an abnormal event stream according to the abnormal monitoring information comprises: determining an index abnormal event according to abnormal index information; wherein the index at least comprises a utilization rate index of a processor and a delay index of a system; determining a log abnormal event according to abnormal log information; determining an abnormal tracking event according to abnormal call chain information; determining an abnormal event stream according to the index abnormal event, the log abnormal event and the abnormal tracking event.

3. The method of claim 2, wherein, The monitoring information is index information, and obtaining abnormal index information of the microservice system comprises: obtaining an observation value and a preset value of a target index; in a case where a deviation between the observation value and the preset value is greater than a preset deviation, determining that the target index is an abnormal index; obtaining abnormal index information; wherein the abnormal index information at least comprises a time when the abnormal index occurs, a service affected by the abnormal index, a service instance affected by the abnormal index, an abnormal index name, an abnormal type and an abnormal score of the abnormal index; the information of the index abnormal event at least comprises a unique identifier of the event and the abnormal index information.

4. The method of claim 2, wherein, determining a log abnormal event according to abnormal log information comprises: extracting entity information and events in the abnormal log information by a pre-trained model; wherein the entity information in the abnormal log information at least comprises a service, a resource, an error code and a request identifier; determining a type of the extracted event according to a pre-set fault mode or event type; wherein the type of the event at least comprises a connection failure event type and a business logic error event type.

5. The method of claim 2, wherein, The monitoring information is call chain information; obtaining abnormal call chain information of the microservice system comprises: According to the call chain data, a service topology graph is constructed, and information of the service topology graph is obtained; wherein the information of the service topology graph at least includes the inter-service dependency direction, the service call frequency and the call delay; According to the information of the service topology graph, abnormal call chain information is determined; The information of the abnormal tracking event at least includes: event type, tracking identifier, root service, bottleneck service and error type.

6. The method of claim 5, wherein, Further comprising: The service topology graph and the information of the service topology graph are stored in a database; And according to the information of the service topology graph, inter-service call events are determined; Wherein, the information of the inter-service call event at least includes: event type, target caller service, source caller service and delay.

7. The method of claim 1, wherein, The information of the abnormal tracking event at least includes: event type, tracking identifier, root service, bottleneck service and error type. The information of the abnormal tracking event at least includes: event type, tracking identifier, root service, bottleneck service and error type. Further comprising: At least the index abnormal value and the log error summary are obtained; And the attribute information of the nodes and the attribute information of the edges between the nodes in the event causal graph are determined based on the index abnormal value and the log error summary. Further comprising:

8. The method of claim 7, wherein, The event causal graph, the attribute information of the nodes and the attribute information of the edges between the nodes in the event causal graph are stored in the database. Before the intelligent agent driven by the pre-training model is used to determine the abnormal service in the event causal graph, the propagation path related to the abnormal service and the fault classification of the abnormal service in combination with the event causal graph and the pre-set fault mode, further comprising:

9. The method of claim 7, wherein, The fault condition of the micro-service system is detected by an intelligent agent coordinator; wherein the fault condition includes the case of appearing fault or the case of not appearing fault; When the intelligent agent coordinator detects that the fault condition of the micro-service system is the case of appearing fault, prompt information for representing fault detection is sent to the target intelligent agent; The intelligent agent driven by the pre-training model is used to determine the abnormal service in the event causal graph, the propagation path related to the abnormal service and the fault classification of the abnormal service in combination with the event causal graph and the pre-set fault mode, comprising: The target intelligent agent driven by the pre-training model is used to determine the abnormal service in the event causal graph, the propagation path related to the abnormal service and the fault classification of the abnormal service in combination with the event causal graph and the pre-set fault mode. The target intelligent agent driven by the pre-training model is used to determine the abnormal service in the event causal graph, the propagation path related to the abnormal service and the fault classification of the abnormal service in combination with the event causal graph and the pre-set fault mode, comprising:

10. The method of claim 9, wherein, In the target intelligent agent, the state of the event causal graph is obtained through the pre-training model; the abnormal signal is recognized according to the state of the event causal graph, and the graph query tool is called to extract the abnormal service related to the abnormal signal from the event causal graph; ​ The call graph query tool and the knowledge base retrieval tool are used to query a propagation path of the abnormal signal in the microservice system, and a causal chain causing the abnormal service to be abnormal is determined according to the propagation path; A fault classification of the abnormal service is determined according to a fault classification stored in a database.

11. The method of claim 9, wherein, The target agent at least includes a first type target agent, a second type target agent and a third type target agent; In a case where the agent coordinator detects that the fault condition of the microservice system is a fault, the prompt information for representing fault detection is sent to the target agent, including: In a case where the agent coordinator detects that the fault condition of the microservice system is a fault, the prompt information for representing starting fault detection is sent to the first type target agent by the agent coordinator; The target agent driven by the pre-trained model, in combination with the event causal graph and the pre-set fault mode, determines the abnormal service in the event causal graph, the propagation path related to the abnormal service and the fault classification of the abnormal service, including: In the first type target agent, the state of the event causal graph is obtained through the pre-trained model; the abnormal signal is identified according to the state of the event causal graph, and the abnormal service related to the abnormal signal is extracted from the event causal graph by calling the graph query tool; the abnormal service related to the abnormal signal is sent to the agent coordinator; so that the agent coordinator determines a candidate causal chain causing the abnormal service to be abnormal according to the event causal graph, and sends prompt information for representing fault detection of the candidate causal chain to the second type target agent; In the second type target agent, the call graph query tool and the knowledge base retrieval tool are used to query a propagation path of the abnormal signal in the microservice system, and a causal chain causing the abnormal service to be abnormal is determined according to the propagation path; A fault classification of the abnormal service is determined according to a fault classification stored in a database.

12. The method of claim 11, wherein, The call graph query tool extracts the abnormal service related to the abnormal signal from the event causal graph, including: In the first type target agent, it is judged whether the result returned by the graph query tool meets the first preset requirement; If not, the thought chain reasoning is performed according to the result returned by the graph query tool and the first preset requirement, to generate a new thinking path and determine the query parameters of the graph query tool based on the new thinking path; the result returned by the graph query tool according to the query parameters is obtained; the judgment of whether the result returned by the graph query tool meets the first preset requirement is returned; If yes, the abnormal service related to the abnormal signal is determined according to the result returned by the graph query tool; The call graph query tool and the knowledge base retrieval tool are used to query a propagation path of the abnormal signal in the microservice system, including: In the second type of target agent, it is judged whether the results returned by the graph query tool and the knowledge base retrieval tool meet the second preset requirement; If not, according to the results returned by the graph query tool and the knowledge base retrieval tool, and the second preset requirement, a new thinking path is generated by thinking chain reasoning, and the query parameters of the results returned by the graph query tool and the knowledge base retrieval tool are determined based on the new thinking path; the results returned by the graph query tool and the knowledge base retrieval tool are queried according to the query parameters; the step of judging whether the results returned by the graph query tool and the knowledge base retrieval tool meet the second preset requirement is returned; If yes, according to the results returned by the graph query tool and the knowledge base retrieval tool, the propagation path of the abnormal signal in the microservice system is determined.

13. The method of claim 11, wherein, Before the agent coordinator sends prompt information for representing fault detection of the candidate causal chain to the second type of target agent, it further includes: acquiring the actual fault determination progress and the load condition of the agent; determining the second type of target agent according to the actual fault determination progress and the load condition of the agent; Before the agent coordinator sends prompt information for representing fault classification to the third type of target agent, it further includes: determining the third type of target agent according to the actual fault determination progress and the load condition of the agent.

14. The method of claim 10 or 11, wherein, The graph query tool is at least used for executing the query of nodes, edges and paths in the event causal graph; The knowledge base retrieval tool is at least used for querying historical fault cases, service metadata, operation and maintenance manuals, fault modes, fault solutions in the database, and providing context and prior knowledge for reasoning.

15. The method of claim 12, wherein, It further includes: acquiring the thinking path of each agent, the called tool, the query parameter of the tool and the result returned by the tool; outputting information containing the thinking path of each agent, the called tool, the query parameter of the tool and the result returned by the tool.

16. A fault determination apparatus of a microservice system, characterized by, It includes: The first acquisition module is used for acquiring abnormal monitoring information of the microservice system, and determining an abnormal event stream according to the abnormal monitoring information; wherein the monitoring information at least includes index information, log information and call chain information; The second acquisition module is used for acquiring entity information in the abnormal event stream and the relationship between entities; wherein the entity information in the abnormal event stream is determined by event description information of the abnormal event stream; The first determination module is used for determining an event causal graph according to the entity information and the relationship between entities; The second determination module is used for determining an abnormal service in the event causal graph, a propagation path related to the abnormal service and a fault classification of the abnormal service by using a pre-trained model driven agent and combining the event causal graph and a pre-set fault mode; The second acquisition module is specifically used for: inputting the abnormal event stream and a service topology graph stored in the database into a pre-trained model; extracting entity information in the abnormal event stream and the relationship between entities by the pre-trained model; wherein the entity information in the abnormal event stream at least includes a service name, an instance identifier, an associated performance index, an abnormal type and a resource extracted from event description of the abnormal event stream. The relationship between the entities at least includes: a dependency relationship determined based on an abnormal tracking event, a causal relationship determined based on an error stack in a log abnormal event, a propagation relationship determined based on an error transmission on a tracking path, and an association relationship determined based on association of different mode events through request identification and thread identification.

17. A computer program product comprising computer programs / instructions, characterized in that, The computer program / instruction is executed by the processor to implement the steps of the fault determination method of the micro-service system in any one of claims 1 to 15.

18. An electronic device, comprising: The computer program / instruction is executed by the processor to implement the steps of the fault determination method of the micro-service system in any one of claims 1 to 15. The computer program / instruction is executed by the processor to implement the steps of the fault determination method of the micro-service system in any one of claims 1 to 15. The computer program / instruction is executed by the processor to implement the steps of the fault determination method of the micro-service system in any one of claims 1 to 15.

19. A computer-readable storage medium, characterized in that, ​

Images