Fault diagnosis methods and devices, storage media and electronic equipment

Abstract

This application discloses a fault diagnosis method, apparatus, storage medium, and electronic device, relating to the field of fault diagnosis technology. The method includes: determining a fault information set based on at least one fault of a target device, wherein the fault information in the fault information set is used to indicate the node where the corresponding fault is located; generating a node topology graph according to the operating architecture of the target device, wherein the nodes are used to indicate the relationship between at least one service node of the target device; generating a fault propagation path corresponding to the target fault in the node topology graph using the fault information set, wherein the target fault is any one of at least one fault; and determining a target node based on the fault propagation path when the fault propagation path includes at least one reference fault, wherein the target node is used to indicate the root cause of the fault propagation path. This solves the technical problem of low accuracy in related fault diagnosis methods.

Description

Technical Field

[0001] This application relates to the field of fault diagnosis, and more particularly to a fault diagnosis method and apparatus, storage medium and electronic equipment. Background Technology

[0002] Private cloud environments typically consist of multi-layered components, including physical hosts, virtual machines, containers, and various microservices and applications, with intricate dependencies between these components. Traditional fault diagnosis relies on the experience and intuition of operations personnel, attempting to find the root cause of the fault by reviewing massive amounts of log files, monitoring metrics, and call chain data. This approach is not only time-consuming and labor-intensive, but also often falls short when faced with multimodal data (metrics, call chains, logs) and complex dependencies, making it difficult to quickly and accurately locate problems.

[0003] Cloud platform failures often propagate across multiple layers, from hardware to virtualization and then to the service layer. However, existing fault diagnosis technologies struggle to trace such cross-layer propagation paths, especially when the fault originates at a lower level. The performance of upper-layer services is often simply attributed to software issues, ignoring potential hardware or network problems. In other words, the fault diagnosis methods in related technologies suffer from low accuracy. Summary of the Invention

[0004] This application provides a fault diagnosis method and apparatus, storage medium and electronic device, to at least solve the problem of low accuracy in fault diagnosis methods in related technologies.

[0005] This application provides a fault diagnosis method, comprising: determining a fault information set based on at least one fault of a target device, wherein the fault information in the fault information set is used to indicate the node where the corresponding fault is located; generating a node topology graph according to the operating architecture of the target device, wherein the nodes are used to indicate the relationship between at least one service node of the target device; generating a fault propagation path corresponding to the target fault in the node topology graph using the fault information set, wherein the target fault is any one of at least one fault; and determining a target node according to the fault propagation path when the fault propagation path includes at least one reference fault, wherein the target node is used to indicate the root cause of the fault propagation path.

[0006] This application also provides a fault diagnosis device, comprising: a set determination module, configured to determine a set of fault information based on at least one fault of a target device, wherein the fault information in the set of fault information is used to indicate the node where the corresponding fault is located; a topology graph generation module, configured to generate a node topology graph according to the operating architecture of the target device, wherein the nodes are used to indicate the relationship between at least one service node of the target device; a path generation module, configured to generate a fault propagation path corresponding to the target fault in the node topology graph using the set of fault information, wherein the target fault is any one of at least one fault; and a target node determination module, configured to determine a target node according to the fault propagation path when the fault propagation path includes at least one reference fault, wherein the target node is used to indicate the root cause of the fault propagation path.

[0007] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of any of the above-described fault diagnosis methods.

[0008] This application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above-described fault diagnosis methods.

[0009] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described fault diagnosis methods.

[0010] This application identifies a set of fault information based on at least one fault of a target device, where the fault information in the set indicates the node where the corresponding fault occurs. A node topology graph is generated based on the target device's operational architecture, where nodes indicate the relationships between at least one service node of the target device. A fault propagation path corresponding to the target fault is generated in the node topology graph using the fault information set, where the target fault is any one of the at least one faults. If the fault propagation path includes at least one reference fault, a target node is determined based on the fault propagation path, where the target node indicates the root cause of the fault propagation path. The automatic generation of the node topology graph based on the target device's operational architecture not only shows the relationships between service nodes but also reflects changes in these relationships in real time, such as new service dependencies or network structure adjustments. This allows the system to dynamically adapt to architectural changes and always maintain the latest and most accurate system view. Based on the node topology graph, the system can construct fault propagation paths using the fault information set, identifying not only individual faults but also revealing how a fault propagates from one node to another. This is crucial for understanding the full picture of complex faults, especially when faults span multiple levels and services. Therefore, it can solve the problem of low accuracy in fault diagnosis methods. Attached Figure Description

[0011] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a schematic diagram of the hardware environment of an optional fault diagnosis method according to an embodiment of this application;

[0013] Figure 2 This is a flowchart of an optional fault diagnosis method according to an embodiment of this application;

[0014] Figure 3 This is a schematic diagram of an optional node layer according to an embodiment of this application;

[0015] Figure 4 This is a schematic diagram of an optional fault diagnosis method according to an embodiment of this application;

[0016] Figure 5 This is a structural block diagram of an optional fault diagnosis device according to an embodiment of this application. Detailed Implementation

[0017] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0018] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0019] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0020] According to one aspect of the embodiments of this application, a fault diagnosis method is provided. As an optional implementation, the above-described fault diagnosis method can be applied to, but is not limited to, situations such as... Figure 1 The fault diagnosis system in the hardware environment shown may include, but is not limited to, terminal device 102, network 110, server 112, and database 114. Terminal device 102 runs a target client (e.g., ...). Figure 1 As shown, taking the target client as an example of a client that can access the server. The terminal device 102 includes a display 108, a processor 106, and a memory 104. The display 108 can be used to display the system interface, and also to provide a human-computer interaction interface to receive human-computer interaction operations on the interface and touch operations on different controls. The processor is used to generate interaction instructions in response to the above-mentioned human-computer interaction operations and send the interaction instructions to the server. The memory is used to store system files.

[0021] In addition, server 112 includes a processing engine 116, which performs storage or read operations on database 114. Specifically, processing engine 116 reads data from database 114. Server 112 can provide cloud services for a private cloud.

[0022] Assumption Figure 1 The terminal device 102 runs a client for accessing the server. The specific process in this embodiment is as follows: As in step S102, the terminal device 102 sends an access operation to the server 112 via network 110. The server 112 executes steps S104-S110, determining a fault information set based on at least one fault of the target device, wherein the fault information in the fault information set is used to indicate the node where the corresponding fault is located; generating a node topology graph according to the operating architecture of the target device, wherein the nodes are used to indicate the relationship between at least one service node of the target device; generating a fault propagation path corresponding to the target fault in the node topology graph using the fault information set, wherein the target fault is any one of at least one fault; and determining a target node based on the fault propagation path when the fault propagation path includes at least one reference fault, wherein the target node is used to indicate the root cause of the fault propagation path.

[0023] Optionally, in this embodiment, the terminal device 102 can be a terminal device configured with a target client, which may include, but is not limited to, at least one of the following: mobile phone (such as Android phone, iOS phone, etc.), laptop computer, tablet computer, PDA, MID (Mobile Internet Devices), PAD, desktop computer, smart TV, etc. The target client can be a client that supports generating text responses. The network may include, but is not limited to, wired networks and wireless networks, wherein the wired network includes: local area network, metropolitan area network, and wide area network, and the wireless network includes: Bluetooth, WIFI, and other networks that enable wireless communication. The server may be a single server, a server cluster composed of multiple servers, or a cloud server. The above is merely an example, and no limitation is made in this embodiment.

[0024] The application scenarios of this application are not limited to the above examples. This application method can be used for any fault diagnosis of private clouds.

[0025] The embodiments of this application provide a fault diagnosis method. Figure 2 This is a flowchart of an optional fault diagnosis method according to an embodiment of this application; such as Figure 2 As shown, the fault diagnosis method includes:

[0026] Step S202: Determine a fault information set based on at least one fault of the target device, wherein the fault information in the fault information set is used to indicate the node where the corresponding fault is located;

[0027] It's important to note that the target device, within a private cloud environment, refers to a physical or virtual resource that has experienced a failure, such as a server, virtual machine, container, or service. The "target device" here can be a single resource or a collection of resources; the key point is that they are the focus of fault diagnosis.

[0028] A fault information set is a collection and organization of all fault information, including but not limited to the time of the fault occurrence, the form of the fault, the components or services involved, the possible fault level, and the scope of impact. The fault information set is an important basis for subsequent fault analysis and root cause localization.

[0029] A node generally refers to a specific location in a system architecture; it can be a physical server, virtual machine, container, service, or any other entity that can run independently or host other components. In a fault information set, a "node" refers to the specific location where the fault occurred or an entity related to the fault.

[0030] In fault diagnosis in a private cloud environment, the first step is fault identification, which involves determining which components among a series of running devices or services are malfunctioning. This is typically achieved through telemetry data collected by monitoring systems (such as KPI time series, call chain traces, logs, etc.). A fault is considered to have occurred when the system detects that the performance indicators of one or more components are outside the normal range, or when abnormal logs and trace data are received. Next, step S202 involves systematically and structurally recording these faults to form a fault information set. Each fault information entry should include the specific location of the fault (i.e., a "node"), the type of fault, the time of occurrence, and the potential impact of the fault on other parts of the system. The process of constructing the fault information set is not limited to recording; it also includes preliminary analysis of the fault information, such as determining the severity and scope of the fault, which helps in subsequent fault propagation path analysis and root cause localization.

[0031] Step S204: Generate a node topology diagram based on the operating architecture of the target device, wherein the nodes are used to indicate the relationship between at least one service node of the target device;

[0032] It's important to note that the runtime architecture refers to the way the target device operates and interacts with other devices, including hardware architecture, software architecture, network architecture, and dependencies between components. A node topology diagram is a graphical representation used to depict the interconnections and hierarchical dependencies between service nodes within the target device, including physical nodes, virtual nodes, and logical nodes (such as services). The relationships between service nodes describe the dependencies, calls, hosting, and other interactions between various service nodes in a private cloud.

[0033] When performing fault diagnosis, constructing a node topology map that accurately reflects the operating architecture of the target device is a crucial task. This step S204 aims to capture and present the complex relationships between service nodes in a private cloud environment, providing a foundational map for subsequent fault propagation path analysis and root cause localization.

[0034] In optional implementations, the system needs to collect information from multiple data sources, including Configuration Management Database (CMDB) information, service registration and discovery information, distributed tracing data (such as Trace Span), and possible network topology information. The CMDB provides information about hardware and software configurations, including component hosting relationships; service registration and discovery information describes dependencies between services; Trace Span data records the time series and relationships of service calls; and network topology information reflects the layout of network devices and connections. Based on the integrated data, the system constructs a dynamic node topology graph. Each node in the graph represents a service entity, such as a physical server, virtual machine, container, or service. Connections represent dependencies or call relationships between services, as well as physical hosting relationships (e.g., virtual machines and containers running on a physical server). Through depth-first search (DFS), breadth-first search (BFS), or other graph traversal algorithms, the system can construct a graph that accurately reflects the component interconnections and hierarchical dependencies in the private cloud environment. One characteristic of a private cloud environment is its dynamic nature, including elastic scaling and service deployment adjustments. Therefore, the node topology graph needs to be able to update in real time to reflect the latest architectural state. The system can ensure that the topology map remains synchronized with the actual environment by periodically refreshing the data source and listening for architecture change events. In the constructed topology map, the system must not only display the relationships between components at the same level but also be able to associate components at different levels. For example, the relationships between service nodes and their running containers, containers and the virtual machines hosting them, and virtual machines and the underlying physical servers. This allows the system to trace the propagation path of faults across different levels. In some cases, the system may need to dynamically generate code to analyze the topology map or extract specific dependencies. For example, using Python and related libraries (such as NetworkX) to generate code snippets to find upstream dependencies of specific services or analyze the failure rate of inter-service calls. The controller component is responsible for logical reasoning and strategy planning, while the executor component executes the generated code in the IPython kernel to obtain the analysis results.

[0035] Step S206: Generate a fault propagation path corresponding to the target fault in the node topology graph using the fault information set, wherein the target fault is any one of at least one fault;

[0036] It should be noted that a fault propagation path refers to the sequence of connections in a node topology graph from the node where the fault initially occurred (the root node) to other nodes affected by the fault (downstream nodes). This path reveals how the fault spreads through dependencies in the system, helping to identify the root cause and scope of the fault.

[0037] In an optional implementation, the system first extracts information about the node where the target fault is located from the fault information set to determine the initial location of the fault. If it is the first analysis or the dependencies have changed, the system updates the node topology graph from the Configuration Management Database (CMDB), service registration and discovery mechanism, and distributed tracing data to ensure the accuracy and timeliness of the graph. On the node topology graph, the system uses a graph traversal algorithm to trace how the fault affects downstream nodes, starting from the initial fault node and through direct or indirect edges. This may involve traversing the call chain between services, the carrying relationship between containers and the host machine, and the allocation chain of hardware resources. The system records each node through which the fault propagates, forming a fault propagation path. During the traversal, the system cross-validates information from multiple data sources, such as KPI metrics, call chain data, and log records, to confirm the accuracy of the fault propagation path. Through the fusion analysis of multimodal data, the credibility of fault judgment can be increased. The system ultimately connects the scattered fault information into a clear fault propagation path and may visualize this path on the node topology graph to help operations personnel quickly understand the fault propagation process.

[0038] Step S208: If the fault propagation path includes at least one reference fault, determine the target node based on the fault propagation path, wherein the target node is used to indicate the root cause of the fault propagation path.

[0039] It should be noted that reference faults are specific faults that are indicative or relevant and identified along the fault propagation path. These may be anomalies that directly or indirectly cause overall failures in the system.

[0040] In an optional implementation, the system performs a depth-first traversal along the edges of the constructed dependency graph, starting with the most severely affected component, examining call chains, bearer relationships, and dependencies until the initial anomaly point unaffected by other failures is found. Combining KPIs, call chains, and log data, the system performs multi-dimensional analysis to confirm the abnormal state of the target node. For example, it examines the node's CPU usage, memory consumption, network latency, and error logs before and after the failure to ensure that the target node's anomaly is the direct cause of the failure propagation. The system uses a comprehensive scoring method to prioritize candidate target nodes, with scoring criteria including the anomaly's timing, scale, scope of impact, and match with historical failure patterns, to intelligently determine the most likely root cause.

[0041] Example 1:

[0042] S202, Data Fusion and Anomaly Detection:

[0043] The system collects KPI time series, call chain traces, and semi-structured log data from components in the private cloud environment. It employs a dynamic threshold adjustment algorithm for anomaly detection; for example, starting with a P95 threshold, if no obvious fault clues are identified, the system automatically adjusts the threshold to P90, or even to P85 if necessary.

[0044] S204, Multi-level Dependency Modeling:

[0045] Construct a dynamic, multi-level topology graph, including components at different levels such as physical nodes, containers, and application services. Analyze the propagation paths of faults between these levels using algorithms such as depth-first search.

[0046] S206, Intelligent Root Cause Analysis:

[0047] When the system detects multiple failures, it traces the anomaly propagation chain, prioritizing the component where the anomaly occurred earliest as a root cause candidate. By combining call chain data, log information, and expert rules, it intelligently identifies and ranks multiple potential root causes to determine the most likely underlying cause.

[0048] S208, Determine the target node:

[0049] Once a series of potential root cause candidates are identified in phase S206, the system further analyzes the failure propagation paths of these candidates. For example, suppose the system detects an increase in the response time of service A, and further analysis reveals that service A depends on service B, which in turn depends on physical server C. If server C experiences abnormal network latency, and this is the earliest abnormality in the propagation path, then server C is identified as the "target node," i.e., the root cause of the entire failure. After the target node is identified, the system provides a detailed analysis report, including the time, location, type, and potential impact of the failure, as well as recommended remediation strategies or automated remediation operations.

[0050] This application identifies a set of fault information based on at least one fault of a target device, where the fault information in the set indicates the node where the corresponding fault occurs. A node topology graph is generated based on the target device's operational architecture, where nodes indicate the relationships between at least one service node of the target device. A fault propagation path corresponding to the target fault is generated in the node topology graph using the fault information set, where the target fault is any one of the at least one faults. If the fault propagation path includes at least one reference fault, a target node is determined based on the fault propagation path, where the target node indicates the root cause of the fault propagation path. The automatic generation of the node topology graph based on the target device's operational architecture not only shows the relationships between service nodes but also reflects changes in these relationships in real time, such as new service dependencies or network structure adjustments. This allows the system to dynamically adapt to architectural changes and always maintain the latest and most accurate system view. Based on the node topology graph, the system can construct fault propagation paths using the fault information set, identifying not only individual faults but also revealing how a fault propagates from one node to another. This is crucial for understanding the full picture of complex faults, especially when faults span multiple levels and services. Therefore, it can solve the problem of low accuracy in fault diagnosis methods.

[0051] In an optional implementation, generating a node topology graph based on the target device's operating architecture includes: obtaining the target device's current operating architecture, wherein the operating architecture indicates the physical nodes, containers, and application services contained in the target device; identifying the physical nodes, containers, and application services included in the operating architecture as at least one graph node; determining the matching relationships between physical nodes, containers, and application services based on the operating architecture, and determining the connection relationships between at least one graph node based on the matching relationships; determining the edges between at least one graph node based on the connection relationships, and generating a node topology graph.

[0052] It should be noted that runtime architecture refers to the internal structure and relationships between components of a target device (including but not limited to physical servers, virtual machines, containers, services, etc.), and how these components interact with other devices or services. In a private cloud environment, runtime architecture may include the hardware configuration of physical servers, the distribution of virtual machines and containers, the calling relationships between services, and network connectivity.

[0053] At least one graph node in the generated node topology graph represents an element of an independent entity (such as a physical server, container, or service) in the system architecture. Each graph node corresponds to an entity in the running architecture, including information such as its status, configuration, and performance metrics.

[0054] Matching relationships and connection relationships describe how graph nodes interact or depend on each other. Matching relationships may be based on the hosting relationship of components (such as containers running on virtual machines, virtual machines running on physical servers), while connection relationships involve more direct or indirect calls between services.

[0055] In a node topology graph, an edge represents a connection between two nodes. Each edge carries relationship attributes, such as the frequency of service calls and the response time of the call chain. These attributes help analyze the path of fault propagation.

[0056] In optional implementations, the system collects architectural information about the target device from sources such as CMDB, service registration and discovery, and call chain tracing data. This data includes hardware configuration, service lists, container and virtual machine hosting relationships, and service call relationships. Information extracted from the data sources is transformed into graph nodes, including physical servers, virtual machines, containers, and application services. CMDB information identifies the virtual machines hosted on physical servers and the containers hosted on virtual machines, establishing hierarchical dependencies. Call chain tracing data is analyzed to determine direct call relationships and indirect dependencies between services. Network device configuration information and traffic data are collected to determine network connectivity between nodes. In the topology graph, each physical node, container, and application service corresponds to a graph node, while hosting relationships, call relationships, and network connections are represented by edges. Due to the dynamic nature of private cloud environments, the node topology graph needs to be updated in real time to reflect architectural changes. This may include the creation and destruction of virtual machines and containers, changes to service registration, and adjustments to the network topology to ensure the accuracy and timeliness of the topology graph. To build and update the topology graph, the system may need to perform a series of data analysis and processing tasks, such as Python code generation in the IPython kernel to read and parse data sources, calculate the dependency strength between nodes, and identify architecture changes.

[0057] Specifically:

[0058] First, import static information containing components such as physical nodes, virtual machines (VMs), containers, and application services, as well as their hosting and relationships, from the Configuration Management Database (CMDB). Then, combine this static information with real-time service location information provided by service registries (such as Consul, Etcd, and Zookeeper) to update dynamic dependencies. Finally, parse tracing data from service call chains to extract information such as call relationships, call latency, and failure rates between services; this is the key data source for constructing the dynamic dependency graph.

[0059] Identify physical servers in the cloud environment, such as host machines, along with their underlying hardware information and network connectivity status. Identify all virtual machines (VMs) and containers running on physical nodes, including their resource usage, running status, and network isolation status. Identify application services running on containers or VMs, obtaining metrics such as their health status and API call success rate, while also identifying dependencies between services.

[0060] Create an empty directed graph using a graph database or graph processing library (such as NetworkX) to represent the dependencies between different components. Based on CMDB, service registration discovery, and tracing data, gradually add physical nodes, containers, and services as nodes in the graph. Establish dependency edges between nodes based on the call relationships in the tracing data and the carrying relationships in the CMDB. The edge weights may be determined based on factors such as call frequency, failure rate, and resource consumption to reflect the strength of the dependency.

[0061] Monitor changes in the cloud environment in real time, such as service additions and deletions, virtual machine migrations, and network topology changes, and update the graph promptly. Synchronize the latest status of components with the cloud management platform, including start-up, stop-up, and restart events, to maintain the timeliness of the graph. When inconsistencies are found in node or edge information, resolve conflicts or mark errors to ensure the accuracy of the graph.

[0062] Upon detecting anomalies, a depth-first search (DFS) or breadth-first search (BFS) is performed using the constructed topology graph to trace the propagation path of the anomalous signal. The dependency strength between different components is calculated based on the edge weights, which helps determine the scope and priority of the fault's impact. For large-scale cloud environments, the algorithm can automatically simplify the graph, focusing on fault-related areas and reducing the complexity of the analysis.

[0063] The constructed multi-level topology diagram is presented in a graphical interface, intuitively displaying the relationships and dependencies between components. Abnormal nodes are highlighted in the diagram, with anomaly indicators and timestamps labeled to help quickly locate faults. An electronic file of the topology diagram is provided to facilitate sharing analysis results between different teams or systems.

[0064] Example 2:

[0065] Suppose the target device is a Kubernetes-based private cloud cluster running multiple microservices. One of the services (Service Y) suddenly experiences a response latency issue, and the system needs to build a topology graph to analyze the failure propagation path.

[0066] CMDB Integration: Obtain basic information such as cluster hardware configuration, network layout, and virtual machine / container distribution from the CMDB, and build a host relationship diagram. For example, host machine Host1 hosts virtual machine VM2, while VM2 runs container Container3, and Service Y is deployed in Container3.

[0067] Service registration and discovery: Through Kubernetes' service discovery mechanism, the system obtains the list of services, port mappings and dependencies in the cluster in real time, and constructs a service call relationship graph of Service Y depending on Service X.

[0068] Trace Span analysis: Analyzing the call chain data of Service Y, it was found that Service Y had a significant delay in each call to Service X, and the delay time matched the response delay time of Service Y, confirming the existence of a fault propagation path.

[0069] Graph construction algorithm: The system integrates the carrier relationship graph, service call relationship graph and call chain information through an algorithm to form a complete, multi-level topology graph, which includes the hierarchical dependencies and direct call relationships between physical servers, virtual machines, containers and microservices.

[0070] Dynamic update mechanism: As the private cloud environment changes, such as during fault diagnosis, if the administrator starts a new virtual machine VM3 to host Service Y, the system immediately updates the topology map, adds VM3 nodes, and adjusts the hosting relationship between Service Y and containers and the host machine to ensure the accuracy and effectiveness of the map.

[0071] Through the above-described implementation methods of this application, the system not only constructs a detailed node topology graph, but also can update and maintain it in real time, thereby providing the most accurate dependency and call path information when a fault occurs, accelerating the fault location and recovery process.

[0072] In an optional implementation, a fault propagation path corresponding to the target fault is generated in the node topology graph using a fault information set, including: traversing the node topology graph and determining the node running path based on the temporal information and edge indication dependencies of at least one node in the node topology graph; determining the target location of the target fault in the node topology graph, and determining the node running path containing the target location as the fault propagation path.

[0073] It should be noted that dependencies can be direct or indirect associations represented by edges between two nodes in a node topology graph, such as one service depending on the functionality or data provided by another service. Node propagation paths, in fault propagation analysis, can be the paths from the fault source node to affected nodes, reflecting how the fault spreads in a private cloud environment.

[0074] In optional implementations, either Depth-First Search (DFS) or Breadth-First Search (BFS) strategies are used to traverse the node topology graph. These strategies start from the source of the fault and explore the graph in depth or layer by layer to find the path related to the fault. Alternatively, the advantages of both strategies can be combined, such as using Iterative Deeper Depth-First Search (IDDFS). During traversal, the system checks the timing information of each node, comparing the fault occurrence time recorded in the fault information set with the event time in the node topology graph. Prioritize nodes that experienced problems earlier than the target fault, as they are likely closer to the fault source. Utilizing dependency information on edges (such as call relationships and bearer relationships), the system can trace the fault propagation path along these relationships. For example, when a service experiences a delayed response time, the system uses call chain data to trace all upstream services that called this service, and then, based on the timing information and running status of these services, gradually traces back to the source. After completing the above analysis, the system reconstructs the fault propagation path, i.e., the complete sequence from the fault source to the affected downstream nodes. Then, multimodal data (KPIs, logs, call chains, etc.) are used to cross-validate the constructed paths to ensure the accuracy of the identified propagation paths. The system may provide the function of marking the fault propagation paths on the node topology graph, presenting how the fault spreads in the system in a visual way, making it easier for operations and maintenance personnel to understand the full picture of the fault.

[0075] Example 3:

[0076] Suppose that in a private cloud cluster, application service S3 experiences a sudden and significant increase in response time over a certain period, impacting the end-user experience. The system has already confirmed the abnormal response time of S3 through anomaly detection algorithms; now, it is necessary to construct a fault propagation path to locate the root cause.

[0077] Previously, by integrating CMDB information, service registration and discovery, and distributed tracing data, a multi-level topology diagram of the private cloud environment was constructed, covering the carrying and calling relationships between components such as physical servers, virtual machines, containers, and application services.

[0078] In the node topology graph, the system first identifies the S3 service, which exhibits abnormal response time, as the location of the target failure and uses it as the starting point for the Depth-First Search (DFS) graph traversal. Starting with service S3, the system employs a depth-first search strategy, analyzing each layer along the edges of the graph. Edges represent dependencies and carrying relationships between services. During the traversal, the system examines the timing information of each relevant component, particularly the time of the failure. For example, if the abnormal response time of S3 occurs at time T3, the system will backtrack to nodes before T3 to check if any abnormalities occurred before S3. In addition to timing information, the system also evaluates the strength of dependencies between components, such as call frequency and failure rate. If an upstream service S2 calls S3 frequently and fails at time T2, the system will consider S2 as a potential source of failure. During the traversal, the system collects information on all abnormal components that directly or indirectly affect S3, including their failure time, failure type, and dependency on S3. Using this information, the system can construct a fault propagation path from the earliest abnormal node to S3: Physical Server N1 -> Virtual Machine VM1 -> Container C1 -> Service S1 -> Service S2 -> Service S3. Finally, the system identifies Service S1, which failed earliest in the service call chain, as the potential source of the fault. Simultaneously, by examining the log data of S1, it verifies that its failure time (T1) is indeed earlier than the abnormal response time (T3) of S3, and evidence of fault propagation is found in the call chain data between T1 and T3, such as a sudden increase in the failure rate of calls from S1 to S2. Based on these analyses, the system confirms that the path from S1 to S3 is the actual fault propagation path.

[0079] Furthermore, the intelligent code generation and execution mechanism plays a crucial role in the analysis of the entire fault propagation path. For example, when the DFS algorithm needs to analyze the call frequency of a specific service, the system automatically generates and executes query code without manual writing or intervention, thus ensuring a high degree of automation and intelligence in fault diagnosis.

[0080] Through the above-described embodiments of this application, particularly the combined use of fault information sets and node topology graphs, the root cause of the fault is effectively located, significantly improving fault response speed and diagnostic accuracy. It not only quickly locates the root cause of the fault but also provides a detailed trajectory of fault propagation, accumulating valuable experience and data for future fault prevention.

[0081] In an optional implementation, determining the node operation path containing the target location as the fault propagation path includes: determining the node operation path containing the target location as a candidate path; determining the service call chain and log data corresponding to the candidate path, wherein the service call chain is used to indicate the call information between services contained in the candidate path, and the log data is used to indicate the log information generated by the operation of the candidate path; comparing the first fault time indicated by the service call chain with the second fault time indicated by the log data; and determining the candidate path as the fault propagation path if the first fault time is before the second fault time.

[0082] It should be noted that candidate paths can be identified during the fault propagation path analysis phase. The system may identify multiple possible propagation paths, which are considered candidates and require further verification to determine which one is the actual fault propagation path.

[0083] A service call chain displays time-series data showing the call relationships between services. It typically includes information such as the source service, target service, call time, and call status, and is used to trace the propagation path of a request in a distributed system.

[0084] Log data includes various log information generated during system operation, such as application logs, system logs, and error logs, which are used to record system status and abnormal events, and help to locate the root cause of failures.

[0085] In an optional implementation, based on the target location, i.e., the suspected origin of the fault, the system constructs multiple possible propagation paths. Each path depicts the system dependencies and call sequence starting from the target location. The system analyzes the service call chains associated with each candidate path, extracting key information such as time, state, source service, and target service from each call. This information is used to construct a call time series to identify the source and path of fault propagation. Simultaneously, the system collects and analyzes all log data involved in the candidate paths, paying particular attention to records marked as faults or anomalies. Log data provides the specific time and contextual information of the fault occurring in the system. The system compares the time of the first occurrence of the fault in the service call chain (first fault time) with the recording time of the fault event in the log data (second fault time). If the first fault time is significantly earlier than the second fault time, this indicates that the fault occurred first in the call chain and was subsequently recorded in the logs, increasing the likelihood that the candidate path is the actual fault propagation path. Based on the results of time consistency verification, the system determines that those candidate paths where the fault occurred in the service call chain before the log recording are the actual fault propagation paths. These paths will be used for subsequent fault root cause localization and impact assessment.

[0086] In one example, within a large private cloud data center, the operations team monitored an abnormally prolonged response time for a critical service (Service Z), a typical service-level failure. During further investigation, the team decided to use the method described in this invention to determine the root cause and propagation path of the failure.

[0087] Service Z, with its prolonged response time, was identified as the target location of the fault. Based on the node topology graph, the system started from Service Z and used a depth-first search (DFS) strategy to find all possible fault propagation paths. These paths involve services directly or indirectly called by Service Z, as well as the containers, virtual machines, and physical hosts hosting Service Z. For each candidate path, the system extracted the corresponding service call chain data, focusing on the call information between Service Z and other services involved in the path, including call time, call status (success / failure), and call latency. Simultaneously, the system also collected and analyzed all log data involved in the candidate paths, especially those anomaly records related to service calls, such as API call failures and connection timeouts.

[0088] First failure time: Determine the time of the first service call that exhibits an anomaly in the service call chain. This is usually a state change point of an upstream service, marked as T1.

[0089] Second failure time: The time when the first abnormal event related to the failure is extracted from the log data, marked as T2.

[0090] The system compares T1 in the service call chain with T2 in the log data. If T1 occurs before T2, it indicates that the fault did indeed propagate through the call chain first, and then left a trace in the log, increasing the likelihood that this candidate path is the real fault propagation path.

[0091] The system further analyzed different levels of anomalies on the candidate paths, including: examining historical CPU usage data of the host machine, revealing that host machine H1's CPU usage reached 95% around time T1, far exceeding the normal average level. It was observed that the memory usage of virtual machine VM1, hosting Service Z, surged to full capacity a few minutes after T1, corresponding to the abnormal CPU usage time of host machine H1, suggesting a potential resource contention issue. Container C1, running on VM1 and hosting Service Z, experienced a significant increase in response time around time T2, and multiple "MemoryError" errors were logged.

[0092] Through in-depth correlation analysis of KPI data (such as CPU utilization and memory utilization) and log data, the system was able to link events such as CPU anomalies on host machine H1, memory overflow anomalies on virtual machine VM1, and prolonged service response times in container C1. Since the host machine's resource pressure (abnormal CPU utilization) preceded the virtual machine's memory problems (VM-level KPI anomalies) and the increased service response time (service-level trace anomalies), the system confirmed the fault propagation path from resource strain on host machine H1 to memory overflow on virtual machine VM1, and then to the abnormally prolonged response time of Service Z in container C1.

[0093] Through the above-described implementation method of this application, the system successfully determined the propagation path of the fault, starting from abnormal resource pressure at the physical node level, gradually spreading to memory anomalies at the virtual machine level, and finally causing a decline in the response performance of container-level services, through detailed time node comparison and cross-modal and cross-level anomaly correlation analysis. This process not only reveals the root cause of the fault (physical host resource overload) but also provides a detailed propagation trajectory, facilitating the operations and maintenance team to quickly understand the full picture of the fault and implement targeted optimization measures, such as adjusting resource allocation strategies, to prevent similar problems from recurring.

[0094] In an optional implementation, determining the target node based on the fault propagation path includes: identifying at least one node layer included in the fault propagation path, wherein the node layer is used to indicate the abstraction level of the target device; determining the most downstream reference fault and / or target fault corresponding to each of the at least one node layer as the first node corresponding to each of the at least one node layer; and determining the first node corresponding to the base layer in the at least one node layer as the target node.

[0095] It should be noted that, Figure 3 This is a schematic diagram of an optional node layer according to an embodiment of this application; as shown Figure 3 As shown, the node layer may include:

[0096] Physical node layer 302: This refers to the most basic hardware infrastructure layer in a private cloud deployment, including servers, storage devices, network equipment, etc. In fault diagnosis, physical node-level faults may include hardware failures, operating system anomalies, network connectivity problems, etc. These faults typically have a direct impact on the upper-layer virtualization environment and applications.

[0097] Container / Virtual Machine Layer 304: In a private cloud virtualization environment, containers (such as Docker) and virtual machines (VMs) are the basic units for hosting applications and services. This layer focuses on the state of containers or virtual machines running on physical nodes, such as resource usage (CPU, memory), running status, and network isolation issues. Container or virtual machine level failures are typically caused by errors in the application within them, improper resource configuration, or problems with the underlying physical nodes.

[0098] Service / Application Layer 306: This is the highest abstraction layer in a private cloud environment, involving specific application services and microservice architecture. Service-level failures may include internal service logic errors, API call failures, inter-service communication anomalies, database connection problems, etc. Troubleshooting at this level often requires in-depth analysis of application code, call chains, and log information to identify abnormal behavior at the service or application level.

[0099] Each layer has its specific metrics, logs, and dependencies. Faults can occur at any layer and propagate upwards or downwards through inter-layer dependencies, affecting the stability and performance of the entire private cloud environment. Therefore, the multi-layer dependency analysis and propagation modeling technology proposed in this invention aims to identify and locate cross-layer fault propagation paths through multi-modal data fusion and intelligent analysis, thereby more accurately determining the root cause of faults and effectively improving the efficiency and accuracy of private cloud operations and maintenance.

[0100] In a fault propagation path, a fault at the end of the call chain or at the outermost layer of the affected scope is considered a "downstream" fault. A reference fault may be another anomaly detected by the system during fault diagnosis, while a target fault is a direct fault that directly affects end users or critical business operations.

[0101] The foundation layer refers to the lowest layer in a private cloud architecture, which is directly related to hardware resources or network connections, such as physical servers and network devices.

[0102] In an optional implementation, within a complex private cloud environment, the operations team detected a significant increase in the response time of the web application Service W accessed by end users, initially identifying it as a typical service-level failure. The system integrates KPI data, call chain data (Trace Span), and log data from the time of the failure to provide sufficient information for fault diagnosis. Starting from Service W, the system traces upstream along the dependencies in the topology graph, while simultaneously performing a deep inspection of lower layers (such as the container, virtual machine, or physical server hosting Service W). The system analyzes the temporal relationship of the failure, prioritizing components with the earliest occurrence, high criticality, and wide impact as candidates for the fault source. The system employs a time-series-based pruning strategy, excluding components that become abnormal after Service W in the fault propagation path, focusing only on anomalies that occur earlier or at lower levels than Service W.

[0103] When multiple failures occur simultaneously at the same level (such as the service layer), the system prioritizes analyzing the service at the lowest position in the call chain, as the failure here is more likely to be the direct result or the starting point of a chain reaction. For failures across levels, the system tends to identify anomalies in the foundational layer (such as the physical node layer) as the most likely root cause, because failures in underlying hardware resources or networks often have a wide-ranging impact on upper-layer services.

[0104] The system calculates priority parameters for each first node, taking into account the timing, criticality (such as core business services), and scope of impact of the fault. The first node located in the foundation layer with the highest priority parameter will be identified as the final target node, i.e., the true source of the fault.

[0105] Through the above-described implementation methods of this application, in a complex and ever-changing private cloud environment, the source of a fault can be quickly located by intelligently analyzing multimodal data, and basic layer problems can be addressed first, so as to minimize business interruption and improve the overall operational efficiency of the system.

[0106] In an optional implementation, determining the most downstream reference fault and / or target fault corresponding to at least one node layer as the first node corresponding to at least one node layer includes: when there are multiple reference faults and / or target faults at the most downstream, determining the first node based on the log data and call data of the reference faults and / or target faults.

[0107] Priority parameters can be quantitative indicators calculated based on factors such as the abnormal timing of the fault, the direction of propagation, the criticality of the node, and the scope of the fault. They are used to assess the importance of the fault and its likelihood of being the root cause.

[0108] When locating complex faults in a private cloud environment, the system employs a bottom-up analysis strategy, starting with the affected "downstream" components and gradually tracing upwards to find the source of the fault. The core of this strategy lies in the granular analysis of the downstream reference and target faults at each level (physical node, container, service), utilizing logs and call chain data as crucial evidence to help determine the direction and intensity of fault propagation. By comparing the timing information (when it occurred), propagation direction (where it came from and where it went), criticality (which core components were involved), and impact scope (how many downstream components were affected) of these anomalies, the system can calculate a priority parameter. This parameter comprehensively assesses the likelihood that the fault is the root cause. When multiple reference and / or target faults exist, the system selects the fault node with the highest priority parameter as the first node, i.e., the most likely fault origin or critical propagation point.

[0109] In an optional implementation, the system integrates multimodal telemetry data, including KPI time series, call chain tracing (Trace Span), and semi-structured log data, ensuring that all data is temporally aligned and formatted into a unified internal representation. Utilizing the previously constructed dynamic multi-level topology graph, the system traces the fault propagation path backward from the target fault using a depth-first search (DFS) algorithm, simultaneously collecting relevant information on all anomalous nodes along the path. The system analyzes the anomalous time of each node in the fault propagation path, combining error stacks and critical events from the log data to determine the propagation order and intensity of the anomalies, providing a basis for subsequent root cause identification. For each reference fault and target fault, the system calculates a comprehensive priority parameter based on its anomalous timing, propagation direction, node criticality, and fault scope. For example, a low-level, highly critical physical node whose resource depletion leads to widespread anomalies in upper-level components may have a very high priority parameter. After analyzing all downstream reference faults and target faults, the system selects the node with the highest priority parameter as the first node, i.e., the component most likely to be the source of the fault or a critical propagation node. Based on the identified first node and its abnormal characteristics, the system will combine adaptive threshold algorithms, multimodal data fusion, and intelligent code self-correction mechanisms to conduct in-depth root cause identification and ultimately provide the most likely root cause of the failure.

[0110] Through the above-described embodiments of this application, the system can not only accurately identify the propagation path of the fault, but also intelligently determine the most likely source of the fault based on the manifestation and impact range of the fault at different levels, providing maintenance personnel with a fast and accurate means of fault location, and greatly saving the time for fault investigation and recovery.

[0111] In an optional implementation, determining the target node based on the fault propagation path includes: determining fault information corresponding to at least one reference fault and the target fault included in the fault propagation path; determining priority parameters corresponding to at least one reference fault and the target fault based on the fault information, wherein the fault information is used to indicate the abnormal timing, propagation direction, node criticality, and fault range of the fault; and determining the node corresponding to the reference fault and / or the target fault with the highest priority parameter as the target node.

[0112] The fault information includes key information such as the time of the fault, the type of fault (e.g., high CPU utilization), the scope of impact (e.g., how many users were affected), and the direction of fault propagation (whether it started from a physical node, container, or service).

[0113] Upon detecting an anomaly in the response time of Service T, the system first collects fault-related KPI time series, call chain tracing data, and log data to form a fault information set. Next, the system uses a depth-first search algorithm to trace the propagation path of the fault in the constructed multi-level topology graph, identifying a series of potential fault points, including components at different levels such as physical nodes, containers, and services. These components are marked as reference faults or target faults during the anomaly detection phase.

[0114] In an optional implementation, the system then proceeds to the fault information analysis phase, calculating the priority of each reference fault and target fault. This process, based on an intelligent root cause prioritization model, involves the following steps:

[0115] The system examines the anomaly occurrence time of each node along the fault propagation path, prioritizing nodes with the earliest anomaly occurrences, as these are more likely to be the starting points of the fault propagation. Simultaneously, considering the direction of anomaly propagation, it prioritizes analyzing components at the starting points of downstream node anomaly propagation paths. The system references component importance information in the CMDB (Customer Management Database) and business operation data to identify components critical to business operations. For example, physical nodes hosting database servers or service components handling critical business logic receive higher priority in the priority ranking. By comparing the current fault mode with records in the historical fault database, the system searches for possible similar cases. If the current fault mode has a high match with a resolved fault mode in the past, the system prioritizes the root cause of the historical fault as a candidate to quickly guide fault localization strategies. The system assesses the severity and impact of each anomaly, such as CPU utilization reaching 99%, memory overflow, and the duration of service unavailability. Faults with higher severity and wider impact receive higher priority parameters. The system has a built-in knowledge base of operation and maintenance experts and fault diagnosis rules. When the priority parameters calculated by the model match the scenario described in the expert rules, the system will give extra weight to ensure that the root cause localization process makes full use of expert experience.

[0116] Through the above-described implementation methods of this application, the intelligent root cause priority ranking model, combined with anomaly time sequence analysis, component criticality assessment, historical pattern matching, consideration of fault severity and impact scope, and integration of expert knowledge rules, accurately locates the most likely root cause of the fault from multi-level and multi-dimensional fault information, significantly improving the accuracy of fault location and operation and maintenance efficiency, and providing strong technical support for fault diagnosis in private cloud environments.

[0117] In an optional implementation, determining a fault information set based on at least one fault of the target device includes: determining at least one fault of the target device; identifying at least one fault, determining the node and node layer where each at least one fault is located; and obtaining a fault information set by combining the at least one fault and the node information set corresponding to each at least one fault.

[0118] In an optional implementation, the system detected an abnormal spike in CPU utilization to 95% on the physical server P1 hosting Service A during the fault period, which was considered a reference fault at the physical node layer. Further investigation revealed that Service A was running on a virtual machine named V1, and V1's memory utilization reached its limit during the same period, representing another reference fault at the container / virtual machine layer. In addition to Service A, high latency was detected in calls to its dependent database service D1, which was considered a reference fault at the service / application layer.

[0119] The identified reference faults are recorded in detail, including the nature of the fault (e.g., "abnormal CPU utilization", "abnormal memory utilization", "database call latency"), the time of occurrence, duration, scope of impact (e.g., whether it affected other services), and the specific fault node (P1, V1, D1) and its node layer (physical node layer, container / virtual machine layer, service / application layer). All this information is combined to form a "fault information set," providing rich context for subsequent fault localization and root cause analysis.

[0120] In an optional implementation, the system automatically collects KPI data for the target device (Service A) from the monitoring system, including response time and processing capacity. Simultaneously, the system integrates call chain data (Trace) and log data (Logs) to ensure comprehensive coverage of fault information at the physical node layer, container / virtual machine layer, and service / application layer. A controller using an intelligent agent layer performs logical reasoning, generating and executing Python code through an executor to dynamically analyze telemetry data streams and identify reference faults related to increased response time in Service A. The controller determines the specific node and layer of the fault based on its nature and timing, such as physical server P1, virtual machine V1, and database service D1. The identified fault information is integrated into a structured dataset, including a detailed description of each fault, its occurrence time, duration, and impact scope, as well as related node information (node ​​ID, node type, and layer). This fault information set provides crucial input for the intelligent root cause localization algorithm, helping the system accurately trace the source of faults across multiple dependency layers.

[0121] Through the above-described embodiments of this application, a comprehensive set of fault information is constructed using intelligent multimodal data analysis and fault hierarchy identification mechanisms, providing a solid data foundation for higher-level fault location and root cause analysis. This process fully utilizes automated data processing and the reasoning capabilities of large language models, significantly improving the efficiency and accuracy of fault identification, which is of great significance for complex fault diagnosis in private cloud environments.

[0122] In an optional implementation, determining at least one fault of the target device includes: generating a fault threshold based on reference data of the target device, wherein the reference data is used to indicate the operating data of the target device when no fault exists; and comparing the current operating data of the target device with the fault threshold to determine at least one fault of the target device.

[0123] It should be noted that anomaly detection is performed on KPI time series data:

[0124] The system first collects time-series data of key performance indicators (KPIs) such as CPU utilization, memory utilization, disk I / O, and network bandwidth from various components (virtual machines, containers, services, etc.) of the private cloud platform. A dynamic threshold is calculated for each KPI using statistical methods (such as calculating the P95, P90, or lower percentiles). This threshold reflects the upper or lower limit of the KPI value under normal operating conditions. The currently collected KPI values ​​are compared with the calculated dynamic thresholds. If a KPI value exceeds (for performance indicators) or falls below (for resource consumption indicators) the threshold, the system marks it as an anomaly, potentially indicating the presence of a fault.

[0125] Anomaly detection for call chain data:

[0126] Acquire distributed tracing data to understand inter-service call relationships and call performance. Similarly, the system calculates dynamic thresholds for expected call duration and success rate based on historical data. The duration and status of the current call are compared to these thresholds. A significant increase in call duration or failure rate may indicate a problem in a service within the call chain.

[0127] Anomaly detection in log data:

[0128] The system uniformly collects and analyzes application logs, system logs, and security logs to look for keywords or patterns related to faults. The system defines rules, such as lists of error keywords or abnormal event patterns, which can be considered a kind of "threshold," albeit a more qualitative one, used to identify specific types of abnormal events. When abnormal keywords or patterns matching the rules appear in the logs, the system records them as signs of a fault.

[0129] Comprehensive analysis and root cause localization: After detecting anomalies in the above three data sources, the system will comprehensively analyze these anomalies, determine the causal relationship between anomalies through multimodal data correlation analysis, backtracking of multi-level dependency graphs, and reasoning capabilities based on large language models, and finally locate the root cause of the failure.

[0130] Through the above-described embodiments of this application, by comparing with dynamic thresholds, the system can detect abnormal activities in the private cloud environment in real time. Combined with in-depth analysis of multimodal data, it can effectively identify and locate faults, thereby improving the speed and accuracy of fault diagnosis.

[0131] In an optional implementation, generating a fault threshold based on reference data of the target device includes: determining fault parameters according to data categories, wherein the fault parameters are used to indicate the proportion of normal data; sorting the parameter data by size, and then determining the fault threshold according to the fault parameters.

[0132] In an optional implementation, the system collects fault-free time-series data from the past week (e.g., 7 days), including key performance indicators (KPIs) such as CPU utilization, memory utilization, and network throughput, as reference data to build a baseline.

[0133] Percentile method: The system calculates the P95 and P90 percentiles of the KPI data as initial fault thresholds for identifying high-value anomalies such as performance spikes and resource exhaustion. For low-value anomalies such as traffic drops and service interruptions, the system calculates P5, P10, and P15 as references.

[0134] Mean plus standard deviation method: Suitable for KPI data distributions that are close to normal, by using the mean ± N The standard deviation is used to define the threshold, where N is a preset constant, usually set to 2 or 3, to cover 95% or 99.7% of the normal data distribution.

[0135] Moving average and exponential smoothing: When dealing with short-term trends and data fluctuations, the system uses these two methods to capture and adjust the sensitivity of the threshold by dynamically calculating the baseline, ensuring that it can adapt to the dynamic changes in system behavior.

[0136] Based on the above statistical analysis results, the system determines an initial strict anomaly threshold for each KPI. For example, P95 or P90 is used as the upper limit of the threshold to identify high-value anomalies such as high resource consumption; while P5 or P10 is used as the lower limit of the threshold to detect low-value anomalies such as decreased traffic or service instability. These thresholds combine large data volumes and multi-method fusion analysis, which can more accurately reflect the data distribution boundaries under normal operating conditions and reduce the risk of false alarms and false negatives.

[0137] In an optional implementation, the operations team periodically collects KPI time-series data from the private cloud environment, covering all critical components, including physical servers, virtual machines, containers, and services. The data is standardized and cleaned to remove noise and extreme values. The system performs statistical analysis on the KPI data from the past week, calculating the P95, P90, P5, and P10 percentiles for each indicator, as well as the baseline and threshold range using the mean plus standard deviation method, providing a reference for subsequent anomaly detection. When analyzing current KPI data, the system uses moving averages and exponential smoothing techniques to dynamically adjust the baseline to adapt to short-term data trends. When a data point is detected to deviate too much from the baseline, the system will determine whether an anomaly exists based on the current fault parameters (P95, P90 or P5, P10), and further verify this using the mean plus standard deviation method. The system continuously monitors KPI data, and triggers an anomaly signal once a data point exceeds the fault threshold. Simultaneously, the system records anomaly events and analyzes their frequency and intensity for subsequent dynamic threshold adjustment and optimization. If the system detects that an initial strict threshold is causing too many false alarms (e.g., frequent alarms during fault-free periods), it will dynamically adjust the threshold, for example, from P95 to P90, to reduce sensitivity. Conversely, if the threshold is causing missed alarms (e.g., failure to detect anomalies in a timely manner during fault periods), it will tighten the threshold to increase sensitivity, for example, from P90 to P95. The system feeds back the anomaly detection results to the operations team, and simultaneously uses an intelligent agent layer for code generation and self-correction to ensure the automation and robustness of the analysis process.

[0138] Through the above-described embodiments of this application, the dynamic threshold adjustment strategy can detect system anomalies more intelligently and accurately in a private cloud environment, effectively avoiding false alarms and missed alarms, improving the efficiency and accuracy of fault diagnosis, and providing strong technical support for the operation and maintenance team. Simultaneously, the intelligent agent architecture based on a large language model enables the system to automatically adjust thresholds according to actual detection results, achieving automated and intelligent fault detection, and bringing a revolutionary solution to operation and maintenance management in a private cloud environment.

[0139] In an optional implementation, after comparing the current operating data of the target device with a fault threshold to determine at least one fault of the target device, the method includes: adjusting the fault threshold if the number of faults of at least one fault is less than a preset number; and determining at least one fault of the target device based on the adjusted fault threshold.

[0140] Suppose that in a private cloud environment, an increase in the response time of the web service Service X is detected. Initial anomaly detection using a pre-set, strictly static threshold reveals that the number of anomalies is too small to support comprehensive fault analysis and root cause localization. To address this issue, a dynamic threshold adjustment mechanism is adopted to discover more meaningful anomalies and provide richer information for subsequent fault diagnosis.

[0141] In an optional implementation, the system first uses an initial strict threshold (e.g., P95) determined for each KPI to perform anomaly detection to confirm whether Service X's response time exceeds the baseline. The system evaluates the number of anomalies detected by the initial threshold. If the number of anomalies is lower than a preset minimum effective number (e.g., 5), the intelligent agent layer (controller) determines that the current threshold may be too strict, leading to missed anomalies. Based on the above evaluation, the controller triggers dynamic adjustment of the threshold, for example, adjusting the P95 threshold down to P90 (for high-value anomalies), or adjusting P10 to P5 for low-value anomalies. This adjustment strategy avoids blindly relaxing the threshold, ensuring that the adjusted threshold can still effectively identify anomalies, rather than simply increasing the number of false alarms. The system again uses the adjusted fault threshold for anomaly detection and evaluates the detection results. If the adjusted threshold can identify more meaningful anomalies, and the number of anomalies reaches the preset minimum effective number, then the system considers the current threshold adjustment effective. If the number of anomalies is still insufficient, the system will perform further threshold adjustments until a reasonable fault threshold that reduces missed alarms and avoids false alarms is found.

[0142] Through the above-described embodiments of this application, the effectiveness and coverage of anomaly detection are effectively increased by dynamically adjusting the threshold, providing more accurate data support for fault diagnosis in private cloud environments. By combining the control and execution mechanism of the intelligent agent layer, this invention can achieve intelligent adjustment of the threshold, ensuring the flexibility and intelligence of fault detection, significantly improving the efficiency and accuracy of fault location, and providing a strong basis for decision-making for the operation and maintenance team.

[0143] In an optional implementation, the current operating data of the target device is compared with a fault threshold to determine at least one fault of the target device, including: if the difference between the current operating data and the fault threshold is greater than a first difference, the node corresponding to the current operating data is faulty; if multiple consecutive current operating data meet the fault conditions, the node corresponding to the current operating data is determined to be faulty, wherein the fault conditions are determined based on the fault threshold.

[0144] The first difference refers to the degree of deviation between the outlier and the fault threshold, expressed as a percentage. For example, if the CPU utilization of Service W exceeds the fault threshold (P95, which is usually 95%) by more than 50%, then this outlier will be considered significant, that is, it deviates from the normal value by more than 50%.

[0145] In private cloud operations and maintenance, relying solely on a single fault threshold to detect anomalies often leads to false alarms or missed alarms. Therefore, a series of noise filtering mechanisms have been introduced to improve the quality of anomaly detection and ensure that the detected anomaly signals are genuine and reliable.

[0146] Deviation Filtering: The system calculates the deviation between abnormal CPU utilization values ​​of Service W and a fault threshold (such as P95). Only when this deviation exceeds a preset first difference (e.g., deviating from the normal value by more than 50%) is the abnormal value marked as a potential fault signal. This step effectively filters out "false positive" anomalies caused by instantaneous data fluctuations or other temporary interference.

[0147] Continuity requirement: The system not only focuses on individual anomalies but also checks whether outliers persist over a period of time. For example, if Service W's CPU utilization exceeds the fault threshold for three consecutive sampling points within 5 minutes, this will be considered a continuous and valid fault signal. This mechanism eliminates isolated, transient noise points, ensuring the robustness and reliability of fault signals.

[0148] Multi-dimensional cross-validation: The system not only focuses on CPU utilization anomalies, but also combines other KPIs (such as memory utilization and disk I / O), call chain data (trace), and log information to comprehensively analyze Service W's overall performance within the failure time window. Only when consistent evidence can be found from data across different dimensions is the anomaly considered highly reliable and confirmed as a genuine failure signal.

[0149] Anomaly scoring and prioritization: For all confirmed anomaly signals, the system scores them based on their deviation, duration, and impact on the overall system or other components. This scoring mechanism allows the operations team to prioritize anomalies, focusing on the most urgent and valuable fault signals, thus improving the efficiency of fault response and handling.

[0150] In an optional implementation, before generating a fault threshold based on reference data of the target device, the method includes: determining the operating parameters of the target device within a reference time as reference data; determining the log data of the target device within a reference time as reference data; and determining the server call chain of the target device within a reference time as reference data.

[0151] In a private cloud environment, the response time of the core transaction service Service C suddenly became longer. In order to accurately locate the cause of the failure, the first task before fault localization is to build a comprehensive reference dataset for subsequent fault threshold setting and anomaly detection.

[0152] The reference time is the time interval used to collect historical operational data, typically a representative period of normal system operation, used to establish a baseline for fault early warning. For example, the period within the past week during which Service C did not experience any faults. Operational parameters refer to the time-series KPI data recorded on the target device within the reference time, such as CPU utilization, memory utilization, disk I / O read / write speed, network throughput, and other metrics.

[0153] Operational parameter collection: The system automatically extracts KPI data of Service C from the monitoring system within a specified reference time (such as a period of stable system operation in the past week), including indicators such as CPU utilization, memory usage, disk activity and network traffic.

[0154] Log data collection: Synchronously acquire the log files of Service C and its related services, covering application logs, system logs and security logs, and record the running status and related events of Service C within a reference time period.

[0155] Server call chain tracing: Using distributed tracing tools, call chain data of Service C is recorded and collected within a reference time period to track its interaction with other services, including information such as call frequency, latency, and success rate.

[0156] In optional implementations, the collected raw data is cleaned to remove invalid entries, outliers, and duplicate records, ensuring data quality and reliability. Timestamp formats are standardized to the UTC standard, and indicator names are standardized to ensure consistency in data formats from different sources, facilitating subsequent analysis and processing. KPI data is aggregated into independent time series, and the timestamps of call chain data and log data are precisely aligned to ensure synchronization and consistency of all data on the timeline, preparing for subsequent cross-modal data correlation analysis.

[0157] Features are derived from collected operational parameters, such as the rate of change in CPU utilization, peak memory usage, and average disk I / O. Pattern analysis is performed on log data to identify common operational processes and abnormal behaviors, providing deeper insights into service operational status. Service dependency features, such as call frequency, failure rate, and latency distribution, are extracted from server call chain data to assess the health status and potential problems between services. All preprocessed and feature-extracted data is integrated into a structured reference dataset and stored in a data warehouse, serving as the basis for setting fault thresholds and anomaly detection.

[0158] Through the above-described implementation methods of this application, not only can the performance bottleneck of Service C be quickly located, but also intelligent algorithms can be used to deeply analyze the call chain and log information, further trace the fault propagation path, accurately locate the root cause, and significantly improve the efficiency of fault response and resolution.

[0159] In an optional implementation, before determining the fault information set based on at least one fault of the target device, the method includes: upon receiving a first instruction, determining the target device according to the first instruction; generating a set of prompt words according to the first instruction; generating an operation logic file based on the set of prompt words, wherein the operation logic file is used to determine the fault information set based on at least one fault of the target device, generate a node topology diagram according to the operating architecture of the target device, generate a fault propagation path corresponding to the target fault in the node topology diagram using the fault information set, determine the target node according to the fault propagation path, and determine the target node according to the operation logic file.

[0160] It should be noted that during the operation and maintenance of a private cloud data center, the monitoring platform noticed a response delay in a core service, Service Z, initially suspecting it was caused by resource contention or a failure of underlying components. To accurately pinpoint the source of the fault, the intelligent root cause localization method proposed in this invention was used, including steps such as generating operation logic files, constructing a node topology graph, and tracing the fault propagation path, to quickly and automatically identify the target node.

[0161] The first instruction refers to a fault diagnosis command issued automatically by maintenance personnel or the system, which aims to conduct in-depth analysis of faults in a specific service (target device).

[0162] The prompt word set is a set of keywords or phrases generated by the intelligent agent (Controller) based on the first instruction, used to guide the executor in focusing on key areas during fault location analysis, such as "CPU utilization", "memory leak", "network latency", etc.

[0163] The operation logic file is a Python script generated by the intelligent agent based on a set of prompt words. It contains the specific logic required for fault diagnosis, such as how to extract and analyze fault-related telemetry data (KPIs), log records and call chain information, and how to construct and analyze fault propagation paths based on the data.

[0164] In an optional implementation, the operations team submits a fault diagnosis task through the monitoring interface, explicitly identifying the target device as Service Z and the fault symptom as response delay. Upon receiving the initial instruction, the intelligent agent's controller performs logical reasoning using its built-in Large Language Model (LLM) to determine key areas for fault localization analysis, such as CPU utilization, memory usage, and network communication status, forming a set of prompt words. Based on this set of prompt words, the controller generates a Python script to automatically extract and analyze telemetry data from Service Z and its associated components, including KPIs, call chains, and log data. The script also includes algorithmic logic for constructing a node topology graph and tracing fault propagation paths. The executor runs the Python script in an interactive execution environment (such as the IPython Kernel), extracting data from the monitoring system for the target device Service Z and its associated components, performing anomaly detection, and constructing a dynamic node topology graph. The executor, through the KPI anomaly detection module, identifies an abnormal increase in Service Z's CPU utilization during the fault period and an extended response time for its dependent database service S1. These anomalies are recorded as part of the fault information set. The intelligent algorithm constructs a dynamic dependency graph for Service Z based on Configuration Management Database (CMDB) information, service registration and discovery data, and real-time call chain information. By analyzing the call relationships and carrying structures in the graph, the intelligent root cause localization algorithm can identify critical paths of fault propagation, such as a call chain failure from S1 to Service Z. Combining the fault information set with the dynamic topology graph, the system, through multi-level dependency analysis and anomaly propagation path reconstruction, identifies that the problem started with network latency anomalies in S1, gradually affecting the CPU utilization of the virtual machine hosting it, and ultimately causing response latency issues in Service Z. Through the above fault propagation path analysis, the system identifies database service S1 as the target node, believing that its network latency anomalies are likely the root cause of the entire fault chain and should be prioritized for troubleshooting and repair. The executor feeds back the diagnostic information of the fault propagation path and target node to the controller. The operations team receives the diagnostic report generated by the system, confirming that the source of the fault is in S1, and can quickly take measures to carry out targeted fault repair.

[0165] In an optional implementation, the intelligent architecture, consisting of a controller and an executor, aims to separate fault diagnosis decision-making from actual data processing and code execution, achieving automated and intelligent fault diagnosis. The controller component receives fault diagnosis instructions, performs logical reasoning and strategy formulation, and does not directly participate in data operations; instead, it generates analysis instructions for the executor. The executor component receives instructions from the controller, translates them into executable Python code, and executes it in the IPython Kernel, while also possessing error handling and self-correction capabilities. The interactive execution environment (IPython Kernel) provides state preservation and variable reuse capabilities, allowing the results of code execution to be directly used by subsequent analysis steps, improving analysis efficiency and coherence.

[0166] In an optional implementation, the operations team sends a fault diagnosis task to the intelligent agent layer through the private cloud operations platform, describing the abnormal response time of Service C and requesting in-depth analysis.

[0167] The controller utilizes an embedded Large Language Model (LLM) to perform semantic understanding and logical analysis on the fault descriptions provided by the operations team, identifying the main objectives and analysis dimensions for fault diagnosis. Based on the inference results of the LLM, the controller generates preliminary analysis strategies, such as "analyzing the CPU and memory usage of Service C", "checking the call chain between Service C and database service D1", and "extracting error logs from Service C during the abnormal time period".

[0168] After receiving the natural language analysis instructions from the controller, the executor uses a code generation engine to parse them and understand the specific requirements of the analysis task. Based on the type of analysis task (e.g., KPI analysis, call chain analysis), data format (e.g., CSV, JSON), and existing analysis results (e.g., intermediate data obtained through IPython Kernel state preservation capabilities), the executor generates corresponding Python code. The generated code executes securely within the IPython Kernel environment, utilizing data processing libraries such as Pandas and NumPy to load, clean, preprocess, and analyze the runtime data of Service C.

[0169] The executor feeds back the analysis results (such as the values ​​of abnormal indicators, abnormal paths in the call chain, and key information in the error log) to the controller. Based on the executor's feedback, the controller adjusts its analysis strategy. For example, if abnormal CPU usage is found in Service C, it will further request an investigation into memory usage; if call chain analysis reveals latency in database service D1, it will request a detailed analysis of D1's runtime status. The controller generates new analysis instructions for the executor, which then optimizes its code or generates new code for subsequent analysis.

[0170] The above-described implementation methods of this application significantly reduce manual intervention and improve the efficiency and accuracy of fault location. The intelligent agent not only provides a clear fault diagnosis path but also ensures the continuity of the analysis process and the efficiency of data processing through the state-preserving capabilities of the interactive environment (IPythonKernel), bringing a revolutionary workflow to fault diagnosis in private cloud environments.

[0171] In an optional implementation, after determining the target node based on the operation logic file, the process includes: determining the exception information in the event of an exception in the operation logic file; generating a first prompt word based on the exception information; and updating the operation logic file based on the first prompt word.

[0172] It should be noted that exception information can be code execution errors caught by the executor when executing the operation logic file, including specific error types (such as KeyError), error messages (such as "'disk_io' column not found"), and stack traces. Updating the operation logic file refers to the executor modifying the erroneous parts of the original Python script based on the first prompt word generated by the controller, in order to bypass or resolve the previous exception in subsequent code execution.

[0173] The intelligent agent's executor possesses powerful error capture and analysis capabilities, enabling it to instantly identify and report anomalies during code execution. This ensures that any potential data processing or logic errors do not affect the correctness and integrity of the entire fault detection process. The controller, based on the anomaly information fed back by the executor, performs in-depth error reasoning and generates targeted repair instructions. This allows for dynamic code correction and retries, ensuring the smooth progress of fault location analysis.

[0174] In an optional implementation, when the executor encounters an error while running the operation logic file, such as a KeyError when processing disk I / O data in Service Y, it captures all exception details, including the error type, error message, and stack trace, providing detailed information for problem localization. Upon receiving the exception information from the executor, the controller uses its embedded Large Language Model (LLM) to perform in-depth analysis of the exception. For example, for a KeyError, the controller might infer a mismatch in column names or a data format issue. Based on the error analysis results, the controller generates a first hint word associated with the error type and cause, such as "data column name mismatch," as guidance for the executor to correct its code. Upon receiving the first hint word from the controller, the executor dynamically adjusts its code generation logic, for example, changing df['disk_io'] to df['Disk IO'] or trying df.columns.str.lower() to resolve the column name mismatch issue. The corrected code is executed again to verify that the error has been successfully resolved. If the corrected code runs successfully without exception, the executor continues to execute subsequent fault analysis instructions. Otherwise, the abnormal information will be fed back to the controller for a new round of error reasoning and correction.

[0175] When the executor attempted to read disk I / O data from Service Y, it incorrectly used `df['disk_io']`. However, because the data column name was "Disk IO," the executor caught a `KeyError` exception and reported detailed error information to the controller. Based on the error information, the controller used LLM analysis to determine that the exception was caused by a mismatched data column name. The controller then generated a first warning message, "data column name mismatch," and issued a correction instruction, requiring the executor to adjust the way the data column name was referenced. After receiving the warning message, the executor dynamically corrected its data processing code, switching to using `df['Disk IO']` to extract disk I / O data. Simultaneously, error handling logic was added to the code to ensure correct data reading under different data formats. The corrected code was re-executed in the IPython Kernel. After correcting the data column name reference, the executor successfully extracted and analyzed Service Y's disk I / O data, confirming that disk read / write latency was a significant cause of Service Y's response delay. The subsequent fault location process proceeded smoothly, and the intelligent agent ultimately identified the disk performance bottleneck on VM1, the virtual machine hosting Service Y, and provided a detailed fault report and repair suggestions.

[0176] Through the above-described implementation methods of this application, by combining the reasoning capabilities of large language models with the dynamic generation and correction mechanism of Python code, anomalies can be effectively handled, ensuring the smooth progress of fault detection and location processes, and providing strong support for intelligent operation and maintenance in a private cloud environment.

[0177] In an optional implementation, after determining the target node based on the operation logic file, the method includes: generating a report file for the target device, the fault propagation path, and the target node, wherein the report file is used to indicate at least one fault.

[0178] After determining the target node based on the operation logic file, the intelligent system will integrate the relevant information of the target device, the fault propagation path, and the target node to generate a report file. This report file can clearly point out the source and scope of impact of at least one fault, providing guidance for operation and maintenance personnel to handle the fault.

[0179] In an optional implementation, the actuator performs data processing and fault propagation path analysis based on the operation logic file generated by the controller, thereby determining the root cause of the fault and the list of affected components.

[0180] The report should first clearly identify the source component or service of the failure, such as "network adapter failure on physical host H1". It should then describe in detail how the failure propagated from the source to other components or services, including direct dependencies and indirect impact chains, such as "network failure on H1 caused a decrease in the network performance of virtual machine VM1 running on it, which in turn affected the response time of service S1 running on VM1". Finally, it should list all critical nodes affected by the failure, including specific component names, affected metrics, and timeframes.

[0181] Reports are typically presented in structured formats, such as PDF or HTML, for easy reading and sharing. They include: a topology diagram visually illustrating the propagation of a fault from its source to the target node; detailed textual explanations of the fault's cause, impact, and system response; and specific remediation steps and recommendations, along with priority ranking, for each fault source and affected node. The intelligent system utilizes a built-in report generation module to integrate this information into a report. This module can structure the analysis results according to preset templates and styles, ensuring clarity and readability. Once generated, the report is saved in the system's output directory and automatically sent to designated operations personnel or fault response teams via email or system notification, enabling them to promptly access fault information and take action.

[0182] Through the above-described implementation method of this application, after determining the source of the fault and the propagation path based on the operation logic file, the generation of the report file provides maintenance personnel with clear fault information and repair guidelines. It is a key link in the fault diagnosis process and helps to achieve rapid response and automated handling of faults.

[0183] Controllers and actuators play crucial roles in each stage of the fault detection and localization process, jointly achieving automation and intelligence in intelligent fault diagnosis. The specific functions of the controller and actuator in each step are detailed below:

[0184] The controller is responsible for receiving fault diagnosis requests from operations and maintenance personnel or the system, and understanding the intent and scope of the requests. For example, if a request is received to "analyze the abnormal response time of Service X yesterday afternoon", the controller will parse out the specific services, metrics, and time windows that need to be monitored.

[0185] Executor: Based on the request parsed by the controller, the executor generates and executes the corresponding data collection and standardization code. For example, the executor might generate a Python code snippet to extract the response time data of Service X from yesterday afternoon from the monitoring system and perform preprocessing operations such as timestamp unification and data type standardization.

[0186] Dynamic threshold adjustment and multi-level dependency analysis:

[0187] Controller: Based on historical data statistical analysis, the controller uses a large language model to infer an initial strict threshold, such as P95, for anomaly detection. Simultaneously, when the number of detected anomalies is insufficient to support further analysis, the controller instructs the executor to adjust the threshold to P90 or P85 based on feedback information, guiding the executor to perform more lenient anomaly detection.

[0188] Actuator: Upon receiving instructions from the controller, the actuator generates Python code to perform dynamic threshold adjustments, such as changing the threshold from P95 to P90. Furthermore, based on the controller's instructions, the actuator also generates code to perform deep analysis of multimodal data (KPIs, logs, call chains) to identify fault propagation paths.

[0189] Multimodal data fusion and fault location:

[0190] The controller is responsible for developing specific multimodal data fusion strategies and fault location algorithms, such as determining when to use KPI data and when to combine call chains and log data for analysis. It generates detailed analysis process instructions based on the nature of the fault and the current diagnostic stage.

[0191] The executor dynamically generates and executes code based on the analysis flow generated by the controller. This could include using Python libraries to integrate KPI data and perform statistical analysis, or analyzing call chain data to build a dependency graph. The executor also executes this code within the IPythonKernel environment, ensuring state continuity and variable reuse.

[0192] Adaptive thresholding and intelligent root cause localization:

[0193] Controller: Utilizes a large language model for logical reasoning to determine when and how to adjust thresholds, and how to optimize root cause localization strategies. For example, if analysis indicates false alarms due to changes in system load, the controller will generate instructions to adjust thresholds and, combined with multi-level dependency analysis, provide a more accurate root cause localization strategy.

[0194] The executor is responsible for translating the controller's policy instructions into executable code, such as code to adjust thresholds, code to execute adaptive thresholding algorithms, and code generated based on the root cause localization strategy. The executor executes this code within the IPython Kernel, ensuring the consistency of the analysis process and the accuracy of the results.

[0195] Results structuring and visualization:

[0196] Controller: After the analysis is completed, the controller is responsible for defining the format and visualization strategy of the output results to ensure that fault location information, analysis trajectory and evaluation indicators can be presented in a way that is easy for maintenance personnel to understand.

[0197] Actuator: Based on the controller's output strategy, it generates code to produce structured results and charts, such as root cause analysis reports, visualizations of anomaly propagation paths, and evaluation metrics for fault diagnosis accuracy and timing precision.

[0198] In a multimodal intelligent root cause localization system based on dynamic threshold adjustment and multi-level dependency analysis, the controller plays a core decision-making and command role. It utilizes the advanced reasoning capabilities of Large Language Models (LLM) to generate a series of targeted indicator words based on the needs and feedback at different stages, guiding the actuators to perform precise data processing and fault analysis. The following is a detailed description of how the controller generates indicator words based on different scenarios throughout the fault diagnosis process:

[0199] Data Input and Preprocessing Stage: In this stage, the controller mainly generates indicator words based on data type, data quality, data consistency, and preprocessing requirements.

[0200] Example of a demonstrative word:

[0201] Data cleaning instructions: "Remove duplicate log entries", "Filter KPI records with low signal-to-noise ratio".

[0202] Data formatting instructions: "Uniform timestamp format to UTC", "Convert Trace Span ID to hexadecimal representation".

[0203] Data fusion indicators: "Merge KPIs and Logs by time series" and "Associate Trace data with CMDB information".

[0204] Time alignment indicator: "KPI, Log and Trace time synchronization".

[0205] Preliminary feature extraction indicators: "Calculate the moving average of CPU utilization" and "Extract key error codes from the logs".

[0206] Anomaly detection phase: based on global threshold calculation results, historical data statistical analysis, and current system status.

[0207] Example of a demonstrative word:

[0208] Dynamic threshold adjustment indicator: "Adjust P95 to P90 according to system load".

[0209] Anomaly detection algorithm indicators: "Use the mean plus standard deviation method to identify memory usage anomalies" and "Apply the percentile method to detect network latency spikes".

[0210] Noise filtering instructions: "Exclude short-term CPU spikes" and "Filter anomalies with less than 5% memory usage".

[0211] Anomaly scoring and ranking indicators: "Score anomalies according to the degree of deviation from the baseline" and "Rank anomalies by the scope of impact".

[0212] Multi-level dependency analysis phase: In this phase, the controller generates indicator words to guide the analysis of fault propagation paths based on the constructed dynamic multi-level topology and the identified anomalies.

[0213] Example of a demonstrative word:

[0214] Hierarchical identification indicators: "Mark physical host resource bottlenecks", "Locate container-level performance fluctuations".

[0215] Fault propagation analysis indicators: "Trace container-level anomalies to the host level", "Analyze the call chain backtracking of service-level faults".

[0216] Cross-level association indicator: "Associate container CPU exception with host machine memory usage".

[0217] The propagation path reconstruction indicator is: "Establish a temporal correlation between call chain anomalies and KPI anomalies".

[0218] Root Cause Locator and Remediation Recommendation Phase: In this phase, the controller generates indicator words based on known anomaly propagation paths, dependency strengths, and historical failure modes to guide intelligent root cause prioritization and intelligent code generation and execution.

[0219] Example of a demonstrative word:

[0220] Root cause priority sorting indicators: "Prioritize service-level API call failures" and "Focus on physical-level network device status anomalies".

[0221] Intelligent code generation instructions: "Generate code to query database connection count", "Write a Python script to analyze error stacks in the logs".

[0222] Self-correcting code execution instructions: "Add exception handling logic to the code", "Try to change the data field name for code that fails to execute".

[0223] Repair suggestion instructions: "It is recommended to restart the network service to resolve the problem" and "It is recommended to optimize service resource configuration to alleviate pressure".

[0224] Output results and operation and maintenance strategy optimization phase: based on fault location results, system performance evaluation and operation and maintenance historical data.

[0225] Example of a demonstrative word:

[0226] Output indicators: "Structured output of fault impact time and components", "Generate a visual report of fault location analysis".

[0227] Operation and maintenance strategy optimization instructions: "Adjust system monitoring thresholds" and "Optimize automated repair processes".

[0228] Knowledge accumulation and training instructions: "Summarize fault cases and add them to the knowledge base", "Generate teaching materials for fault diagnosis process".

[0229] Figure 4 This is a schematic diagram of an optional fault diagnosis method according to an embodiment of this application; as shown... Figure 4 As shown, the system architecture is divided into five layers from bottom to top: data input layer 402, intelligent agent layer 404 (including controller 406 and actuator 408), core algorithm layer 412, analysis and processing layer 410, and output result layer 414. Each layer undertakes specific data processing and fault analysis tasks, together constructing a complete fault diagnosis and root cause localization process.

[0230] Data Input Layer: Receives KPI time-series data, dependency tracing graph data (Trace), and semi-structured log data (Log) from the private cloud environment, providing a data source for subsequent fault analysis. It is responsible for data cleaning, format conversion, and time alignment to ensure data consistency and availability.

[0231] Controller: Utilizes a large language model for logical reasoning and decision-making, receives fault diagnosis tasks, and generates analysis instructions.

[0232] Executor: Transforms the controller's instructions into executable Python code, executes it in the IPython Kernel, and has the ability to self-correct errors.

[0233] Analysis and Processing Layer: Preprocessing Module: Aggregates component KPIs and calculates global thresholds. Anomaly Detection Module: Detects abnormal indicators based on an adaptive threshold algorithm. Fault Identification Module: Performs noise filtering and identifies fault subsequences. Root Cause Analysis Module: Performs multi-level dependency analysis, identifies fault propagation paths, and determines the most likely root cause.

[0234] Core Algorithm Layer: Adaptive Threshold Algorithm: Dynamically adjusts the detection threshold of KPI data to reduce false positives and false negatives. Multi-level Fault Propagation Modeling: Constructs dependency graphs between physical nodes, containers, and services to analyze fault propagation paths. Intelligent Code Generation and Execution: Transforms natural language analysis requirements into Python code to achieve automated data processing and fault diagnosis.

[0235] Output Results Layer: This layer outputs the root cause analysis process and results in a structured format, including time, affected components, and cause of failure. It also outputs evaluation metrics such as accuracy and timing precision, as well as detailed analysis logs, facilitating review and knowledge accumulation for operations and maintenance personnel.

[0236] Multimodal telemetry data flows from the private cloud platform into the data input layer for preprocessing, generating data in a unified format. The preprocessed data is then input into the intelligent agent layer, where the controller generates analysis instructions based on a large language model, and the executor runs the code in an interactive environment. The analysis and processing layer and the core algorithm layer work together to perform preliminary fault identification, anomaly detection, multi-level dependency analysis, and intelligent root cause localization. The analysis results and process are presented in a structured manner in the output results layer, including detailed fault location reports, analysis indicators, and process trajectories, facilitating understanding and corrective action by operations and maintenance personnel.

[0237] In optional implementations, by introducing intelligent diagnostic assistants and proactive defense mechanisms, the system can not only efficiently and accurately locate faults, but also provide early warnings before faults occur, and even automatically take measures to prevent faults from occurring or spreading, thereby achieving a higher level of operation and maintenance automation and intelligent security protection.

[0238] In an optional implementation, the intelligent diagnostic assistant leverages the predictive capabilities of a large language model, combining historical data, fault propagation models, and the current system status to predict potential failure modes. For example, it can analyze the resource usage trend of a component, predict the time point before resource exhaustion, and provide early warnings. Operations personnel can interact with the intelligent diagnostic assistant using natural language to inquire about fault analysis results, fault causes, and remediation strategies. By constructing an operations and maintenance knowledge graph, the intelligent diagnostic assistant can provide personalized consulting suggestions based on a large number of historical fault cases, expert experience, and the latest technical literature. The intelligent diagnostic assistant continuously monitors system status and fault analysis results, constantly optimizing its predictive model through machine learning algorithms to improve the accuracy of fault prediction and the quality of intelligent consultation. Based on the intelligent diagnostic assistant's predictive results and dynamic threshold adjustment algorithms, the system can assess the risk level of each component or service in real time, identifying high-risk points that may become sources of failure. Once the system identifies a high-risk component, the proactive defense mechanism will automatically adjust its configuration, such as increasing resource allocation, activating redundant services, or adjusting load balancing strategies, to reduce the likelihood or impact of failure. The system can intelligently schedule resources, pre-optimizing resource allocation based on predicted failure modes and risk assessment results. This ensures critical components receive sufficient resource support at critical moments, preventing failures caused by resource bottlenecks. Combined with a security policy library and automated operation and maintenance tools, when the intelligent diagnostic assistant predicts a security threat, the proactive defense mechanism can automatically execute remediation processes, such as isolating abnormal components, updating system patches, and optimizing firewall rules, thereby enhancing system security and stability.

[0239] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0240] Embodiments of this application also provide a fault diagnosis device. Figure 5 This is a structural block diagram of an optional fault diagnosis device according to an embodiment of this application, such as... Figure 5 As shown, the device includes:

[0241] The set determination module 502 is used to determine a set of fault information based on at least one fault of the target device, wherein the fault information in the set of fault information is used to indicate the node where the corresponding fault is located;

[0242] The topology generation module 504 is used to generate a node topology diagram based on the operating architecture of the target device, wherein the nodes are used to indicate the relationship between at least one service node of the target device.

[0243] The path generation module 506 is used to generate a fault propagation path corresponding to the target fault in the node topology graph using the fault information set, wherein the target fault is any one of at least one fault;

[0244] The target node determination module 508 is used to determine the target node based on the fault propagation path when the fault propagation path includes at least one reference fault, wherein the target node is used to indicate the root cause of the fault propagation path.

[0245] Optionally, the aforementioned topology graph generation module 504 is further configured to obtain the current operating architecture of the target device, wherein the operating architecture is used to indicate the physical nodes, containers, and application services contained in the target device; determine the physical nodes, containers, and application services included in the operating architecture as at least one graph node; determine the matching relationship between the physical nodes, containers, and application services according to the operating architecture, and determine the connection relationship between at least one graph node based on the matching relationship; determine the edges between at least one graph node based on the connection relationship, and generate a node topology graph.

[0246] Optionally, the path generation module 506 is further configured to: traverse the node topology graph and determine the node running path based on the temporal information and edge indication dependency of at least one node in the node topology graph; determine the target location of the target fault in the node topology graph and determine the node running path containing the target location as the fault propagation path.

[0247] Optionally, the path generation module 506 is further configured to: determine the node running path containing the target location as a candidate path; determine the service call chain and log data corresponding to the candidate path, wherein the service call chain is used to indicate the call information between services contained in the candidate path, and the log data is used to indicate the log information generated by the running of the candidate path; compare the first failure time indicated by the service call chain with the second failure time indicated by the log data; and if the first failure time is before the second failure time, determine the candidate path as a failure propagation path.

[0248] Optionally, the target node determination module 508 is further configured to: identify at least one node layer included in the fault propagation path, wherein the node layer is used to indicate the abstraction level of the target device; determine the most downstream reference fault and / or target fault corresponding to each of the at least one node layer as the first node corresponding to each of the at least one node layer; and determine the first node corresponding to the base layer in the at least one node layer as the target node.

[0249] Optionally, the target node determination module 508 is further configured to: determine the first node based on the log data and call data of the reference faults and / or target faults when there are multiple reference faults and / or target faults at the downstream end.

[0250] Optionally, the target node determination module 508 is further configured to: determine the fault information corresponding to at least one reference fault and the target fault included in the fault propagation path; determine the priority parameters corresponding to at least one reference fault and the target fault based on the fault information, wherein the fault information is used to indicate the abnormal timing, propagation direction, node criticality, and fault range of the fault; and determine the node corresponding to the reference fault and / or the target fault with the highest priority parameter as the target node.

[0251] Optionally, the above-mentioned set determination module 502 is further configured to: determine at least one fault of the target device; identify the at least one fault, determine the node and node layer where each of the at least one fault is located; and obtain a fault information set by combining the at least one fault and the node information set corresponding to each of the at least one fault.

[0252] Optionally, the above-mentioned set determination module 502 is further configured to: generate a fault threshold based on reference data of the target device, wherein the reference data is used to indicate the operating data of the target device when there is no fault; compare the current operating data of the target device with the fault threshold to determine at least one fault of the target device.

[0253] Optionally, the above-mentioned set determination module 502 is further configured to: determine fault parameters according to data categories, wherein the fault parameters are used to indicate the proportion of normal data; and after sorting the parameter data by size, determine the fault threshold according to the fault parameters.

[0254] Optionally, the above-mentioned set determination module 502 is further configured to: adjust the fault threshold when the number of faults of at least one fault is less than a preset number; and determine at least one fault of the target device based on the adjusted fault threshold.

[0255] Optionally, the above-mentioned set determination module 502 is further configured to: determine that the node corresponding to the current running data is faulty when the difference between the current running data and the fault threshold is greater than a first difference; and determine that the node corresponding to the current running data is faulty when multiple consecutive current running data meet the fault conditions, wherein the fault conditions are determined according to the fault threshold.

[0256] Optionally, the above-mentioned set determination module 502 is further configured to: determine the operating parameters of the target device within the reference time as reference data; determine the log data of the target device within the reference time as reference data; and determine the server call chain of the target device within the reference time as reference data.

[0257] Optionally, the aforementioned set determination module 502 is further configured to: upon receiving a first instruction, determine a target device according to the first instruction; generate a set of prompt words according to the first instruction; generate an operation logic file based on the set of prompt words, wherein the operation logic file is used to determine a set of fault information based on at least one fault of the target device, generate a node topology diagram according to the operating architecture of the target device, generate a fault propagation path corresponding to the target fault in the node topology diagram using the set of fault information, determine a target node according to the fault propagation path, and determine a target node according to the operation logic file.

[0258] Optionally, the target node determination module 508 is further configured to: determine the exception information when an exception occurs in the operation logic file; generate a first prompt word based on the exception information; and update the operation logic file according to the first prompt word.

[0259] Optionally, the target node determination module 508 is further configured to: generate a report file for the target device, fault propagation path and target node, wherein the report file is used to indicate at least one fault.

[0260] For a description of the features in the embodiment corresponding to the fault diagnosis device, please refer to the relevant description of the embodiment corresponding to the fault diagnosis method, which will not be repeated here.

[0261] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above-described fault diagnosis method embodiments.

[0262] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described fault diagnosis method embodiments when it is run.

[0263] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0264] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described fault diagnosis method embodiments.

[0265] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described fault diagnosis method embodiments.

[0266] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0267] The foregoing has provided a detailed description of a fault diagnosis method, apparatus, storage medium, and electronic device provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A fault diagnosis method, characterized in that, include: A set of fault information is determined based on at least one fault of the target device, wherein the fault information in the set of fault information is used to indicate the node where the corresponding fault is located; A node topology diagram is generated based on the operating architecture of the target device, wherein the nodes are used to indicate the relationship between at least one service node of the target device; The fault information set is used to generate a fault propagation path corresponding to the target fault in the node topology graph, wherein the target fault is any one of the at least one fault; In cases where the fault propagation path includes at least one reference fault, a target node is determined based on the fault propagation path, wherein the target node is used to indicate the root cause of the fault propagation path; The fault information set determined based on at least one fault of the target device includes: Identify at least one fault in the target device; Identify the at least one fault and determine the node and node layer where each of the at least one faults is located; The fault information set is obtained by combining the at least one fault and the set of node information corresponding to each at least one fault. The step of determining the target node based on the fault propagation path includes: Identify at least one node layer included in the fault propagation path, wherein the node layer is used to indicate the abstraction level of the target device; The most downstream reference fault and / or target fault corresponding to each of the at least one node layer is determined as the first node corresponding to each of the at least one node layer; The first node corresponding to the base layer in the at least one node layer is determined as the target node; The step of determining the target node based on the fault propagation path includes: The controller performs logical analysis based on the first instruction and sends natural language analysis instructions to the actuator; The executor uses a code generation engine to obtain analysis results and sends the analysis results to the controller; The controller determines the target node.

2. The method according to claim 1, characterized in that, The step of generating a node topology map based on the operating architecture of the target device includes: Obtain the current operating architecture of the target device, wherein the operating architecture is used to indicate the physical nodes, containers, and application services contained in the target device; The physical nodes, containers, and application services included in the runtime architecture are identified as at least one graph node; The matching relationship between the physical nodes, containers, and application services is determined based on the operational architecture, and the connection relationship between the at least one graph node is determined based on the matching relationship; Based on the connection relationship, the edges between the at least one graph node are determined, and the node topology graph is generated.

3. The method according to claim 2, characterized in that, The step of generating a fault propagation path corresponding to the target fault in the node topology graph using the fault information set includes: The node topology graph is traversed, and the node running path is determined based on the temporal information of at least one node in the node topology graph and the dependency relationship indicated by the edge. The target location of the target fault in the node topology graph is determined, and the node running path containing the target location is determined as the fault propagation path.

4. The method according to claim 3, characterized in that, The step of determining the node's running path containing the target location as the fault propagation path includes: The node running path containing the target location is determined as a candidate path; Determine the service call chain and log data corresponding to the candidate path, wherein the service call chain is used to indicate the call information between services contained in the candidate path, and the log data is used to indicate the log information generated by the candidate path during operation; Compare the first failure time indicated by the service call chain with the second failure time indicated by the log data; If the first fault time is before the second fault time, the candidate path is determined as the fault propagation path.

5. The method according to claim 1, characterized in that, The step of determining the most downstream reference fault and / or target fault corresponding to each of the at least one node layer as the first node corresponding to each of the at least one node layer includes: In the case where multiple reference faults and / or target faults exist at the downstream end, the first node is determined based on the log data and call data of the reference faults and / or target faults.

6. The method according to claim 1, characterized in that, Determining the target node based on the fault propagation path includes: Determine the fault information corresponding to at least one of the reference faults and the target fault contained in the fault propagation path; The priority parameters corresponding to the at least one reference fault and the target fault are determined based on the fault information, wherein the fault information is used to indicate the abnormal timing, propagation direction, node criticality, and fault range of the fault. The node corresponding to the reference fault and / or target fault with the highest priority parameter is determined as the target node.

7. The method according to claim 1, characterized in that, Determining at least one fault in the target device includes: A fault threshold is generated based on reference data of the target device, wherein the reference data is used to indicate the operating data of the target device when there is no fault; The current operating data of the target device is compared with the fault threshold to determine at least one fault of the target device.

8. The method according to claim 7, characterized in that, The generation of fault thresholds based on reference data from the target device includes: Fault parameters are determined based on data categories, wherein the fault parameters are used to indicate the proportion of normal data; After sorting the parameter data by size, the fault threshold is determined according to the fault parameters.

9. The method according to claim 7, characterized in that, After comparing the current operating data of the target device with the fault threshold to determine at least one fault of the target device, the process includes: If the number of at least one fault is less than a preset number, the fault threshold is adjusted. At least one fault of the target device is determined based on the adjusted fault threshold.

10. The method according to claim 7, characterized in that, The step of comparing the current operating data of the target device with the fault threshold to determine at least one fault of the target device includes: If the difference between the current running data and the fault threshold is greater than a first difference, the node corresponding to the current running data is considered to be faulty. If multiple consecutive current running data meet the fault conditions, it is determined that the node corresponding to the current running data is faulty, wherein the fault conditions are determined based on the fault threshold.

11. The method according to claim 7, characterized in that, Before generating the fault threshold based on the reference data of the target device, the following steps are included: The operating parameters of the target device within the reference time period are determined as the reference data; The log data of the target device within the reference time period is determined as the reference data; The server call chain of the target device within the reference time period is determined as the reference data.

12. The method according to any one of claims 1 to 11, characterized in that, Before determining the set of fault information based on at least one fault of the target device, the following are included: Upon receiving a first instruction, the target device is determined based on the first instruction; Generate a set of prompt words based on the first instruction; An operation logic file is generated based on the set of prompt words. The operation logic file is used to determine a set of fault information based on at least one fault of the target device, generate a node topology map according to the operating architecture of the target device, generate a fault propagation path corresponding to the target fault in the node topology map using the set of fault information, and determine the target node according to the fault propagation path. The target node is determined based on the operation logic file.

13. The method according to claim 12, characterized in that, After determining the target node according to the operation logic file, the process includes: In the event of an anomaly in the operation logic file, determine the anomaly information. A first prompt word is generated based on the aforementioned anomaly information; The operation logic file is updated based on the first prompt word.

14. The method according to claim 12, characterized in that, After determining the target node according to the operation logic file, the process includes: A report file is generated for the target device, the fault propagation path, and the target node, wherein the report file is used to indicate the at least one fault.

15. A fault diagnosis device, characterized in that, include: A set determination module is used to determine a set of fault information based on at least one fault of the target device, wherein the fault information in the set of fault information is used to indicate the node where the corresponding fault is located; A topology graph generation module is used to generate a node topology graph based on the operating architecture of the target device, wherein the nodes are used to indicate the relationship between at least one service node of the target device; The path generation module is used to generate a fault propagation path corresponding to a target fault in the node topology graph using the fault information set, wherein the target fault is any one of the at least one fault; A target node determination module is configured to determine a target node based on the fault propagation path when the fault propagation path includes at least one reference fault, wherein the target node is used to indicate the root cause of the fault propagation path; The set determination module is further configured to: determine at least one fault of the target device; identify the at least one fault, determine the node and node layer where each of the at least one fault is located; and obtain the fault information set by combining the at least one fault and the node information set corresponding to each of the at least one fault. The target node determination module is further configured to: identify at least one node layer included in the fault propagation path, wherein the node layer is used to indicate the abstraction level of the target device; determine the most downstream reference fault and / or target fault corresponding to each of the at least one node layer as the first node corresponding to each of the at least one node layer; and determine the first node corresponding to the base layer in the at least one node layer as the target node. The target node determination module is further configured to: perform logical analysis according to the first instruction and send the natural language analysis instruction to the executor; the executor obtains the analysis result using the code generation engine and sends the analysis result to the controller; and the controller determines the target node.

16. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the fault diagnosis method as described in any one of claims 1 to 14 when executing the computer program.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the fault diagnosis method as described in any one of claims 1 to 14.

18. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the steps of the fault diagnosis method as described in any one of claims 1 to 14.

Images