Distributed system anomaly detection method, electronic equipment and computer readable storage medium

Abstract

The invention provides a distributed system anomaly detection method, electronic equipment and a computer readable storage medium, and the method can comprise the steps: obtaining monitoring data for a target distributed system; performing preliminary screening on the monitoring data to determine an initial abnormal node set; constructing state data based on the initial abnormal node set; the state data is input into a reinforcement learning model, so that the reinforcement learning model searches and determines a fault propagation chain in a pre-constructed system dependency graph, the system dependency graph is a dependency relationship in the target distributed system constructed based on the target distributed system, and the fault propagation chain is determined based on the dependency relationship in the target distributed system. The fault propagation chain comprises one or more fault nodes and a propagation relationship of the fault nodes, and the fault nodes are one of a server, a container and a service instance in the target distributed system. According to the method, fault detection of the distributed system can be clearer and more visual.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and more specifically, to a method for detecting anomalies in a distributed system, an electronic device, and a computer-readable storage medium. Background Technology

[0002] Current solutions for distributed system monitoring and fault diagnosis include rule-based systems, traditional machine learning models, and simple graph algorithms. However, these solutions lack a global perspective and fail to understand the causal relationships between metrics. Summary of the Invention

[0003] The purpose of this application is to provide a method, electronic device, and computer-readable storage medium for detecting anomalies in distributed systems, which enables the detection of faults in distributed systems to be clearer and more intuitive.

[0004] In a first aspect, the present invention provides a method for detecting anomalies in a distributed system, comprising: obtaining monitoring data for a target distributed system; performing preliminary screening on the monitoring data to determine an initial set of anomalous nodes; constructing state data based on the initial set of anomalous nodes; and inputting the state data into a reinforcement learning model to enable the reinforcement learning model to search a pre-constructed system dependency graph to determine a fault propagation chain, wherein the system dependency graph is a dependency relationship in the target distributed system constructed based on the target distributed system, and the fault propagation chain includes one or more faulty nodes and the propagation relationship of multiple faulty nodes, wherein the faulty node is one of a server, a container, or a service instance in the target distributed system.

[0005] In the above implementation, during the detection of the distributed system, preliminary screening is performed based on the data obtained from monitoring to determine the possible initial set of abnormal nodes. After determining the initial set of abnormal nodes, a pre-constructed system dependency graph can be further combined with a reinforcement learning model to search for the fault propagation path, thereby determining the fault propagation chain. This fault propagation chain not only presents the fault itself but also identifies the chain reactions caused by the fault.

[0006] In an optional implementation, the actions of the reinforcement learning model include: the next node to be traversed; the reward of the reinforcement learning model includes one or more of the following: the proportion of abnormal nodes among the nodes connected in the output propagation chain exceeds a threshold, the number of abnormal nodes connected in the output propagation chain exceeds a first threshold, and the proportion of abnormal nodes connected in the output propagation chain to the total number of abnormal nodes exceeds a second threshold; the step of inputting the state data into the reinforcement learning model so that the reinforcement learning model can search the pre-built system dependency graph to determine the fault propagation chain includes: inputting the state data into the reinforcement learning model so that the reinforcement learning model can search the pre-built system dependency graph to determine the node access path that maximizes the cumulative reward, and the node access path that maximizes the cumulative reward is used as the fault propagation chain.

[0007] In the above implementation, for the reinforcement learning model, the reward content and learning actions can be set in combination with the requirements related to anomaly detection in distributed systems. This can enable the determined reinforcement learning model to better identify faults and the chain reactions caused by faults, thereby better finding the complete chain of abnormal nodes.

[0008] In an optional implementation, the initial screening of the monitoring data to determine an initial set of abnormal nodes includes: determining a target identification method based on the data types in the monitoring data, wherein the data types include numerical types and log types, and the target identification method includes a threshold comparison method for the numerical types and an abnormal log matching method for the log types; identifying abnormal data in the monitoring data based on the target identification method; and determining the initial set of abnormal nodes based on the abnormal data.

[0009] In the above implementation methods, different identification methods can be used to identify anomalies for different types of monitoring data, and anomaly identification can be more accurate.

[0010] In an optional implementation, the method further includes: generating a fault diagnosis report based on the fault propagation chain; and displaying the fault node and fault path on a preset visualization topology map of the target distributed system.

[0011] In the above implementation method, a fault diagnosis report can also be generated, which can make the presentation of faults more intuitive and readable. Furthermore, fault nodes and fault paths can be presented in a visual topology map, which can make it easier for users to intuitively understand the fault distribution through the topology map.

[0012] In an optional implementation, the system dependency graph is constructed by: obtaining a system-related dataset of the target distributed system, wherein the system-related dataset includes multiple system performance data and a device label corresponding to each system performance data, and the system performance data includes one or more of the following: application layer data of the distributed system, system layer data of the distributed system, and system setting data of the distributed system; and constructing a system dependency graph based on the system-related dataset.

[0013] In the above implementation, considering that different distributed systems may have different nodes and relationships between nodes, the above implementation constructs a system dependency graph based on the system association dataset of the target distributed system. This allows the constructed system dependency graph to better represent the relationships of the target distributed system. Based on a reliable system dependency graph, the identified fault dependency chains can also be made more reliable.

[0014] In an optional implementation, the method further includes: monitoring real-time events of the target distributed system; and updating the system dependency graph based on the real-time events.

[0015] In the above implementation, as the target distributed system is used, some changes in dependencies may occur. Therefore, the system dependency graph can be updated based on real-time data, so that the system dependency graph can better represent the real-time dependencies of the target distributed system.

[0016] In an optional implementation, constructing a system dependency graph based on the system association dataset includes: constructing system hint data based on the system association dataset; inputting the system association dataset and the system hint data into a large language model for recognition to determine the three-dimensional data relationships, wherein the three-dimensional data relationships include a subject node, a relationship type, and an object node; the subject node is one of a server, a container, or a service instance in the target distributed system, and the object node is one of a server, a container, or a service instance in the target distributed system; and constructing a system dependency graph based on the three-dimensional data relationships.

[0017] In the above implementation method, a large language model can be combined to learn and identify the relationships between nodes (servers, containers and service instances) in the target distributed system, which can make the learned relationships between nodes more accurate.

[0018] In a second aspect, the present invention provides an electronic device, comprising: a processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the machine-readable instructions are executed by the processor to perform the steps of the method described in any of the foregoing embodiments.

[0019] Thirdly, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the method described in any of the foregoing embodiments.

[0020] Fourthly, the present invention provides a computer program product, the computer program product comprising a computer program, which, when executed by a processor, implements the method described in any one of the foregoing embodiments. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 A schematic diagram illustrating the interaction of the distributed system provided in the embodiments of this application; Figure 2 A block diagram illustrating an electronic device provided in an embodiment of this application; Figure 3 A flowchart of a distributed system anomaly detection method provided in an embodiment of this application; Figure 4 This is a partial flowchart of a distributed system anomaly detection method provided in an embodiment of this application. Detailed Implementation

[0023] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0024] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0025] A distributed system is a system consisting of a group of components (software or hardware) located on different network computer nodes, communicating and coordinating through message passing to jointly complete a specific task.

[0026] Existing distributed system monitoring and fault diagnosis solutions mainly rely on the following types of technologies: (1) Rule-based detection of distributed system faults: such as various APM (application performance monitoring) tools, which trigger alarms by setting preset thresholds (such as CPU utilization > 90%). These systems require maintenance personnel to pre-set a large number of rules based on experience, which is difficult to cover complex and hidden fault scenarios and cannot understand the causal relationship between faults. (2) Traditional machine learning models: using isolated forests, clustering algorithms, etc. to detect anomalies in single indicators or single services. These methods are usually "flat", lacking the ability to model the dependency relationship between services and the cross-layer propagation path of faults, resulting in alarm storms and difficulty in root cause localization. The data sources of the first method mentioned above are usually isolated and lack internal correlation, resulting in relatively isolated fault judgment results. Lacking a global perspective, it is impossible to understand the causal relationship between indicators and to associate various faults. Furthermore, when a low-level fault (such as a slow database) occurs, it will affect all upstream services that depend on it (such as application services and front-end APIs) like dominoes, causing each affected service to trigger its own alarm. In this scenario, operations and maintenance personnel will receive a large number of alerts within a short period. Furthermore, operations and maintenance experts need to rely on their personal experience to manually sift through, correlate, and analyze these massive amounts of alerts in an attempt to pinpoint the initial point of failure. This process is time-consuming, labor-intensive, and highly prone to misjudgment.

[0027] Based on the above research, the embodiments of this application can provide a distributed system anomaly detection method, electronic device, and computer-readable storage medium, which can improve the accuracy of fault detection in distributed systems.

[0028] like Figure 1 As shown, Figure 1 The schematic diagram of the distributed system provided in the embodiments of this application is shown. The distributed system 100 may include multiple electronic devices 110. The electronic devices 110 may be network servers, database servers, etc.

[0029] Optionally, the distributed system may include a central server for allocating tasks that the distributed system needs to process. The central server may also interact with front-end devices to receive requests from them. The central server can then be used to generate tasks based on these requests.

[0030] like Figure 2 The diagram shown is a block illustration of an electronic device. The electronic device 110 may include a memory 111 and a processor 113. Those skilled in the art will understand that... Figure 2 The structure shown is for illustrative purposes only and does not limit the structure of the electronic device 110. For example, the electronic device 110 may also include components that are larger than... Figure 2 The more or fewer components shown, or having the same Figure 2 The different configurations shown.

[0031] The memory 111 and processor 113 described above are electrically connected to each other directly or indirectly to enable data transmission or interaction. For example, these components can be electrically connected to each other via one or more communication buses or signal lines. The processor 113 described above is used to execute executable modules stored in the memory.

[0032] The memory 111 can be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), etc. The memory 111 stores programs, and the processor 113 executes these programs upon receiving execution instructions. The methods executed by the electronic device 110, as defined in any embodiment of this application, can be applied to or implemented by the processor 113.

[0033] The aforementioned processor 113 may be an integrated circuit chip with signal processing capabilities. The processor 113 may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a digital signal processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor.

[0034] The electronic device 110 in this embodiment can be used to execute various steps in the various methods provided in the embodiments of this application. The implementation process of the distributed system anomaly detection method is described in detail below through several embodiments.

[0035] Please see Figure 3 This is a flowchart of a distributed system anomaly detection method provided in an embodiment of this application. The distributed system anomaly detection method provided in this application can be applied to electronic devices, through which the electronic devices execute the steps in the distributed system anomaly detection method. The following will describe... Figure 3 The specific process shown will be explained in detail.

[0036] Step 210: Obtain monitoring data for the target distributed system.

[0037] Alternatively, a monitoring agent can be used to monitor the target distributed system and obtain monitoring data. For example, the monitoring agent could be a Prometheus, ELK, or similar agent.

[0038] Monitoring data can include data monitored for each node (server, container, service instance). A monitoring agent can collect various monitoring data from each node of the target distributed system.

[0039] For example, monitoring data may include application-layer metrics such as Queries Per Second (QPS), request latency, task execution error rate, and logs (e.g., slow queries, error stacks). Monitoring data may include system-layer metrics such as CPU utilization, memory utilization, and disk I / O. Monitoring data may include network-layer metrics such as network traffic (NIC packet loss rate, TCP retransmissions) and kernel events. Monitoring data may include infrastructure-layer metrics such as middleware (Redis, MySQL) status indicators. Monitoring data may also include text data, such as system configuration documents and deployment files (YAML).

[0040] Step 220: Perform initial screening on the monitoring data to determine the initial set of abnormal nodes.

[0041] Optionally, if an anomaly is identified, some alarm information can be output based on the anomaly.

[0042] Optionally, the initial screening of monitoring data can be achieved by numerical comparison or by model recognition.

[0043] Optionally, the identification in step 220 can be a lightweight, interpretable hybrid rule and model approach. Step 220 can quickly narrow down the range of potentially anomalous nodes, providing a high-quality initial set of anomalous nodes for subsequent analysis of refined reinforcement learning model implementations.

[0044] In this embodiment, step 220 may include steps 221 to 223.

[0045] Step 221: Determine the target identification method based on the data type in the monitoring data.

[0046] The data types include numerical data and log data, and the target identification methods include threshold comparison for numerical data and abnormal log matching for log data.

[0047] Numerical monitoring data can include CPU utilization, memory utilization, request latency, etc.

[0048] Optionally, for each numerical monitoring data category, a corresponding numerical threshold can be set for each numerical monitoring data category (e.g., the numerical threshold for CPU utilization is 80%).

[0049] Alternatively, for each type of numerical monitoring data, a dynamic baseline can be established using a time series decomposition algorithm (such as STL or Twitter's Robust PCA) instead of a fixed numerical threshold. This time series decomposition algorithm decomposes the series into trend, periodicity, and residuals. Anomalies are defined as points where the residual portion exceeds its own historical distribution (e.g., the 3-sigma criterion). For example, the baseline value of the numerical monitoring data may differ across different time periods. For instance, the baseline may be relatively higher during peak usage periods of the target distributed system and relatively lower during off-peak periods.

[0050] Optionally, for each type of numerical monitoring data, fixed numerical thresholds can be used instead of fixed ones. Different numerical thresholds can be configured for different time periods. For example, during peak usage periods of the target distributed system, the numerical threshold can be set to a first threshold (e.g., a CPU utilization threshold of 80%). Conversely, during off-peak periods, the numerical threshold can be set to a second threshold (e.g., a CPU utilization threshold of 70%). The first threshold is a value greater than the second threshold. By setting different values, the system can better adapt to different usage needs at different times, preventing frequent alarms during peak periods and reducing the target distributed system's utilization during off-peak periods, allowing it to rest and extending its lifespan.

[0051] For monitoring data in the form of logs, log templates can be used to process the logs. For example, the Drain algorithm can be used for log template processing. Then, the frequency of each template within a time window is counted. If an error template (such as "Connection refused to...") is encountered... <host>If the frequency of ") spikes within a short period of time, it is marked as abnormal.

[0052] Alternatively, log identification can also be performed using a classification model to determine whether the logs contain anomalies. For example, the classification model could be a binary classification model, which categorizes logs into normal logs and abnormal logs.

[0053] Step 222: Identify abnormal data in the monitoring data based on the target identification method.

[0054] In one example, the CPU utilization of a certain service is usually 10% at 2:00 AM, but suddenly rises to 50% at 2:00 AM on a certain day. This is a significant anomaly compared to its historical baseline (10%) for the same period, and it can be determined that there is an anomaly in the CPU utilization of the service.

[0055] Step 223: Based on the abnormal data, determine the initial set of abnormal nodes.

[0056] By identifying different types of monitoring data, anomaly scores can be determined for each monitoring node (service, database, etc.) and its monitoring data.

[0057] Anomaly scores are compared to anomaly thresholds to determine whether a node is classified as an anomaly. For example, if a node's anomaly score exceeds the anomaly threshold, it can be identified as an anomaly node.

[0058] Taking the example of an anomaly score for each node being out of 100, 100 points can be allocated to various monitoring data points of the node. For instance, the score corresponding to the i-th type of monitoring data is Ni. Given a total of m possible values, N1 + N2 + ... + Nm = 100. The anomaly threshold can be set to a value greater than 0 and less than 100, such as 70, 80, 75, or 85. If the i-th type of monitoring data is determined to be an anomaly, the node receives a score of Ni. If the i-th type of monitoring data is determined to be non-anomaly, the node receives a score of 0, and so on, obtaining a score for each type of monitoring data. The total score obtained by the node is then considered its anomaly score.

[0059] The above is merely an example. In reality, the anomaly score for each node could be out of ten, five, or other scales. The scores for various monitoring data can be the same or different. Understandably, the scores for various monitoring data, as well as the total score for all monitoring data, can be set based on actual needs.

[0060] Step 230: Construct state data based on the initial set of abnormal nodes.

[0061] For example, the initial set of anomalous nodes can be converted into the format required by the reinforcement learning model and used as state data.

[0062] Step 240: Input the state data into the reinforcement learning model so that the reinforcement learning model can search the pre-built system dependency graph to determine the fault propagation chain.

[0063] Among them, the system dependency graph is the dependency relationship in the target distributed system built on the target distributed system, and the fault propagation chain includes one or more fault nodes and the propagation relationship of multiple fault nodes. The fault node is one of the servers, containers, and service instances in the target distributed system.

[0064] Alternatively, the reinforcement learning model can use a Deep Q-Network (DQN). A Deep Q-Network is a combination of a deep neural network and the Q-Learning algorithm.

[0065] The state of a reinforcement learning model can represent the initial set of anomalous nodes among all nodes at the current time. The action of a reinforcement learning model can be the selection of the next node to be traversed. The reward of a reinforcement learning model can be dynamically calculated based on whether the path explored by the agent can well explain the global anomalous phenomena (e.g., a higher reward for a path connecting a large number of anomalous nodes).

[0066] For example, the reward for a reinforcement learning model includes one or more of the following: the proportion of anomalous nodes among the nodes connected in the output propagation chain exceeds a threshold, the number of anomalous nodes connected in the output propagation chain exceeds a first threshold, and the proportion of anomalous nodes connected in the output propagation chain to the total number of anomalous nodes exceeds a second threshold.

[0067] Step 240 above may include: inputting state data into a reinforcement learning model so that the reinforcement learning model searches a pre-built system dependency graph to determine the node access path that maximizes the cumulative reward, and the node access path that maximizes the cumulative reward is used as a fault propagation chain.

[0068] In this embodiment, the reinforcement learning model searches for the node access path that maximizes the cumulative reward within the modeled Markov Decision Process (MDP).

[0069] The specific implementation is illustrated using a deep Q-network as an example of a reinforcement learning model: The reward function of a deep learning model can be represented as R(s, a, s'). This reward function can consider the following factors: causal consistency reward. Based on the causal consistency reward, if the action performed by the agent of the deep learning model is to move from node A to node B in the target distributed system, and there is a representation of node A -> node B in the system dependency graph (node ​​A -> node B indicates that A may affect B), then a positive reward is given. This positive reward can encourage the agent of the reinforcement learning model to explore along the directions where dependencies exist in the system dependency graph.

[0070] The reward function of a deep learning model can also consider factors such as anomaly score increase rewards. For example, if the anomaly score of a newly visited node s' is higher than that of the previous node s, a positive reward is given. This positive reward can guide the agent of the reinforcement learning model to trace back to the source of the more serious anomaly.

[0071] Deep learning models can also include a penalty function, which can take into account factors such as path length penalty. Based on the path length penalty, the reinforcement learning agent is given a small negative reward for each step it takes. This negative reward encourages the agent to find the root cause using the shortest possible path, preventing ineffective wandering.

[0072] Deep learning models can also include a reward function, which can consider the following factors: Goal Reward. Based on the goal reward, when the agent of the reinforcement learning model visits a node that has no parent node or whose parent nodes have been checked and are normal, a very large positive reward is given, suggesting that the starting point (root cause) of the path may have been found.

[0073] In one example, the formula for the reward function could be: ; Where CC represents the causal consistency reward; ASI represents the abnormal score growth reward; PLP represents the path length penalty; GR represents the target reward; λ1, λ2, λ3 and λ4 represent weights, which are hyperparameters; and R represents the reward value.

[0074] Each parameter (λ1, λ2, λ3, and λ4) is a weight hyperparameter used to balance the relative importance of different behavioral objectives. Their values ​​are typically determined experimentally (e.g., through grid search) before model training.

[0075] In this embodiment, λ1 can be a positive weight (typically λ1>0), which determines the importance of the reinforcement learning agent exploring along known dependency directions. This rewards the agent's "following the map" behavior. In the system dependency graph, the direction of the edge represents the possible causal influence direction (for example, if node A calls node B, then a failure of node A may affect node B, but the reverse is not true). When the reinforcement learning agent moves from one node to another, if the direction of action is consistent with the direction of the edge in the system dependency graph (i.e., from the "cause" node to the "effect" node), it receives a positive reward.

[0076] When λ1 is large, it strongly encourages the agent of the reinforcement learning model to explore along the most likely path of fault propagation (i.e. the direction of the dependency edge) and avoid reverse or random jumps. In this embodiment, λ2 can also be a positive weight (i.e., λ2 > 0), and λ1 determines the importance of the agent in the reinforcement learning model pursuing anomaly strength cues. In fault diagnosis of distributed systems, the root cause is usually the most severe and earliest source of the anomaly. The reward corresponding to λ2 can be based on the difference in anomaly scores between adjacent nodes. If the agent moves from the current node (lower anomaly score) to a new node (higher anomaly score), it will receive a reward proportional to the increase in score.

[0077] When λ2 is large, it will guide the agent of the reinforcement learning model to explore nodes with stronger anomalies. This helps the agent of the reinforcement learning model to quickly approach the source of the fault in the system dependency graph, rather than wandering between affected edge nodes.

[0078] In this embodiment, λ3 can be a negative weight (i.e., λ3<0), and λ3 represents the constraint on exploration efficiency.

[0079] Under this constraint, wherever the reinforcement learning agent goes, the cumulative reward decreases with each additional step it takes. The introduction of λ3 is to prevent the reinforcement learning agent from engaging in meaningless and lengthy explorations. λ3 encourages the agent to find the shortest and most direct path to the root cause node.

[0080] Under this constraint, the agent in the reinforcement learning model can quickly find the root cause by taking a shorter path in the system dependency graph, and the obtained fault path can also be optimal. The target reward can be set to a fixed value or a variable value. For example, if the target reward is a fixed value, λ4 can be 1. The target reward can be a maximum positive number, representing the reward for the agent in the reinforcement learning model to complete the task.

[0081] Target rewards can provide a clear and strong termination signal for the agent in a reinforcement learning model throughout the search process.

[0082] By configuring various rewards and penalties as described above, reinforcement learning models can find fault propagation chains more quickly and accurately.

[0083] To make the identified fault results more intuitive and to enable relevant personnel (such as operations and maintenance personnel) to understand the fault more clearly, the distributed system anomaly detection method may also include the following steps.

[0084] Step 250: Generate a fault diagnosis report based on the fault propagation chain.

[0085] For example, the fault diagnosis report could be a more readable report. For instance, it could be a report described using natural language. The fault diagnosis report could describe the causes of each fault and the potential consequences of each fault.

[0086] In one instance, the fault diagnosis report could be: "The root cause is a cache breakdown in Redis node R1, which leads to excessive load on database DB1, causing slow queries, which in turn causes application service S1 and S2 threads to be blocked, ultimately resulting in gateway response timeout."

[0087] Step 260: Display the faulty node and faulty path in the preset visualization topology map of the target distributed system.

[0088] A visualized topology diagram can present a graph showing the distribution and relationships of devices in a target distributed system. For example, this visualized topology diagram can include schematic diagrams of all nodes in the target distributed system, as well as schematic diagrams of the relationships between the nodes. For instance, the node schematic diagrams can be represented by electronic device graphics, and the schematic diagrams of the relationships between the nodes can be represented by lines, unidirectional lines, bidirectional lines, etc.

[0089] For example, the schematic diagrams of faulty nodes can be displayed differently in the visualized topology map. This differentiation could involve highlighting or bolding the nodes. Similarly, the schematic diagrams illustrating the relationships between faulty nodes can also be displayed differently, such as highlighting or bolding the relationships.

[0090] For example, a visual topology diagram can be displayed on an electronic device with a display screen in the target distributed system, such as a display screen connected to a central server. The visual topology diagram can also be displayed on a diagnostic device connected to the target distributed system, which may include a display screen.

[0091] Steps 250 and 260 above make the presentation of the fault more intuitive, which is convenient for relevant personnel to directly trust and quickly handle the fault based on the report.

[0092] In this embodiment, the system dependency graph used for searching the fault propagation chain can be determined based on the usage scenario of the target distributed system. For example... Figure 4 As shown, the construction of the system dependency graph includes steps 310 to 320.

[0093] Step 310: Obtain the system association dataset of the target distributed system.

[0094] The system-related dataset includes multiple system performance data and the device label corresponding to each system performance data. The system performance data includes one or more of the following: application layer data of the distributed system, system layer data of the distributed system, and system setting data of the distributed system.

[0095] Step 320: Construct a system dependency graph based on the system association dataset.

[0096] In this embodiment, step 320 may include: constructing system prompt data based on the system association dataset; inputting the system association dataset and system prompt data into a large language model for recognition to determine the three-dimensional data relationships; and constructing a system dependency graph based on the three-dimensional data relationships.

[0097] The three-dimensional data relationship includes the subject node, the relationship type, and the object node; the subject node is one of the servers, containers, and service instances in the target distributed system, and the object node is one of the servers, containers, and service instances in the target distributed system.

[0098] The aforementioned large language model can be a trained large language model that can recognize language and vocabulary related to operations and maintenance.

[0099] In this embodiment, the Large Language Model (LLM) serves as the core engine, enabling the LLM to understand the language of the operations and maintenance domain and automatically extract three-dimensional data relationships from the text.

[0100] In one example, the three-element metadata relationship can be <subject node, relationship, object node>. For example, extracting <Service A1, calls, Service B1> from the log "Service A1 called Service B1" indicates that Service A1 called Service B1.

[0101] In this embodiment, a specific prompting process can be configured for the input of the large language model. This specific prompting process can be a structured prompt word template that includes domain knowledge. The prompt word template takes unstructured log text as input and guides the large language model to output in a specified JSON format. The prompt word template can serve as system prompt data.

[0102] In one example, a specific prompt might be: You are a senior operations expert. Please analyze the following system log snippet and extract all involved services, infrastructure components, and their dependencies or influence relationships. Log entry: [2023-10-27 10:05:23] ERROR [service-api-gateway] Request to user-service timed out after 3000ms; caused by: java.net.ConnectException:Connection refused to user-service-host:8080.

[0103] The large language model can identify all nodes mentioned in the logs. It can also determine the relationship types between nodes, such as calls, dependencies, hosted on, and causes.

[0104] Optionally, the output of the large language model can be in a specified format, such as JSON. Furthermore, the output of the large language model can always be a three-dimensional data relation.

[0105] In one example, the output JSON format could be: { "triples_extracted": [ ["service-api-gateway", "calls", "user-service"], ["user-service", "hosted_on", "user-service-host"], ["service-api-gateway", "experiences", "timeout"], ["connection_refused", "causes", "timeout"]]}.

[0106] The above design guides large language models to follow strict output specifications. Even without fine-tuning, it can extract the relationships between nodes from new logs and present them as three-dimensional data relationships, providing a data foundation for building system dependency graphs.

[0107] Alternatively, the three-dimensional data relationships output by the large language model can be demonstrated by sampling and evaluating the extracted precision and recall.

[0108] To enable larger language models to output more stably and professionally, labeled operation and maintenance log data can be used to fine-tune the basic large language model, which can also reduce computational costs.

[0109] Alternatively, LoRA technology can be used to fine-tune large language models.

[0110] Optionally, the operation and maintenance logs of the target distributed system, along with labels provided for each log, can be collected in advance to form a training dataset. The labels can include nodes and the relationships between them.

[0111] LoRA (Local Alternate Array) technique is used for fine-tuning. Instead of fully fine-tuning the numerous parameters of the original model, LoRA injects a small number of trainable "bypass" matrices (low-rank adapters). During training, only these newly added parameters are updated, significantly reducing training overhead and memory usage.

[0112] The fine-tuning process implemented by LoRA technology involves adapting the underlying parameters of a large language model to the vocabulary distribution and grammatical structure of the operations and maintenance domain. For example, a large language model can learn the specific meanings of terms such as "Timeout," "Connection Refused," and "CPUSpike," as well as the causal relationship patterns between them, thereby transforming them into accurate three-dimensional data relationships.

[0113] The specific prompting engineering described above provides flexibility and rapid startup capabilities, while LoRA-based fine-tuning enables deep specialization. The combination of these two ensures that the large language model can accurately understand operational language.

[0114] To enable the system dependency graph to more accurately represent the target distributed system, the system dependency graph can be dynamically updated according to requirements. Based on this, the method in this application embodiment may further include the following steps.

[0115] Step 340: Monitor real-time events of the target distributed system.

[0116] For example, a monitoring agent can be used to listen to a target distributed system. It can monitor real-time events on each node of the target distributed system. The monitoring agent can continuously listen for events from different sources.

[0117] Step 350: Update the system dependency graph based on real-time events.

[0118] For example, updated ternary metadata relationships can be determined based on real-time events, and the system dependency graph can be updated based on these ternary metadata relationships.

[0119] Optionally, step 350 can be executed according to a predefined time pattern. The predefined time pattern can be every set interval. The set interval can be a week, a day, or a specific time point.

[0120] Alternatively, when a change event in the Configuration Management Database (CMDB) is detected, such as when a new service is registered or an old service is taken offline, the system dependency graph can be updated through step 350 above.

[0121] Alternatively, when a deployment event of the continuous integration / continuous deployment (CI / CD) system is detected, such as when a new version of the application is deployed, the system dependency graph can be updated through step 350 above.

[0122] Alternatively, when a significant change is detected in the log stream, the system dependency graph can be updated using step 350 described above. A significant change in the log stream could be the first appearance of a new error pattern or call chain in the log.

[0123] For example, when it is necessary to trigger step 350, relevant logs that trigger step 350 can be collected to update the system dependency graph based on these logs. In one example, for instance, when a deployment-event indicates that the user-service has been upgraded from v1.0 to v2.0, the deployment description file (such as Kubernetes YAML) of the new version and logs related to the service in the recent past can be collected.

[0124] Optionally, step 350 above can also use a large language model to identify real-time events, determine the three-dimensional data relationships, and update the system dependency graph based on the newly identified three-dimensional data relationships.

[0125] For example, the newly identified three-dimensional data relationships are used to modify the system dependency graph through incremental update operations to achieve the update of the system dependency graph.

[0126] To improve the traceability of the system dependency graph, the system dependency graph for each version can be saved and a corresponding version number can be set.

[0127] To improve the reliability of each updated system dependency graph, a consistency check can be performed on the updated graph. This check detects whether the updated graph contains dependencies on services that are no longer operational. For example, if the check finds dependencies on offline services in the updated graph, the check fails, and a rollback or alert can be triggered.

[0128] The system dependency graph update method described above can keep the system dependency graph up-to-date and better reflect the latest state of the target distributed system.

[0129] The method in this application embodiment can be used to perform anomaly analysis on the indicators of each layer (application layer, system layer or network layer) in the target distributed system. Furthermore, it can use a reinforcement learning model to search for fault propagation chains, thereby associating faults existing in each layer. For example, it can effectively associate "slow database query", "network packet loss" with cross-node and cross-layer fault phenomena such as "front-end page timeout" experienced by the end user.

[0130] Compared to existing technologies that output massive amounts of alarms for various anomalies, this application embodiment correlates numerous anomalies and finally presents each fault node and fault path through a more readable fault diagnosis report and a visual representation, making it easier for operation and maintenance personnel to take quick action based on the fault diagnosis report and the visualized fault nodes and fault paths.

[0131] Furthermore, in this embodiment, a large language model is used to automatically construct and update a cross-layer system dependency graph, which can understand the complex dependencies between applications, systems, and networks, and reduces the need for manual intervention. The search based on this system dependency graph can reveal fault nodes and fault paths. The root cause localization problem is modeled as a reinforcement learning (RL) problem, and fault propagation paths are simulated on the dependency graph, thereby achieving automated and intelligent localization of fault points and fault paths. Combined with path search using reinforcement learning, it can penetrate multiple layers of indirect influence and accurately locate the root cause at the lowest level (such as packet loss on a specific network card), rather than remaining at surface phenomena (such as application timeouts).

[0132] Furthermore, in constructing the system dependency graph and identifying abnormal nodes, multi-source heterogeneous data (application logs, kernel events, network traffic, performance metrics) are integrated for joint analysis to break down data silos. Ultimately, this generates an interpretable and visualized fault reasoning chain, rather than simply providing a single root cause node, making the presentation of faults clearer.

[0133] The method described in this application can reduce fault location time from hours to minutes, significantly improving system availability.

[0134] Furthermore, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the distributed system anomaly detection method described in the above method embodiments.

[0135] The computer program product of the distributed system anomaly detection method provided in this application includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the steps of the distributed system anomaly detection method described in the above method embodiments. For details, please refer to the above method embodiments, which will not be repeated here.

[0136] In the several embodiments provided in this application, it should be understood that the disclosed methods can also be implemented in other ways. The method embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of methods and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0137] In addition, the method steps in the various embodiments of this application can be integrated together to form an independent part for execution, or each method step can be executed by a separate module, or two or more steps can be formed into an independent part for execution.

[0138] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks. It should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0139] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application. It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0140] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.< / host>

Claims

1. A method for anomaly detection in a distributed system, characterized in that, include: Obtain monitoring data for the target distributed system; The monitoring data is initially screened to determine the initial set of abnormal nodes; State data is constructed based on the initial set of abnormal nodes; The state data is input into a reinforcement learning model so that the reinforcement learning model searches a pre-built system dependency graph to determine the fault propagation chain. The system dependency graph is the dependency relationship in the target distributed system constructed based on the target distributed system. The fault propagation chain includes one or more fault nodes and the propagation relationship of multiple fault nodes. The fault node is one of the following in the target distributed system: server, container, or service instance.

2. The method according to claim 1, characterized in that, The actions of the reinforcement learning model include: the next node to be traversed; the rewards of the reinforcement learning model include: the proportion of abnormal nodes in the nodes connected in the output propagation chain exceeds a threshold, the number of abnormal nodes connected in the output propagation chain exceeds a first threshold, and the proportion of abnormal nodes connected in the output propagation chain to the total number of abnormal nodes exceeds a second threshold, or one or more of the following: The step of inputting the state data into a reinforcement learning model, so that the reinforcement learning model can search a pre-built system dependency graph to determine the fault propagation chain, includes: The state data is input into a reinforcement learning model so that the reinforcement learning model searches a pre-built system dependency graph to determine the node access path that maximizes the cumulative reward, and the node access path that maximizes the cumulative reward serves as a fault propagation chain.

3. The method according to claim 1, characterized in that, The initial screening of the monitoring data to determine the initial set of abnormal nodes includes: For the data types in the monitoring data, a target identification method is determined, wherein the data types include numerical data and log data, and the target identification method includes a threshold comparison method for the numerical data and an abnormal log matching method for the log data. Based on the target identification method, abnormal data in the monitoring data is identified; Based on the abnormal data, an initial set of abnormal nodes is determined.

4. The method according to claim 1, characterized in that, The method further includes: A fault diagnosis report is generated based on the fault propagation chain. The preset visualization topology of the target distributed system displays the faulty nodes and fault paths.

5. The method according to any one of claims 1-4, characterized in that, The system dependency graph is constructed in the following ways: Obtain the system-related dataset of the target distributed system, wherein the system-related dataset includes multiple system performance data and a device label corresponding to each system performance data, and the system performance data includes one or more of the following: application layer data of the distributed system, system layer data of the distributed system, and system setting data of the distributed system. A system dependency graph is constructed based on the system-related dataset.

6. The method according to claim 5, characterized in that, The method further includes: Monitor real-time events of the target distributed system; The system dependency graph is updated based on the real-time events.

7. The method according to claim 5, characterized in that, The construction of the system dependency graph based on the system association dataset includes: System prompt data is constructed based on the system-related dataset; The system-related dataset and the system prompt data are input into a large language model for recognition to determine the three-dimensional data relationship, wherein the three-dimensional data relationship includes a subject node, a relationship type, and an object node; the subject node is one of the servers, containers, and service instances in the target distributed system, and the object node is one of the servers, containers, and service instances in the target distributed system; Based on the aforementioned three-dimensional data relationships, a system dependency graph is constructed.

8. An electronic device, characterized in that, include: The processor and memory, wherein the memory stores machine-readable instructions executable by the processor, wherein when the electronic device is running, the machine-readable instructions are executed by the processor to perform the steps of the method as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the method as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method described in any one of claims 1 to 7.

Images