Fault identification method and device of distributed system and storage medium

Abstract

The application discloses a fault identification method and device of a distributed system and a storage medium. It relates to the field of distributed systems, and the method comprises: identifying abnormal transactions in the distributed system; performing clustering analysis on abnormal index vectors corresponding to the abnormal transactions according to a density clustering algorithm, determining a common fault cluster, and converting the abnormal index vectors in the common fault cluster into topology vectors through a graph embedding algorithm; performing weighted processing on the fused feature vectors through a self-attention mechanism to obtain target feature vectors; mapping the target feature vectors into candidate node probability distributions, determining a candidate node with the maximum probability in the candidate nodes as a root cause node, determining a self-healing suggestion according to the root cause type of the root cause node, and performing a self-healing recovery operation on the root cause node according to the self-healing suggestion. Through the application, the problem that the root cause node cannot be accurately identified when a fault occurs in the distributed system in the related art is solved.

Description

Technical Field

[0001] This application relates to the field of distributed systems, and more specifically, to a method, apparatus, and storage medium for fault identification in a distributed system. Background Technology

[0002] In existing distributed systems, microservice architectures, due to their fine-grained service specifications, complex call chains, and dynamically changing dependencies, make it difficult to accurately pinpoint the root cause node and its type when a failure occurs. Traditional monitoring and diagnostic methods often rely on independent metric threshold alarms or shallow link tracing, which can only identify single-point performance anomalies and cannot effectively capture hidden fault propagation patterns across services and call layers. Especially in high-concurrency, heterogeneous deployment environments, subtle but common anomaly patterns in a large number of transaction chains are masked by noise. Existing technologies lack the ability to model the semantic associations of multi-dimensional time-series anomaly indicators, making it difficult to extract diagnostically valuable fault pattern clusters from massive link fingerprints. Furthermore, because service topology relationships and call context information are fragmented, root cause nodes are often misjudged as high-load or high-latency end services, while the true root cause node is often misdiagnosed or missed due to the lack of a global context fusion analysis mechanism. This leads to delayed operational response, high mean time to repair, and severely restricts the stability and observability of distributed systems.

[0003] There is currently no effective solution to the problem of accurately identifying the root cause node when a fault occurs in a distributed system in related technologies. Summary of the Invention

[0004] The main objective of this application is to provide a method, apparatus, and storage medium for fault identification in a distributed system, so as to solve the problem in the related art of being unable to accurately identify the root cause node when a fault occurs in a distributed system.

[0005] To achieve the above objectives, according to one aspect of this application, a fault identification method for a distributed system is provided. The method includes: aggregating abnormal indicators in the link fingerprint vector corresponding to each transaction in the distributed system to obtain an aggregated abnormal indicator vector for each transaction, and integrating the aggregated abnormal indicator vectors for each transaction into an abnormal indicator vector set; identifying abnormal transactions in the distributed system based on the abnormal indicators in the link fingerprint vector corresponding to each transaction; performing cluster analysis on the abnormal indicator vectors corresponding to the abnormal transactions using a density clustering algorithm to determine common fault clusters, and converting the abnormal indicator vectors in the common fault clusters into topological vectors using a graph embedding algorithm, wherein the common fault clusters include at least one abnormal indicator vector; fusing the topological vectors, the abnormal indicator vector set, and the feature vectors of historical fault cases to obtain a fused feature vector; weighting the fused feature vector using a self-attention mechanism to obtain a target feature vector; mapping the target feature vector to a candidate node probability distribution, determining the candidate node with the highest probability among the candidate nodes as the root cause node, determining a self-healing suggestion based on the root cause type of the root cause node, and performing a self-healing recovery operation on the root cause node according to the self-healing suggestion.

[0006] Optionally, before aggregating the abnormal indicators in the link fingerprint vector corresponding to each transaction in the distributed system, the method further includes: obtaining the full-link data corresponding to each transaction in the distributed system; encrypting and encoding the service name and interface path in the full-link data corresponding to each transaction using a hash algorithm to obtain the encrypted and encoded vector corresponding to each transaction; normalizing the response time in the full-link data corresponding to each transaction using a normalization algorithm to obtain the response time normalized vector corresponding to each transaction; performing one-hot encoding on the error codes in the full-link data corresponding to each transaction to obtain the error code encoding vector corresponding to each transaction; and concatenating the encrypted and encoded vector, the response time normalized vector, and the error code encoding vector corresponding to each transaction to obtain the link fingerprint vector corresponding to each transaction.

[0007] Optionally, the aggregation of abnormal indicators in the link fingerprint vector corresponding to each transaction in the distributed system is performed to obtain an aggregated abnormal indicator vector corresponding to each transaction, and the aggregation of the abnormal indicator vectors corresponding to each transaction is integrated into an abnormal indicator vector set. This includes: aggregating the abnormal indicators in the link fingerprint vector corresponding to each transaction within a preset time window according to preset aggregation rules to obtain an aggregated abnormal indicator vector corresponding to each transaction, wherein the preset aggregation rules include at least: abnormal response time aggregation rules, error code aggregation rules, and abnormal dependent resource indicator aggregation rules; and integrating the aggregated abnormal indicator vectors corresponding to each transaction into an abnormal indicator vector set.

[0008] Optionally, transforming the abnormal indicator vectors in the common fault clusters into topological vectors using a graph embedding algorithm includes: converting each common fault cluster into a node graph, where the nodes in the node graph are the microservice instances corresponding to the abnormal indicator vectors, the edges in the node graph are the call relationships between microservice instances, and the edge weights in the node graph are the number of calls between microservice instances per unit time; generating a node sequence corresponding to each node using the random walk node sequence generation formula of the graph embedding algorithm; and transforming the node sequence corresponding to each node into a topological vector corresponding to each node using a feature mapping function.

[0009] Optionally, the target feature vector is obtained by weighting the fused feature vector through a self-attention mechanism, including: mapping the fused feature vector to a query vector based on the query projection matrix, mapping the fused feature vector to a key vector based on the key projection matrix, and mapping the fused feature vector to a value vector based on the value projection matrix; calculating the attention weight matrix based on the query vector and the key vector, and then weighting and summing the value vectors based on the attention weight matrix to obtain the target feature vector.

[0010] Optionally, after converting the node sequence corresponding to each node into the topological vector corresponding to each node through the feature mapping function, the method further includes: calculating the similarity between the topological vectors corresponding to any two nodes in the node graph; determining nodes with a similarity greater than a preset threshold as common fault nodes; and determining the fault type of the common fault nodes based on the abnormal behavior characteristics of the common fault nodes.

[0011] Optionally, after determining self-healing recommendations based on the root cause type of the root cause node and performing self-healing recovery operations on the root cause node based on the self-healing recommendations, the method further includes: generating a fault analysis report of the distributed system based on the root cause node; adding the root cause node and self-healing recommendations to the historical fault case library after the root cause node successfully recovers based on the self-healing recommendations; and generating alarm information corresponding to the root cause node and sending the alarm information to the operation and maintenance personnel after the root cause node fails to recover based on the self-healing recommendations.

[0012] To achieve the above objectives, according to another aspect of this application, a fault identification device for a distributed system is provided. The device includes: an integration unit for aggregating abnormal indicators in the link fingerprint vector corresponding to each transaction in the distributed system to obtain an aggregated abnormal indicator vector for each transaction, and integrating the aggregated abnormal indicator vectors for each transaction into an abnormal indicator vector set; an identification unit for identifying abnormal transactions in the distributed system based on the abnormal indicators in the link fingerprint vector corresponding to each transaction; a clustering unit for performing cluster analysis on the abnormal indicator vectors corresponding to abnormal transactions according to a density clustering algorithm to determine common fault clusters, and converting the abnormal indicator vectors in the common fault clusters into topology vectors through a graph embedding algorithm, wherein the common fault clusters include at least one abnormal indicator vector; a fusion unit for fusing the topology vectors, the abnormal indicator vector set, and the feature vectors of historical fault cases to obtain a fused feature vector; a processing unit for weighting the fused feature vectors through a self-attention mechanism to obtain a target feature vector; and a first determination unit for mapping the target feature vector to a candidate node probability distribution, determining the candidate node with the highest probability among the candidate nodes as the root cause node, determining a self-healing suggestion based on the root cause type of the root cause node, and performing a self-healing recovery operation on the root cause node according to the self-healing suggestion.

[0013] According to another aspect of this application, a computer-readable storage medium is provided, which includes a stored program, wherein, when the program is executed, it controls the device where the computer-readable storage medium is located to execute any kind of fault identification method for a distributed system.

[0014] According to another aspect of this application, an electronic device is provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include a fault identification method for performing any kind of distributed system.

[0015] According to another aspect of this application, a computer program product is provided, including computer instructions that, when executed by a processor, implement the steps of a fault identification method for a distributed system according to any one of the above.

[0016] In this embodiment, by aggregating the abnormal indicators in the link fingerprint vector corresponding to each transaction in the distributed system, an aggregated abnormal indicator vector corresponding to each transaction is obtained, and the aggregated abnormal indicator vectors corresponding to each transaction are integrated into an abnormal indicator vector set. Abnormal transactions in the distributed system are identified based on the abnormal indicators in the link fingerprint vector corresponding to each transaction. Cluster analysis is performed on the abnormal indicator vectors corresponding to abnormal transactions using a density clustering algorithm to determine common fault clusters. The abnormal indicator vectors in the common fault clusters are then transformed into topological vectors using a graph embedding algorithm, wherein each common fault cluster includes at least one abnormal indicator vector. The topological vectors, the abnormal indicator vector set, and the feature vectors of historical fault cases are fused to obtain a fused feature vector. A self-attention mechanism is used to weight the fused feature vector to obtain a target feature vector. The target feature vector is mapped to a candidate node probability distribution, and the candidate node with the highest probability is determined as the root cause node. A self-healing suggestion is determined based on the root cause type of the root cause node, and a self-healing recovery operation is performed on the root cause node based on the self-healing suggestion. This solves the technical problem of not being able to accurately identify the root cause node when a fault occurs in a distributed system.

[0017] In this application, by aggregating the abnormal indicators in the link fingerprint vector corresponding to each transaction in the distributed system, an aggregated abnormal indicator vector corresponding to each transaction is obtained. This vector is then integrated to form an abnormal indicator vector set. Density clustering algorithm is used to perform cluster analysis on the abnormal indicator vectors to identify common fault clusters with similar abnormal patterns. A graph embedding algorithm is then used to transform the abnormal indicator vectors in the common fault clusters into topology vectors that reflect the service call structure and dependencies. The feature vectors of the topology vector, the abnormal indicator vector set, and historical fault cases are fused to obtain a fused feature vector. An attention mechanism is introduced to weight the fused feature vector, which can improve the discriminative power of the feature expression and obtain the target feature vector. The target feature vector is mapped to the probability distribution of candidate nodes. The node with the highest probability is the most likely root cause node, and the root cause type is inferred. Based on the root cause type, a self-healing suggestion is determined, and a recovery operation is automatically triggered. This achieves the technical effect of accurately identifying the root cause node of a fault in the distributed system. Attached Figure Description

[0018] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0019] Figure 1 A hardware structure block diagram of a computer terminal for implementing a fault identification method for a distributed system is shown.

[0020] Figure 2This is a flowchart of a fault identification method for a distributed system provided according to an embodiment of this application;

[0021] Figure 3 This is a schematic diagram of a fault identification method for a distributed system provided according to an embodiment of this application;

[0022] Figure 4 This is a schematic diagram of fault clustering provided according to an embodiment of this application;

[0023] Figure 5 This is a schematic diagram of self-healing recovery provided according to an embodiment of this application;

[0024] Figure 6 This is a schematic diagram of distributed system fault identification and fault recovery provided according to an embodiment of this application;

[0025] Figure 7 This is a schematic diagram of a fault identification device for a distributed system provided according to an embodiment of this application;

[0026] Figure 8 This is a structural block diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0027] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0028] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0029] It should be noted that all relevant information (including at least one of the following: transaction information, node information, system information, etc.) and data (including at least one of the following: collected data, analyzed data, etc.) involved in this application are information and data authorized by the user or fully authorized by all parties. For example, if there is an interface between this system and the relevant user or institution, before obtaining the relevant information, it is necessary to send an acquisition request to the aforementioned user or institution through the interface, and obtain the relevant information after receiving the consent information from the aforementioned user or institution.

[0030] Example 1

[0031] According to an embodiment of this application, a method embodiment for fault identification in a distributed system is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0032] The method embodiment provided in Embodiment 1 of this application can be executed on a mobile terminal, computer terminal, or similar computing device. Figure 1 A hardware block diagram of a computer terminal (or mobile device) for implementing a fault identification method in a distributed system is shown. Figure 1 As shown, the computer terminal 10 (or mobile device) may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) 102 (processor 102 may include at least one of the following: a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a Universal Serial Bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0033] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0034] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the fault identification method of the distributed system in this embodiment. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the aforementioned fault identification method of the distributed system. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include at least one of the following: the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0035] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0036] The display can be, for example, a touchscreen liquid crystal display (LCD), which allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0037] Under the aforementioned operating environment, this application provides the following: Figure 2 The method for fault identification in a distributed system is shown. Figure 2 This is a flowchart of a fault identification method for a distributed system according to Embodiment 1 of this application.

[0038] Step S201: Aggregate the abnormal indicators in the link fingerprint vector corresponding to each transaction in the distributed system to obtain the aggregated abnormal indicator vector corresponding to each transaction, and integrate the aggregated abnormal indicator vector corresponding to each transaction into an abnormal indicator vector set.

[0039] Optionally, a microservice monitoring system can collect operational metrics of each microservice node in real time. To ensure the real-time performance and accuracy of metric collection, the collection frequency should be no less than once per second. The collected operational metrics include key indicators such as response time P95 / P99 quantiles, error rate, number of slow database queries, cache hit rate, third-party API call latency, and connection status, providing data support for anomaly transaction identification. Anomaly metrics from the link fingerprint vector corresponding to each transaction can be aggregated according to a preset time window. The duration of the preset time window can be set according to the microservice transaction frequency, ranging from 5 to 30 seconds. Aggregation rules for anomaly metrics can include rules for anomaly response time metrics, error code aggregation rules, and rules for anomaly dependent resource metrics.

[0040] Step S202: Identify abnormal transactions in the distributed system based on the abnormal indicators in the link fingerprint vector corresponding to each transaction.

[0041] Optionally, abnormal transactions can be identified by analyzing whether each abnormal indicator in the link fingerprint vector corresponding to each transaction exceeds the corresponding threshold, and whether there is an error code other than 200 in the link fingerprint vector corresponding to each transaction. If an abnormal indicator in the link fingerprint vector corresponding to a transaction exceeds the corresponding threshold, or if there is an error code other than 200 in the link fingerprint vector corresponding to a transaction, then the transaction corresponding to that link fingerprint vector is marked as an abnormal transaction.

[0042] Step S203: Perform cluster analysis on the abnormal indicator vectors corresponding to abnormal transactions according to the density clustering algorithm to determine common fault clusters, and transform the abnormal indicator vectors in the common fault clusters into topological vectors through the graph embedding algorithm. The common fault clusters include at least one abnormal indicator vector.

[0043] Optionally, a spatial clustering algorithm (Density-Based Spatial Clustering of Applications with Noise, DBSCAN) can be used to cluster the anomalous indicator vectors corresponding to anomalous transactions to identify common fault clusters. Common fault clusters are formed by clustering multiple anomalous transaction vectors, and each cluster has a similar anomalous pattern, representing a type of fault. For example, a payment service might fail due to a timeout in the inventory service cascading. A graph embedding algorithm can be used to transform the anomalous indicator vectors into low-dimensional topology vectors, where the dimension of the topology vector is positively correlated with the number of microservice nodes.

[0044] Step S204: The feature vectors of the topology vector, the abnormal indicator vector set, and the historical fault cases are fused to obtain the fused feature vector.

[0045] Optionally, feature vectors for historical fault cases can be generated by semantically parsing, extracting key information, and structurally encoding unstructured or semi-structured text such as historical work orders, fault reports, and operation and maintenance logs. These feature vectors characterize the root cause, scope of impact, conditions of occurrence, and handling methods of the fault. The feature vectors of historical fault cases are then fused with topology vectors and anomaly indicator vector sets. The fused feature vectors incorporate the handling methods and context of the historical faults, allowing the root cause reasoning process to operate independently of isolated indicators or human experience, thus improving the accuracy, interpretability, and reliability of root cause localization.

[0046] Step S205: The fused feature vector is weighted using a self-attention mechanism to obtain the target feature vector.

[0047] Optionally, the self-attention mechanism can automatically learn the semantic relationships between nodes. It maps the fused feature vector to a query vector based on the query projection matrix, to a key vector based on the key projection matrix, and to a value vector based on the value projection matrix. An attention weight matrix is ​​then calculated based on the query and key vectors, and the value vectors are weighted and summed according to this matrix to obtain the target feature vector. The resulting target feature vector is an enhanced representation after global inference, achieving context awareness, dynamic weighting, and semantic enhancement of the fused feature vector. Simultaneously, it suppresses irrelevant or affected interfering nodes, contributing to improved accuracy and interpretability of root cause localization.

[0048] Step S206: Map the target feature vector to the probability distribution of candidate nodes, determine the candidate node with the highest probability among the candidate nodes as the root cause node, determine the self-healing suggestion according to the root cause type of the root cause node, and perform the self-healing recovery operation on the root cause node according to the self-healing suggestion.

[0049] Optionally, the target feature vector can be input into a fully connected layer and a Softmax classifier, which outputs the probability of each service node being the root cause node (i.e., the candidate node probability distribution mentioned above). The candidate node with the highest probability can be identified as the root cause node, i.e., the true source of the fault.

[0050] Optionally, by matching with historical failure cases, the root cause type corresponding to the root cause node can be determined. Based on the root cause type, corresponding self-healing suggestions can be determined. For example, if the root cause type is a database anomaly, the self-healing suggestion is to automatically execute index optimization scripts, rebuild the database indexes corresponding to slow queries, and adjust the minimum and maximum number of connections and timeout parameters of the database connection pool. If the root cause type is a cache anomaly, the self-healing suggestion is to automatically clean up expired or invalid data in the cache, update the cache expiration policy to proactive expiration, and adjust the cache capacity threshold. If the root cause type is a third-party interface anomaly, the self-healing suggestion is to automatically switch to a preset backup node for the third-party interface, trigger interface health checks, and record interface anomaly logs. If the root cause type is an application-specific anomaly, the self-healing suggestion is to automatically restart the abnormal service instance; if the restart fails, roll back to the latest stable version, and collect application logs for subsequent analysis. Performing self-healing recovery operations on the root cause node according to the self-healing suggestions helps to automate the fault repair of distributed systems, thereby improving the stability and security of distributed systems.

[0051] Optionally, Figure 3 The diagram illustrates a fault identification method for a distributed system. The input layer receives data including link fingerprint topology features, anomaly indicator vectors, and historical fault case features. The feature fusion layer uses vector concatenation and normalization to fuse these features, resulting in a fused feature vector. The attention mechanism layer uses multi-head attention to calculate the attention weights of each feature in the fused feature vector, highlighting key anomaly features and suppressing interference from irrelevant features, outputting the target feature vector. The output layer uses a fully connected layer and a softmax function to map the target feature vector output by the attention mechanism layer to a root cause node probability distribution, selecting the node with the highest probability as the root cause node and outputting the root cause type, thus completing the root cause inference process. This method overcomes the limitations of traditional coarse-grained localization based on statistical thresholds, improving the accuracy of root cause localization.

[0052] The fault identification method for distributed systems provided in this application aggregates abnormal indicators from the link fingerprint vector corresponding to each transaction in the distributed system to obtain an aggregated abnormal indicator vector for each transaction. This aggregated vector is then integrated to form an abnormal indicator vector set. A density clustering algorithm is used to cluster these vectors, identifying common fault clusters with similar abnormal patterns. A graph embedding algorithm transforms the abnormal indicator vectors within these common fault clusters into topology vectors that reflect the service call structure and dependencies. The topology vectors, the abnormal indicator vector set, and the feature vectors from historical fault cases are fused to obtain a fused feature vector. An attention mechanism is introduced to weight the fused feature vector, enhancing its discriminative power and yielding a target feature vector. This target feature vector is then mapped to a probability distribution of candidate nodes. The node with the highest probability is identified as the most likely root cause node, and the root cause type is inferred. Based on the root cause type, a self-healing suggestion is determined, and recovery operations are automatically triggered. This method solves the technical problem of accurately identifying the root cause node when a fault occurs in a distributed system, achieving the technical effect of accurately identifying the root cause node of a fault in a distributed system.

[0053] To provide a unique link identifier for aggregating abnormal indicators and locating root causes, optionally, in the fault identification method for a distributed system provided in this application embodiment, before aggregating the abnormal indicators in the link fingerprint vector corresponding to each transaction in the distributed system, the method further includes:

[0054] The first step is to obtain the full-chain data corresponding to each transaction in the distributed system.

[0055] Optionally, the full-link data corresponding to each transaction in the distributed system includes a global serial number, service name, interface path, response time, error code, dependent resource identifier (e.g., database instance ID, cache node ID, third-party interface address), and upstream and downstream call relationships, which can provide data support for generating the link fingerprint vector corresponding to each transaction.

[0056] The second step involves using a hash algorithm to encrypt and encode the service name and interface path in the end-to-end data corresponding to each transaction, resulting in an encrypted vector for each transaction.

[0057] Optionally, the service name and interface path in the end-to-end data can be encrypted using the SHA-256 hash algorithm, and the first 128 bits can be used as the encrypted vector corresponding to each transaction.

[0058] The third step is to normalize the response time in the end-to-end data corresponding to each transaction using a normalization algorithm, thereby obtaining a normalized response time vector for each transaction.

[0059] Optionally, the response time can be mapped to the [0,1] interval using a min-max normalization algorithm, which can eliminate the impact of differences in the magnitude of response time between different services and obtain a normalized response time vector for each transaction.

[0060] The fourth step is to perform one-hot encoding on the error codes in the end-to-end data corresponding to each transaction to obtain the error code encoding vector for each transaction.

[0061] Optionally, an error code is a standardized, structured numerical or string identifier actively returned by service components in a distributed system during request processing to identify the execution status, exception type, or reason for failure. It can be used to quickly and accurately determine whether service behavior is normal without relying on log text. One-hot encoding of error codes can convert discrete error codes into binary vector form, realizing the conversion of text information into numerical information, and obtaining the error code encoding vector corresponding to each transaction.

[0062] The fifth step involves concatenating the encrypted vector, the response time normalization vector, and the error code encoding vector corresponding to each transaction to obtain the link fingerprint vector for each transaction.

[0063] Optionally, by concatenating the encrypted vector corresponding to each transaction, the normalized response time vector corresponding to each transaction, and the error code vector corresponding to each transaction, a unique link fingerprint (i.e., the link fingerprint vector corresponding to each transaction mentioned above) can be generated, corresponding to each transaction one-to-one. The link fingerprint vector corresponding to each transaction can be associated with a global serial number and persistently stored in a distributed database, providing a unique link identifier for subsequent anomaly indicator aggregation and root cause localization, thus enabling traceability and investigation of the link for each transaction.

[0064] For example, by concatenating the encrypted vector corresponding to each transaction, the normalized response time vector corresponding to each transaction, and the error code vector corresponding to each transaction, the link fingerprint vector corresponding to each transaction is obtained as follows:

[0065]

[0066] in, This is the link fingerprint vector. The encrypted vector. For the response time normalized vector, Error code encoding vector, This is a vector concatenation operation, where The calculation formula is:

[0067]

[0068] in, This is the current response time. To preset the minimum response time threshold, This is the preset maximum response time threshold.

[0069] In summary, by concatenating the encrypted vector, response time normalization vector, and error code encoding vector corresponding to each transaction, the link fingerprint vector corresponding to each transaction is obtained. This provides a unique link identifier for subsequent aggregation of abnormal indicators and root cause localization, and makes the link of each transaction traceable and verifiable.

[0070] To provide a data foundation for density clustering, common fault identification, and root cause localization, optionally, in the fault identification method for distributed systems provided in this application embodiment, the abnormal indicators in the link fingerprint vector corresponding to each transaction in the distributed system are aggregated to obtain an aggregated abnormal indicator vector corresponding to each transaction, and the aggregated abnormal indicator vectors corresponding to each transaction are integrated into an abnormal indicator vector set, including:

[0071] The first step is to aggregate the abnormal indicators in the link fingerprint vector corresponding to each transaction within a preset time window according to the preset aggregation rules, so as to obtain the aggregated abnormal indicator vector corresponding to each transaction. The preset aggregation rules include at least: abnormal response time aggregation rules, error code aggregation rules, and abnormal dependent resource indicator aggregation rules.

[0072] Optionally, abnormal indicators from the link fingerprint vector corresponding to each transaction can be aggregated according to a preset time window. The duration of the preset time window can be set according to the microservice transaction frequency, ranging from 5 to 30 seconds. The aggregation rules for abnormal indicators can include rules for abnormal response time indicators, error code aggregation rules, and rules for abnormal dependent resource indicators. The abnormal response time indicator aggregation rule calculates the P95 and P99 quantiles of the response time of each node in the link fingerprint vector corresponding to each transaction. The result can be determined using the following formula; if the criteria are met, the response time indicator is marked as abnormal:

[0073]

[0074]

[0075] in, For the P95 quantile, The 99th quantile, , They are respectively Corresponding preset threshold, The corresponding preset threshold;

[0076] Error code aggregation rules are implemented by statistically analyzing the number and type of error codes in the link fingerprint vector corresponding to each transaction. The anomaly determination rules are as follows: if there are error codes other than 200 or 500, they are directly marked as errors; if the proportion of 500 error codes exceeds a preset threshold, the following formula can be used for determination, and if it is satisfied, it is marked as an error code anomaly:

[0077]

[0078] in, The preset threshold corresponding to the percentage of 500 error codes;

[0079] The aggregation rules for abnormal dependency resource metrics use real-time monitoring of database connection count, cache hit count, and third-party API call success rate. The following formula can be used to determine anomalies; if any one of these conditions is met, the dependency resource metric is marked as abnormal:

[0080]

[0081]

[0082]

[0083] in, This is a preset threshold for the number of database connections. This is a preset threshold corresponding to the percentage of cache hits. A preset threshold for the success rate of third-party interface calls.

[0084] The second step is to integrate the aggregated abnormal indicator vectors corresponding to each transaction into an abnormal indicator vector set.

[0085] Optionally, the aggregated anomaly indicator vectors corresponding to each transaction can be stacked row by row to form a two-dimensional matrix (i.e., the anomaly indicator vector set mentioned above), which can provide a data foundation for subsequent density clustering, identification of common faults, and root cause localization, and help identify root cause nodes and root cause types in distributed systems.

[0086] In summary, through the above steps, the abnormal indicators in the link fingerprint vector corresponding to each transaction are aggregated within a preset time window according to the preset aggregation rules, resulting in the aggregated abnormal indicator vector corresponding to each transaction. This is then integrated into an abnormal indicator vector set, which provides a data foundation for subsequent density clustering, identification of common faults, and root cause localization. This helps to identify root cause nodes and root cause types in distributed systems.

[0087] To accurately identify root cause nodes in a distributed system, optionally, the fault identification method for a distributed system provided in this application embodiment transforms the abnormal indicator vectors in common fault clusters into topological vectors using a graph embedding algorithm, including:

[0088] The first step is to transform each common fault cluster into a node graph, where the nodes in the node graph are the microservice instances corresponding to the anomaly indicator vectors, the edges in the node graph are the call relationships between microservice instances, and the edge weights in the node graph are the number of calls between microservice instances per unit time.

[0089] Optionally, each common fault cluster can be transformed into a weighted directed graph (i.e., the node graph mentioned above), where the set of nodes represents microservice instances, the set of edges represents the call relationships between microservice instances, and the weight is the number of times that edge is called within a unit of time (e.g., 10 seconds), which is the number of calls between microservice instances.

[0090] The second step is to generate the node sequence corresponding to each node using the random walk node sequence generation formula of the graph embedding algorithm.

[0091] Optionally, the graph embedding algorithm can map nodes in a node graph to low-dimensional dense vectors. The formula for generating the random walk node sequence of the graph embedding algorithm is as follows:

[0092]

[0093] Where Seq(p) represents the random walk sequence of node p (i.e. the node sequence mentioned above), G is the node graph, and k is the number of random walk steps (which can be set to 10).

[0094] The third step is to use a feature mapping function to transform the node sequence corresponding to each node into a topological vector corresponding to each node.

[0095] Alternatively, the transformation formula for topological vectors can be:

[0096]

[0097] in, For nodes topological vectors, This is the topology vector dimension (which can be adapted to the number of microservice nodes, for example, set to 32 dimensions). This is a feature mapping function used to transform a random walk sequence into a low-dimensional dense vector.

[0098] In summary, by transforming each common fault cluster into a node graph and then converting each node into a topology vector using a graph embedding algorithm, the dependencies between microservice instances in a distributed system can be reflected, providing a data foundation for identifying root cause nodes in the distributed system.

[0099] To highlight key anomaly features and suppress interference from irrelevant features, optionally, the fault identification method for distributed systems provided in this application embodiment uses a self-attention mechanism to weight the fused feature vector to obtain the target feature vector, which includes:

[0100] The first step is to map the fused feature vector to a query vector based on the query projection matrix, the fused feature vector to a key vector based on the key projection matrix, and the fused feature vector to a value vector based on the value projection matrix.

[0101] Optionally, based on the query projection matrix Map the fused feature vector X to the query vector. According to the key projection matrix The fused feature vector X is mapped to a key vector. According to the value projection matrix Map the fused feature vector X to a value vector. :

[0102]

[0103] The second step is to calculate the attention weight matrix based on the query vector and the key vector, and then perform a weighted summation of the value vectors based on the attention weight matrix to obtain the target feature vector.

[0104] Optionally, an attention weight matrix is ​​calculated, and the value vectors are weighted and summed based on the attention weight matrix to obtain a target feature vector that highlights key anomalous features (e.g., indicators with the largest exceedance of the threshold, frequently occurring error codes) and suppresses interference from irrelevant features. Attention weight matrix A The target feature vector is obtained by weighting and summing the value vector V according to the attention weight matrix A. :

[0105]

[0106] In summary, by using the above steps to calculate the attention weight matrix through the self-attention mechanism and performing a weighted summation on the value vector, the resulting target feature vector can highlight the key abnormal features in the fused feature vector and suppress the interference of irrelevant features.

[0107] To improve the accuracy of identifying root cause types, optionally, in the fault identification method for distributed systems provided in this application embodiment, after converting the node sequence corresponding to each node into a topology vector corresponding to each node through a feature mapping function, the method further includes:

[0108] The first step is to calculate the similarity between the topological vectors corresponding to any two nodes in the node graph.

[0109] Optionally, the cosine similarity between the topological vectors corresponding to any two nodes in the node graph can be calculated. The formula for calculating the cosine similarity is:

[0110]

[0111] in, Let be any two topological vectors in the node graph. For vector dot product, These are the magnitudes of the two topological vectors. The similarity calculation result ranges from [-1, 1]. The closer the value is to 1, the higher the similarity of the abnormal features corresponding to the two topological vectors.

[0112] The second step is to identify nodes with a similarity greater than a preset threshold as common fault nodes.

[0113] For example, a preset threshold of 0.7 can be set. When the similarity between two topological vectors is greater than 0.7, the two nodes corresponding to the two topological vectors are determined to be common fault nodes.

[0114] The third step is to determine the fault type of the common fault node based on the abnormal behavior characteristics of the common fault node.

[0115] Optionally, the abnormal behavior characteristics of common fault nodes can be determined by the abnormal indicator vector corresponding to the common fault nodes. Then, they can be matched with a preset rule set to automatically identify the fault type of the common fault nodes. The root cause types include database anomalies, cache anomalies, third-party interface anomalies, or application anomalies, which can provide a data foundation for subsequently determining the root cause type corresponding to the root cause node.

[0116] In summary, by calculating the similarity between the topological vectors corresponding to any two nodes through the above steps, nodes with a similarity greater than a preset threshold are identified as common fault nodes, and the fault type of the common fault nodes is determined, which can provide a data foundation for subsequently determining the root cause type corresponding to the root cause node.

[0117] Optionally, Figure 4The diagram illustrates fault clustering. First, an abnormal transaction indicator vector set is input. The Density-Based Spatial Clustering of Applications with Noise (DBSCAN) algorithm is used to identify multiple common fault clusters. Then, each common fault cluster is transformed into a node graph. A graph embedding algorithm is used to transform each node in the node graph into a topological vector. The similarity between any two topological vectors is calculated. Nodes with a similarity greater than a preset threshold are identified as common fault nodes. The fault type of the common fault nodes is determined based on their abnormal behavior characteristics and pushed to the root cause module, thus providing a data foundation for subsequently determining the root cause type corresponding to the root cause node.

[0118] To improve the operational security of distributed systems, optionally, in the fault identification method for distributed systems provided in this application embodiment, after determining a self-healing suggestion based on the root cause type of the root cause node and performing a self-healing recovery operation on the root cause node based on the self-healing suggestion, the method further includes:

[0119] The first step is to generate a fault analysis report for the distributed system based on the root cause nodes;

[0120] Optionally, the fault analysis report may include root cause information, the scope of impact of the root cause, detailed information on abnormal indicators, self-healing suggestions, etc.

[0121] For example, a fault analysis report includes:

[0122] Root cause node information: CACHE001 (caching service instance, IP address: 192.168.1.100).

[0123] The root cause node's impact scope: the core accounting system's account-service, affecting 120 transactions per hour, including account inquiries and fund reconciliation.

[0124] Detailed information on abnormal metrics: account-service interface response time P95=180ms (80% over threshold), P99=250ms (25% over threshold), cache hit rate=90% (5.3% below threshold), other metrics are normal;

[0125] Self-healing suggestion: Clean up expired data in CACHE001 that has been expired for more than 24 hours, update the cache expiration policy to "active expiration", and adjust the cache capacity threshold to 100GB.

[0126] The second step is to add the root cause node and the self-healing suggestion to the historical failure case library after the root cause node has been successfully restored according to the self-healing suggestion.

[0127] Optionally, after performing the self-healing recovery operation, the corresponding abnormal indicators can be monitored in real time to verify whether the indicators have returned to the normal range. The monitoring duration can be flexibly configured (e.g., 5-10 minutes). If the abnormal indicators corresponding to the root cause node are successfully recovered, the root cause node and self-healing suggestions are added to the historical fault case library to complete the handling of this anomaly.

[0128] For example, the system reads the self-healing recommendations from the fault analysis report, automatically calls the cache cleanup script, cleans up expired data (approximately 50GB) in CACHE001 that has been expired for more than 24 hours, updates the cache expiration policy to "active expiration," and adjusts the cache capacity threshold to 100GB. After the self-healing script is executed, it monitors the changes in abnormal indicators in real time. The monitoring period is set to 5 minutes. Within 5 minutes, the response time of the account-service interface is P95=80ms, P99=150ms, and the cache hit rate is 96%, all of which have returned to the normal range. This indicates that the root cause node has successfully recovered according to the self-healing recommendations. The results of this repair (including link fingerprint, root cause node, self-healing recommendations, and indicator recovery status) are recorded in the historical fault case library, completing the handling of this anomaly. The entire process requires no manual intervention.

[0129] The third step is to generate an alarm message corresponding to the root cause node after the root cause node fails to recover according to the self-healing suggestion, and send the alarm message to the operation and maintenance personnel.

[0130] Optionally, if the abnormal indicator corresponding to the root cause node fails to recover, an alarm message corresponding to the root cause node is generated and immediately pushed to the operation and maintenance personnel via SMS, email, or other means to remind them to intervene manually, so that the fault can be handled in a timely manner and the anomaly can be prevented from escalating.

[0131] In summary, by taking the above steps and generating a fault analysis report, if the root cause node successfully recovers according to the self-healing suggestions, the root cause node and the self-healing suggestions are added to the historical fault case library. If the root cause node fails to recover according to the self-healing suggestions, an alarm message is generated and pushed to the operation and maintenance personnel, thereby improving the operation and maintenance security and stability of the distributed system.

[0132] Figure 5The diagram illustrates self-healing recovery. First, a fault report is received. The self-healing suggestions in the report are read, and the root cause type of the root cause node is determined. If the root cause type is a database anomaly, index optimization and connection pool parameter adjustment are performed. If the root cause type is a cache anomaly, expired data is cleaned and cached data is adjusted. If the root cause type is a third-party interface anomaly, a backup node is switched and a health check is performed. If the root cause type is an application-specific anomaly, the service is restarted or the version is rolled back. After execution, anomaly indicators are monitored in real time to determine if they have returned to normal. If they have returned to normal, the repair result is recorded in the fault case database. If recovery fails, an alert is sent via SMS or email to remind manual intervention. This achieves automated recovery from faults in the distributed system, improving the operational security and stability of the distributed system.

[0133] Figure 6 The diagram illustrates fault identification and recovery in a distributed system. The distributed system comprises a microservice gateway layer, a monitoring system, a distributed database, and an operations and maintenance platform. The microservice gateway layer pushes transaction link data to the link fingerprint generation module, and the monitoring system pushes operational metrics to the same module. The link fingerprint generation module generates link fingerprints based on the transaction link data and operational metrics, inputs these fingerprints into the distributed database, and stores them. The anomaly aggregation module receives the link fingerprints and operational metrics, aggregates anomalies, and outputs abnormal transaction data to the common fault clustering module. This module performs cluster analysis, outputs common faults and their corresponding root cause types, and pushes them to the root cause reasoning and report generation module. This module identifies the root cause node of the fault, generates a fault report, pushes the report to the self-healing execution module, stores the repair results in the distributed database, and calls the self-healing script in the fault report. The operations and maintenance platform then performs self-healing recovery on the root cause node based on the self-healing script. In summary, by automatically collecting link and operational data from the microservice gateway and monitoring system, storing and transferring it uniformly through a distributed database, accurately locating the root cause node and generating a fault report, and having the operation and maintenance platform automatically perform repairs and write back the results, the system can achieve fully automated root cause localization and self-healing, thereby reducing the average fault handling time and improving the security and stability of the distributed system.

[0134] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0135] Example 2

[0136] This application also provides a fault identification device for a distributed system. It should be noted that the fault identification device for a distributed system in this application can be used to execute the fault identification method for a distributed system provided in this application. The following describes the fault identification device for a distributed system provided in this application.

[0137] According to embodiments of this application, an apparatus for implementing the above-described fault identification method for a distributed system is also provided, such as... Figure 7 As shown, the device includes: an integration unit 701, an identification unit 702, a clustering unit 703, a fusion unit 704, a processing unit 705, and a first determination unit 706.

[0138] Integration unit 701 is used to aggregate the abnormal indicators in the link fingerprint vector corresponding to each transaction in the distributed system to obtain the aggregated abnormal indicator vector corresponding to each transaction, and integrate the aggregated abnormal indicator vector corresponding to each transaction into an abnormal indicator vector set.

[0139] The identification unit 702 is used to identify abnormal transactions in the distributed system based on the abnormal indicators in the link fingerprint vector corresponding to each transaction.

[0140] Clustering unit 703 is used to perform cluster analysis on the abnormal indicator vectors corresponding to abnormal transactions according to the density clustering algorithm, determine the common fault clusters, and transform the abnormal indicator vectors in the common fault clusters into topological vectors through the graph embedding algorithm, wherein the common fault clusters include at least one abnormal indicator vector.

[0141] The fusion unit 704 is used to fuse the feature vectors of the topology vector, the abnormal indicator vector set, and the historical fault cases to obtain the fused feature vector.

[0142] The processing unit 705 is used to perform weighted processing on the fused feature vector through a self-attention mechanism to obtain the target feature vector;

[0143] The first determining unit 706 is used to map the target feature vector to a candidate node probability distribution, determine the candidate node with the highest probability among the candidate nodes as the root cause node, determine a self-healing suggestion based on the root cause type of the root cause node, and perform a self-healing recovery operation on the root cause node based on the self-healing suggestion.

[0144] The fault identification device for a distributed system provided in this application embodiment aggregates abnormal indicators in the link fingerprint vector corresponding to each transaction in the distributed system through an integration unit 701, obtaining an aggregated abnormal indicator vector corresponding to each transaction, and integrating the aggregated abnormal indicator vectors corresponding to each transaction into an abnormal indicator vector set; an identification unit 702 identifies abnormal transactions in the distributed system based on the abnormal indicators in the link fingerprint vector corresponding to each transaction; a clustering unit 703 performs cluster analysis on the abnormal indicator vectors corresponding to abnormal transactions according to a density clustering algorithm, determines common fault clusters, and transforms the abnormal indicator vectors in the common fault clusters into topological vectors through a graph embedding algorithm, wherein the common fault clusters include at least one Anomaly indicator vector; fusion unit 704 fuses the topology vector, the set of anomaly indicator vectors, and the feature vectors of historical fault cases to obtain the fused feature vector; processing unit 705 uses a self-attention mechanism to weight the fused feature vector to obtain the target feature vector; the first determination unit 706 maps the target feature vector to a probability distribution of candidate nodes, determines the candidate node with the highest probability among the candidate nodes as the root cause node, and determines a self-healing suggestion based on the root cause type of the root cause node, and performs a self-healing recovery operation on the root cause node according to the self-healing suggestion. This solves the technical problem of not being able to accurately identify the root cause node when a fault occurs in a distributed system, and achieves the technical effect of accurately identifying the root cause node of a fault in a distributed system.

[0145] Optionally, in the fault identification device for a distributed system provided in this application embodiment, the device further includes: an acquisition unit, used to acquire full-link data corresponding to each transaction in the distributed system before aggregating the abnormal indicators in the link fingerprint vector corresponding to each transaction; an encryption unit, used to encrypt and encode the service name and interface path in the full-link data corresponding to each transaction using a hash algorithm to obtain an encrypted and encoded vector corresponding to each transaction; a normalization unit, used to normalize the response time in the full-link data corresponding to each transaction using a normalization algorithm to obtain a response time normalized vector corresponding to each transaction; an encoding unit, used to perform one-hot encoding on the error codes in the full-link data corresponding to each transaction to obtain an error code encoding vector corresponding to each transaction; and a concatenation unit, used to concatenate the encrypted and encoded vector corresponding to each transaction, the response time normalized vector corresponding to each transaction, and the error code encoding vector corresponding to each transaction to obtain a link fingerprint vector corresponding to each transaction.

[0146] Optionally, in the fault identification device of the distributed system provided in this application embodiment, the integration unit 701 includes: an aggregation module, used to aggregate the abnormal indicators in the link fingerprint vector corresponding to each transaction within a preset time window according to preset aggregation rules, to obtain the aggregated abnormal indicator vector corresponding to each transaction, wherein the preset aggregation rules include at least: abnormal response time aggregation rules, error code aggregation rules, and abnormal dependent resource indicator aggregation rules; and an integration module, used to integrate the aggregated abnormal indicator vector corresponding to each transaction into an abnormal indicator vector set.

[0147] Optionally, in the fault identification device for a distributed system provided in this application embodiment, the clustering unit 703 includes: a first transformation module, used to transform each common fault cluster into a node graph, wherein the nodes in the node graph are microservice instances corresponding to the anomaly indicator vectors, the edges in the node graph are the call relationships between microservice instances, and the edge weights in the node graph are the number of calls between microservice instances per unit time; a generation module, used to generate a node sequence corresponding to each node through a random walk node sequence generation formula of a graph embedding algorithm; and a second transformation module, used to transform the node sequence corresponding to each node into a topology vector corresponding to each node through a feature mapping function.

[0148] Optionally, in the fault identification device for a distributed system provided in this application embodiment, the processing unit 705 includes: a mapping module, used to map the fused feature vector to a query vector based on a query projection matrix, to a key vector based on a key projection matrix, and to a value vector based on a value projection matrix, based on a self-attention mechanism; and a weighted summation module, used to calculate an attention weight matrix based on the query vector and the key vector, and to perform a weighted summation of the value vector based on the attention weight matrix to obtain the target feature vector.

[0149] Optionally, in the fault identification device for a distributed system provided in this application embodiment, the device further includes: a calculation unit, used to calculate the similarity between the topological vectors corresponding to any two nodes in the node graph after converting the node sequence corresponding to each node into the topological vector corresponding to each node through a feature mapping function; a second determination unit, used to determine nodes with similarity greater than a preset threshold as common fault nodes; and a third determination unit, used to determine the fault type of the common fault nodes based on the abnormal behavior characteristics of the common fault nodes.

[0150] Optionally, in the fault identification device for a distributed system provided in this application embodiment, the device further includes: a generation unit, configured to generate a fault analysis report for the distributed system based on the root cause node after determining a self-healing suggestion based on the root cause type of the root cause node and performing a self-healing recovery operation on the root cause node based on the self-healing suggestion; an adding unit, configured to add the root cause node and the self-healing suggestion to a historical fault case library after the root cause node successfully recovers based on the self-healing suggestion; and a sending unit, configured to generate alarm information corresponding to the root cause node and send the alarm information to the operation and maintenance personnel after the root cause node fails to recover based on the self-healing suggestion.

[0151] It should be noted that the integration unit 701, identification unit 702, clustering unit 703, fusion unit 704, processing unit 705, and first determination unit 706 mentioned above correspond to steps S201 to S206 in Embodiment 1. The six units and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules or units can be hardware or software components stored in memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n). The above modules can also be part of the device and run in the computer terminal 10 provided in Embodiment 1.

[0152] Example 3

[0153] Embodiments of this application may provide a computer terminal, which may be any computer terminal device in a group of computer terminals. Optionally, in this embodiment, the aforementioned computer terminal may also be replaced with a mobile terminal or an electronic device, etc.

[0154] Optionally, in this embodiment, the computer terminal may be located in at least one of a plurality of network devices in a computer network.

[0155] In this embodiment, the computer terminal described above can execute the program code for the following steps in the fault identification method of a distributed system: aggregating the abnormal indicators in the link fingerprint vector corresponding to each transaction in the distributed system to obtain an aggregated abnormal indicator vector corresponding to each transaction, and integrating the aggregated abnormal indicator vectors corresponding to each transaction into an abnormal indicator vector set; identifying abnormal transactions in the distributed system based on the abnormal indicators in the link fingerprint vector corresponding to each transaction; performing cluster analysis on the abnormal indicator vectors corresponding to abnormal transactions using a density clustering algorithm to determine common fault clusters, and converting the abnormal indicator vectors in the common fault clusters into topological vectors using a graph embedding algorithm, wherein the common fault clusters include at least one abnormal indicator vector; fusing the topological vectors, the abnormal indicator vector set, and the feature vectors of historical fault cases to obtain a fused feature vector; weighting the fused feature vectors using a self-attention mechanism to obtain a target feature vector; mapping the target feature vector to a candidate node probability distribution, determining the candidate node with the highest probability among the candidate nodes as the root cause node, determining a self-healing suggestion based on the root cause type of the root cause node, and performing a self-healing recovery operation on the root cause node based on the self-healing suggestion.

[0156] Optionally, the aforementioned computer terminal can execute the program code for the following steps in the fault identification method for a distributed system: Before aggregating the abnormal indicators in the link fingerprint vector corresponding to each transaction in the distributed system, the method further includes: obtaining the full-link data corresponding to each transaction in the distributed system; encrypting and encoding the service name and interface path in the full-link data corresponding to each transaction using a hash algorithm to obtain the encrypted and encoded vector corresponding to each transaction; normalizing the response time in the full-link data corresponding to each transaction using a normalization algorithm to obtain the response time normalized vector corresponding to each transaction; performing one-hot encoding on the error codes in the full-link data corresponding to each transaction to obtain the error code encoding vector corresponding to each transaction; and concatenating the encrypted and encoded vector, the response time normalized vector, and the error code encoding vector corresponding to each transaction to obtain the link fingerprint vector corresponding to each transaction.

[0157] Optionally, the aforementioned computer terminal can execute the program code for the following steps in the fault identification method for a distributed system: aggregating the abnormal indicators in the link fingerprint vector corresponding to each transaction in the distributed system to obtain the aggregated abnormal indicator vector corresponding to each transaction, and integrating the aggregated abnormal indicator vector corresponding to each transaction into an abnormal indicator vector set, including: aggregating the abnormal indicators in the link fingerprint vector corresponding to each transaction within a preset time window according to preset aggregation rules to obtain the aggregated abnormal indicator vector corresponding to each transaction, wherein the preset aggregation rules include at least: abnormal response time aggregation rules, error code aggregation rules, and abnormal dependent resource indicator aggregation rules; and integrating the aggregated abnormal indicator vector corresponding to each transaction into an abnormal indicator vector set.

[0158] Optionally, the aforementioned computer terminal can execute the program code for the following steps in the fault identification method of a distributed system: converting the abnormal indicator vectors in the common fault clusters into topological vectors using a graph embedding algorithm, including: converting each common fault cluster into a node graph, where the nodes in the node graph are the microservice instances corresponding to the abnormal indicator vectors, the edges in the node graph are the call relationships between microservice instances, and the edge weights in the node graph are the number of calls between microservice instances per unit time; generating a node sequence corresponding to each node using the random walk node sequence generation formula of the graph embedding algorithm; and converting the node sequence corresponding to each node into a topological vector corresponding to each node using a feature mapping function.

[0159] Optionally, the aforementioned computer terminal can execute the program code for the following steps in the fault identification method of a distributed system: weighting the fused feature vectors using a self-attention mechanism to obtain the target feature vector, including: mapping the fused feature vectors to a query vector based on the query projection matrix, mapping the fused feature vectors to a key vector based on the key projection matrix, and mapping the fused feature vectors to a value vector based on the value projection matrix; calculating the attention weight matrix based on the query vector and the key vector, and performing a weighted summation of the value vectors based on the attention weight matrix to obtain the target feature vector.

[0160] Optionally, the aforementioned computer terminal can execute the program code for the following steps in the fault identification method of a distributed system: after converting the node sequence corresponding to each node into the topology vector corresponding to each node through the feature mapping function, the method further includes: calculating the similarity between the topology vectors corresponding to any two nodes in the node graph; determining nodes with a similarity greater than a preset threshold as common fault nodes; and determining the fault type of the common fault nodes based on the abnormal behavior characteristics of the common fault nodes.

[0161] Optionally, the aforementioned computer terminal can execute program code for the following steps in the fault identification method for a distributed system: after determining a self-healing suggestion based on the root cause type of the root cause node and performing a self-healing recovery operation on the root cause node based on the self-healing suggestion, the method further includes: generating a fault analysis report for the distributed system based on the root cause node; after the root cause node successfully recovers based on the self-healing suggestion, adding the root cause node and the self-healing suggestion to the historical fault case library; after the root cause node fails to recover based on the self-healing suggestion, generating alarm information corresponding to the root cause node and sending the alarm information to the operation and maintenance personnel.

[0162] Optionally, Figure 8 This is a structural block diagram of an electronic device according to an embodiment of this application. Figure 8 As shown, the electronic device may include: one or more ( Figure 8 (Only one is shown) processor 802, memory 804, memory controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.

[0163] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the fault identification method and apparatus for the distributed system in this application embodiment. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby realizing the aforementioned fault identification method for the distributed system. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the terminal via a network. Examples of the aforementioned networks include at least one of the following: the Internet, corporate intranet, local area network, mobile communication network, and combinations thereof.

[0164] The processor can access the information and application programs stored in the memory via the transmission device to execute the steps described above in the fault identification method for the distributed system.

[0165] The present application provides a scheme for fault identification in a distributed system. By aggregating abnormal indicators from the link fingerprint vectors corresponding to each transaction in a distributed system, an aggregated abnormal indicator vector for each transaction is obtained, and these aggregated abnormal indicator vectors are integrated into an abnormal indicator vector set. Abnormal transactions in the distributed system are identified based on the abnormal indicators in the link fingerprint vectors corresponding to each transaction. Density clustering is used to perform cluster analysis on the abnormal indicator vectors corresponding to abnormal transactions to determine common fault clusters. A graph embedding algorithm is then used to transform the abnormal indicator vectors in these common fault clusters into topological vectors, where each common fault cluster includes at least one abnormal indicator vector. The topological vectors, the abnormal indicator vector set, and the feature vectors of historical fault cases are fused to obtain a fused feature vector. A self-attention mechanism is used to weight the fused feature vector to obtain a target feature vector. This target feature vector is mapped to a probability distribution of candidate nodes, and the candidate node with the highest probability is identified as the root cause node. A self-healing suggestion is determined based on the root cause type of the root cause node, and a self-healing recovery operation is performed on the root cause node according to the self-healing suggestion. This solves the technical problem of inaccurate identification of root cause nodes when faults occur in distributed systems, achieving the technical effect of accurately identifying the root cause nodes of faults in distributed systems.

[0166] Those skilled in the art will understand that Figure 8 The structure shown is for illustrative purposes only. Electronic devices can also be smartphones, tablets, handheld computers, mobile internet devices (MIDs), PADs, and other terminal devices. Figure 8 This does not limit the structure of the aforementioned electronic device. For example, electronic devices may also include components that are more... Figure 8 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 8 The different configurations shown.

[0167] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0168] Example 4

[0169] Embodiments of this application also provide a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the fault identification method of the distributed system provided in Embodiment 1.

[0170] Optionally, in this embodiment, the storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.

[0171] This application also provides a computer program product that, when executed on a data processing device, is suitable for performing steps of a fault identification method for a distributed system.

[0172] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0173] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0174] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between units or modules may be electrical or other forms.

[0175] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0176] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0177] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0178] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A fault identification method for a distributed system, characterized in that, include: The abnormal indicators in the link fingerprint vector corresponding to each transaction in the distributed system are aggregated to obtain the aggregated abnormal indicator vector corresponding to each transaction, and the aggregated abnormal indicator vector corresponding to each transaction is integrated into an abnormal indicator vector set. Abnormal transactions in the distributed system are identified based on the abnormal indicators in the link fingerprint vector corresponding to each transaction. Cluster analysis is performed on the abnormal indicator vectors corresponding to abnormal transactions based on the density clustering algorithm to determine common fault clusters. The abnormal indicator vectors in the common fault clusters are then transformed into topological vectors using the graph embedding algorithm. The common fault clusters include at least one abnormal indicator vector. The topology vector, the set of abnormal indicator vectors, and the feature vectors of historical fault cases are fused to obtain the fused feature vector. The fused feature vector is weighted using a self-attention mechanism to obtain the target feature vector. The target feature vector is mapped to a candidate node probability distribution. The candidate node with the highest probability among the candidate nodes is determined as the root cause node. A self-healing suggestion is determined based on the root cause type of the root cause node. A self-healing recovery operation is performed on the root cause node based on the self-healing suggestion.

2. The method according to claim 1, characterized in that, Before aggregating the abnormal indicators in the link fingerprint vector corresponding to each transaction in the distributed system, the method further includes: Obtain the full-link data corresponding to each transaction in the distributed system; The service name and interface path in the full-link data corresponding to each transaction are encrypted and encoded using a hash algorithm to obtain the encrypted vector corresponding to each transaction. The response time in the end-to-end data corresponding to each transaction is normalized using a normalization algorithm to obtain a normalized response time vector for each transaction. One-hot encoding is performed on the error codes in the end-to-end data corresponding to each transaction to obtain the error code encoding vector corresponding to each transaction; The encrypted vector, the response time normalized vector, and the error code encoding vector corresponding to each transaction are concatenated to obtain the link fingerprint vector corresponding to each transaction.

3. The method according to claim 1, characterized in that, The abnormal indicators in the link fingerprint vector corresponding to each transaction in the distributed system are aggregated to obtain the aggregated abnormal indicator vector for each transaction. These aggregated abnormal indicator vectors are then integrated into an abnormal indicator vector set, including: According to preset aggregation rules, the abnormal indicators in the link fingerprint vector corresponding to each transaction are aggregated within a preset time window to obtain the aggregated abnormal indicator vector corresponding to each transaction. The preset aggregation rules include at least: abnormal response time aggregation rules, error code aggregation rules, and abnormal dependent resource indicator aggregation rules. The aggregated abnormal indicator vectors corresponding to each transaction are integrated into an abnormal indicator vector set.

4. The method according to claim 1, characterized in that, The transformation of the abnormal indicator vectors in the common fault clusters into topological vectors using a graph embedding algorithm includes: Each common fault cluster is transformed into a node graph, where the nodes in the node graph are microservice instances corresponding to the anomaly indicator vectors, the edges in the node graph are the call relationships between microservice instances, and the edge weights in the node graph are the number of calls between microservice instances per unit time. The node sequence corresponding to each node is generated using the random walk node sequence generation formula of the graph embedding algorithm. The feature mapping function transforms the node sequence corresponding to each node into a topological vector corresponding to each node.

5. The method according to claim 1, characterized in that, By employing a self-attention mechanism, the fused feature vector is weighted to obtain the target feature vector, which includes: Based on the self-attention mechanism, the fused feature vector is mapped to a query vector according to the query projection matrix, the fused feature vector is mapped to a key vector according to the key projection matrix, and the fused feature vector is mapped to a value vector according to the value projection matrix. The attention weight matrix is ​​calculated based on the query vector and the key vector, and the value vector is weighted and summed based on the attention weight matrix to obtain the target feature vector.

6. The method according to claim 4, characterized in that, After transforming the node sequence corresponding to each node into a topological vector corresponding to each node through a feature mapping function, the method further includes: Calculate the similarity between the topological vectors corresponding to any two nodes in the node graph; Nodes with a similarity greater than a preset threshold are identified as common fault nodes; The fault type of the common fault node is determined based on the abnormal behavior characteristics of the common fault node.

7. The method according to claim 1, characterized in that, After determining a self-healing suggestion based on the root cause type of the root cause node, and performing a self-healing recovery operation on the root cause node based on the self-healing suggestion, the method further includes: A fault analysis report of the distributed system is generated based on the root cause node; After the root cause node is successfully restored according to the self-healing suggestion, the root cause node and the self-healing suggestion are added to the historical failure case library. After the root cause node fails to recover according to the self-healing suggestion, an alarm message corresponding to the root cause node is generated and sent to the operation and maintenance personnel.

8. A fault identification device for a distributed system, characterized in that, include: The integration unit is used to aggregate the abnormal indicators in the link fingerprint vector corresponding to each transaction in the distributed system to obtain the aggregated abnormal indicator vector corresponding to each transaction, and integrate the aggregated abnormal indicator vector corresponding to each transaction into an abnormal indicator vector set. The identification unit is used to identify abnormal transactions in the distributed system based on the abnormal indicators in the link fingerprint vector corresponding to each transaction. The clustering unit is used to perform clustering analysis on the abnormal indicator vectors corresponding to abnormal transactions according to the density clustering algorithm, determine the common fault clusters, and transform the abnormal indicator vectors in the common fault clusters into topological vectors through the graph embedding algorithm, wherein the common fault clusters include at least one abnormal indicator vector. The fusion unit is used to fuse the feature vectors of the topology vector, the abnormal indicator vector set, and the historical fault cases to obtain the fused feature vector. The processing unit is used to perform weighted processing on the fused feature vector through a self-attention mechanism to obtain the target feature vector; The first determining unit is configured to map the target feature vector to a candidate node probability distribution, determine the candidate node with the highest probability among the candidate nodes as the root cause node, determine a self-healing suggestion based on the root cause type of the root cause node, and perform a self-healing recovery operation on the root cause node based on the self-healing suggestion.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored executable program, wherein, when the executable program is executed, it controls the device where the computer-readable storage medium is located to perform the fault identification method of the distributed system according to any one of claims 1 to 7.

10. An electronic device, characterized in that, include: Memory, which stores executable programs; A processor for running the program, wherein the program executes the fault identification method for a distributed system according to any one of claims 1 to 7.

11. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the steps of the fault identification method for a distributed system according to any one of claims 1 to 7.

Images