Multi-node system exception root cause positioning method and device, equipment and storage medium

Abstract

The application discloses a multi-node system abnormal root cause positioning method and device, equipment and storage medium, and belongs to the technical field of computers. The method comprises the following steps: when it is detected that an abnormal node exists in a multi-node system, the performance data and a first network topology graph of a plurality of nodes in the multi-node system are processed through a space-time graph neural network to obtain an attention weight matrix of the plurality of nodes, thereby adjusting the first network topology graph, retaining the edges between nodes with similar performance data, restoring the real network topology of the multi-node system, obtaining a second network topology graph, and realizing abnormal root cause positioning. Since the real network topology of the multi-node system is restored according to the performance data of the plurality of nodes, and the nodes with similar performance data are usually on a communication connection path, the abnormal root cause can be quickly positioned according to the abnormal node and the second network topology graph, and the accuracy of the abnormal root cause positioning result is improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, device and storage medium for locating the root cause of anomalies in a multi-node system. Background Technology

[0002] With the continuous development of computer technology, computers can interconnect and form multi-node systems to meet various business needs. Taking a data center as an example, data centers are now equipped with increasingly diverse and larger-scale devices. A typical data center includes tens of thousands or even millions of nodes, which are connected via wired or wireless means to form a complex network topology. When a node in a data center malfunctions, it often leads to a chain reaction of malfunctions, making accurate root cause localization crucial.

[0003] In related technologies, taking data centers as an example, when an anomaly is detected in a node of the data center, the network topology of the data center is used to find the associated nodes of that node, and check whether there are any anomalies in the log information of these associated nodes. If anomalies are found, the associated nodes of these associated nodes are further searched until the root cause node of the anomaly is found, thus realizing the root cause localization of the data center anomaly.

[0004] However, due to the complexity of data center network topology and the dynamic changes in connections between some nodes, it is difficult to obtain the true network topology, resulting in low accuracy in locating the root cause of anomalies. Summary of the Invention

[0005] This application provides a method, apparatus, device, and storage medium for locating the root cause of anomalies in a multi-node system, which can improve the accuracy of the root cause location results. The technical solution is as follows:

[0006] Firstly, a method for locating the root cause of anomalies in a multi-node system is provided, the method comprising:

[0007] When an abnormal node is detected in a multi-node system, the performance data of multiple nodes in the multi-node system and the first network topology graph are processed based on a spatiotemporal graph neural network to obtain the attention weight matrix of the multiple nodes. The spatiotemporal graph neural network is trained based on the historical performance data and historical network topology graph of the multiple nodes. The first network topology graph includes the multiple nodes and the edges between the nodes. The edges between the nodes indicate that there is a communication connection between the nodes. The attention weight matrix indicates the similarity between the performance data of any two nodes.

[0008] Based on the attention weight matrix, the first network topology is adjusted to obtain the second network topology of the multiple nodes. The edges between nodes in the second network topology indicate that there is a communication connection between the nodes and that the similarity between the performance data of the nodes meets the target conditions.

[0009] Based on the second network topology and the abnormal node, the root cause localization result of the anomaly in the multi-node system is determined.

[0010] The performance data can be at least one of latency, input / output operations per second (IOPS), bandwidth, network traffic, and port access volume, etc., without limitation. In the above method, when an abnormal node is detected in a multi-node system, a spatiotemporal graph neural network is used to process the performance data of multiple nodes in the multi-node system and a first network topology graph to obtain an attention weight matrix for multiple nodes. This adjusts the first network topology graph, preserving the edges between nodes with similar performance data, thus restoring the true network topology of the multi-node system and obtaining a second network topology graph, achieving anomaly root cause localization. Since the true network topology of the multi-node system is restored based on the performance data of multiple nodes, and nodes with similar performance data are usually on the same communication connection path, the root cause of the anomaly can be quickly located based on the abnormal node and the second network topology graph, improving the accuracy of the anomaly root cause localization results. In particular, in the case of a multi-node system that is a data center, the actual network topology is often difficult to obtain due to the large number of nodes in the data center. The above method can quickly reconstruct the actual network topology of the data center based on the performance data of the massive number of nodes in the data center, thereby realizing the rapid location of the root cause of the anomaly. This not only improves the efficiency of the root cause location of the anomaly in the data center, but also improves the accuracy of the root cause location results.

[0011] In some embodiments, adjusting the first network topology based on the attention weight matrix to obtain a second network topology of the plurality of nodes includes:

[0012] Based on the weights of the first number of targets in the target row of the attention weight matrix, the target number of associated nodes of the target node is determined. The target row refers to any row in the attention weight matrix, and the weights in the target row are used to indicate the degree of similarity between the performance data of the target node and other nodes.

[0013] Delete the edges between the target node and other nodes except for the number of associated nodes of the target.

[0014] It should be understood that since the attention weight matrix is ​​a symmetric matrix, the first network topology can also be adjusted based on "columns," and the process is similar and will not be elaborated here. Using the above method, since the weights in the attention weight matrix indicate the similarity between the performance data of nodes, the larger the weight, the greater the similarity between the performance data of two nodes. Correspondingly, these two nodes are more likely to belong to the same communication connection path in the multi-node system. Through the above adjustment method, the edges between nodes with similar performance data are retained, while the edges between nodes with significantly different performance data are deleted, thus obtaining the true network topology of the multi-node system, providing technical support for subsequent anomaly root cause localization.

[0015] In some embodiments, determining the root cause localization result of the anomaly in the multi-node system based on the second network topology and the anomalous node includes:

[0016] Based on the connected component in which the abnormal node is located in the second network topology graph, the nodes in the connected component are sorted topologically to obtain the topology sorting result, which indicates the communication connection path of the node in the connected component in the multi-node system.

[0017] Based on the topological sorting results, the root cause of the anomaly was located.

[0018] In this topology sorting result, the order of the nodes is the order of their communication connections in the multi-node system. By using the above method, since nodes with similar performance data are usually on the same communication connection path, the root cause of the anomaly can be quickly located based on the connected component where the abnormal node is located, thus improving the accuracy of the root cause location result.

[0019] In some embodiments, the topological sorting of nodes in the connected component where the abnormal node is located in the second network topology graph, to obtain a topological sorting result, includes:

[0020] Starting with the node with an in-degree of zero in the connected component, traverse the nodes in the connected component based on the direction of the edges between the nodes in the connected component to obtain the topological sorting result.

[0021] In some embodiments, determining the root cause localization result based on the topological sorting result includes at least one of the following:

[0022] The node at the top of the topology sorting result is identified as the root cause node, thus obtaining the root cause localization result. This topology sorting result directly locates the root cause node, improving the efficiency of root cause localization.

[0023] The order of nodes in the topology sorting result is used to determine the root cause path, thus obtaining the root cause localization result for the anomaly. Since the root cause node of an anomaly is not necessarily the first node in the communication connection path of the anomaly node, locating the root cause path can provide more information, enabling relevant personnel to conduct anomaly investigations and improving the accuracy of root cause localization.

[0024] In some embodiments, determining the root cause localization result of the anomaly in the multi-node system based on the second network topology and the anomalous node includes:

[0025] Based on the node types of these multiple nodes, the second network topology is adjusted so that the communication connections between the nodes meet the operational requirements of the multi-node system, resulting in a third network topology for these multiple nodes.

[0026] Based on the connected component where the abnormal node is located in the third network topology graph, the root cause localization result of the anomaly in the multi-node system is determined.

[0027] By further trimming the second network topology to obtain a third network topology that better reflects the actual situation, the accuracy of the anomaly root cause localization results can be further improved.

[0028] In some embodiments, the method further includes:

[0029] Obtain the initial network topology of the multiple nodes, in which any two nodes are connected.

[0030] Based on the node types of these multiple nodes, the initial network topology is adjusted so that the communication connections between the nodes meet the operational requirements of the multi-node system, thus obtaining the first network topology.

[0031] In this way, the number of edges in the first network topology graph can be reduced, thereby reducing the amount of data processing on the server and improving the efficiency of anomaly root cause localization.

[0032] In some embodiments, the method further includes:

[0033] If the performance data of the first node is not obtained, the performance data of the first node will be configured as the default performance data. The first node can be any one of the multiple nodes.

[0034] In this way, for nodes where performance data cannot be obtained, the performance data of these nodes is configured to the default performance data, so that subsequent processing can still be carried out to achieve the root cause of the anomaly.

[0035] In some embodiments, the spatiotemporal graph neural network-based processing of performance data of multiple nodes in the multi-node system and a first network topology graph yields an attention weight matrix for the multiple nodes, including:

[0036] The performance data of these multiple nodes and the first network topology graph are input into the spatiotemporal graph neural network.

[0037] Based on the spatial attention layer in the spatiotemporal graph neural network, the spatial attention weight matrix of the multiple nodes is obtained. The spatial attention weight matrix indicates the degree of spatial similarity of the performance data of any two nodes.

[0038] Based on the temporal attention layer in the spatiotemporal graph neural network, the temporal attention weight matrix of the multiple nodes is obtained. The temporal attention weight matrix indicates the degree of temporal similarity of the performance data of any two nodes.

[0039] The attention weight matrix is ​​obtained by weighted summing of the spatial attention weight matrix and the temporal attention weight matrix.

[0040] By utilizing the attention mechanism in spatiotemporal graph neural networks, attention weight matrices for multiple nodes in a multi-node system were obtained. Since these attention weight matrices can indicate the similarity between the performance data of any two nodes, they can serve as the basis for adjusting the first network topology graph. This allows for the removal of edges between nodes with significantly different performance data, while retaining edges between nodes with similar performance data. Consequently, nodes with similar performance data are presented as a connected component (or clustered into a group), providing technical support for anomaly root cause localization.

[0041] Secondly, embodiments of this application provide a multi-node system anomaly root cause localization device, which includes at least one functional module for performing the multi-node system anomaly root cause localization method provided by the first aspect or any possible implementation of the first aspect.

[0042] Thirdly, embodiments of this application provide a computing device including a processor and a memory; the memory is used to store at least one piece of program code, which is loaded by the processor and executed as the multi-node system anomaly root cause localization method provided in the first aspect or any possible implementation thereof.

[0043] Fourthly, embodiments of this application provide a computer-readable storage medium for storing at least one piece of program code, which is used to implement the multi-node system anomaly root cause localization method provided in the first aspect or any possible implementation thereof. The storage medium includes, but is not limited to, volatile memory, such as random access memory, and non-volatile memory, such as flash memory, hard disk drive (HDD), and solid state drive (SSD).

[0044] Fifthly, embodiments of this application provide a computer program product that, when run on a computing device, enables the computing device to implement the multi-node system anomaly root cause localization method provided in the first aspect or any possible implementation thereof. The computer program product can be a software installation package; when the aforementioned multi-node system anomaly root cause localization method needs to be implemented, the computer program product can be downloaded and executed on the computing device. Attached Figure Description

[0045] Figure 1 This is a schematic diagram of a data center architecture provided in an embodiment of this application;

[0046] Figure 2 This is a schematic diagram of an implementation environment provided in an embodiment of this application;

[0047] Figure 3 This is a schematic diagram of the hardware structure of a computing device provided in an embodiment of this application;

[0048] Figure 4 This is a flowchart of a method for locating the root cause of anomalies in a multi-node system provided in an embodiment of this application;

[0049] Figure 5 This is a schematic diagram of a first network topology provided in an embodiment of this application;

[0050] Figure 6 This is a schematic diagram of a second network topology provided in an embodiment of this application;

[0051] Figure 7 This is a schematic diagram illustrating an embodiment of obtaining a second network topology map provided in this application;

[0052] Figure 8 This is a schematic diagram of a third network topology provided in an embodiment of this application;

[0053] Figure 9 This is a schematic diagram of a data center anomaly root cause localization method provided in an embodiment of this application;

[0054] Figure 10 This is a schematic diagram of a data center network topology provided in an embodiment of this application;

[0055] Figure 11 This is a schematic diagram of the structure of a multi-node system anomaly root cause localization device provided in an embodiment of this application. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0057] To facilitate understanding, the key terms and concepts involved in this application will be explained below.

[0058] Graph neural networks (GNNs) are a general term for algorithms that use neural networks to learn graph-structured data, extract and discover features and patterns in graph-structured data, and meet the needs of graph learning tasks such as clustering, classification, prediction, segmentation, and generation.

[0059] Attention mechanisms refer to the methods used to quickly sift out high-value information from a large amount of data using limited attentional resources. They are widely used in various deep learning tasks, including natural language processing, image recognition, and speech recognition, and are one of the core technologies in machine learning that deserve the most attention and in-depth understanding. Typically, attention mechanisms have two main aspects: first, determining which part of the input to focus on; and second, allocating limited information processing resources to the important parts. In essence, attention mechanisms in machine learning are similar to the selective visual attention mechanisms in humans; their core goal is also to select the information most critical to the current task from a large amount of information.

[0060] A connected component is a graph that consists of multiple nodes that are connected to each other. Illustratively, two nodes are said to be connected if a path exists between them.

[0061] Topological sorting is a linear sequence of all nodes in a directed acyclic graph (DAG). This sequence satisfies the following two conditions: (1) Each node appears exactly once. (2) If there is a path from node A to node B, then node A appears before node B in the sequence. Illustratively, for a directed edge (u, v), a node sequence satisfying u < v in all cases is defined as a topological sequence, that is, a node with an in-degree of zero (i.e., no incoming edges) is found from the graph, the node is output, the node and all its outgoing edges are deleted from the graph, the above steps are repeated until all nodes with an in-degree of zero have been output, and a linear sequence of all nodes in the graph is obtained, that is, the topological sorting result of all nodes in the graph is obtained.

[0062] The application scenarios and implementation environments involved in the present application are introduced below.

[0063] The technical solutions provided by the embodiments of the present application can be applied in the field of computer technology, and can perform root cause localization for abnormal nodes detected in a multi-node system, or perform root cause localization for abnormal conditions of a multi-node system. Illustratively, taking a multi-node system as a data center as an example, when an abnormal node is detected in the data center, localizing the root cause node of the abnormal node facilitates relevant personnel to troubleshoot the abnormal cause in time and restore the normal operation of the data center. Reference Figure 1 , Figure 1 is a schematic architectural diagram of a data center provided by an embodiment of the present application. As shown in Figure 1 , the data center includes a plurality of nodes, and the plurality of nodes have a hierarchical structure, such as an application layer, a virtualization layer, a host layer, a switch layer, and a storage layer. These nodes are communicatively connected by wired or wireless means to form a complex network topology. For example, when an abnormal node is detected in the application layer, its root cause node may be located in the switch layer and has a relatively serious fault, so there is an urgent need to quickly localize the root cause node according to the detected abnormal node, so that relevant personnel can troubleshoot the abnormal cause in time.

[0064] Figure 2 is a schematic diagram of an implementation environment provided by an embodiment of the present application. As shown in Figure 2 , the implementation environment includes a multi-node system 100, the multi-node system 100 includes a plurality of nodes 101 and an anomaly detection server 102, and the plurality of nodes 101 and the anomaly detection platform 102 are communicatively connected by wired or wireless means.

[0065] The multiple nodes 101 include hosts, virtual machines, network devices (such as switches, routers, etc.), and storage devices, etc., and are not limited thereto. These multiple nodes 101 are interconnected via wired or wireless means, forming a multi-node system 100. For example, the multiple nodes 101 may be interconnected to form a large data center; or, for example, the multiple nodes 101 may be interconnected within a limited geographical area to form a local area network. The scale of the multi-node system 100 is not limited in this embodiment. Furthermore, the number and form of the nodes shown in the figures are merely illustrative; the number and form of the multiple nodes 101 may be more or fewer, and are not limited thereto.

[0066] An anomaly detection server 102 is used to detect the operating status of the multi-node system 100 to ensure its normal operation. The anomaly detection server 102 collects performance data from each node 101 at preset intervals (e.g., 10 seconds, but this is not limited and can be set according to actual needs) to determine whether any node 101 is experiencing anomalies. If an anomaly is found, the multi-node system anomaly root cause localization method provided in this application embodiment is executed. Based on the detected abnormal nodes, the root cause of the anomaly is located, and the anomaly root cause localization result of the multi-node system is obtained, facilitating timely investigation of the cause of the anomaly and restoration of the normal operation of the multi-node system. In some embodiments, the anomaly detection server 102 can also be understood as a management platform for the multi-node system 100, capable of maintaining the normal operation of the multi-node system 100, without limitation. The anomaly detection server 102 can be a standalone physical server, a server cluster consisting of multiple physical servers, a distributed file system, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0067] In some embodiments, the aforementioned wireless or wired networks use standard communication technologies and / or protocols. The network is typically the Internet, but can be any network, including but not limited to local area networks (LANs), metropolitan area networks (MANs), wide area networks (WANs), mobile, wired or wireless networks, private networks, or any combination of virtual private networks. In some implementations, technologies and / or formats, including Hypertext Markup Language (HTML), Extensible Markup Language (XML), etc., are used to represent data exchanged over the network. Furthermore, conventional encryption technologies such as Secure Sockets Layer (SSL), Transport Layer Security (TLS), Virtual Private Networks (VPNs), and Internet Protocol Security (IPsec) can be used to encrypt all or some links. In other embodiments, custom and / or dedicated data communication technologies can be used to replace or supplement the aforementioned data communication technologies.

[0068] The hardware structure of the anomaly detection server 102 in the above implementation environment is described below.

[0069] This application provides a computing device that can be configured as the above-described anomaly detection server 102. (Illustratively, refer to...) Figure 3 , Figure 3 This is a schematic diagram of the hardware structure of a computing device provided in an embodiment of this application. Figure 3 As shown, the computing device 300 includes a memory 301, a processor 302, a communication interface 303, and a bus 304. The memory 301, processor 302, and communication interface 303 are interconnected via the bus 304.

[0070] The memory 301 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. Illustratively, taking the computing device configured as the aforementioned terminal as an example, the memory 301 is used to store at least a piece of program code. When the program code stored in the memory 301 is executed by the processor 302, the processor 302 is used to perform the steps performed by the anomaly detection server in the following method embodiments.

[0071] Processor 302 may be a network processor (NP), a central processing unit (CPU), an application-specific integrated circuit (ASIC), or an integrated circuit used to control the execution of the program in this application. Processor 302 may be a single-core processor or a multi-core processor. There may be one or more processors 302. Communication interface 303 uses a transceiver module, such as a transceiver, to enable communication between computing device 300 and other devices or communication networks. For example, data can be acquired through communication interface 303.

[0072] The memory 301 and the processor 302 can be set separately or integrated together.

[0073] Bus 304 may include a pathway for transmitting information between various components of computing device 300 (e.g., memory 301, processor 302, communication interface 303).

[0074] The following section introduces the multi-node system anomaly root cause localization method provided in this application.

[0075] Figure 4This is a flowchart of a multi-node system anomaly root cause localization method provided in an embodiment of this application. Figure 4 As shown, taking the anomaly root cause localization method of the multi-node system as an example applied to the anomaly detection server involved in the above implementation environment (hereinafter referred to as the server), the multi-node system anomaly root cause localization method includes the following steps 401 to 405.

[0076] 401. When the server detects an abnormal node in a multi-node system, it obtains the performance data of multiple nodes in the multi-node system and the first network topology diagram.

[0077] In this embodiment, when the server detects an abnormal node in a multi-node system, it acquires the performance data of multiple nodes in the multi-node system within a target time period and a first network topology diagram of these nodes. The target time period refers to the time during which the abnormal node experiences an anomaly, or the period during which the multi-node system is abnormal. For example, taking a data center as an example, if the server detects an anomaly in the performance data of a virtual machine between 0:00 and 1:00 today, such as an abnormal latency, the server acquires the performance data of multiple nodes in the data center between 0:00 and 1:00 today and a first network topology diagram of these nodes. This performance data can be at least one of latency, input / output operations per second (IOPS), bandwidth, network traffic, and port access volume, etc., without limitation.

[0078] For any one of these multiple nodes, hereinafter referred to as the first node, if performance data for the first node is not available, its performance data will be configured as the default performance data. For example, the default performance data could be zero, or it could be the node's historical performance data, etc., without limitation. It should be understood that when an abnormal node is detected in a multi-node system, its root cause node often experiences a severe failure, preventing the server from obtaining its performance data. Therefore, by configuring the performance data of nodes for which performance data cannot be obtained as the default performance data, subsequent processing can still be performed to locate the root cause of the anomaly.

[0079] The first network topology graph includes the plurality of nodes and the edges connecting them, whereby the edges indicate that there is a communication connection between the nodes. In some embodiments, any two nodes in the first network topology graph are connected, that is, for any given node, there is an edge connecting it to any other node; thus, the first network topology graph can be understood as a hypothetical complete graph. Furthermore, the first network topology graph can be a network topology graph generated by the server based on the multi-node system anomaly root cause localization method provided in this application embodiment when the server last detected an abnormal node; it can also be a network topology graph regenerated by the server when an abnormal node is detected this time, and this application embodiment does not limit this.

[0080] For example, refer to Figure 5 , Figure 5 This is a schematic diagram of a first network topology provided in an embodiment of this application. For example... Figure 5 As shown, the first network topology graph includes multiple nodes and the edges connecting the nodes, and for any given node, there is an edge connecting the node to any other node.

[0081] 402. The server uses a spatiotemporal graph neural network to process the performance data of the multiple nodes and the first network topology graph to obtain the attention weight matrix of the multiple nodes. The spatiotemporal graph neural network is trained based on the historical performance data and historical network topology graph of the multiple nodes.

[0082] In this embodiment of the application, the server invokes a spatiotemporal graph neural network, inputting the performance data of the multiple nodes and a first network topology graph into the spatiotemporal graph neural network. The spatiotemporal graph neural network is used to calculate the similarity between the performance data of the multiple nodes, thereby obtaining the attention weight matrix of the multiple nodes. The attention weight matrix indicates the similarity between the performance data of any two nodes.

[0083] This spatiotemporal graph neural network employs an attention mechanism, using spatial and temporal attention layers to extract the dependencies between node performance data, thus obtaining the attention weight matrix for these multiple nodes. For example, this spatiotemporal graph neural network includes an input layer, parallel-connected spatial and temporal attention layers, and an output layer. The process of obtaining the attention weight matrix based on the spatiotemporal graph neural network is described below, including steps A to D:

[0084] Step A: Input the performance data of the multiple nodes and the first network topology graph into the spatiotemporal graph neural network.

[0085] Step B: Based on the spatial attention layer in the spatiotemporal graph neural network, obtain the spatial attention weight matrix of the multiple nodes. The spatial attention weight matrix indicates the degree of spatial similarity of the performance data of any two nodes.

[0086] The spatial attention layer is used to calculate the spatial similarity of the performance data of multiple nodes based on the attention mechanism, so as to obtain the spatial attention weight matrix of the multiple nodes. The spatial similarity of the performance data of the nodes is also the similarity between the performance data of the nodes at the same time (or the same time window). Illustratively, this process is implemented by the following formulas (1) to (3).

[0087]

[0088]

[0089]

[0090] In the above formula, W s [i, j] represents the spatial similarity of the performance data of the i-th node and the j-th node, k represents any one of the multiple nodes, N is the number of the multiple nodes, and i, j, k, and N are all integers. This represents the performance data of the i-th node and its adjacent nodes at the same time (or within the same time window). In formula (2), n is a positive integer, representing the number of nodes adjacent to the i-th node. In other words, this... It can also be understood as a matrix, in which multiple dimensions represent the performance data of multiple nodes at a certain moment. i The feature representation of the performance data of the i-th node and its neighboring nodes can also be understood as an embedding vector. Similarly, p j p represents the feature representation of the performance data of the j-th node and its neighboring nodes. k Let p represent the feature representation of the performance data of any node and its neighboring nodes. CONV stands for graph convolution operation, which is used to perform a weighted sum of the performance data of the i-th node and its neighboring nodes, and then add an activation function to obtain the feature representation p of the i-th node. i After obtaining the feature representation of each node, the inner product of the feature representations of the i-th node and the j-th node is calculated using formula (3) to obtain the spatial similarity of the performance data of the i-th node and the j-th node. Finally, the spatial attention weight matrix of multiple nodes is obtained using formula (1).

[0091] Step C: Based on the temporal attention layer in the spatiotemporal graph neural network, obtain the temporal attention weight matrix of the multiple nodes. The temporal attention weight matrix indicates the degree of temporal similarity of the performance data of any two nodes.

[0092] The temporal attention layer is used to calculate the temporal similarity of the performance data of multiple nodes based on an attention mechanism, in order to obtain the temporal attention weight matrix of these nodes. The temporal similarity of the node performance data can be understood as the degree of similarity between the performance data of nodes at different times (or different time windows).

[0093] Indicatively, this process is achieved through the following formulas (4) to (6).

[0094]

[0095]

[0096]

[0097] In the above formula, W t [i, j] represents the temporal similarity of the performance data of the i-th node and the j-th node, k represents any one of the multiple nodes, N is the number of the multiple nodes, and i, j, k, and N are all integers. This represents the performance data of the i-th node at multiple times (or multiple time windows). In formula (4), n is a positive integer, representing the number of times. In other words, the... It can also be understood as a matrix, where each dimension represents the performance data of a node at multiple time points. p′ i The feature representation of the performance data of the i-th node at multiple time points can also be understood as an embedding vector. Similarly, p′ j The feature representation of the performance data of the j-th node and its neighboring nodes, p′ k This represents the feature representation of the performance data of any node and its neighboring nodes. GRU stands for Gate Recurrent Unit, which processes the performance data of the i-th node at multiple time points to obtain the feature representation p′ of the i-th node. i After obtaining the feature representation of each node, the inner product of the feature representations of the i-th node and the j-th node is calculated using formula (6) to obtain the temporal similarity of the performance data of the i-th node and the j-th node. Finally, the temporal attention weight matrix of multiple nodes is obtained using formula (4).

[0098] Steps B and C can be executed in parallel, or step C can be executed first and then step B; there is no limitation on this.

[0099] Step D: Perform a weighted summation of the spatial attention weight matrix and the temporal attention weight matrix to obtain the attention weight matrix.

[0100] The attention weight matrix is ​​obtained by the following formula (7).

[0101] W = αW s +βW t (7)

[0102] In the formula, α and β are hyperparameters of the spatiotemporal graph neural network, and there is no limit to the specific values, such as α = 1 and β = 0.5.

[0103] After steps A to D above, the server invokes the spatiotemporal graph neural network to obtain the attention weight matrix of multiple nodes in the multi-node system. Since this attention weight matrix can indicate the similarity between the performance data of any two nodes, it can serve as the basis for adjusting the first network topology graph. This allows for the removal of edges between nodes with significantly different performance data, while retaining edges between nodes with similar performance data. As a result, nodes with similar performance data are presented as a connected component (or clustered into a group), providing technical support for anomaly root cause localization.

[0104] Furthermore, the spatiotemporal graph neural network of this application embodiment is not limited to the above structure. For example, other processing layers can be added, and there is no limitation in this regard. Moreover, the spatiotemporal graph neural network can be pre-trained by the server based on the historical performance data and historical network topology of multiple nodes, or it can be re-trained by the server by acquiring the historical performance data and historical network topology of multiple nodes when an abnormal node is detected, and there is no limitation in this regard.

[0105] 403. Based on the attention weight matrix, the server adjusts the first network topology to obtain the second network topology of the multiple nodes.

[0106] In this embodiment, the edges between nodes in the second network topology graph indicate that the nodes have a communication connection and that the similarity between the performance data of the nodes meets the target conditions. The server adjusts the first network topology graph based on the weights of each row in the attention weight matrix to obtain the second network topology graph with the multiple nodes. The following describes this process using any row in the attention weight matrix (hereinafter referred to as the target row) as an example (it should be understood that since the attention weight matrix is ​​a symmetric matrix, the first network topology graph can also be adjusted based on "columns," and the process is similar and will not be repeated here). This includes the following steps 1 and 2:

[0107] Step 1: Based on the weight of the target node in the target row of the attention weight matrix, determine the target number of associated nodes.

[0108] In this context, the weights in the target row indicate the similarity between the performance data of the target node and other nodes. The target number is a preset number and can be set according to actual needs; it is not limited, for example, the target number can be 5. This process involves the server selecting the K nodes with the highest weights in the target row as the associated nodes of the target node in that row, where K is a positive integer. Referring to Table 1, which shows an attention weight matrix provided in an embodiment of this application, as shown in Table 1, taking the target row as the first row as an example, with a target number of 5 and the target node being node 0, the nodes corresponding to the top 5 weights in this target row are node 0, node 4, node 5, node 6, and node 7, respectively. Based on this, the associated nodes of node 0 are determined to be nodes 4, 5, 6, and 7.

[0109] Table 1

[0110] 0 0.28303 0.09314 0.8737 0.08947 0.11102 0.11935 0.11686 0.09976 1 0.08944 … … … … … … … 2 0.08726 … … … … … … … 3 0.08523 … … … … … … … 4 0.10140 … … … … … … … 5 0.10469 … … … … … … … 6 0.10539 … … … … … … … 7 0.09706 … … … … … … …

[0111] Step 2: Delete the edges between the target node and other nodes except for the number of associated nodes of the target.

[0112] Taking Table 1 above as an example, after determining that the associated nodes of node 0 are node 4, node 5, node 6 and node 7, delete the edges between node 0 and node 1, node 2 and node 3.

[0113] It should be understood that since the weights in the attention weight matrix above can indicate the similarity between the performance data of nodes, the larger the weight, the greater the similarity between the performance data of two nodes. Correspondingly, these two nodes are more likely to belong to the same communication connection path in the multi-node system. Through the above adjustment method, the edges between nodes with similar performance data are retained, and the edges between nodes with significantly different performance data are deleted, thereby obtaining the real network topology of the multi-node system, which provides technical support for subsequent anomaly root cause localization.

[0114] Alternatively, the server can directly obtain a second network topology graph based on the first network topology graph by deleting some edges, meaning the positions of nodes in the topology graph remain unchanged before and after adjustment; or it can regenerate a second network topology graph based on the first network topology graph by deleting some edges and then regenerating the second network topology graph based on the communication connections between multiple nodes, meaning the positions of nodes in the topology graph differ before and after adjustment. In this second network topology graph, nodes with similar performance data are closer together, forming multiple clusters. This application does not limit the specific implementation of this method. (Illustratively, refer to...) Figure 6 , Figure 6 This is a schematic diagram of a second network topology provided in an embodiment of this application. For example... Figure 6As shown, the first network topology graph includes 6 nodes and 30 edges. The server adjusts the first network topology graph based on the attention weight matrix, removing the edges between nodes with significantly different performance data, to obtain the second network topology graph, which includes 6 nodes and 6 edges. Each edge indicates that there is a communication connection between the nodes and that the similarity between the performance data of the nodes meets the target conditions. This process can also be understood as pruning a complete graph into multiple connected components.

[0115] After steps 402 and 403 above, when an abnormal node is detected, the server processes the performance data of multiple nodes in the multi-node system and the first network topology diagram using a spatiotemporal graph neural network. This yields the true network topology diagram of the multi-node system during the target time period (i.e., the period of abnormality in the multi-node system), facilitating root cause localization based on the location of the abnormal node in the network topology diagram. (The above process is referenced...) Figure 7 , Figure 7 This is a schematic diagram of obtaining a second network topology map provided in an embodiment of this application, such as... Figure 7 As shown, the server inputs the performance data of multiple nodes and the first network topology graph into the spatiotemporal graph neural network. Through the spatial attention layer in the spatiotemporal graph neural network, the spatial attention weight matrix of the multiple nodes is obtained. Through the temporal attention layer in the spatiotemporal graph neural network, the temporal attention weight matrix of the multiple nodes is obtained. The two matrices are weighted and summed to obtain the attention weight matrix of the multiple nodes. Furthermore, based on the attention weight matrix, the first network topology graph is adjusted (which can also be understood as a pruning process) to obtain the second network topology graph.

[0116] 404. Based on the connected component in which the abnormal node is located in the second network topology graph, the server performs a topology sort on the nodes in that connected component to obtain the topology sorting result.

[0117] In this embodiment, the server assigns directions to the edges in the connected component based on the node type of the nodes in the connected component, where the abnormal node is located in the second network topology graph. For example, taking a data center as an example of a multi-node system, the direction of the edge between the host and the virtual machine in the data center is from the host to the virtual machine, not from the virtual machine to the host. After assigning directions to the edges in the connected component, a directed acyclic graph is obtained. By performing a topological sort on the nodes in the connected component, a topological sorting result can be obtained. Schematic, the server starts with a node in the connected component with an in-degree of zero, traverses the nodes in the connected component based on the direction of the edges between the nodes in the connected component, and obtains the topological sorting result. The topological sorting result indicates the communication connection path of the nodes in the connected component in the multi-node system; in other words, the order of the nodes in the topological sorting result is the communication connection order of these nodes in the multi-node system. For example, if the abnormal node is node 0, and the connected components include node 0, node 1, node 3, and node 5, and the topology sorting result indicates that the order of these nodes is node 1, node 5, node 0, node 3, then the communication connection path of these nodes in the multi-node system is also node 1, node 5, node 0, node 3.

[0118] In some embodiments, the server can also pre-assign directions to the edges in the second network topology graph based on the node types of multiple nodes in the multi-node system, provided that a second network topology graph is obtained. That is, the server can pre-assign directions to the edges of the entire second network topology graph before locating the connected components to improve the efficiency of anomaly root cause localization, or it can assign directions to the edges in the connected components after locating the abnormal node to reduce the amount of data processing.

[0119] Furthermore, step 404 above is illustrated using a single abnormal node as an example. In some embodiments, the server detects multiple abnormal nodes. In such cases, the server performs topological sorting on each connected component in the second network topology graph based on the location of each abnormal node to obtain the topological sorting result. This is not limited in this respect.

[0120] 405. Based on the topology sorting result, the server determines the root cause location of the anomaly.

[0121] In this application embodiment, the server determines the anomaly root cause location result based on the topology sorting result in the following ways:

[0122] (1) The node that ranks first in the topology sorting result is identified as the root cause node, thus obtaining the root cause localization result. That is, the root cause node is directly located through the topology sorting result, improving the efficiency of root cause localization. For example, if the abnormal node is node 0, and the topology sorting result indicates that the order of nodes in the corresponding connected component is node 1, node 5, node 0, node 3, then node 1 is identified as the root cause node, thus obtaining the root cause localization result.

[0123] (2) The order of nodes in the topology sorting result is determined as the root cause path to obtain the root cause location result of the anomaly. Since the root cause node of an abnormal node is not necessarily the first node in the communication connection path where the abnormal node is located, locating the root cause path can provide more information so that relevant personnel can conduct anomaly investigation and improve the accuracy of root cause location. In addition, in some embodiments, the server can also mark the root cause node in the root cause location result of the anomaly to prompt relevant personnel to focus on investigating the node and improve the efficiency of root cause location. For example, if the abnormal node is node 0, and the topology sorting result indicates that the order of nodes in the corresponding connected component is node 1, node 5, node 0, node 3, then node 1 is determined as the root cause node and marked to obtain the root cause location result of the anomaly (root cause path: node 1 (root cause node), node 5, node 0, node 3).

[0124] Steps 404 and 405 described above are the process of determining the root cause localization result of the multi-node system based on the second network topology diagram and the abnormal nodes. After steps 401 to 405, when the server detects the presence of abnormal nodes in the multi-node system, it uses a spatiotemporal graph neural network to obtain the real network topology during the abnormal period of the multi-node system, thereby achieving root cause localization and improving the accuracy of the root cause localization result.

[0125] In the above embodiments, the server performs anomaly root cause localization based on a second network topology graph. In some embodiments, the server can further trim the second network topology graph to obtain a third network topology graph that better reflects the actual situation, thereby further improving the accuracy of the anomaly root cause localization results. Illustratively, the server adjusts the second network topology graph based on the node types of multiple nodes to ensure that the communication connections between nodes meet the operational requirements of a multi-node system, resulting in a third network topology graph with multiple nodes. Based on the connected components where the abnormal node is located in the third network topology graph, the anomaly root cause localization result of the multi-node system is determined. Adjusting the second network topology graph refers to deleting obviously meaningless edges based on the node types of multiple nodes. For example, taking a data center as an example in a multi-node system, the second network topology graph may show instances where upper-layer application nodes and storage nodes are directly connected, even though they are not physically directly connected. Therefore, this method can improve the accuracy of the network topology graph, thereby further improving the accuracy of the anomaly root cause localization results. It should be noted that the process by which the server determines the root cause of the anomaly based on the third network topology map is the same as step 404 above, that is, referring to... Figure 8 , Figure 8 This is a schematic diagram of a third network topology provided in an embodiment of this application, such as... Figure 8 As shown, after obtaining the second network topology, the server adjusts the second network topology based on the node types of multiple nodes to make the communication connection relationship between nodes meet the operating requirements of the multi-node system, thus obtaining the third network topology. Then, based on the node types of multiple nodes, the server assigns directions to the edges in the third network topology to achieve anomaly root cause localization.

[0126] Furthermore, in the above embodiments, any two nodes in the first network topology graph are connected; that is, the first network topology graph can be understood as a hypothetical complete graph. In other embodiments, the first network topology graph is a topology graph adjusted based on an initial network topology graph. Specifically, the server obtains the initial network topology graph of the multiple nodes and adjusts it based on the node types of the multiple nodes to ensure that the communication connections between the nodes meet the operational requirements of the multi-node system, thus obtaining the first network topology graph. In this initial network topology graph, any two nodes are connected. This method reduces the number of edges in the first network topology graph, thereby reducing the server's data processing load and improving the efficiency of anomaly root cause localization.

[0127] Using the above method, when an abnormal node is detected in a multi-node system, a spatiotemporal graph neural network is invoked to process the performance data of multiple nodes in the system and the first network topology graph. This process yields an attention weight matrix for each node, reconstructing the true network topology of the multi-node system and enabling root cause localization, thus improving the accuracy of the root cause localization results. In this process, because the attention weight matrix indicates the similarity between the performance data of any two nodes, it can serve as the basis for adjusting the first network topology graph, preserving the edges between nodes with similar performance data. Since nodes with similar performance data are usually on a communication connection path, the root cause node or root cause path can be quickly found based on the connected component where the abnormal node is located, improving the accuracy of the root cause localization results.

[0128] The following is for reference. Figure 9 Taking a multi-node system as an example, this paper illustrates the above-mentioned method for locating the root cause of anomalies in a multi-node system. Figure 9 This is a schematic diagram of a data center anomaly root cause localization method provided in an embodiment of this application. Figure 9 As shown, when the server detects an abnormal node in the data center, it acquires performance data of multiple nodes in the data center and a first network topology map. For nodes for which no performance data is acquired, the performance data of these nodes is set to zero. Further, based on a spatiotemporal graph neural network, the performance data of these multiple nodes and the first network topology map are processed to obtain a second network topology map. Based on the connected components where the abnormal node is located in the second network topology map, the nodes in those connected components are topologically sorted to obtain a topology sorting result. Based on this topology sorting result, the root cause localization result of the anomaly is obtained. Of course, in some embodiments, the server can also directly output the second network topology map for manual investigation by relevant personnel; this is not limited.

[0129] As can be seen, using the above method, the server can promptly reconstruct the true network topology of the data center during the period of an anomaly when abnormal nodes are detected. This means automatically obtaining the dynamic network topology of the data center from a complex network, and even when performance data cannot be collected from some nodes, the above method can still be used to locate the root cause of the anomaly. Furthermore, because the number of nodes in a data center is enormous, its true network topology is often difficult to obtain. The above method can quickly reconstruct the true network topology of the data center based on the performance data of massive numbers of nodes, thereby achieving rapid location of the root cause of the anomaly. This not only improves the efficiency of root cause location in data center anomalies but also enhances the accuracy of the root cause location results.

[0130] Next, refer to Figure 10 , Figure 10This is a schematic diagram of a data center network topology provided in an embodiment of this application. Figure 10 As shown, taking a multi-node system as an example of a data center, the method provided in this application can utilize a spatiotemporal graph neural network to reconstruct the real network topology of the data center, providing technical support for anomaly root cause localization. In contrast, related technologies often encounter severe anomalies in upper-layer application nodes, leading to performance data acquisition failures, making root cause localization impossible. Furthermore, the method provided in this application, by obtaining the attention weight matrix of multiple nodes, can represent nodes with similar performance data as a connected component (or cluster them into a group), facilitating the localization of the root cause node based on the anomaly node, narrowing the localization range, and improving the accuracy of the anomaly root cause localization results.

[0131] In summary, this application provides a method for locating the root cause of anomalies in a multi-node system. When an abnormal node is detected in the multi-node system, a spatiotemporal graph neural network is used to process the performance data of multiple nodes in the system and a first network topology graph to obtain an attention weight matrix for multiple nodes. This adjusts the first network topology graph, preserving the connections between nodes with similar performance data, thus restoring the true network topology of the multi-node system and obtaining a second network topology graph, thereby achieving root cause localization. Since the true network topology of the multi-node system is restored based on the performance data of multiple nodes, and nodes with similar performance data are usually on the same communication connection path, the root cause of the anomaly can be quickly located based on the abnormal node and the second network topology graph, improving the accuracy of the root cause localization results.

[0132] Figure 11 This is a schematic diagram of a multi-node system anomaly root cause localization device provided in an embodiment of this application. This multi-node system anomaly root cause localization device can be implemented using software, hardware, or a combination of both to achieve the aforementioned multi-node system anomaly root cause localization method. For example... Figure 11 As shown, the multi-node system anomaly root cause localization device includes a processing module 1101, an adjustment module 1102, and a determination module 1103.

[0133] The processing module 1101 is used to process the performance data of multiple nodes in the multi-node system and a first network topology graph based on a spatiotemporal graph neural network when an abnormal node is detected in the multi-node system, to obtain the attention weight matrix of the multiple nodes. The spatiotemporal graph neural network is trained based on the historical performance data and historical network topology graph of the multiple nodes. The first network topology graph includes the multiple nodes and the edges between the nodes. The edges between the nodes indicate that there is a communication connection between the nodes. The attention weight matrix indicates the similarity between the performance data of any two nodes.

[0134] The adjustment module 1102 is used to adjust the first network topology based on the attention weight matrix to obtain a second network topology of the multiple nodes. The edges between nodes in the second network topology indicate that there is a communication connection between the nodes and that the similarity between the performance data of the nodes meets the target conditions.

[0135] The determination module 1103 is used to determine the root cause localization result of the anomaly in the multi-node system based on the second network topology and the abnormal node.

[0136] In some embodiments, the adjustment module 1102 is configured to:

[0137] Based on the weights of the first number of targets in the target row of the attention weight matrix, the target number of associated nodes of the target node is determined. The target row refers to any row in the attention weight matrix, and the weights in the target row are used to indicate the degree of similarity between the performance data of the target node and other nodes.

[0138] Delete the edges between the target node and other nodes except for the number of associated nodes of the target.

[0139] In some embodiments, the determining module 1103 includes:

[0140] The sorting unit is used to sort the nodes in the connected component based on the location of the abnormal node in the second network topology graph, and obtain the topology sorting result, which indicates the communication connection path of the node in the connected component in the multi-node system.

[0141] The determination unit is used to determine the root cause location result of the anomaly based on the topological sorting result.

[0142] In some embodiments, the sorting unit is used for:

[0143] Starting with the node with an in-degree of zero in the connected component, and based on the direction of the edges between the nodes in the connected component, traverse the nodes in the connected component to obtain the topological sorting result.

[0144] In some embodiments, the determining unit is used for at least one of the following:

[0145] The node that ranks first in the topological sorting result is identified as the root cause node, thus obtaining the root cause localization result of the anomaly.

[0146] The order of nodes in the topological sorting result is determined as the root cause path, thus obtaining the root cause localization result of the anomaly.

[0147] In some embodiments, the determining module 1103 includes:

[0148] The adjustment unit is used to adjust the second network topology based on the node types of the multiple nodes, so that the communication connection relationship between the nodes meets the operating requirements of the multi-node system, and obtains the third network topology of the multiple nodes.

[0149] The determination unit is used to determine the connected component in the third network topology graph where the abnormal node is located, and to determine the root cause localization result of the abnormality in the multi-node system.

[0150] In some embodiments, the apparatus further includes an acquisition module for:

[0151] Obtain the initial network topology of the multiple nodes, in which any two nodes are connected.

[0152] Based on the node types of these multiple nodes, the initial network topology is adjusted so that the communication connections between the nodes meet the operational requirements of the multi-node system, thus obtaining the first network topology.

[0153] In some embodiments, the device further includes a configuration module for:

[0154] If the performance data of the first node is not obtained, the performance data of the first node will be configured as the default performance data. The first node can be any one of the multiple nodes.

[0155] In some embodiments, the processing module 1101 is configured to:

[0156] The performance data of these multiple nodes and the first network topology graph are input into the spatiotemporal graph neural network.

[0157] Based on the spatial attention layer in the spatiotemporal graph neural network, the spatial attention weight matrix of the multiple nodes is obtained. The spatial attention weight matrix indicates the degree of spatial similarity of the performance data of any two nodes.

[0158] Based on the temporal attention layer in the spatiotemporal graph neural network, the temporal attention weight matrix of the multiple nodes is obtained. The temporal attention weight matrix indicates the degree of temporal similarity of the performance data of any two nodes.

[0159] The attention weight matrix is ​​obtained by weighted summing of the spatial attention weight matrix and the temporal attention weight matrix.

[0160] The aforementioned multi-node system anomaly root cause localization device, upon detecting an abnormal node in the multi-node system, processes the performance data of multiple nodes and a first network topology graph using a spatiotemporal graph neural network. This process yields attention weight matrices for multiple nodes, thereby adjusting the first network topology graph. Edges between nodes with similar performance data are retained, restoring the true network topology of the multi-node system and obtaining a second network topology graph, thus achieving anomaly root cause localization. Since the true network topology of the multi-node system is restored based on the performance data of multiple nodes, and nodes with similar performance data are typically on the same communication connection path, the root cause of the anomaly can be quickly located based on the abnormal node and the second network topology graph, improving the accuracy of the anomaly root cause localization results.

[0161] It should be noted that the multi-node system anomaly root cause localization device provided in the above embodiments is only illustrated by the division of the above functional modules when performing multi-node system anomaly root cause localization. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the multi-node system anomaly root cause localization device and the multi-node system anomaly root cause localization method embodiments belong to the same concept, and the specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0162] It should be noted that all information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this application have been authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the performance data involved in this application were obtained with full authorization.

[0163] In this application, the terms "first," "second," etc., are used to distinguish identical or similar items with substantially the same function. It should be understood that there is no logical or temporal dependency between "first," "second," and "nth," nor does it limit the quantity or execution order. It should also be understood that although the following description uses the terms "first," "second," etc., to describe various elements, these elements should not be limited by the terms. These terms are merely used to distinguish one element from another. For example, without departing from the various examples described, a first node can be referred to as a second node, and similarly, a second node can be referred to as a first node. Both first and second nodes can be nodes, and in some cases, they can be separate and distinct nodes.

[0164] In this application, the term "at least one" means one or more, and the term "multiple" means two or more. For example, multiple nodes means two or more nodes.

[0165] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0166] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, in the form of program structure information. This program structure information includes one or more program instructions. When these program instructions are loaded and executed on a computing device, the processes or functions according to the embodiments of this application are generated, in whole or in part.

[0167] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0168] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A method for locating the root cause of anomalies in a multi-node system, characterized in that, The method includes: When an abnormal node is detected in a multi-node system, the performance data of multiple nodes in the multi-node system and a first network topology graph are processed based on a spatiotemporal graph neural network to obtain the attention weight matrix of the multiple nodes. The spatiotemporal graph neural network is trained based on the historical performance data and historical network topology graph of the multiple nodes. The first network topology graph includes the multiple nodes and the edges between the nodes. The edges between the nodes indicate that there is a communication connection between the nodes. The attention weight matrix indicates the similarity between the performance data of any two nodes. Based on the attention weight matrix, the first network topology graph is adjusted to obtain a second network topology graph of the plurality of nodes. The edges between nodes in the second network topology graph indicate that there is a communication connection between the nodes and that the similarity between the performance data of the nodes meets the target conditions. Based on the connected component where the abnormal node is located in the second network topology graph, the root cause localization result of the anomaly in the multi-node system is determined.

2. The method according to claim 1, characterized in that, The step of adjusting the first network topology graph based on the attention weight matrix to obtain the second network topology graph of the plurality of nodes includes: Based on the weights of the target nodes in the target row of the attention weight matrix, the target number of associated nodes of the target node is determined. The target row refers to any row in the attention weight matrix, and the weights in the target row are used to indicate the degree of similarity between the performance data of the target node and other nodes. Delete the edges between the target node and other nodes except for the target number of associated nodes.

3. The method according to claim 1 or 2, characterized in that, The step of determining the root cause localization result of the multi-node system based on the connected component where the abnormal node is located in the second network topology graph includes: Based on the connected component where the abnormal node is located in the second network topology graph, the nodes in the connected component are topologically sorted to obtain a topology sorting result. The topology sorting result indicates the communication connection path of the nodes in the connected component in the multi-node system. Based on the topological sorting results, the abnormal root cause localization results are determined.

4. The method according to claim 3, characterized in that, The step of performing topological sorting on the nodes in the connected component where the abnormal node is located in the second network topology graph, to obtain the topological sorting result, includes: Starting with the node with an in-degree of zero in the connected component, the nodes in the connected component are traversed based on the direction of the edges between the nodes in the connected component to obtain the topological sorting result.

5. The method according to claim 3, characterized in that, The determination of the anomaly root cause localization result based on the topological sorting result includes at least one of the following: The node that ranks first in the topological sorting result is identified as the root cause node, and the abnormal root cause localization result is obtained. The order of nodes in the topological sorting result is determined as the root cause path to obtain the abnormal root cause localization result.

6. The method according to claim 1 or 2, characterized in that, The step of determining the root cause localization result of the multi-node system based on the connected component where the abnormal node is located in the second network topology graph includes: Based on the node types of the multiple nodes, the second network topology is adjusted so that the communication connection relationship between the nodes meets the operating requirements of the multi-node system, thereby obtaining a third network topology of the multiple nodes. Based on the connected component where the abnormal node is located in the third network topology graph, the root cause localization result of the anomaly in the multi-node system is determined.

7. The method according to any one of claims 1 to 6, characterized in that, The method further includes: Obtain an initial network topology diagram of the plurality of nodes, wherein any two nodes in the initial network topology diagram are connected; Based on the node types of the multiple nodes, the initial network topology is adjusted so that the communication connections between the nodes meet the operational requirements of the multi-node system, thus obtaining the first network topology.

8. The method according to any one of claims 1 to 7, characterized in that, The method further includes: If the performance data of the first node is not obtained, the performance data of the first node will be configured as the default performance data. The first node can be any one of the plurality of nodes.

9. The method according to any one of claims 1 to 8, characterized in that, The spatiotemporal graph neural network-based method processes the performance data of multiple nodes in the multi-node system and the first network topology graph to obtain the attention weight matrix of the multiple nodes, including: The performance data of the multiple nodes and the first network topology graph are input into the spatiotemporal graph neural network; Based on the spatial attention layer in the spatiotemporal graph neural network, the spatial attention weight matrix of the multiple nodes is obtained, and the spatial attention weight matrix indicates the degree of spatial similarity of the performance data of any two nodes. Based on the temporal attention layer in the spatiotemporal graph neural network, the temporal attention weight matrix of the multiple nodes is obtained, and the temporal attention weight matrix indicates the degree of temporal similarity of the performance data of any two nodes. The attention weight matrix is ​​obtained by weighted summation of the spatial attention weight matrix and the temporal attention weight matrix.

10. A multi-node system anomaly root cause localization device, characterized in that, The device includes: The processing module is used to process the performance data of multiple nodes in the multi-node system and a first network topology graph based on a spatiotemporal graph neural network when an abnormal node is detected in the multi-node system, to obtain the attention weight matrix of the multiple nodes. The spatiotemporal graph neural network is trained based on the historical performance data and historical network topology graph of the multiple nodes. The first network topology graph includes the multiple nodes and the edges between the nodes. The edges between the nodes indicate that there is a communication connection between the nodes. The attention weight matrix indicates the similarity between the performance data of any two nodes. An adjustment module is used to adjust the first network topology graph based on the attention weight matrix to obtain a second network topology graph of the plurality of nodes. The edges between nodes in the second network topology graph indicate that there is a communication connection between the nodes and that the similarity between the performance data of the nodes meets the target conditions. The determination module is used to determine the root cause localization result of the anomaly in the multi-node system based on the connected component where the abnormal node is located in the second network topology graph.

11. The apparatus according to claim 10, characterized in that, The adjustment module is used for: Based on the weights of the target nodes in the target row of the attention weight matrix, the target number of associated nodes of the target node is determined. The target row refers to any row in the attention weight matrix, and the weights in the target row are used to indicate the degree of similarity between the performance data of the target node and other nodes. Delete the edges between the target node and other nodes except for the target number of associated nodes.

12. The apparatus according to claim 10 or 11, characterized in that, The determining module includes: The sorting unit is used to perform topological sorting on the nodes in the connected component based on the location of the abnormal node in the second network topology graph, and obtain a topological sorting result, wherein the topological sorting result indicates the communication connection path of the nodes in the connected component in the multi-node system. The determining unit is used to determine the abnormal root cause localization result based on the topological sorting result.

13. The apparatus according to claim 12, characterized in that, The sorting unit is used for: Starting with the node with an in-degree of zero in the connected component, the nodes in the connected component are traversed based on the direction of the edges between the nodes in the connected component to obtain the topological sorting result.

14. The apparatus according to claim 12, characterized in that, The determining unit is used for at least one of the following: The node that ranks first in the topological sorting result is identified as the root cause node, and the abnormal root cause localization result is obtained. The order of nodes in the topological sorting result is determined as the root cause path to obtain the abnormal root cause localization result.

15. The apparatus according to claim 11 or 12, characterized in that, The determining module includes: An adjustment unit is used to adjust the second network topology based on the node types of the plurality of nodes, so that the communication connection relationship between the nodes meets the operating requirements of the multi-node system, thereby obtaining a third network topology of the plurality of nodes. The determining unit is used to determine the root cause localization result of the anomaly in the multi-node system based on the connected component where the abnormal node is located in the third network topology graph.

16. The apparatus according to any one of claims 10 to 15, characterized in that, The device further includes an acquisition module for: Obtain an initial network topology diagram of the plurality of nodes, wherein any two nodes in the initial network topology diagram are connected; Based on the node types of the multiple nodes, the initial network topology is adjusted so that the communication connections between the nodes meet the operational requirements of the multi-node system, thus obtaining the first network topology.

17. The apparatus according to any one of claims 10 to 16, characterized in that, The device further includes a configuration module for: If the performance data of the first node is not obtained, the performance data of the first node will be configured as the default performance data. The first node can be any one of the plurality of nodes.

18. The apparatus according to any one of claims 10 to 17, characterized in that, The processing module is used for: The performance data of the multiple nodes and the first network topology graph are input into the spatiotemporal graph neural network; Based on the spatial attention layer in the spatiotemporal graph neural network, the spatial attention weight matrix of the multiple nodes is obtained, and the spatial attention weight matrix indicates the degree of spatial similarity of the performance data of any two nodes. Based on the temporal attention layer in the spatiotemporal graph neural network, the temporal attention weight matrix of the multiple nodes is obtained, and the temporal attention weight matrix indicates the degree of temporal similarity of the performance data of any two nodes. The attention weight matrix is ​​obtained by weighted summation of the spatial attention weight matrix and the temporal attention weight matrix.

19. A computing device, characterized in that, The computing device includes a processor and a memory, the memory being used to store at least one piece of program code, the at least one piece of program code being loaded by the processor and executed as described in any one of claims 1 to 9, the multi-node system anomaly root cause localization method.

20. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store at least one piece of program code, which is used to execute the multi-node system anomaly root cause localization method as described in any one of claims 1 to 9.

21. A computer program product, characterized in that, When the computer program product is run on a computing device, the computing device performs the multi-node system anomaly root cause localization method as described in any one of claims 1 to 9.

Images