Method, apparatus, device, storage medium and product for determining a faulty node

Abstract

The present application provides a method, apparatus, device, storage medium and product for determining a faulty node, belonging to the field of Internet technology. The method includes: obtaining at least one transmission failure event during data processing by a computing cluster, where the computing cluster includes a plurality of computing nodes and a plurality of data exchange nodes; obtaining the fault tracing information of each of the at least one transmission failure event, where the fault tracing information includes the node information of the sending node and the receiving node of the data stream indicated by the transmission failure event; determining at least one candidate data exchange node for each of the at least one transmission failure event from the computing cluster based on the fault tracing information of each of the at least one transmission failure event; and obtaining the faulty node in the computing cluster based on the sending node, receiving node and candidate data exchange node of each of the at least one transmission failure event. This method improves the efficiency of determining the faulty node while ensuring the accuracy of the determined faulty node.

Description

[0001] This application claims the priority of a Chinese patent application with the application number 202410117741.1 and the invention title "Method, apparatus, device, storage medium and product for determining a faulty node" filed on January 26, 2024, the entire content of which is incorporated herein by reference. Technical Field

[0002] This application relates to the field of Internet technologies, and particularly to a method, apparatus, device, storage medium and product for determining a faulty node. Background Art

[0003] With the development of large model technologies, the scale of clusters used for large model training and computing is getting larger and larger. A computing cluster includes multiple computing nodes and data exchange nodes that provide communication functions between the computing nodes. The computing nodes are, for example, servers, and the data exchange nodes are, for example, network devices. A fault in a computing node or a data exchange node in the computing cluster will cause the computing to be interrupted. Due to the extremely large scale of the computing cluster, it is very difficult to quickly locate the faulty node when an interruption occurs.

[0004] In related technologies, generally, operation and maintenance personnel use the dichotomy method for fault location, that is, by continuously dividing a large cluster into two parts and restarting the task test until the final faulty node is found. However, this method takes a long time, and the time consumption increases as the scale of the cluster increases. Usually, it takes several hours to locate the faulty node, which is time-consuming and laborious. Summary of the Invention

[0005] Embodiments of this application provide a method, apparatus, device, storage medium and product for determining a faulty node. This method improves the efficiency of determining the faulty node while ensuring the accuracy of the determined faulty node. The technical solution is as follows:

[0006] On the one hand, a method for determining a faulty node is provided. The method includes:

[0007] Obtain at least one transmission failure event during the data processing of a computing cluster. The computing cluster includes multiple computing nodes and multiple data exchange nodes. The multiple computing nodes are used to execute computing tasks, and the multiple data exchange nodes are used to transmit data streams between the multiple computing nodes;

[0008] Obtain the fault tracing information of each of the at least one transmission failure event. The fault tracing information includes the node information of the sending node and the receiving node of the data stream indicated by the transmission failure event;

[0009] Determine the respective candidate data exchange nodes of the at least one transmission failure event from the computing cluster based on the respective failure traceability information of the at least one transmission failure event, where the candidate data exchange node is a data exchange node that transmits the data stream indicated by the transmission failure event between the sending node and the receiving node;

[0010] Obtain the faulty nodes in the computing cluster based on the respective sending nodes, receiving nodes, and candidate data exchange nodes of the at least one transmission failure event.

[0011] On the other hand, a method for determining a faulty node is provided, and the method includes:

[0012] In response to the failure of the data stream transmission corresponding to the computing node, generate a transmission failure event. The computing cluster includes multiple computing nodes and multiple data exchange nodes, where the multiple computing nodes are used to execute computing tasks, and the multiple data exchange nodes are used to transmit data streams between the multiple computing nodes;

[0013] Obtain the failure traceability information of the transmission failure event, where the failure traceability information includes the node information of the sending node and the receiving node of the data stream indicated by the transmission failure event;

[0014] Send the failure traceability information to the management node of the computing cluster. The management node is used to determine the candidate data exchange node of the transmission failure event, and determine the faulty node in the computing cluster based on the sending node, the receiving node, and the candidate data exchange node. The candidate data exchange node is a data exchange node that transmits the data stream indicated by the transmission failure event between the sending node and the receiving node.

[0015] On the other hand, a device for determining a faulty node is provided, and the device includes:

[0016] An acquisition module, configured to acquire at least one transmission failure event during the data processing of the computing cluster. The computing cluster includes multiple computing nodes and multiple data exchange nodes, where the multiple computing nodes are used to execute computing tasks, and the multiple data exchange nodes are used to transmit data streams between the multiple computing nodes;

[0017] The acquisition module is further configured to acquire the respective failure traceability information of the at least one transmission failure event, where the failure traceability information includes the node information of the sending node and the receiving node of the data stream indicated by the transmission failure event;

[0018] A determination module, configured to determine, based on the failure tracing information of each of the at least one transmission failure event, the candidate data exchange nodes corresponding to each of the at least one transmission failure event from the computing cluster, where the candidate data exchange node is a data exchange node for transmitting the data stream indicated by the transmission failure event between the sending node and the receiving node;

[0019] The determination module is further configured to obtain the faulty nodes in the computing cluster based on the sending node, the receiving node, and the candidate data exchange nodes corresponding to each of the at least one transmission failure event.

[0020] In some embodiments, the determination module is configured to:

[0021] Sort the sending node, the receiving node, and the candidate data exchange nodes corresponding to each of the at least one transmission failure event based on the total number of occurrences of each of the sending node, the receiving node, and the candidate data exchange nodes in the at least one transmission failure event, and the order of any node is positively correlated with the total number of occurrences of the node;

[0022] Use the nodes ranked in the preset first positions as the faulty nodes.

[0023] In some embodiments, the obtaining module is configured to:

[0024] For any transmission failure event, obtain the failure tracing information of the transmission failure event from the failure log of the target computing node, where the target computing node is the computing node that has the transmission failure event among the multiple computing nodes, the failure log is the log generated by the target computing node for the transmission failure event, and the failure log includes the failure tracing information.

[0025] In some embodiments, the determination module is configured to:

[0026] For any transmission failure event, determine the first data exchange node corresponding to the transmission failure event based on the failure tracing information of the transmission failure event, where the first data exchange node is the data exchange node connected to the sending node indicated by the failure tracing information;

[0027] Determine multiple first equivalent paths based on the next-hop address of the first data exchange node, where both ends of each first equivalent path are the first data exchange node and a second data exchange node indicated by the next-hop address, each first equivalent path is a path between the first data exchange node and a second data exchange node, and the bandwidths of the multiple first equivalent paths are the same;

[0028] For any second data exchange node, determine multiple second equivalent paths based on the next-hop address of the second data exchange node. Each end of each second equivalent path is the second data exchange node and a third data exchange node indicated by the next-hop address of the second data exchange node;

[0029] Based on the multiple first equivalent paths and the multiple second equivalent paths, determine a target path between the sending node and the receiving node indicated by the fault tracing information, and use the data exchange nodes on the target path as candidate data exchange nodes for the transmission failure event. Each second equivalent path is a path between the second data exchange node and a third data exchange node, and the multiple second equivalent paths have the same bandwidth.

[0030] In some embodiments, the determining module is configured to:

[0031] For a transmission failure event of transmitting a data stream through a two-level network, combine the multiple first equivalent paths and the multiple second equivalent paths to obtain multiple first combined paths. Each end of each first combined path is a first data exchange node and a third data exchange node;

[0032] Determine the target path from the multiple first combined paths. The third data exchange node in the target path is the data exchange node connected to the receiving node indicated by the fault tracing information.

[0033] In some embodiments, the determining module is configured to:

[0034] For a transmission failure event of transmitting a data stream through a three-level network, for any third data exchange node, determine multiple third equivalent paths based on the next-hop address of the third data exchange node. Each end of each third equivalent path is the third data exchange node and a fourth data exchange node indicated by the next-hop address of the third data exchange node. Each third equivalent path is a path between the third data exchange node and a fourth data exchange node, and the multiple third equivalent paths have the same bandwidth;

[0035] For any fourth data exchange node, determine multiple fourth equivalent paths based on the next-hop address of the fourth data exchange node. Each end of each fourth equivalent path is the fourth data exchange node and a fifth data exchange node indicated by the next-hop address of the fourth data exchange node. Each fourth equivalent path is a path between the fourth data exchange node and a fifth data exchange node, and the multiple fourth equivalent paths have the same bandwidth;

[0036] Based on the multiple first equivalent paths, the multiple second equivalent paths, the multiple third equivalent paths, and the multiple fourth equivalent paths, determine the target path between the sending node and the receiving node indicated by the fault tracing information.

[0037] In some embodiments, the determining module is configured to:

[0038] Combine the multiple first equivalent paths, the multiple second equivalent paths, the multiple third equivalent paths, and the multiple fourth equivalent paths to obtain multiple second combined paths, with each end of each second combined path being a first data exchange node and a fifth data exchange node respectively;

[0039] Determine multiple candidate paths from the multiple second combined paths, where the fifth data exchange nodes in the multiple candidate paths are data exchange nodes connected to the receiving node indicated by the fault tracing information;

[0040] Determine the target path from the multiple candidate paths, where the target path is the candidate path with the fewest data exchange nodes among the multiple candidate paths.

[0041] On the other hand, a device for determining a faulty node is provided, and the device includes:

[0042] A generating module, configured to generate a transmission failure event in response to a failure in the data stream transmission corresponding to a computing node. The computing cluster includes multiple computing nodes and multiple data exchange nodes, where the multiple computing nodes are used to execute computing tasks, and the multiple data exchange nodes are used to transmit data streams between the multiple computing nodes;

[0043] An obtaining module, configured to obtain the fault tracing information of the transmission failure event, where the fault tracing information includes the node information of the sending node and the receiving node of the data stream indicated by the transmission failure event;

[0044] A sending module, configured to send the fault tracing information to the management node of the computing cluster. The management node is configured to determine candidate data exchange nodes for the transmission failure event, and determine the faulty node in the computing cluster based on the sending node, the receiving node, and the candidate data exchange nodes. The candidate data exchange nodes are data exchange nodes that transmit the data stream indicated by the transmission failure event between the sending node and the receiving node.

[0045] In some embodiments, the device further includes an establishing module, configured to establish a data stream flow table during the initialization process of the computing cluster. The data stream flow table includes the identification numbers of at least one data stream to be transmitted by the computing node and the corresponding fault tracing information of the data stream;

[0046] The obtaining module is configured to obtain, from the data flow table, fault tracing information that matches the identification number based on the identification number indicated by the transmission failure event.

[0047] In some embodiments, the sending module is configured to:

[0048] generate a fault log based on the fault tracing information of the transmission failure event, where the fault log includes the fault tracing information of the transmission failure event and the occurrence time of the transmission failure event;

[0049] send the fault log to the management node.

[0050] On the other hand, a computer device is provided. The computer device includes a processor and a memory. The memory is used to store at least one segment of program, and the at least one segment of program is loaded and executed by the processor to implement the method for determining a fault node in the embodiments of the present application.

[0051] On the other hand, a computer-readable storage medium is provided. At least one segment of program is stored in the computer-readable storage medium, and the at least one segment of program is loaded and executed by a processor to implement the method for determining a fault node in the embodiments of the present application.

[0052] On the other hand, a computer program product is provided. The computer program product includes at least one segment of program. The at least one segment of program is stored in a computer-readable storage medium. A processor of a computer device reads the at least one segment of program from the computer-readable storage medium, and the processor executes the at least one segment of program, so that the computer device executes the method for determining a fault node in any of the above implementation manners.

[0053] The embodiments of the present application provide a method for determining a fault node. For a transmission failure event in which any data flow transmission fails in a computing cluster, corresponding fault tracing information is obtained. Since the fault tracing information includes node information of the sending node and the receiving node of the data flow, the transmission path of the data flow can be restored based on the fault tracing information, that is, a candidate data exchange node for transmitting the data flow between the sending node and the receiving node can be found. Since the sending node, the receiving node, and the candidate data exchange node are all related to the data flow with transmission failure, these nodes may all be fault nodes. And because multiple nodes communicate with each other, a fault node may cause multiple transmission failure events. Then, by comprehensively considering these nodes in multiple transmission failure events, an accurate fault node can be found. Moreover, this method determines the fault node only from the nodes related to the data flow failure, rather than screening all nodes indiscriminately, which can significantly improve the efficiency of determining the fault node. That is, this method improves the efficiency of determining the fault node on the basis of ensuring the accuracy of the determined fault node. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0055] Figure 1 is a schematic diagram of an implementation environment provided by an embodiment of the present application;

[0056] Figure 2 is a flowchart of a method for determining a faulty node provided by an embodiment of the present application;

[0057] Figure 3 is a flowchart of another method for determining a faulty node provided by an embodiment of the present application;

[0058] Figure 4 is a flowchart of another method for determining a faulty node provided by an embodiment of the present application;

[0059] Figure 5 is a flowchart of another method for determining a faulty node provided by an embodiment of the present application;

[0060] Figure 6 is a flowchart of the operation of a computing node provided by an embodiment of the present application;

[0061] Figure 7 is a flowchart of the operation of another computing node provided by an embodiment of the present application;

[0062] Figure 8 is a schematic diagram of the aggregation of multiple fault logs provided by an embodiment of the present application;

[0063] Figure 9 is a schematic diagram of the aggregated information of transmission failure events provided by an embodiment of the present application;

[0064] Figure 10 is a schematic diagram of determining a target path provided by an embodiment of the present application;

[0065] Figure 11 is a schematic diagram of another method for determining a target path provided by an embodiment of the present application;

[0066] Figure 12 is a schematic diagram of the target path corresponding to a five-tuple provided by an embodiment of the present application;

[0067] Figure 13 is a schematic diagram of the reporting result of a faulty node provided by an embodiment of the present application;

[0068] Figure 14 It is a flowchart of another method for determining a faulty node provided by an embodiment of the present application;

[0069] Figure 15 It is a flowchart for processing a faulty node in a model training cluster provided by an embodiment of the present application;

[0070] Figure 16 It is a block diagram of a device for determining a faulty node provided by an embodiment of the present application;

[0071] Figure 17 It is a block diagram of another device for determining a faulty node provided by an embodiment of the present application;

[0072] Figure 18 It is a block diagram of a terminal provided by an embodiment of the present application;

[0073] Figure 19 It is a block diagram of a server provided by an embodiment of the present application. Detailed implementation manners

[0074] To make the objectives, technical solutions, and advantages of the present application clearer, the following will further describe the embodiments of the present application in detail with reference to the accompanying drawings.

[0075] In the present application, terms such as "first" and "second" are used to distinguish identical or similar items with basically the same functions and effects. It should be understood that there is no logical or temporal dependence relationship among "first", "second", and "nth", nor are the quantity and execution order limited.

[0076] In the present application, the term "at least one" means one or more, and the meaning of "a plurality" means two or more.

[0077] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data for analysis, stored data, displayed data, etc.), and signals involved in the present application are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data need to comply with the relevant laws, regulations, and standards of relevant countries and regions. For example, the fault tracing information involved in the present application is obtained under full authorization.

[0078] The following introduces the professional terms involved in the present application:

[0079] Artificial Intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a way similar to human intelligence. Artificial intelligence also studies the design principles and implementation methods of various intelligent machines, enabling the machines to have the functions of perception, reasoning, and decision-making. Artificial intelligence technology is an interdisciplinary subject that involves a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, pre-trained model technology, operation / interaction systems, mechatronics, etc. Among them, the pre-trained model, also known as the large model or the foundation model, can be widely applied to downstream tasks in various directions of artificial intelligence after fine-tuning. The artificial intelligence software technology mainly includes several major directions such as computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0080] Machine Learning (ML) is an interdisciplinary subject that involves multiple disciplines such as probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize the existing knowledge structure to continuously improve their own performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent, and its applications cover all fields of artificial intelligence. Machine learning and deep learning usually include technologies such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning from demonstration. The pre-trained model is the latest development result of deep learning, integrating the above technologies.

[0081] Remote Direct Memory Access (RDMA) technology is a direct memory access technology that transfers data directly from the memory of one computer to another without the intervention of the operating systems of both parties. This allows for high-throughput and low-latency network communication, especially suitable for use in large-scale parallel computer clusters. RDMA technology can significantly improve the data transfer speed, reduce network latency, and at the same time relieve the burden on the CPU (Central Processing Unit) and the operating system.

[0082] Next, the implementation environment involved in this application will be introduced:

[0083] The method for determining a faulty node provided by an embodiment of the present application can be executed by a computer device, which can be provided as a server or a terminal. The following introduces a schematic diagram of the implementation environment of the method for determining a faulty node provided by an embodiment of the present application.

[0084] See Figure 1 , Figure 1 FIG. is a schematic diagram of the implementation environment of a method for determining a faulty node provided by an embodiment of the present application. The implementation environment includes multiple computing nodes 101, multiple data exchange nodes 102, and a management node 103. The multiple computing nodes 101 and the multiple data exchange nodes 102 are used to form a computing cluster. The data exchange node 102 is used to transmit data streams between multiple computing nodes. The management node 103 is used to uniformly manage the nodes in the computing cluster. The computing node can be a terminal or a server. The data exchange node can be a network device. For example, the data exchange node can be a switch, a router, etc. The management node can be a terminal or a server. The multiple computing nodes 101, the multiple data exchange nodes 102, and the management node 103 can be directly or indirectly connected through wired or wireless communication methods, which are not limited in this application.

[0085] In an embodiment of the present application, the computing cluster is used for data processing to complete various computing tasks. The data processing in the computing cluster is also the joint processing of multiple computing nodes. There may be data dependencies between the computing nodes. Therefore, the computing nodes need to transmit data streams through the data exchange node. The data stream includes the data obtained after the computing nodes process the data, that is, the computing cluster transmits data in the form of data streams to complete the computing tasks.

[0086] In some embodiments, the computing cluster is used for model training. The multiple computing nodes are computing nodes participating in model training, and the multiple data exchange nodes are used to transmit data streams during the model training process between the multiple computing nodes. In other embodiments, the computing cluster is used for model calculation, that is, for providing computing services for tasks such as prediction or classification of a model. The multiple data exchange nodes are used to transmit data streams during the model calculation process between the multiple computing nodes.

[0087] In some embodiments, the computing cluster is used for medical digital processing. For example, the medical data to be processed is distributed to multiple computing nodes for processing, and after each computing node processes its own medical data, it can transmit data streams with other computing nodes through the data exchange node.

[0088] In some embodiments, the computing cluster is used for game data processing. For example, the game data to be processed is distributed to multiple computing nodes for processing, and after each computing node processes its own game data, it can transmit data streams with other computing nodes through the data exchange node.

[0089] The computing cluster in the embodiments of the present application can also be a cluster used for other computing tasks. For example, the computing cluster is used to process traffic data, seismic data, live broadcast data, etc. to complete corresponding computing tasks.

[0090] In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart voice interaction device, a smart home appliance, a vehicle-mounted terminal, an aircraft, a VR (Virtual Reality) device, an AR (Augmented Reality) device, etc., but is not limited thereto. In some embodiments, the server can be an independent server, or a server cluster or a distributed system composed of multiple servers, and can also be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. In some embodiments, the server mainly undertakes computing work, and the terminal undertakes secondary computing work; or, the server undertakes secondary computing services, and the terminal undertakes primary computing work; or, the server and the terminal adopt a distributed computing architecture for collaborative computing.

[0091] See Figure 2 , Figure 2 is a flowchart of a method for determining a faulty node provided by an embodiment of the present application. The execution subject of this method is the management node of the computing cluster, and the management node can be a terminal or a server. This method includes the following steps.

[0092] 201. The management node obtains at least one transmission failure event during the data processing of the computing cluster. The computing cluster includes multiple computing nodes and multiple data exchange nodes. The multiple computing nodes are used to execute computing tasks, and the multiple data exchange nodes are used to transmit data streams between the multiple computing nodes.

[0093] In the embodiments of the present application, the multiple computing nodes can be terminals or servers. Taking the computing node as a server as an example, for example, the computing node is a GPU (Graphics Processing Unit) server. Optionally, the data exchange node can be a network device, such as a switch, an optical module, a router, a cable, etc. Optionally, a network card is embedded in the computing node, and the computing node sends the data stream to the data exchange node through the network card, and then the data exchange node sends it to another computing node.

[0094] In the embodiments of the present application, a transmission failure event (ERRCQE event) is used to indicate that the data stream transmission fails. The node that causes the data stream transmission failure may be a computing node or a data exchange node, that is, any node on the data stream transmission path may be a faulty node.

[0095] In the embodiments of the present application, the data stream is transmitted between two computing nodes, and the two computing nodes can transmit the data stream through one or more data exchange nodes.

[0096] In the embodiments of the present application, the transmission failure event occurs on a computing node, and the computing node may be the sending node of the data stream with the transmission failure or the receiving node of the data stream with the transmission failure.

[0097] 202. The management node obtains the fault tracing information of each of at least one transmission failure event, and the fault tracing information includes the node information of the sending node and the receiving node of the data stream indicated by the transmission failure event.

[0098] In the embodiments of the present application, the node information is used to restore the data stream transmission path between the sending node and the receiving node, and the data exchange node on the data stream transmission path is also the data exchange node that transmits the data stream indicated by the transmission failure event between the two computing nodes. The node information may include the IP address of the sending node, the port number of the sending node, the IP address of the receiving node, the port number of the receiving node, and the transport layer protocol between the two computing nodes. Optionally, the node information is a five-tuple.

[0099] In some embodiments, the transport layer protocols and the port numbers of the receiving nodes used by multiple data streams transmitted in the computing cluster are the same, and only the IP address of the sending node, the port number of the sending node, and the IP address of the receiving node are different. Therefore, the fault tracing information may be a triple that only includes the IP address of the sending node, the port number of the sending node, and the IP address of the receiving node, so as to reduce the data transmission volume and save resources.

[0100] 203. The management node determines at least one candidate data exchange node of each of at least one transmission failure event from the computing cluster based on the fault tracing information of each of at least one transmission failure event, and the candidate data exchange node is the data exchange node that transmits the data stream indicated by the transmission failure event between the sending node and the receiving node.

[0101] In the embodiments of the present application, the data stream transmission path between the sending node and the receiving node can be restored based on the fault tracing information to obtain the data exchange node that transmits the data stream indicated by the transmission failure event between the sending node and the receiving node.

[0102] 204. The management node obtains the faulty nodes in the computing cluster based on the sending node, receiving node, and candidate data exchange nodes of each of at least one transmission failure event.

[0103] In the embodiments of the present application, since multiple computing nodes and multiple data exchange nodes communicate with each other, for a faulty node, it will cause all data streams associated with it to fail in transmission, and thus the faulty node is associated with multiple transmission failure events. Therefore, if there are multiple transmission failure events, by combining the nodes in multiple transmission failure events, the faulty node can be found. If there is one transmission failure event, the nodes in this transmission failure event are screened one by one to find the faulty node; or all the nodes in this transmission failure event are regarded as faulty nodes to first remove the faulty nodes and restore the normal operation of the computing cluster, and then the nodes in this transmission failure event are screened one by one later.

[0104] The embodiments of the present application provide a method for determining a faulty node. For a transmission failure event in which any data stream in the computing cluster fails in transmission, the corresponding fault tracing information is obtained. Since the fault tracing information includes the node information of the sending node and receiving node of the data stream, the transmission path of the data stream can be restored based on this fault tracing information, that is, the candidate data exchange nodes that transmit the data stream between the sending node and the receiving node can be found. Since the sending node, receiving node, and candidate data exchange nodes are all related to the data stream that fails in transmission, these nodes may all be faulty nodes. And because multiple nodes communicate with each other, one faulty node may cause multiple transmission failure events. Therefore, by combining these nodes in multiple transmission failure events, the accurate faulty node can be found. Moreover, this method determines the faulty node only from the nodes related to the data stream failure, rather than screening all nodes indiscriminately, which can significantly improve the efficiency of determining the faulty node. That is, this method improves the efficiency of determining the faulty node on the basis of ensuring the accuracy of the determined faulty node.

[0105] The above Figure 2 is the basic process of the method for determining a faulty node. Next, based on Figure 3 the method for determining a faulty node will be further introduced. Refer to Figure 3 , Figure 3 which is a flowchart of a method for determining a faulty node provided by an embodiment of the present application. The execution subject of this method is any computing node in the computing cluster. In the embodiments of the present application, the computing node can be a terminal or a server. This method includes the following steps.

[0106] 301. In response to the failure of data stream transmission corresponding to a computing node, the computing node generates a transmission failure event. The computing cluster includes multiple computing nodes and multiple data exchange nodes. The multiple computing nodes are used to execute computing tasks, and the multiple data exchange nodes are used to transmit data streams between the multiple computing nodes.

[0107] In an embodiment of the present application, the transmission failure event is used to indicate the failure of data stream transmission. The computing node can be the sending node of the data stream or the receiving node of the data stream.

[0108] 302. The computing node obtains the fault tracing information of the transmission failure event. The fault tracing information includes the node information of the sending node and the receiving node of the data stream indicated by the transmission failure event.

[0109] In an embodiment of the present application, the fault tracing information is the same as that in step 202 and will not be elaborated here.

[0110] 303. The computing node sends the fault tracing information to the management node of the computing cluster. The management node is used to determine the candidate data exchange nodes of the transmission failure event, and determine the faulty nodes in the computing cluster based on the sending node, the receiving node, and the candidate data exchange nodes. The candidate data exchange nodes are the data exchange nodes that transmit the data stream indicated by the transmission failure event between the sending node and the receiving node.

[0111] In an embodiment of the present application, the computing node can actively send the fault tracing information to the management node, or send the fault tracing information to the management node in response to a data request from the management node.

[0112] In an embodiment of the present application, there can be multiple computing nodes where transmission failure events occur. And since each computing node can transmit multiple data streams, multiple transmission failure events can also occur on each computing node. Correspondingly, the management node obtains the fault tracing information of multiple transmission failure events of multiple computing nodes, obtains the candidate data exchange nodes of each transmission failure event, and then determines the faulty nodes in the computing cluster based on the sending node, the receiving node, and the candidate data exchange nodes of each transmission failure event.

[0113] The embodiment of the present application provides a method for determining a faulty node. For any transmission failure event where data stream transmission fails in a computing cluster, the corresponding failure tracing information is obtained. Since the failure tracing information includes the node information of the sending node and the receiving node of the data stream, the transmission path of the data stream can be restored based on this failure tracing information, that is, the candidate data exchange nodes that transmit the data stream between the sending node and the receiving node can be found. Since the sending node, the receiving node, and the candidate data exchange nodes are all related to the data stream with transmission failure, these nodes may all be faulty nodes. And because multiple nodes communicate with each other, a faulty node may cause multiple transmission failure events. Therefore, by comprehensively considering these nodes in multiple transmission failure events, the accurate faulty node can be found. Moreover, this method determines the faulty node only from the nodes related to the data stream failure, rather than screening all nodes indiscriminately, which can significantly improve the efficiency of determining the faulty node. That is, this method improves the efficiency of determining the faulty node on the basis of ensuring the accuracy of the determined faulty node.

[0114] Through the above Figure 2 and Figure 3 are the basic processes of the method for determining a faulty node. The following further introduces the process of determining a faulty node based on the Figure 4 embodiment. Refer to Figure 4 , Figure 4 is a flowchart of a method for determining a faulty node provided by an embodiment of the present application. The execution subject of this method is the management node of the computing cluster, and this method includes the following steps.

[0115] 401. The management node obtains at least one transmission failure event during the data processing of the computing cluster. The computing cluster includes multiple computing nodes and multiple data exchange nodes. The multiple computing nodes are used to execute computing tasks, and the multiple data exchange nodes are used to transmit data streams between the multiple computing nodes.

[0116] In the embodiment of the present application, for the transmission failure event Occurs at on the computing node, after the transmission failure event occurs, the computing node can actively alarm the transmission failure event to the management node. Correspondingly, the management node takes the computing node that alarms the transmission failure event as the computing node where the transmission failure event occurs. Or, the transmission failure event carries the node information of the corresponding computing node, and the management node takes the computing node indicated by the node information as the computing node where the transmission failure event occurs.

[0117] Alternatively, the management node actively detects whether there is a transmission failure event on each computing node to determine the computing node where the transmission failure event occurs. Further, the management node detects whether a failure log is generated on each computing node to determine whether a transmission failure event has occurred on the computing node, that is, the management node regards the computing node with a generated failure log as the computing node where the transmission failure event occurs.

[0118] It should be noted that the timing for determining the faulty node is generally the timing when the computing cluster interrupts data processing. The interruption of the computing cluster may be caused by the transmission failure of a single data stream or the transmission failures of multiple data streams. Correspondingly, at least one transmission failure event obtained by the management node includes any transmission failure event that occurred before the computing cluster interrupted data processing.

[0119] 402. For any transmission failure event, the management node obtains the fault tracing information of the transmission failure event from the failure log of the target computing node. The target computing node is the computing node where the transmission failure event occurs among multiple computing nodes. The failure log is the log generated by the target computing node for the transmission failure event. The failure log includes fault tracing information, and the fault tracing information includes the node information of the sending node and the receiving node of the data stream indicated by the transmission failure event.

[0120] In some embodiments, the transmission failure event occurs on a computing node. Any transmission failure event carries the node information of its corresponding computing node. The management node determines the target computing node where the transmission failure event occurs based on this node information, and then obtains the failure log from the target computing node. In this way, it is only necessary to obtain the failure log from the computing node where the transmission failure event occurs, without having to traverse all computing nodes, which improves efficiency and reduces resource waste.

[0121] In some other embodiments, the management node traverses all computing nodes to obtain the failure log to ensure the comprehensiveness of the obtained failure log. In some other embodiments, the management node directly traverses all computing nodes to obtain the failure log. When obtaining the failure log, the corresponding transmission failure event is also obtained, without having to separately execute the step of obtaining the transmission failure event. In this way, multiple pieces of information are obtained through one communication, which improves efficiency and reduces resource waste.

[0122] In some embodiments, the management node obtains the failure log of the transmission failure event through a log collection platform, and then aggregates the failure logs on multiple computing nodes to the management node for unified processing. Optionally, this log collection platform is called the ERRCQE log collection platform.

[0123] Since many computing nodes usually participate in data processing in a computing cluster during the data processing process, when a failure occurs, many computing nodes usually communicate with the failed node. All computing nodes that experience transmission failure events will record the fault tracing information of their respective transmission failure events. Correspondingly, this fault tracing information will be distributed among many computing nodes. Therefore, the management node performs real-time collection of logs through a log collection platform and aggregates the fault logs at time intervals according to the occurrence time, that is, arranges each fault log according to the occurrence time.

[0124] It should be noted that since multiple computing nodes and multiple data exchange nodes communicate with each other, a node failure may cause multiple data stream transmissions to fail. During the same period, it is highly likely that the transmission failures of multiple data streams are due to the same failed node, while the possibility that the transmission failures of different data streams in different periods are due to the same failed node is relatively low. Therefore, by arranging multiple fault logs based on the occurrence time, the fault logs can be processed by time period, the failed node in each time period can be found, and the accuracy of the found failed node is high.

[0125] For example, refer to Figure 8 , Figure 8 is a schematic diagram of the aggregation of multiple fault logs provided by an embodiment of the present application. Among them, the ERRCQE log collection platform collects the fault logs on multiple computing nodes and aggregates multiple fault logs according to the occurrence time. Among them, SIP, DIP, and SRCPORT respectively represent the IP address of the sending node, the IP address of the receiving node, and the port number of the sending node. ERRCQE represents a transmission failure event.

[0126] Again, for example, refer to Figure 9 , Figure 9 is a schematic diagram of the aggregation information of a transmission failure event provided by an embodiment of the present application. Among them, the fault logs of multiple transmission failure events at the same occurrence time are aggregated to obtain this aggregation information. This aggregation information includes the same occurrence period of multiple transmission failure events, the fault tracing information of multiple transmission failure events, and includes a link address for indicating this aggregation information. It should be noted that since the transmission failure event causes an alarm, the information included in the aggregation information is also various alarm information.

[0127] In the embodiment of the present application, the process of the computing node generating a fault log refers to the embodiments of Figure 3 or Figure 5 , which will not be elaborated here. Among them, the fault tracing information can be a five-tuple or a three-tuple. Further, the five-tuple is stored in the data stream flow table so that the computing node stores rich information for other purposes. And the management node obtains the three-tuple from the five-tuple and then determines the failed node based on the three-tuple.

[0128] In the embodiment of the present application, the process of obtaining the fault traceability information is implemented through the above step 402. This process can also be implemented through other optional implementation manners, which will not be elaborated here.

[0129] 403. For any transmission failure event, the management node determines a first data exchange node corresponding to the transmission failure event based on the fault traceability information of the transmission failure event, where the first data exchange node is the data exchange node connected to the sending node indicated by the fault traceability information.

[0130] It should be noted that for any data stream with a transmission failure, a transmission failure event occurs not only at the sending node of the data stream but also at the receiving node of the data stream. Correspondingly, for two transmission failure events corresponding to the same data stream with a transmission failure, their fault traceability information is the same. Therefore, before executing step 403, for any fault traceability information, the management node searches for fault traceability information with the same IP address based on the IP address of the sending node and the IP address of the receiving node in the fault traceability information. If found, duplicate removal processing is performed. If not found, the next fault traceability information is returned for processing.

[0131] In the embodiment of the present application, the first data exchange node is determined based on the IP address and port number of the sending node, etc. However, since the fault traceability information may be recorded incorrectly, correspondingly, the data exchange node indicated by the fault traceability information may not be included in the computing cluster. Therefore, if the corresponding data exchange node is not found based on the node information of the sending node, the next fault traceability information is processed and this fault traceability information is no longer processed. Similarly, if the corresponding data exchange node is not found based on the node information of the receiving node, this fault traceability information is also no longer processed, and the node information of the receiving node indicates the data exchange node connected to the receiving node.

[0132] 404. The management node determines multiple first equivalent paths based on the next-hop address of the first data exchange node. Both ends of each first equivalent path are the first data exchange node and a second data exchange node indicated by the next-hop address. Each first equivalent path is a path between the first data exchange node and a second data exchange node, and the bandwidths of the multiple first equivalent paths are the same.

[0133] In some embodiments, the computing cluster is preset with a routing table. For each data exchange node, the next-hop address of the data exchange node can be obtained from the routing table. The next-hop address is used to indicate multiple data exchange nodes that are adjacent to the data exchange node and form an equivalent path with the data exchange node. Optionally, the next-hop address is the next-hop BGP (Border Gateway Protocol) neighbor routing address.

[0134] Among them, the next-hop address of the first data exchange node can indicate multiple second data exchange nodes. Correspondingly, the first data exchange node can form multiple first equivalent paths (ECMP, Equal-cost multi-path routing) with the multiple second data exchange nodes, and the multiple first equivalent paths form an ECMP group.

[0135] Any data exchange node can establish a connection with an adjacent data exchange node, and this connection is also the path between the two data exchange nodes, which is used to transmit data streams between the two data exchange nodes.

[0136] Among them, each data exchange node can have paths with multiple adjacent data exchange nodes respectively, that is, the data exchange node can correspond to multiple paths. Among the multiple paths corresponding to the data exchange node, there are multiple paths with the same bandwidth. When the data exchange node transmits data streams with another data exchange node on these multiple paths respectively through these multiple paths with the same bandwidth, the transmission performances such as network delay and transmission duration corresponding to these multiple paths are the same. These multiple paths with the same bandwidth are also the multiple equivalent paths corresponding to the data exchange node.

[0137] Among them, the multiple first equivalent paths are the multiple equivalent paths corresponding to the first data exchange node. The starting points of the multiple first equivalent paths are all the first data exchange node, and the end points are different second data exchange nodes. Optionally, the multiple second data exchange nodes indicated by the next-hop address of the first data exchange node are the multiple data exchange nodes that form equivalent paths with the first data exchange node, and the multiple second data exchange nodes are adjacent to the first data exchange node.

[0138] In the embodiments of the present application, taking the data exchange node as a switch as an example for illustration, the first data exchange node is an access layer switch (Lan Access, LA), and the second data exchange node is a convergence layer switch (Lan Core, LC).

[0139] 405. For any second data exchange node, the management node determines multiple second equivalent paths based on the next-hop address of the second data exchange node. Both ends of each second equivalent path are respectively the second data exchange node and a third data exchange node indicated by the next-hop address of the second data exchange node. Each second equivalent path is a path between the second data exchange node and a third data exchange node, and the bandwidths of the multiple second equivalent paths are the same.

[0140] Among them, the next-hop address of the second data exchange node can indicate multiple third data exchange nodes. Correspondingly, the second data exchange node can form multiple second equivalent paths with multiple third data exchange nodes.

[0141] Among them, the multiple second equivalent paths are the equivalent paths corresponding to the second data exchange node. The starting points of the multiple second equivalent paths are all the second data exchange node, and the end points are different third data exchange nodes. Optionally, the multiple third data exchange nodes indicated by the next-hop address of the second data exchange node are the multiple data exchange nodes that form equivalent paths with the second data exchange node, and the multiple third data exchange nodes are adjacent to the second data exchange node.

[0142] 406. The management node determines the target path between the sending node and the receiving node indicated by the fault tracing information based on the multiple first equivalent paths and the multiple second equivalent paths, and uses the data exchange nodes on the target path as the candidate data exchange nodes for the transmission failure event.

[0143] In the embodiments of the present application, the data stream can be transmitted through a two-level network or a three-level network.

[0144] The two-level network means that the data stream is transmitted between the sending node and the receiving node through two levels of data exchange nodes. The sending node is connected to the data exchange nodes of the first level, the data exchange nodes of the first level are connected to the data exchange nodes of the second level, and the data exchange nodes of the second level are connected to the data exchange nodes of the first level to which the receiving node is connected. That is, the data exchange nodes of the first level are the data exchange nodes connected to the computing nodes. The data exchange nodes of the second level are the data exchange nodes connected to the data exchange nodes of the first level and are used to converge the data streams transmitted by the data exchange nodes of the first level. For example, if the data exchange node is a switch, the data exchange nodes of the first level are access layer switches, and the data exchange nodes of the second level are aggregation layer switches.

[0145] Three-level networking means that the data stream is transmitted between the sending node and the receiving node through data exchange nodes at three levels. That is, the sending node is connected to the data exchange node at the first level, the data exchange node at the first level is connected to the data exchange node at the second level, the data exchange node at the second level is connected to the data exchange node at the third level, the data exchange node at the third level is then connected to another data exchange node at the second level, and this data exchange node at the second level is then connected to the data exchange node at the first level to which the receiving node is connected. That is, the data exchange node at the first level is connected to the computing node, the data exchange node at the second level is connected to the data exchange node at the first level to aggregate the data stream transmitted by the data exchange node at the first level, and the data exchange node at the third level is connected to the data exchange node at the second level to aggregate the data stream transmitted by the data exchange node at the second level. For example, if the data exchange node is a switch, then the data exchange node at the first level is an access layer switch, the data exchange node at the second level is an aggregation layer switch, and the data exchange node at the third level is a core layer switch (Super GPU Lan Core, SGLC).

[0146] Correspondingly, the process by which the above management node determines the target path between the sending node and the receiving node indicated by the fault tracing information based on multiple first equivalent paths and multiple second equivalent paths includes the following two cases.

[0147] For a transmission failure event of transmitting the data stream through two-level networking, the management node combines multiple first equivalent paths and multiple second equivalent paths to obtain multiple first combined paths, with each end of each first combined path being a first data exchange node and a third data exchange node respectively; the target path is determined from the multiple first combined paths, and the third data exchange node in the target path is the data exchange node connected to the receiving node indicated by the fault tracing information.

[0148] Among them, after combining multiple first equivalent paths and multiple second equivalent paths, at this time, a structure of an undirected graph will be formed starting from the first data exchange node. The management node obtains the path from the first data exchange node to the data exchange node indicated by the fault tracing information from this undirected graph, and thus obtains the target path between the sending node and the receiving node. It should be noted that since multiple first equivalent paths and multiple second equivalent paths are all undirected paths, multiple first equivalent paths and multiple second equivalent paths may form detour paths, that is, each first combined path includes not only one first equivalent path and one second equivalent path. Therefore, in this case, a depth-first search (DFS) is performed on this undirected graph to find the shortest path from the first data exchange node to the data exchange node indicated by the fault tracing information.

[0149] Among them, the data exchange nodes on the target path include a first data exchange node connected to the sending node, a second data exchange node, and a third data exchange node connected to the receiving node.

[0150] In this embodiment, for the data stream transmitted through the two-level network formation, it passes through data exchange nodes at two different levels, and each level of data exchange nodes can communicate with multiple other data exchange nodes to generate a transmission path. In this embodiment, multiple equivalent paths of each data exchange node are respectively determined, and then the multiple equivalent paths are combined. In this way, various possible paths are taken into account, and the target path between the data exchange node connected to the sending node and the data exchange node connected to the receiving node can be found from them, improving the comprehensiveness of path search, avoiding omission, and then accurately finding the data exchange node between the sending node and the receiving node, improving the accuracy of the found path.

[0151] For the transmission failure event of the data stream transmitted through the three-level network formation, for any third data exchange node, the management node determines multiple third equivalent paths based on the next-hop address of the third data exchange node. Both ends of each third equivalent path are the third data exchange node and a fourth data exchange node indicated by the next-hop address of the third data exchange node. Each third equivalent path is the path between the third data exchange node and a fourth data exchange node, and the bandwidths of the multiple third equivalent paths are the same; for any fourth data exchange node, the management node determines multiple fourth equivalent paths based on the next-hop address of the fourth data exchange node. Both ends of each fourth equivalent path are the fourth data exchange node and a fifth data exchange node indicated by the next-hop address of the fourth data exchange node. Each fourth equivalent path is the path between the fourth data exchange node and a fifth data exchange node, and the bandwidths of the multiple fourth equivalent paths are the same; based on the multiple first equivalent paths, multiple second equivalent paths, multiple third equivalent paths, and multiple fourth equivalent paths, the target path between the sending node and the receiving node indicated by the fault tracing information is determined.

[0152] Among them, multiple third equivalent paths are equivalent paths corresponding to a third data exchange node. The starting points of multiple third equivalent paths are all the third data exchange node, and the end points are different fourth data exchange nodes. Optionally, multiple fourth data exchange nodes indicated by the next-hop address of the third data exchange node are multiple data exchange nodes that form equivalent paths with the third data exchange node, and multiple fourth data exchange nodes are adjacent to the third data exchange node. Multiple fourth equivalent paths are equivalent paths corresponding to a fourth data exchange node. The starting points of multiple fourth equivalent paths are all the fourth data exchange node, and the end points are different fifth data exchange nodes. Optionally, multiple fifth data exchange nodes indicated by the next-hop address of the fourth data exchange node are multiple data exchange nodes that form equivalent paths with the fourth data exchange node, and multiple fifth data exchange nodes are adjacent to the fourth data exchange node.

[0153] In some embodiments, the process of determining the target path between the sending node and the receiving node indicated by the fault tracing information based on multiple first equivalent paths, multiple second equivalent paths, multiple third equivalent paths, and multiple fourth equivalent paths includes the following steps.

[0154] The management node combines multiple first equivalent paths, multiple second equivalent paths, multiple third equivalent paths, and multiple fourth equivalent paths to obtain multiple second combined paths. Both ends of each second combined path are a first data exchange node and a fifth data exchange node respectively; multiple candidate paths are determined from multiple second combined paths, and the fifth data exchange nodes in multiple candidate paths are data exchange nodes connected to the receiving node indicated by the fault tracing information; the target path is determined from multiple candidate paths, and the target path is the candidate path with the fewest data exchange nodes among multiple candidate paths.

[0155] Among them, after combining multiple equivalent paths, at this time, a structure of an undirected graph will be formed starting from the first data exchange node. The management node obtains the path from the first data exchange node to the data exchange node indicated by the fault tracing information from this undirected graph to obtain the target path between the sending node and the receiving node. It should be noted that since multiple equivalent paths are all undirected paths, free combination of multiple equivalent paths may form a detour path, that is, the second combined path does not only include a first equivalent path, a second equivalent path, a third equivalent path, and a fourth equivalent path. Therefore, in this case, a depth-first traversal is performed on the undirected graph to find the shortest path from the first data exchange node to the data exchange node indicated by the fault tracing information.

[0156] Among them, the data exchange nodes on the target path include a first data exchange node connected to the sending node, a second data exchange node, a third data exchange node, a fourth data exchange node, and a fifth data exchange node connected to the receiving node.

[0157] In this embodiment, for the data stream transmitted through a three-level network architecture, it passes through data exchange nodes at three different levels, and each level of data exchange nodes can communicate with multiple other data exchange nodes to generate transmission paths. In this embodiment, multiple equivalent paths of each data exchange node are respectively determined, and then the multiple equivalent paths are combined. By taking into account various possible paths in this way, a path can be found between the data exchange node connected to the sending node and the data exchange node connected to the receiving node, improving the comprehensiveness of path search and avoiding omission. Moreover, since data streams are generally transmitted between two computing nodes only through the shortest path, the shortest path is used as the target path between the sending node and the receiving node, improving the accuracy of the found path, and thus enabling accurate candidate data exchange nodes to be found.

[0158] In the embodiment of the present application, the process of determining at least one candidate data exchange node of each transmission failure event from the computing cluster is implemented through the above steps 403-406 based on the respective fault tracing information of at least one transmission failure event. In this embodiment, data streams are transmitted between the sending node and the receiving node through data exchange nodes, and each data exchange node can communicate with multiple other data exchange nodes to generate multiple transmission paths. In this embodiment, based on the next-hop address of the data exchange node, multiple data exchange nodes connected to each data exchange node are determined, and then multiple equivalent paths are determined. These multiple equivalent paths are all possible transmission paths. Then, based on the multiple possible transmission paths of each of the multiple data exchange nodes, the target path between the sending node and the receiving node is determined, improving the comprehensiveness of path search and avoiding omission. Thus, the target path between the sending node and the receiving node can be accurately found. Then, the data exchange nodes on this target path are used as candidate data exchange nodes, improving the accuracy of the candidate data exchange nodes. It should be noted that the management node can also implement this process through other optional implementation manners, which will not be elaborated here.

[0159] For example, refer to Figure 10 , Figure 10It is a schematic diagram for determining a target path provided by an embodiment of the present application. First, for any fault tracing information, determine the first data exchange node connected to the sending node according to the fault tracing information, and then find multiple first equivalent paths according to the next-hop address of the first data exchange node. The opposite end of the first equivalent path is the second data exchange node. Then, find multiple second equivalent paths according to the next-hop address of the second data exchange node. For a two-level network, the opposite end of the second equivalent path is the third data exchange node, and multiple third data exchange nodes include the data exchange node connected to the receiving node. For a three-level network, multiple third data exchange nodes do not include the data exchange node connected to the receiving node. Then, find multiple third equivalent paths according to the next-hop address of the third data exchange node. The opposite end of the third equivalent path is the fourth data exchange node. Find multiple fourth equivalent paths according to the next-hop address of the fourth data exchange node, and its opposite end is the fifth data exchange node. Multiple fifth data exchange nodes include the data exchange node connected to the receiving node. Finally, for any network mode, combine the respective multiple equivalent paths, traverse all the paths obtained by the combination, and find the target path with the first data exchange node and the target data exchange node at both ends. The target data exchange node is also the data exchange node connected to the receiving node.

[0160] For example, refer to Figure 11 , Figure 11This is a schematic diagram for determining a target path provided by an embodiment of the present application. Herein, it is described by taking the fault tracing information as a five-tuple and the data exchange node as a switch as an example. First, for any five-tuple, check whether there is a five-tuple with the same source IP and destination IP according to the source IP and destination IP in the five-tuple. The source IP refers to the IP of the sending node, and the destination IP refers to the IP of the receiving node. If not found, return to process the next five-tuple. If found, perform deduplication processing. Then, for any five-tuple, check whether there is a corresponding access layer switch (LA) based on the source IP of the five-tuple. If not, return, that is, do not process this five-tuple. Similarly, check whether there is a corresponding access layer switch based on the destination IP of the five-tuple. If not, return and also do not process this five-tuple. If corresponding access layer switches can be found for both the source IP and destination IP of the five-tuple, continue the processing. According to the next-hop BGP route of the source device, find the ECMP group starting from the source device. The source device refers to the access layer switch connected to the sending node. The ECMP group includes multiple first equivalent paths, and the peer end is the aggregation layer switch (LC). Then, according to the next-hop BGP route of the aggregation layer switch, find the ECMP group starting from the aggregation layer switch. This ECMP group includes multiple second equivalent paths. For a two-level network architecture, the peer end of the ECMP group is the access layer switch. For a three-level network architecture, the peer end of the ECMP group is the core layer switch (SGLC). According to the next-hop BGP route of the core layer switch, find the ECMP group starting from the core layer switch. This ECMP group includes multiple third equivalent paths, and the peer end of this ECMP group is the aggregation layer switch. Then, according to the next-hop BGP route of the aggregation layer switch, find the ECMP group starting from the aggregation layer switch. This ECMP group includes multiple fourth equivalent paths, and the peer end of this ECMP group is the access layer switch. Then, abstract all the ECMP groups in this process into an undirected graph, where the nodes of the undirected graph are switches, and then perform a depth-first traversal to determine whether the destination device is found. The destination device is also the access layer switch connected to the receiving node. If not found, repeat the traversal. If found, the target path can be determined and the process ends.

[0161] For example, refer to Figure 12 , Figure 12 This is a schematic diagram of the target path corresponding to a five-tuple provided by an embodiment of the present application. Herein, it is described by taking a two-level network architecture as an example. The management node restores the target path corresponding to the transmission failure event based on the five-tuple information of any transmission failure event. The target path includes an access layer switch and an aggregation layer switch.

[0162] In some embodiments, the management node can also collect the path information of each data stream by means of int (inband-network telemetry). Int is an inband network telemetry technology used for measuring key network metrics. In a complex network, it can quickly locate faulty nodes. However, this method requires intermediate network devices to support it, and the end-side machine needs to install an int agent (driver) to collect accurate path information.

[0163] In some embodiments, the management node uses a flow path restoration module to find candidate data exchange nodes based on the fault tracing information. This flow path restoration module can be called the SFLOW flow path restoration module. SFLOW (Solution For Network Monitoring) is a network monitoring technology standard of the IEEE (Institute of Electrical and Electronics Engineers).

[0164] 407. The management node obtains the faulty nodes in the computing cluster based on the sending node, receiving node, and candidate data exchange node of each of at least one transmission failure event.

[0165] In some embodiments, the process by which the management node obtains the faulty nodes in the computing cluster based on the sending node, receiving node, and candidate data exchange node of each of at least one transmission failure event includes the following steps: The management node sorts the sending node, receiving node, and candidate data exchange node of each of at least one transmission failure event according to the total number of occurrences of each in at least one transmission failure event. The order of any node is positively correlated with the total number of occurrences of the node. The management node uses the nodes ranked in the previous preset positions as faulty nodes.

[0166] Among them, the previous preset positions can be set and changed as needed. For example, the previous preset position can be the first position, or the first two positions or the first three positions, etc.

[0167] Among them, the process in which the management node sorts the sending nodes, receiving nodes, and candidate data exchange nodes of at least one transmission failure event respectively based on the total number of occurrences of each of the sending nodes, receiving nodes, and candidate data exchange nodes in at least one transmission failure event includes: The management node sorts the sending nodes, receiving nodes, and candidate data exchange nodes of at least one transmission failure event in descending order according to the total number of occurrences of the sending nodes, receiving nodes, and candidate data exchange nodes of each of the at least one transmission failure event. That is, the node with the larger total number of occurrences has a more forward sorting position.

[0168] In the embodiment of the present application, since multiple computing nodes communicate with each other among multiple data exchange nodes, after any node fails, it will cause the transmission failure of multiple data streams. That is, one faulty node corresponds to multiple transmission failure events. That is, if a node appears in multiple transmission failure events, there is a high probability that these multiple transmission failure events are caused by this node. Therefore, in this embodiment, by summarizing the relevant nodes in multiple transmission failure events and taking the node with the most occurrences in multiple transmission failure events as the faulty node, it is not only convenient, but also ensures the rationality and accuracy of the determined faulty node.

[0169] In some cases, the first data exchange node is uniquely connected to the sending node. Therefore, when determining the faulty node, only the first data exchange node or the sending node needs to be screened. The failure of either of them will cause transmission failure. Therefore, the faulty node can be determined only based on one of them. If any one of them is the faulty node, the other one is directly taken as the faulty node. For example, only sort the candidate data exchange nodes. If the nodes in the preset front positions in the sorting include the first data exchange node, then directly take the sending node connected to the first data exchange node and the first data exchange node as the faulty nodes to quickly eliminate the fault and restore the normal operation of the computing cluster. Subsequently, a detailed investigation can be carried out on the first data exchange node and the sending node. Similarly, only the receiving node or the data exchange node connected to the receiving node needs to be screened.

[0170] For example, see Figure 13 , Figure 13It is a schematic diagram of the reporting result of a faulty node provided by an embodiment of the present application. Among them, taking the interruption of a computing cluster as an alarm event as an example for illustration, the computing node is a server, and the data exchange node is a switch. The reporting result includes the number, time, module, level, and diagnosis completion time of the alarm event, etc. And the reporting result includes the relevant information of the diagnosed faulty node, such as the servers, access layer switches, and IP addresses whose total occurrence times ranked in the top three in the diagnosis, and indicates the total number of data streams with transmission failures and the respective total occurrence times of the diagnosed faulty nodes, and also indicates the proportion of the server ranked first in multiple transmission failure events, which is clear and straightforward.

[0171] In some embodiments, the management node determines the faulty node through a fault hot spot calculation module, that is, the fault hot spot calculation module calculates the fault hot spot according to the probabilities of different nodes appearing in transmission failure events to obtain the faulty node, and the fault hot spot is the node that appears multiple times. Among them, the flow path restoration module sends the determined candidate data exchange nodes, sending nodes, receiving nodes, etc. to the fault hot spot calculation module, and the fault hot spot calculation module determines the faulty node. After determining the faulty node, it is reported to the computing platform of the computing cluster, so that the computing platform removes the faulty node from the computing cluster to enable the computing cluster to continue data processing.

[0172] The method for determining a faulty node provided by the embodiment of the present application is mainly implemented through three modules, namely the ERRCQE log collection platform, the SFLOW flow path restoration module, and the fault hot spot calculation module. For example, see Figure 14 , Figure 14 It is a flowchart of a method for determining a faulty node provided by an embodiment of the present application. Among them, the computing cluster includes multiple computing nodes and multiple data exchange nodes, the computing node is a GPU server, and the data exchange node is a switch. For any sending node, the sending process of its data stream is shown in Figure 7 the embodiment of. The ERRCQE log collection platform collects the fault logs on the computing nodes to obtain the fault traceability information of each of the multiple transmission failure events, and the fault traceability information is a triple. Then the flow path restoration module finds the target path corresponding to each transmission failure event through these fault traceability information. Then the fault hot spot calculation module finds the faulty node through the candidate faulty nodes, sending nodes, and receiving nodes on the above multiple target paths.

[0173] For example, see Figure 15 , Figure 15It is a flowchart for handling faulty nodes in a model training cluster provided by an embodiment of the present application. Herein, taking a computing cluster for large model AI training as an example. In the case of training anomalies, the method provided by the embodiment of the present application is used to quickly detect faulty nodes, and this method is mainly implemented through four modules. First, the ERRCQE log collection platform collects the fault logs recorded by TCCL, then the abnormal flow path restoration module restores the communication paths of all abnormal data streams, and then calculates the fault hotspots to find the faulty nodes; then the faulty nodes are pushed to the training platform, and the training platform automatically excludes the faulty nodes and continues the training.

[0174] For computing services, the method provided by the embodiment of the present application can quickly detect faulty nodes, that is, reduce the detection time of faulty nodes. The detection time of faulty nodes is shortened from an average of 4 hours to within 3 minutes, and the troubleshooting efficiency is increased by 80 times. For network operation and maintenance, during the process of large model training, many faults are suspected to be network problems. Since this method can not only detect faulty computing nodes, but also detect faulty data exchange nodes, it can then be used to determine whether it is a network problem or a server-side problem, and can be used for troubleshooting in AI large model training to achieve rapid resumption of training for the service.

[0175] The embodiment of the present application provides a method for determining a faulty node. For any transmission failure event where data stream transmission fails in a computing cluster, the corresponding fault tracing information is obtained. Since the fault tracing information includes the node information of the sending node and the receiving node of the data stream, the transmission path of the data stream can then be restored based on this fault tracing information, that is, the candidate data exchange nodes that transmit the data stream between the sending node and the receiving node are found. Since the sending node, the receiving node, and the candidate data exchange nodes are all related to the data stream with transmission failure, these nodes may all be faulty nodes. And because multiple nodes communicate with each other, a faulty node may cause multiple transmission failure events. Then, by comprehensively considering these nodes in multiple transmission failure events, the accurate faulty node can be found. Moreover, this method determines the faulty node only from the nodes related to the data stream failure, rather than screening all nodes indiscriminately, which can significantly improve the efficiency of determining faulty nodes, that is, this method improves the efficiency of determining faulty nodes on the basis of ensuring the accuracy of the determined faulty nodes.

[0176] See Figure 5 , Figure 5 It is a flowchart of a method for determining a faulty node provided by an embodiment of the present application. This method introduces the process of a computing node generating a fault log, and a management node obtains the fault log from the computing node to determine the faulty node. This method takes the computing node as the execution entity, and this method includes the following steps.

[0177] 501. During the initialization process of the computing cluster, the computing nodes establish data flow tables, where the data flow tables include the identification numbers of at least one data flow to be transmitted by the computing nodes and the fault tracing information corresponding to the data flows. The computing cluster includes multiple computing nodes and multiple data exchange nodes. The multiple computing nodes are used to execute computing tasks, and the multiple data exchange nodes are used to transmit data flows between the multiple computing nodes.

[0178] In some embodiments, in order to improve the communication efficiency between computing nodes in the computing cluster, the computing nodes communicate using network cards that support the RDMA technology. Among them, the sending node of the data flow sends the data flow to the receiving node by means of RDMA WRITE. When any node on the path from the sending node to the receiving node fails, it will cause the data flow transmission to fail, and thus the communication to fail.

[0179] In the embodiments of the present application, during the initialization process of the computing cluster, each computing node learns the data flows to be transmitted by itself. After the computing cluster completes initialization, the computing nodes in the computing cluster start to communicate.

[0180] For any computing node, the computing node obtains the identification number and fault tracing information of the data flow to be transmitted by it, and then establishes the data flow table of this computing node. Among them, the data flows to be transmitted by this computing node include at least one of the data flows to be sent out by this computing node and the data flows to be received by this computing node.

[0181] Optionally, the identification number is the QP (Queue Pair) number, and this identification number includes the QP numbers of the sending node and the receiving node of the data flow respectively.

[0182] Among them, the computing node registers the data buffer (buffer stream) to be sent into a virtual address and forms a WQE (Work Queue Event) and hands it to the network card. Among them, there are multiple data flows to be sent by the computing node, and the corresponding WQEs are multiple. The multiple WQEs are stored in a queue, such as stored in the SQ (Send Queue) send queue. One WQE corresponds to one data flow sending task. The network card obtains the data flow from this SQ send queue and sends it. For any data flow that has been sent, the CQE (Complete Queue Event, send completion event) corresponding to this data flow is stored in the CQ (Complete Queue) complete queue. When a failure occurs, the network card of the sending node will generate a transmission failure event due to the uncompleted data flow sending task. It should be noted that this CQ complete queue stores not only the transmission completion events of successful transmissions, but also the transmission failure events of failed transmissions.

[0183] In some embodiments, the computing node establishes a data flow table through TCCL (Tencent Collective Communication Library, a collective communication library). TCCL is used to manage data communication between computing nodes during the data processing of the computing cluster and is a functional module. In the embodiments of the present application, each computing node may be embedded with the TCCL, such as being embedded in the computing node in a software manner or a hardware manner. Furthermore, the computing node executes steps 501-505 through the TCCL.

[0184] 502. In response to the failure of the data flow transmission corresponding to the computing node, the computing node generates a transmission failure event. The computing cluster includes multiple computing nodes and multiple data exchange nodes. The multiple computing nodes are used to execute computing tasks, and the multiple data exchange nodes are used to transmit data flows between the multiple computing nodes.

[0185] In the embodiments of the present application, the transmission failure event is used to indicate the failure of the data flow transmission. Optionally, the event information of the transmission failure event includes the flow information of the data flow, and the flow information includes an identification number.

[0186] 503. The computing node obtains the fault tracing information matching the identification number from the data flow table based on the identification number indicated by the transmission failure event.

[0187] In some embodiments, the identification number is a QP number, including the QPN (Queue Pair Number) of the sending node and the QPN of the receiving node. The computing node parses the transmission failure event, obtains the QPN of the corresponding QP, and then finds the matching fault tracing information by looking up the data flow table.

[0188] It should be noted that since any computing node can transmit multiple data flows, and the identification number is used to identify the data flow. Correspondingly, for different data flows, their identification numbers are different, that is, the QPN of the sending node and the QPN of the receiving node are respectively different.

[0189] In the embodiments of the present application, the process of the computing node obtaining the fault tracing information of the transmission failure event is realized through the above step 503. In this embodiment, by obtaining the fault tracing information of each data flow in advance and establishing a data flow table, for any data flow with a transmission failure, the fault tracing information of the data flow can be quickly obtained based on the data flow table, improving the efficiency and convenience of obtaining the fault tracing information. It should be noted that the computing node can also implement this process through other optional implementation methods, which will not be elaborated here.

[0190] 504. The computing node generates a fault log based on the fault tracing information of the transmission failure event. The fault log includes the fault tracing information of the transmission failure event and the occurrence time of the transmission failure event.

[0191] Optionally, the fault log further includes the event identifier of the transmission failure event. Optionally, the computing node may have multiple transmission failure events, and correspondingly, there may be multiple fault logs. Optionally, the multiple fault logs are stored in a log list. Alternatively, the computing node pre-sets a blank log list, and the computing node records the event identifier, fault tracing information, and occurrence time of the transmission failure event as a record in the blank log list.

[0192] Optionally, the computing node generates a fault log through the log function provided by TCCL. The log function is used to record the fault tracing information of the data stream with transmission failure in a log manner. This log function can be called the TCCL ERRCQE log function, and correspondingly, the fault log can be called the TCCL ERRCQE log.

[0193] For example, refer to Figure 6 , Figure 6 , which is a flowchart of the operation of a computing node provided by an embodiment of the present application. First, the computing node establishes a data stream flow table, and the data stream flow table includes the identification number and fault tracing information of the data stream to be transmitted by the computing node. Then, multiple data streams to be sent are recorded in a list and submitted to the network card. The computing node transmits these data streams through the network card and records the transmitted data streams in a list. For the data streams that have been transmitted but the transmission fails, the corresponding fault tracing information is obtained from the data stream flow table, and the fault tracing information of the data stream with transmission failure is recorded.

[0194] Again, for example, refer to Figure 7 , Figure 7 , which is a flowchart of the operation of a computing node provided by an embodiment of the present application. Among them, taking the fault tracing information as a five-tuple as an example for illustration. The computing node establishes a data stream flow table, and the data stream flow table includes the QP number and five-tuple information of each data stream to be transmitted. Then, the data buffer to be sent is registered as a virtual address and formed into a WQE and submitted to the SQ send queue. The network card obtains the WQE from the SQ send queue and sends it to the receiving node through the RDMA WRITE method. If any node fails during the sending process, it will cause the communication between the sending node and the receiving node to fail. Among them, the computing node stores the send completion event in the CQ completion queue. The CQ completion queue includes the transmission completion event of the successful data stream transmission and the transmission failure event of the data stream transmission failure. For the transmission failure event, the corresponding triple is obtained from the data stream flow table, and then a fault log is generated. If there are multiple transmission failure events, there are multiple fault logs, and the multiple fault logs are stored in the log list.

[0195] 505. The computing node sends the fault log to the management node, which is used to determine the candidate data exchange nodes for the transmission failure event, and determine the faulty node in the computing cluster based on the sending node, the receiving node, and the candidate data exchange nodes. The candidate data exchange node is the data exchange node that transmits the data stream indicated by the transmission failure event between the sending node and the receiving node.

[0196] It should be noted that the computing node can send the fault log to the management node after generating any fault log. The computing node can also store the fault log on the computing node after generating any fault log, and only send the fault log to the management node in response to the data acquisition request of the management node.

[0197] In the embodiment of the present application, the process of the computing node sending the fault tracing information to the management node of the computing cluster is implemented through the above steps 504-505. In this embodiment, for any transmission failure event, a fault log is generated based on its fault tracing information, so that relevant information can be recorded in time when a transmission failure event occurs, which is convenient for data storage; furthermore, by monitoring the log, the transmission failure event can be found in time and the faulty node can be located. It should be noted that the computing node can also implement this process through other optional implementation methods, which will not be elaborated here.

[0198] The embodiment of the present application provides a method for quickly locating a faulty node when a computing cluster is interrupted. By adding a log module to the collective communication, the connection information of all abnormal data streams when a fault occurs is recorded, and the transmission paths of all abnormal data streams are restored through the abnormal flow path restoration module. Finally, the central analysis module performs intelligent analysis on all relevant data to ultimately achieve the quick location of the faulty node. If the computing cluster is a model training cluster, then through the linkage with the training task, when the training is interrupted, the located faulty node can be automatically removed and the training can be quickly resumed. This method can reduce the training interruption caused by the interruption type of high-performance computing from several hours to within 5 minutes, greatly improving the training efficiency and reducing the waste of computing resources caused by training termination. Moreover, through the method provided by the embodiment of the present application, during the process of determining the faulty node, there is no need to restart the training task, which can improve the processing efficiency during business training faults and save training costs.

[0199] Figure 16 It is a block diagram of a device for determining a faulty node provided according to an embodiment of the present application. Refer to Figure 16 , the device includes:

[0200] An acquisition module 1601, configured to acquire at least one transmission failure event during the data processing of a computing cluster, where the computing cluster includes a plurality of computing nodes and a plurality of data exchange nodes, the plurality of computing nodes are used to execute computing tasks, and the plurality of data exchange nodes are used to transmit data streams between the plurality of computing nodes;

[0201] The acquisition module 1601 is further configured to acquire the fault tracing information of each of the at least one transmission failure event, where the fault tracing information includes the node information of the sending node and the receiving node of the data stream indicated by the transmission failure event;

[0202] A determination module 1602, configured to determine at least one candidate data exchange node corresponding to each of the at least one transmission failure event from the computing cluster based on the fault tracing information of each of the at least one transmission failure event, where the candidate data exchange node is a data exchange node that transmits the data stream indicated by the transmission failure event between the sending node and the receiving node;

[0203] The determination module 1602 is further configured to obtain the faulty nodes in the computing cluster based on the sending node, the receiving node, and the candidate data exchange node of each of the at least one transmission failure event.

[0204] In some embodiments, the determination module 1602 is configured to:

[0205] Based on the total number of occurrences of the sending node, the receiving node, and the candidate data exchange node of each of the at least one transmission failure event in the at least one transmission failure event, sort the sending node, the receiving node, and the candidate data exchange node of each of the at least one transmission failure event, and the order of any node is positively correlated with the total number of occurrences of the node;

[0206] Take the nodes ranked in the preset positions as the faulty nodes.

[0207] In some embodiments, the acquisition module 1601 is configured to:

[0208] For any transmission failure event, obtain the fault tracing information of the transmission failure event from the fault log of the target computing node, where the target computing node is the computing node that has a transmission failure event among the plurality of computing nodes, the fault log is the log generated by the target computing node for the transmission failure event, and the fault log includes the fault tracing information.

[0209] In some embodiments, the determination module 1602 is configured to:

[0210] For any transmission failure event, determine the first data exchange node corresponding to the transmission failure event based on the fault tracing information of the transmission failure event, where the first data exchange node is the data exchange node connected to the sending node indicated by the fault tracing information;

[0211] Determine multiple first equivalent paths based on the next-hop address of the first data exchange node. Each end of each first equivalent path is the first data exchange node and a second data exchange node indicated by the next-hop address. Each first equivalent path is a path between the first data exchange node and a second data exchange node, and the bandwidths of the multiple first equivalent paths are the same;

[0212] For any second data exchange node, determine multiple second equivalent paths based on the next-hop address of the second data exchange node. Each end of each second equivalent path is the second data exchange node and a third data exchange node indicated by the next-hop address of the second data exchange node. Each second equivalent path is a path between the second data exchange node and a third data exchange node, and the bandwidths of the multiple second equivalent paths are the same;

[0213] Based on the multiple first equivalent paths and the multiple second equivalent paths, determine the target path between the sending node and the receiving node indicated by the fault tracing information, and use the data exchange nodes on the target path as candidate data exchange nodes for the transmission failure event.

[0214] In some embodiments, the determining module 1602 is configured to:

[0215] For a transmission failure event of transmitting a data stream through a two-level network, combine the multiple first equivalent paths and the multiple second equivalent paths to obtain multiple first combined paths. Each end of each first combined path is the first data exchange node and a third data exchange node;

[0216] Determine the target path from the multiple first combined paths. The third data exchange node in the target path is the data exchange node connected to the receiving node indicated by the fault tracing information.

[0217] In some embodiments, the determining module 1602 is configured to:

[0218] For a transmission failure event of transmitting a data stream through a three-level network, for any third data exchange node, determine multiple third equivalent paths based on the next-hop address of the third data exchange node. Each end of each third equivalent path is the third data exchange node and a fourth data exchange node indicated by the next-hop address of the third data exchange node. Each third equivalent path is a path between the third data exchange node and a fourth data exchange node, and the bandwidths of the multiple third equivalent paths are the same;

[0219] For any fourth data exchange node, based on the next-hop address of the fourth data exchange node, determine multiple fourth equivalent paths. Each end of each fourth equivalent path is respectively the fourth data exchange node and a fifth data exchange node indicated by the next-hop address of the fourth data exchange node. Each fourth equivalent path is a path between the fourth data exchange node and a fifth data exchange node, and the bandwidths of the multiple fourth equivalent paths are the same;

[0220] Based on the multiple first equivalent paths, multiple second equivalent paths, multiple third equivalent paths, and multiple fourth equivalent paths, determine the target path between the sending node and the receiving node indicated by the fault tracing information.

[0221] In some embodiments, the determining module 1602 is configured to:

[0222] Combine the multiple first equivalent paths, multiple second equivalent paths, multiple third equivalent paths, and multiple fourth equivalent paths to obtain multiple second combined paths. Each end of each second combined path is respectively a first data exchange node and a fifth data exchange node;

[0223] Determine multiple candidate paths from the multiple second combined paths. The fifth data exchange nodes in the multiple candidate paths are the data exchange nodes connected to the receiving node indicated by the fault tracing information;

[0224] Determine the target path from the multiple candidate paths. The target path is the candidate path with the fewest data exchange nodes among the multiple candidate paths.

[0225] The embodiments of the present application provide a device for determining a faulty node. For any transmission failure event in which a data stream transmission fails in a computing cluster, the corresponding fault tracing information is obtained. Since the fault tracing information includes the node information of the sending node and the receiving node of the data stream, the transmission path of the data stream can be restored based on this fault tracing information, that is, the candidate data exchange nodes for transmitting the data stream between the sending node and the receiving node can be found. Since the sending node, the receiving node, and the candidate data exchange nodes are all related to the failed data stream transmission, these nodes may all be faulty nodes. And because multiple nodes communicate with each other, a faulty node may cause multiple transmission failure events. Therefore, by comprehensively considering these nodes in multiple transmission failure events, the accurate faulty node can be found. Moreover, the device determines the faulty node only from the nodes related to the data stream failure, rather than screening all nodes indiscriminately, which can significantly improve the efficiency of determining the faulty node. That is, the device improves the efficiency of determining the faulty node on the basis of ensuring the accuracy of the determined faulty node.

[0226] Figure 17 It is a block diagram of a device for determining a faulty node provided according to an embodiment of the present application. SeeFigure 17 , the apparatus includes:

[0227] A generation module 1701, configured to generate a transmission failure event in response to a data stream transmission failure corresponding to a computing node. The computing cluster includes multiple computing nodes and multiple data exchange nodes. The multiple computing nodes are used to execute computing tasks, and the multiple data exchange nodes are used to transmit data streams between the multiple computing nodes;

[0228] An acquisition module 1702, configured to acquire fault tracing information of the transmission failure event. The fault tracing information includes node information of the sending node and the receiving node of the data stream indicated by the transmission failure event;

[0229] A sending module 1703, configured to send the fault tracing information to the management node of the computing cluster. The management node is configured to determine candidate data exchange nodes for the transmission failure event, and determine the faulty nodes in the computing cluster based on the sending node, the receiving node, and the candidate data exchange nodes. The candidate data exchange node is a data exchange node that transmits the data stream indicated by the transmission failure event between the sending node and the receiving node.

[0230] In some embodiments, the apparatus further includes an establishment module, configured to establish a data stream flow table during the initialization process of the computing cluster. The data stream flow table includes the identification numbers of at least one data stream to be transmitted by the computing node and the fault tracing information corresponding to the data stream;

[0231] The acquisition module 1702 is configured to acquire the fault tracing information matching the identification number from the data stream flow table based on the identification number indicated by the transmission failure event.

[0232] In some embodiments, the sending module 1703 is configured to:

[0233] Generate a fault log based on the fault tracing information of the transmission failure event. The fault log includes the fault tracing information of the transmission failure event and the occurrence time of the transmission failure event;

[0234] Send the fault log to the management node.

[0235] An embodiment of the present application provides an apparatus for determining a faulty node. For any transmission failure event in which a data stream transmission fails in a computing cluster, the apparatus obtains corresponding fault tracing information. Since the fault tracing information includes node information of the sending node and the receiving node of the data stream, it is possible to restore the transmission path of the data stream based on the fault tracing information, that is, to find candidate data exchange nodes that transmit the data stream between the sending node and the receiving node. Since the sending node, the receiving node, and the candidate data exchange nodes are all related to the data stream with transmission failure, these nodes may all be faulty nodes. And because multiple nodes communicate with each other, a faulty node may cause multiple transmission failure events. Therefore, by comprehensively considering these nodes in multiple transmission failure events, the accurate faulty node can be found. Moreover, the apparatus determines the faulty node only from the nodes related to the data stream failure, rather than screening all nodes indiscriminately, which can significantly improve the efficiency of determining the faulty node. That is, the apparatus improves the efficiency of determining the faulty node on the basis of ensuring the accuracy of the determined faulty node.

[0236] In an embodiment of the present application, the computer device may be a terminal or a server. When the computer device is a terminal, the terminal is used as the execution entity to implement the technical solution provided by the embodiment of the present application; when the computer device is a server, the server is used as the execution entity to implement the technical solution provided by the embodiment of the present application; or, the technical solution provided by the present application is implemented through the interaction between the terminal and the server. The embodiment of the present application does not limit this.

[0237] Figure 18 The block diagram of a terminal 1800 provided by an exemplary embodiment of the present application is shown.

[0238] Generally, the terminal 1800 includes a processor 1801 and a memory 1802.

[0239] The processor 1801 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. The processor 1801 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), or PLA (Programmable Logic Array). The processor 1801 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the wake state, also known as the CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 1801 may be integrated with a GPU (Graphics Processing Unit), and the GPU is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1801 may further include an AI (Artificial Intelligence) processor, and the AI processor is used to process computational operations related to machine learning.

[0240] The memory 1802 may include one or more computer-readable storage media, and the computer-readable storage media may be non-transitory. The memory 1802 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash storage devices. In some embodiments, the non-transitory computer-readable storage media in the memory 1802 is used to store at least one program code, and the at least one program code is used to be executed by the processor 1801 to implement the method for determining a faulty node provided in the method embodiments of the present application.

[0241] In some embodiments, the terminal 1800 may further optionally include: a peripheral device interface 1803 and at least one peripheral device. The processor 1801, the memory 1802, and the peripheral device interface 1803 may be connected through a bus or signal lines. Each peripheral device may be connected to the peripheral device interface 1803 through a bus, signal lines, or a circuit board. Specifically, the peripheral devices include at least one of a radio frequency circuit 1804, a display screen 1805, a camera assembly 1806, an audio circuit 1807, and a power supply 1808.

[0242] The peripheral device interface 1803 can be used to connect at least one I / O (Input / Output) related peripheral device to the processor 1801 and the memory 1802. In some embodiments, the processor 1801, the memory 1802, and the peripheral device interface 1803 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1801, the memory 1802, and the peripheral device interface 1803 can be implemented on a separate chip or circuit board, and this embodiment does not limit this.

[0243] The radio frequency circuit 1804 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 1804 communicates with a communication network and other communication devices through electromagnetic signals. The radio frequency circuit 1804 converts an electrical signal into an electromagnetic signal for transmission, or converts the received electromagnetic signal into an electrical signal. Optionally, the radio frequency circuit 1804 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and so on. The radio frequency circuit 1804 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes but is not limited to: the World Wide Web, a metropolitan area network, an intranet, various generations of mobile communication networks (2G, 3G, 4G, and 5G), a wireless local area network, and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 1804 may further include a circuit related to NFC (Near Field Communication), and this application does not limit this.

[0244] The display screen 1805 is used to display the UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 1805 is a touch display screen, the display screen 1805 also has the ability to collect touch signals on or above the surface of the display screen 1805. The touch signals can be input as control signals to the processor 1801 for processing. At this time, the display screen 1805 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 1805, which is disposed on the front panel of the terminal 1800; in other embodiments, there may be at least two display screens 1805, which are respectively disposed on different surfaces of the terminal 1800 or are in a foldable design; in other embodiments, the display screen 1805 may be a flexible display screen, which is disposed on a curved surface or a folding surface of the terminal 1800. Even further, the display screen 1805 can also be set to an irregular non-rectangular shape, that is, a special-shaped screen. The display screen 1805 can be prepared using materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0245] The camera module 1806 is used to capture images or videos. Optionally, the camera module 1806 includes a front camera and a rear camera. Generally, the front camera is disposed on the front panel of the terminal, and the rear camera is disposed on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth-of-field camera, a wide-angle camera, and a telephoto camera, to implement functions such as background blurring by fusing the main camera and the depth-of-field camera, panoramic shooting by fusing the main camera and the wide-angle camera, and VR (Virtual Reality) shooting functions or other fused shooting functions. In some embodiments, the camera module 1806 may further include a flash. The flash can be a single-color temperature flash or a two-color temperature flash. A two-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation under different color temperatures.

[0246] The audio circuit 1807 may include a microphone and a speaker. The microphone is used to collect sound waves of the user and the environment, and convert the sound waves into electrical signals for input to the processor 1801 for processing, or input to the radio frequency circuit 1804 to achieve voice communication. For the purpose of stereo collection or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the terminal 1800. The microphone may also be an array microphone or an omnidirectional collection microphone. The speaker is used to convert the electrical signal from the processor 1801 or the radio frequency circuit 1804 into sound waves. The speaker may be a traditional thin film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for uses such as ranging. In some embodiments, the audio circuit 1807 may further include a headphone jack.

[0247] The power supply 1808 is used to supply power to each component in the terminal 1800. The power supply 1808 may be alternating current, direct current, a disposable battery or a rechargeable battery. When the power supply 1808 includes a rechargeable battery, the rechargeable battery may be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery charged through a wired line, and a wireless rechargeable battery is a battery charged through a wireless coil. The rechargeable battery may also be used to support fast charging technology.

[0248] In some embodiments, the terminal 1800 further includes one or more sensors 1809. The one or more sensors 1809 include but are not limited to: an acceleration sensor 1810, a gyroscope sensor 1811, a pressure sensor 1812, an optical sensor 1813, and a proximity sensor 1814.

[0249] The acceleration sensor 1810 can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established with the terminal 1800. For example, the acceleration sensor 1810 can be used to detect the components of the gravitational acceleration on the three coordinate axes. The processor 1801 can control the display screen 1805 to display the user interface in a landscape view or a portrait view according to the gravitational acceleration signal collected by the acceleration sensor 1810. The acceleration sensor 1810 can also be used for collecting game or user's motion data.

[0250] The gyroscope sensor 1811 can detect the body direction and rotation angle of the terminal 1800. The gyroscope sensor 1811 can cooperate with the acceleration sensor 1810 to collect the 3D actions of the user on the terminal 1800. According to the data collected by the gyroscope sensor 1811, the processor 1801 can achieve the following functions: motion sensing (such as changing the UI according to the user's tilting operation), image stabilization during shooting, game control, and inertial navigation.

[0251] The pressure sensor 1812 can be disposed on the side frame of the terminal 1800 and / or the lower layer of the display screen 1805. When the pressure sensor 1812 is disposed on the side frame of the terminal 1800, it can detect the holding signal of the user on the terminal 1800, and the processor 1801 can perform left / right hand recognition or quick operation according to the holding signal collected by the pressure sensor 1812. When the pressure sensor 1812 is disposed on the lower layer of the display screen 1805, the processor 1801 can control the operable controls on the UI interface according to the pressure operation of the user on the display screen 1805. The operable controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0252] The optical sensor 1813 is used to collect the ambient light intensity. In one embodiment, the processor 1801 can control the display brightness of the display screen 1805 according to the ambient light intensity collected by the optical sensor 1813. Specifically, when the ambient light intensity is high, the display brightness of the display screen 1805 is increased; when the ambient light intensity is low, the display brightness of the display screen 1805 is decreased. In another embodiment, the processor 1801 can also dynamically adjust the shooting parameters of the camera module 1806 according to the ambient light intensity collected by the optical sensor 1813.

[0253] The proximity sensor 1814, also known as the distance sensor, is usually disposed on the front panel of the terminal 1800. The proximity sensor 1814 is used to collect the distance between the user and the front of the terminal 1800. In one embodiment, when the proximity sensor 1814 detects that the distance between the user and the front of the terminal 1800 is gradually decreasing, the processor 1801 controls the display screen 1805 to switch from the lit state to the off state; when the proximity sensor 1814 detects that the distance between the user and the front of the terminal 1800 is gradually increasing, the processor 1801 controls the display screen 1805 to switch from the off state to the lit state.

[0254] Those skilled in the art can understand that Figure 18 the structure shown in does not constitute a limitation on the terminal 1800, and may include more or fewer components than shown in the figure, or combine some components, or adopt different component arrangements.

[0255] Figure 19It is a schematic structural diagram of a server provided by an embodiment of the present application. The server 1900 may vary greatly due to different configurations or performances, and may include one or more processors (Central Processing Units, CPUs) 1901 and one or more memories 1902. Among them, the memory 1902 is used to store executable program codes, and the processor 1901 is configured to execute the above-mentioned executable program codes to implement the method for determining a faulty node provided by each of the above method embodiments. Of course, the server may also have components such as wired or wireless network interfaces, keyboards, and input / output interfaces for input / output. The server may also include other components for implementing device functions, which will not be elaborated here.

[0256] An embodiment of the present application also provides a computer-readable storage medium, in which at least one segment of program is stored. The at least one segment of program is loaded and executed by a processor to implement the method for determining a faulty node in any of the above implementation manners.

[0257] An embodiment of the present application also provides a computer program product, which includes at least one segment of program. The at least one segment of program is stored in a computer-readable storage medium. The processor of a computer device reads the at least one segment of program from the computer-readable storage medium, and the processor executes the at least one segment of program, so that the computer device executes the method for determining a faulty node in any of the above implementation manners.

[0258] In some embodiments, the computer program product involved in the embodiments of the present application may be deployed to be executed on one computer device, or on multiple computer devices located at one place. Or, it may be executed on multiple computer devices distributed at multiple places and interconnected through a communication network. The multiple computer devices distributed at multiple places and interconnected through a communication network may form a blockchain system.

[0259] All the above optional technical solutions can be combined arbitrarily to form optional embodiments of the present application, which will not be elaborated one by one here. The above are only optional embodiments of the present application, and are not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for determining a faulty node, characterized in that, The method includes: Processing the fault logs in time periods, where the fault logs are the logs generated by computing nodes in a computing cluster for transmission failure events, and the fault logs include the fault tracing information of the transmission failure events and the occurrence time of the transmission failure events; Obtaining at least one transmission failure event in the same time period during the data processing of the computing cluster, where the computing cluster includes a plurality of computing nodes and a plurality of data exchange nodes, the plurality of computing nodes are used to execute computing tasks, and the plurality of data exchange nodes are used to transmit data streams between the plurality of computing nodes; Obtaining the fault tracing information of each of the at least one transmission failure event, where the fault tracing information includes the node information of the sending node and the receiving node of the data stream indicated by the transmission failure event; For any one of the at least one transmission failure events, based on the fault tracing information of the transmission failure event, determining a first data exchange node corresponding to the transmission failure event, where the first data exchange node is the data exchange node connected to the sending node indicated by the fault tracing information; Based on the next-hop address of the first data exchange node, determining a plurality of first equivalent paths, with each end of each first equivalent path being the first data exchange node and a second data exchange node indicated by the next-hop address; For any second data exchange node, based on the next-hop address of the second data exchange node, determining a plurality of second equivalent paths, with each end of each second equivalent path being the second data exchange node and a third data exchange node indicated by the next-hop address of the second data exchange node; Based on the plurality of first equivalent paths and the plurality of second equivalent paths, determining a target path between the sending node and the receiving node indicated by the fault tracing information, where the target path is the shortest path between the sending node and the receiving node; and taking the data exchange nodes on the target path as the candidate data exchange nodes of the transmission failure event; Sorting the candidate data exchange nodes of each of the at least one transmission failure event based on the total number of occurrences of each candidate data exchange node in each of the at least one transmission failure event, and the order of any candidate data exchange node is positively correlated with the total number of occurrences of the candidate data exchange node; Taking the plurality of candidate data exchange nodes ranked in the preset front positions as the fault nodes in the computing cluster.

2. The method according to claim 1, wherein The method further includes: Sorting the sending nodes, receiving nodes, and candidate data exchange nodes of each of the at least one transmission failure event based on the total number of occurrences of each of the sending nodes, receiving nodes, and candidate data exchange nodes in each of the at least one transmission failure event; Taking the plurality of nodes ranked in the preset front positions as the fault nodes.

3. The method according to any one of claims 1-2, characterized in that, The obtaining the fault tracing information of each of the at least one transmission failure event includes: For any one of the at least one transmission failure event, obtain the fault tracing information of the transmission failure event from the fault log of the target computing node, where the target computing node is the computing node that has the transmission failure event among the multiple computing nodes.

4. The method according to any one of claims 1-3, characterized in that Each first equivalent path is a path between the first data exchange node and a second data exchange node, and the bandwidths of the multiple first equivalent paths are the same; Each second equivalent path is a path between the second data exchange node and a third data exchange node, and the bandwidths of the multiple second equivalent paths are the same.

5. The method according to claim 4, characterized in that The determining of the target path between the sending node and the receiving node indicated by the fault tracing information based on the multiple first equivalent paths and the multiple second equivalent paths includes: For a transmission failure event of transmitting data flow through a two-level network, combine the multiple first equivalent paths and the multiple second equivalent paths to obtain multiple first combined paths, with each end of each first combined path being the first data exchange node and a third data exchange node respectively; Determine the target path from the multiple first combined paths, where the third data exchange node in the target path is the data exchange node connected to the receiving node indicated by the fault tracing information.

6. The method according to claim 4 or 5, characterized in that, The determining of the target path between the sending node and the receiving node indicated by the fault tracing information based on the multiple first equivalent paths and the multiple second equivalent paths includes: For a transmission failure event of transmitting data flow through a three-level network, for any third data exchange node, determine multiple third equivalent paths based on the next-hop address of the third data exchange node, with each end of each third equivalent path being the third data exchange node and a fourth data exchange node indicated by the next-hop address of the third data exchange node respectively, and each third equivalent path being a path between the third data exchange node and a fourth data exchange node, and the bandwidths of the multiple third equivalent paths being the same; For any fourth data exchange node, determine multiple fourth equivalent paths based on the next-hop address of the fourth data exchange node, with each end of each fourth equivalent path being the fourth data exchange node and a fifth data exchange node indicated by the next-hop address of the fourth data exchange node respectively, and each fourth equivalent path being a path between the fourth data exchange node and a fifth data exchange node, and the bandwidths of the multiple fourth equivalent paths being the same; Based on the multiple first equivalent paths, the multiple second equivalent paths, the multiple third equivalent paths, and the multiple fourth equivalent paths, determine the target path between the sending node and the receiving node indicated by the fault tracing information.

7. The method according to claim 6, wherein The determining of the target path between the sending node and the receiving node indicated by the fault tracing information based on the multiple first equivalent paths, the multiple second equivalent paths, the multiple third equivalent paths, and the multiple fourth equivalent paths includes: Combine the multiple first equivalent paths, the multiple second equivalent paths, the multiple third equivalent paths, and the multiple fourth equivalent paths to obtain multiple second combined paths, with each end of each second combined path being a first data exchange node and a fifth data exchange node respectively; Determine multiple candidate paths from the multiple second combined paths, where the fifth data exchange nodes in the multiple candidate paths are the data exchange nodes connected to the receiving node indicated by the fault tracing information; Determine the target path from the multiple candidate paths, where the target path is the candidate path with the fewest data exchange nodes among the multiple candidate paths.

8. A method for determining a faulty node, characterized in that, Applied to a computing node in a computing cluster, the method includes: In response to a data stream transmission failure corresponding to the computing node, generate a transmission failure event. The computing cluster includes multiple computing nodes and multiple data exchange nodes. The multiple computing nodes are used to execute computing tasks, and the multiple data exchange nodes are used to transmit data streams between the multiple computing nodes; Obtain the fault tracing information of the transmission failure event, where the fault tracing information includes the node information of the sending node and the receiving node of the data stream indicated by the transmission failure event; Process the fault log in time segments. The fault log is the log generated by the computing node for the transmission failure event, and the fault log includes the fault tracing information of the transmission failure event and the occurrence time of the transmission failure event; Obtain at least one transmission failure event in the same time segment during the data processing of the computing cluster; Obtain the fault tracing information of each of the at least one transmission failure event; Send the fault tracing information to the management node of the computing cluster; Wherein, the management node is used for: For any one of the at least one transmission failure event, based on the fault tracing information of the transmission failure event, determine the first data exchange node corresponding to the transmission failure event. The first data exchange node is the data exchange node connected to the sending node indicated by the fault tracing information; Based on the next-hop address of the first data exchange node, determine multiple first equivalent paths, with each end of each first equivalent path being the first data exchange node and a second data exchange node indicated by the next-hop address respectively; For any second data exchange node, based on the next-hop address of the second data exchange node, determine multiple second equivalent paths, with each end of each second equivalent path being the second data exchange node and a third data exchange node indicated by the next-hop address of the second data exchange node respectively; Based on the multiple first equivalent paths and the multiple second equivalent paths, determine the target path between the sending node and the receiving node indicated by the fault tracing information. The target path is the shortest path between the sending node and the receiving node; Use the data exchange nodes on the target path as the candidate data exchange nodes for the transmission failure event; Sort the candidate data exchange nodes corresponding to the at least one transmission failure event respectively according to the total number of occurrences of each candidate data exchange node in the at least one transmission failure event, and the order of any candidate data exchange node is positively correlated with the total number of occurrences of the candidate data exchange node; Use the multiple candidate data exchange nodes sorted in the preset first positions as the faulty nodes in the computing cluster.

9. The method according to claim 8, wherein The method further includes: During the initialization process of the computing cluster, establish a data flow table, where the data flow table includes the identification numbers of at least one data flow to be transmitted by the computing nodes and the fault tracing information corresponding to the data flow; The obtaining the fault tracing information of the at least one transmission failure event includes: Based on the identification number indicated by the transmission failure event, obtain the fault tracing information matching the identification number from the data flow table.

10. The method according to claim 8 or 9, characterized in that, The sending the fault tracing information to the management node of the computing cluster includes: Send the fault log to the management node.

11. An apparatus for determining a faulty node, characterized in that, The device includes: A module for performing the following steps: processing the fault log in time periods, where the fault log is the log generated by the computing nodes in the computing cluster for the transmission failure event, and the fault log includes the fault tracing information of the transmission failure event and the occurrence time of the transmission failure event; An obtaining module, configured to obtain at least one transmission failure event in the same time period during the data processing process of the computing cluster, where the computing cluster includes multiple computing nodes and multiple data exchange nodes, the multiple computing nodes are used to execute computing tasks, and the multiple data exchange nodes are used to transmit data flows between the multiple computing nodes; The obtaining module is further configured to obtain the fault tracing information of each of the at least one transmission failure event, where the fault tracing information includes the node information of the sending node and the receiving node of the data flow indicated by the transmission failure event; A determination module, configured to, for any one of the at least one transmission failure event, based on the fault tracing information of the transmission failure event, determine a first data exchange node corresponding to the transmission failure event, where the first data exchange node is a data exchange node connected to the sending node indicated by the fault tracing information; based on the next-hop address of the first data exchange node, determine multiple first equivalent paths, where both ends of each first equivalent path are the first data exchange node and a second data exchange node indicated by the next-hop address; for any second data exchange node, based on the next-hop address of the second data exchange node, determine multiple second equivalent paths, where both ends of each second equivalent path are the second data exchange node and a third data exchange node indicated by the next-hop address of the second data exchange node; based on the multiple first equivalent paths and the multiple second equivalent paths, determine a target path between the sending node and the receiving node indicated by the fault tracing information, where the target path is the shortest path between the sending node and the receiving node; use the data exchange nodes on the target path as the candidate data exchange nodes for the transmission failure event; The determination module is further configured to, based on the total number of occurrences of the candidate data exchange nodes of each of the at least one transmission failure event in the at least one transmission failure event, sort the candidate data exchange nodes of each of the at least one transmission failure event, where the order of any candidate data exchange node is positively correlated with the total number of occurrences of the candidate data exchange node; use the multiple candidate data exchange nodes ranked in the top preset positions as the faulty nodes in the computing cluster.

12. The device according to claim 11, wherein The determination module is further configured to: Based on the total number of occurrences of the sending nodes, receiving nodes, and candidate data exchange nodes of each of the at least one transmission failure event in the at least one transmission failure event, sort the sending nodes, receiving nodes, and candidate data exchange nodes of each of the at least one transmission failure event; Use the multiple nodes ranked in the top preset positions as the faulty nodes.

13. The device according to any one of claims 11-12, characterized in that, The acquisition module is configured to: For any one of the at least one transmission failure event, obtain the fault tracing information of the transmission failure event from the fault log of the target computing node, where the target computing node is the computing node that has the transmission failure event among the multiple computing nodes.

14. The device according to any one of claims 11 - 13, characterized in that Each first equivalent path is a path between the first data exchange node and a second data exchange node, and the multiple first equivalent paths have the same bandwidth; Each second equivalent path is a path between the second data exchange node and a third data exchange node, and the multiple second equivalent paths have the same bandwidth.

15. The device according to claim 14, characterized in that, The determination module is configured to: For a transmission failure event of transmitting data streams through a two-level network, combine the multiple first equivalent paths and the multiple second equivalent paths to obtain multiple first combined paths, where both ends of each first combined path are the first data exchange node and a third data exchange node; Determine the target path from the multiple first combined paths, where the third data exchange node in the target path is the data exchange node connected to the receiving node indicated by the fault tracing information.

16. The device according to claim 14 or 15, characterized in that The determining module is configured to: For a transmission failure event of transmitting a data stream through a three-level network, for any third data exchange node, based on the next-hop address of the third data exchange node, determine multiple third equivalent paths, with each end of each third equivalent path being the third data exchange node and a fourth data exchange node indicated by the next-hop address of the third data exchange node. Each third equivalent path is a path between the third data exchange node and a fourth data exchange node, and the bandwidths of the multiple third equivalent paths are the same; For any fourth data exchange node, based on the next-hop address of the fourth data exchange node, determine multiple fourth equivalent paths, with each end of each fourth equivalent path being the fourth data exchange node and a fifth data exchange node indicated by the next-hop address of the fourth data exchange node. Each fourth equivalent path is a path between the fourth data exchange node and a fifth data exchange node, and the bandwidths of the multiple fourth equivalent paths are the same; Based on the multiple first equivalent paths, the multiple second equivalent paths, the multiple third equivalent paths, and the multiple fourth equivalent paths, determine the target path between the sending node and the receiving node indicated by the fault tracing information.

17. The device according to claim 16, characterized in that, The determining module is configured to: Combine the multiple first equivalent paths, the multiple second equivalent paths, the multiple third equivalent paths, and the multiple fourth equivalent paths to obtain multiple second combined paths, with each end of each second combined path being a first data exchange node and a fifth data exchange node; Determine multiple candidate paths from the multiple second combined paths, where the fifth data exchange node in the multiple candidate paths is the data exchange node connected to the receiving node indicated by the fault tracing information; Determine the target path from the multiple candidate paths, where the target path is the candidate path with the fewest data exchange nodes among the multiple candidate paths.

18. A device for determining a faulty node, characterized in that Applied to a computing node in a computing cluster, the apparatus includes: A generating module, configured to generate a transmission failure event in response to the transmission failure of the data stream corresponding to the computing node. The computing cluster includes multiple computing nodes and multiple data exchange nodes. The multiple computing nodes are used to execute computing tasks, and the multiple data exchange nodes are used to transmit data streams between the multiple computing nodes; A module for performing the following steps: Process the fault log in time periods. The fault log is the log generated by the computing node for the transmission failure event, and the fault log includes the fault tracing information of the transmission failure event and the occurrence time of the transmission failure event; Obtain at least one transmission failure event in the same time period during the data processing of the computing cluster; An acquisition module, configured to acquire fault tracing information of the at least one transmission failure event, where the fault tracing information includes node information of a sending node and a receiving node of a data stream indicated by the transmission failure event; A sending module, configured to send the fault tracing information to a management node of the computing cluster; Wherein, the management node is configured to: For any one of the at least one transmission failure event, based on the fault tracing information of the transmission failure event, determine a first data exchange node corresponding to the transmission failure event, where the first data exchange node is a data exchange node connected to the sending node indicated by the fault tracing information; Based on a next-hop address of the first data exchange node, determine multiple first equivalent paths, where both ends of each first equivalent path are the first data exchange node and a second data exchange node indicated by the next-hop address; For any second data exchange node, based on a next-hop address of the second data exchange node, determine multiple second equivalent paths, where both ends of each second equivalent path are the second data exchange node and a third data exchange node indicated by the next-hop address of the second data exchange node; Based on the multiple first equivalent paths and the multiple second equivalent paths, determine a target path between the sending node and the receiving node indicated by the fault tracing information, where the target path is the shortest path between the sending node and the receiving node; and use the data exchange nodes on the target path as candidate data exchange nodes for the transmission failure event; Based on the total number of occurrences of the candidate data exchange nodes of the at least one transmission failure event in the at least one transmission failure event respectively, sort the candidate data exchange nodes of the at least one transmission failure event respectively, and the order of any candidate data exchange node is positively correlated with the total number of occurrences of the candidate data exchange node; Use multiple candidate data exchange nodes ranked in the top preset positions as fault nodes in the computing cluster.

19. The device according to claim 18, characterized in that, The apparatus further includes: A building module, configured to build a data stream flow table during the initialization process of the computing cluster, where the data stream flow table includes identification numbers of at least one data stream to be transmitted by the computing node and fault tracing information corresponding to the data stream; The acquisition module, configured to acquire, based on the identification number indicated by the transmission failure event, fault tracing information matching the identification number from the data stream flow table.

20. The device according to claim 18 or 19, characterized in that, The sending module is configured to: Send the fault log to the management node.

21. A computer device, characterized in that, The computer device includes a processor and a memory, where the memory is used to store at least one program, and the at least one program is loaded and executed by the processor to execute the method for determining a fault node according to any one of claims 1-7 or the method for determining a fault node according to any one of claims 8-10.

22. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store at least one program, and the at least one program is used to execute the method for determining a fault node according to any one of claims 1-7 or the method for determining a fault node according to any one of claims 8-10.

23. A computer program product, characterized in that, The computer program product includes at least one program, the at least one program is stored in a computer-readable storage medium, a processor of a computer device reads the at least one program from the computer-readable storage medium, and the processor executes the at least one program, so that the computer device executes the method for determining a faulty node according to any one of claims 1-7 or the method for determining a faulty node according to any one of claims 8-10.

Images