Fault detection method, device and equipment for distributed cluster

Abstract

The embodiment of the invention provides a fault detection method, device and equipment for a distributed cluster, nodes in the distributed cluster parallelly and cooperatively process the same task, and relates to the technical field of artificial intelligence. Obtaining the latest network state of the node; and comparing the newest network state with the stored network state, and if the newest network state is consistent with the stored network state, performing inter-node communication through the communication link established by the node. Fault detection is carried out on the network state of the nodes in the distributed cluster before communication, so that data transmission delay and even transmission failure caused by network faults at the beginning of communication are avoided; by comparing the detected newest network state with the stored network state, whether the newest end-side network state fails or not can be judged.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a fault detection method, apparatus, and device for distributed clusters. Background Art

[0002] Currently, training machine learning or deep learning models typically involves using distributed training systems, which often include multiple nodes. Parallel training across these nodes accelerates training. However, data transmission between these nodes requires reliable network communication. Failures in network communication between these nodes can significantly reduce model training speed and may even lead to training failure.

[0003] Therefore, for network fault detection or recovery between multiple nodes, the following methods are generally used: 1) Set a time threshold between communications to determine whether the communication time exceeds the set time threshold; 2) Set a management node, which will send or receive status information of other nodes to determine whether the communication between the management node and other nodes has failed; 3) Perform regular backups during model training to enable timely recovery after a failure; 4) Add additional equipment, so that the data information on all devices is consistent. When one of the devices fails, other devices are used to continue training.

[0004] However, the above method will bring the following problems: 1) The time threshold is set too short, which affects normal model training; the time threshold is set too long and cannot effectively identify faults; 2) When a management node fails, a more complex mechanism is required to handle it; 3) Scheduled backups occupy a high amount of memory, affecting device performance; 4) Adding additional equipment increases training costs. Summary of the Invention

[0005] The embodiments of the present application provide a fault detection method, apparatus, and device for a distributed cluster, which can quickly detect and restore network fault nodes during model training.

[0006] In a first aspect, an embodiment of the present application provides a fault detection method for a distributed cluster, wherein nodes in the distributed cluster collaboratively process the same task in parallel, the method comprising:

[0007] Before any node communicates with another node, it performs fault detection on the network status of the node to obtain the latest network status of the node;

[0008] The latest network status is compared with the stored network status, and if they are consistent, inter-node communication is performed through the communication link established by the nodes.

[0009] In an embodiment of the present application, by performing fault detection on the network status of nodes in a distributed cluster before communication, data transmission delays or even transmission failures due to network faults at the start of communication are avoided; by comparing the latest detected network status with the stored network status, it is possible to determine whether the latest network status has failed. The embodiment of the present application detects network status when each node in a distributed cluster is collaborating in parallel to process the same task, thereby avoiding the need for regular backups to prevent data loss between nodes after the task starts, which would result in excessive memory usage, and would not require additional equipment to back up the data of each node, which would increase costs.

[0010] Optionally, it also includes:

[0011] If the latest network state is inconsistent with the stored network state, generating a network fault marker;

[0012] Notifying the network fault marker in the distributed cluster and using the latest network state as the stored network state;

[0013] Inter-node communication is performed by avoiding the communication link corresponding to the network fault represented by the network fault marker.

[0014] In an embodiment of the present application, when the latest network status is inconsistent with the stored network status, a network fault marker is generated and then the network fault marker is notified to all nodes in the distributed cluster, thereby avoiding the problem that other nodes are not aware of the faulty node and data is transmitted again through the faulty node.

[0015] Optionally, the network failure of the node includes a first communication link failure;

[0016] Optionally, performing inter-node communication by avoiding the communication link indicated by the network fault marker includes:

[0017] Determining a second communication link having the same transceiver node as the first communication link;

[0018] The data to be transmitted corresponding to the first communication link is transmitted through the second communication link.

[0019] In an embodiment of the present application, by determining a second communication link having the same transceiver node as the first communication link, the data to be transmitted can be transmitted through the second communication link, thereby avoiding the problem that the data to be transmitted cannot be transmitted after the original communication link fails.

[0020] Optionally, before determining the second communication link having the same transceiver node as the first communication link, the method further includes:

[0021] After interrupting all communication links between the transceiver nodes associated with the first communication link, a second communication link is re-established between the transceiver nodes.

[0022] In an embodiment of the present application, after the first communication link fails, other communication links between the same transceiver nodes as the first communication link are simultaneously interrupted, and then a second communication link is re-established between the transceiver nodes, so that the data to be transmitted that originally required the first communication link to be transmitted can be reallocated and thus transmitted through the second communication link.

[0023] Optionally, re-establishing the second communication link between the transceiver nodes includes:

[0024] The communication links between all nodes in the distributed cluster are re-planned in a unified manner to rebuild the communication links between the nodes.

[0025] In the embodiment of the present application, by uniformly planning the communication links between all nodes in a distributed cluster, the reconstructed communication links are more appropriate communication links, thereby ensuring the efficiency of data transmission after a network failure.

[0026] Optionally, it also includes:

[0027] If the latest network status is consistent with the stored network status of the node, a network normal marker is generated and announced in the distributed cluster.

[0028] In the embodiment of the present application, by notifying other nodes in the distributed cluster of the network normal identifier, other nodes know which nodes are normal, which facilitates the selection of nodes for data transmission.

[0029] Optionally, performing fault detection on the network status of the node to obtain the latest network status of the node includes:

[0030] determining a network type of the node;

[0031] Determining each network attribute to be detected according to the network type of the node;

[0032] Fault detection is performed on each of the network attributes to obtain the latest network status of the node in each of the network attributes.

[0033] In the embodiment of the present application, by determining the network type of each node and then detecting the network type, misjudgment caused by different items or indicators detected for different network types is avoided.

[0034] In a second aspect, an embodiment of the present application provides a fault detection device for a distributed cluster, the device comprising:

[0035] A detection module is used to perform fault detection on the network status of any node before any node performs inter-node communication to obtain the latest network status of the node;

[0036] A comparison module is used to compare the latest network status with the stored network status, and if they are consistent, perform inter-node communication via the communication link established by the node. Optionally, the device also includes:

[0037] a status notification module, configured to generate a network fault marker if the latest network status is inconsistent with the stored network status, wherein the network fault marker is used to indicate a network fault of a node;

[0038] Notifying the network fault marker in the distributed cluster and using the latest network state as the stored network state;

[0039] The link avoidance module is used to perform inter-node communication by avoiding the communication link corresponding to the network fault represented by the network fault identifier.

[0040] Optionally, the link avoidance module is specifically configured to:

[0041] Determining a second communication link having the same transceiver node as the first communication link;

[0042] The data to be transmitted corresponding to the first communication link is transmitted through the second communication link.

[0043] Optionally, the link avoidance module is further configured to:

[0044] Before determining the second communication link having the same transceiver node as the first communication link, all communication links between the transceiver nodes associated with the first communication link are interrupted, and then the second communication link is re-established between the transceiver nodes.

[0045] Optionally, the link avoidance module is specifically configured to:

[0046] The communication links between all nodes in the distributed cluster are re-planned in a unified manner to rebuild the communication links between the nodes.

[0047] Optionally, the status notification module is further configured to:

[0048] If the latest network state is consistent with the initial network state of the node, a network normal marker is generated and announced in the distributed cluster.

[0049] Optionally, the detection module is specifically configured to:

[0050] determining a network type of the node;

[0051] Determining each network attribute to be detected according to the network type of the node;

[0052] Fault detection is performed on each of the network attributes to obtain the latest network status of the node in each of the network attributes.

[0053] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the above-described methods when executing the program.

[0054] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium storing a computer program executable by a computer device, which enables the computer device to perform the steps of any of the above-mentioned methods when the program is run on the computer device.

[0055] In the embodiment of the present application, by performing fault detection on the network status of the nodes in the distributed cluster before communication, data transmission delays or even transmission failures due to network faults at the beginning of communication are avoided; by comparing the latest detected network status with the stored network status, it can be determined whether the latest network status is abnormal. The embodiment of the present application detects the network status when each node in the distributed cluster is processing the same task in parallel and collaboratively, avoiding the need for regular backup after the task starts to prevent data loss between nodes, which would result in excessive memory usage, and would not require additional equipment to back up the data of each node, which would increase costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0057] Figure 1 A system architecture diagram provided for an embodiment of the present application;

[0058] Figure 2 A schematic diagram of a fault detection method for a distributed cluster provided in an embodiment of the present application (I);

[0059] Figure 3 A schematic diagram of a fault detection method for a distributed cluster provided in an embodiment of the present application (II);

[0060] Figure 4A schematic diagram of the structure of a fault detection device for a distributed cluster provided in an embodiment of the present application;

[0061] Figure 5 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solutions and beneficial effects of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0063] For ease of understanding, the terms involved in the embodiments of the present invention are explained below.

[0064] A distributed cluster is a system consisting of multiple computer nodes that communicate and collaborate over a network to complete a task or provide a service. Distributed clusters offer advantages such as high availability, high performance, and scalability, making them widely used in large-scale computing, storage, and processing scenarios.

[0065] To facilitate understanding of this solution, the following describes its application scenarios.

[0066] Model training in machine learning and other applications often utilizes distributed cluster training systems, which demonstrate high efficiency when processing large amounts of data and complex models. A distributed cluster training system typically consists of multiple nodes, which communicate over a network to transmit data. However, in large-scale distributed cluster training systems, network failures can cause communication timeouts, leading to delays in model training or even training failure. Therefore, when training models in a distributed cluster training system, it is necessary to detect whether network failures exist between nodes. If so, the location of the failure must be determined and timely recovery measures implemented. Traditional methods for fault detection and recovery include: 1) using timeouts for fault detection. Based on experience, a communication time threshold can be set. Checkpoints are set before and after communication begins. The time difference between these checkpoints is the communication duration. When the communication duration exceeds the time threshold, a node is considered to have experienced a network failure. However, the time threshold should not be set too low, to avoid the system mistakenly identifying a failure despite normal data communication; nor should it be set too high, to avoid failure detection in a timely manner. 2) Using a management node to detect failures. In a distributed cluster training system, some nodes are set as management nodes, and the management nodes send and receive status information of other nodes to determine whether other nodes have failed. However, when a management node fails, a more complex mechanism must be adopted to handle it. 3) Save model parameters. In order to be able to recover from a failure in a timely manner after a failure occurs, the model parameters can be saved during the training process, but this will require a higher memory usage and affect machine performance. 4) Add additional storage devices. During model training, the data information on multiple devices is kept consistent. When the network of a device fails, other devices are used to continue training, but adding devices will increase the training cost. The following describes the process of the fault detection method provided in the embodiment of the present application.

[0067] First, before model training begins, each node is fault-checked. If no nodes are faulty, a network health marker is generated and notified to each node in the distributed cluster. If a node is faulty, a network failure marker is generated and notified to each node in the distributed cluster. All communication links are then interrupted, and the data originally transmitted by the faulty node is redistributed. After determining the data transmitted by each communication link, the communication links are reestablished, and data transmission is finally completed.

[0068] See also Figure 1 , is a system architecture diagram provided by an embodiment of the present application, the system includes: a terminal device 101 and multiple servers 102. The terminal device 101 responds to the operation of the business development personnel.

[0069] Multiple servers 102 are servers used to implement a distributed cluster fault detection method. Multiple servers 102 can be independent physical servers, or a server cluster or distributed system composed of multiple physical servers. They can also be cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0070] The terminal device 101 can be directly or indirectly connected to the multiple servers 102 via a wired or wireless manner; the multiple servers 102 can be directly or indirectly connected to each other via a wired manner, which is not limited in this application.

[0071] based on Figure 1 The system architecture diagram of the present application provides a process of a fault detection method for a distributed cluster, in which each node in the distributed cluster collaborates to process the same task in parallel, such as Figure 2 As shown, the process of the method is executed by a computer device, which may be Figure 1 The server 102 shown executes the following steps:

[0072] Step 201: Before any node performs inter-node communication, it performs fault detection on the node's network status to obtain the node's latest network status.

[0073] Specifically, in a distributed cluster, before data communication begins between multiple nodes, network fault detection is performed on each node to obtain the latest network status of each node. For example, in a distributed cluster used for machine learning model training, a network fault detection program is deployed before training begins, before each node's communication program. This allows the network status of each node to be obtained before data communication begins.

[0074] Step 202: Compare the latest network status with the stored network status. If they are consistent, perform inter-node communication via the communication link established by the nodes.

[0075] Specifically, each node will also have its own stored network status, and the stored network status is the network status detected last time. If the network status stored by each node during the first detection is the initialized network status, the initialized network status is generally the standard network status of each node. The latest network status of each node is compared with the stored network status. If they are consistent, it is considered that there is no network failure in the network node, and communication can be carried out through the corresponding communication link established between the nodes. In a distributed cluster, there is generally at least one communication link between every two nodes. Based on business needs, data volume and other factors, before data transmission begins, the number of communication links between the two nodes that need to transmit data and the size of the data required to be transmitted by each communication link are determined so that data transmission is carried out according to the preset communication link. There is a corresponding communication link between every two nodes.

[0076] Fault detection of any node's network status can be performed by testing the status of the node's network interface card (NIC), its path and number within the node, and the port attributes and traffic rate of the communication link between nodes. Detecting a node's NIC requires determining whether the node itself has a NIC, as well as the device path and NIC number corresponding to each NIC. Detecting the presence of a NIC is a prerequisite for determining the node's network status. A node typically has multiple NICs, each with its own storage path and number. Each NIC generates a corresponding communication link. If a NIC is absent or damaged, there's no need to determine the corresponding communication link. The communication link ports are located on both nodes, so the node's network status can also be determined by detecting the status of the port attributes. After a communication link is established, the node's network status can also be determined by detecting the traffic rate of the communication link. The above node network status detection can be customized as needed.

[0077] When comparing the latest network status with the stored network status, you can compare whether the network card path in the latest network status is consistent with the network card path in the stored network status, whether the network card number is consistent, whether the number of communication links between nodes is consistent, and other information.

[0078] In an embodiment of the present application, by performing fault detection on the network status of nodes in a distributed cluster before communication, data transmission delays or even transmission failures due to network faults at the start of communication are avoided; by comparing the latest detected network status with the stored network status, it is possible to determine whether the latest network status has failed. The embodiment of the present application detects network status when each node in a distributed cluster is collaborating in parallel to process the same task, thereby avoiding the need for regular backups to prevent data loss between nodes after the task starts, which would result in excessive memory usage, and would not require additional equipment to back up the data of each node, which would increase costs.

[0079] In some embodiments, if the latest network status is inconsistent with the stored network status, a network fault marker is generated, the network fault marker is used to characterize the network fault of the node, the network fault marker is announced in the distributed cluster, and the latest network status is used as the stored network status; inter-node communication is performed by avoiding the communication link corresponding to the network fault represented by the network fault marker.

[0080] Specifically, if a network item is detected where the latest network state of a node is inconsistent with the stored network state, a network fault marker is first generated for the node. The network marker is used to indicate that a node has a network fault. Then, the latest network state replaces the stored network state. When data is subsequently transmitted, the network fault marker can be used to determine which node has a network fault, thereby avoiding the communication link corresponding to the node with the network fault. A communication link is a link for data transmission between nodes. Finally, in a distributed cluster, the network fault marker is notified to all nodes in the cluster so that all nodes in the cluster know which node has a fault, avoiding the misuse of the faulty node during subsequent data transmission, which may lead to data transmission failure.

[0081] When notifying the network fault marker, there is no limitation on the notification method, which can be a communication method similar to Allreduce or a communication method similar to Broadcast. If you want to send the data of your own node to other nodes in the cluster, you can use the Broadcast communication method, in which the sending node serves as the master node and the other receiving nodes are designated nodes. Allreduce is a many-to-many communication method with multiple senders and multiple receivers. Reduce is called a reduction operation, which is a general term for a series of calculation operations. Specifically, it includes SUM, MIN, MAX, PROD, LOR, etc. The Reduce operation obtains an input element group on each process, and after the Reduce operation, the simplified elements will be obtained. Allreduce applies the same Reduce operation on all nodes.

[0082] After notifying the abnormal network fault marker in the distributed cluster, the method further includes: after the network item fault is repaired, performing inter-node communication through the network item.

[0083] Specifically, the network item is a first communication link. After a failure occurs in the first communication link, the failure is repaired. The repair may be to re-establish a second communication link, and then continue communication between nodes through the re-established second communication link. The network item may also be a second communication link.

[0084] After the first communication link fails, the network status information of the failed node must be stored in the local fault log. The fault log records the specific node of the fault, which link between the nodes is faulty, and the type of fault. Through the upper-level management application, the failed node can be located or automatically repaired. It also makes it easier for maintenance personnel to collect statistics and check the network status of the failed node.

[0085] In an embodiment of the present application, when the latest network status is inconsistent with the stored network status, a network fault marker is generated and then the network fault marker is notified to all nodes in the distributed cluster, thereby avoiding the problem that other nodes are not aware of the faulty node and data is transmitted again through the faulty node.

[0086] In some embodiments, the network failure of a node includes a failure of a first communication link; communicating between nodes by avoiding the communication link indicated by the network failure marker includes: determining a second communication link having the same transceiver node as the first communication link; and transmitting the data to be transmitted corresponding to the first communication link through the second communication link.

[0087] Specifically, there are multiple communication links within a distributed cluster, and the network fault marker indicates a first communication link failure, that is, the first communication link is a communication link with a network fault, and there is at least one first communication link. Inter-node communication is performed by circumventing the communication link indicated by the network fault marker. Specifically, the following steps are performed: first, determine the nodes connected to the first communication link, that is, the nodes that send data and the nodes that receive data via the first communication link. Then, the data to be transmitted that originally needed to be transmitted via the first communication link is transmitted via a second communication link, wherein the sending node and the receiving node of the second communication link are the same as those of the first communication link, and the second communication link is a communication link without a network fault.

[0088] In an embodiment of the present application, by determining a second communication link having the same transceiver node as the first communication link, the data to be transmitted can be transmitted through the second communication link, thereby avoiding the problem that the data to be transmitted cannot be transmitted after the original communication link fails.

[0089] In some embodiments, before determining the second communication link having the same transceiver node as the first communication link, the method further includes: interrupting all communication links between the transceiver nodes associated with the first communication link, and then re-establishing the second communication link between the transceiver nodes.

[0090] Specifically, before determining the second communication link having the same transceiver node as the first communication link, all communication links having the same transceiver node as the first communication link are interrupted, and then the second communication link is re-established between the same transceiver nodes. There is at least one second communication link.

[0091] For example, when the first communication link a between node 1 and node 2 fails, the data block is the data to be transmitted that was originally required to be transmitted through the first communication link a. According to the type and size of the data block, the second communication link b is re-established between node 1 and node 2, and the data block is continued to be transmitted through the second communication link b.

[0092] In an embodiment of the present application, after the first communication link fails, other communication links between the same transceiver nodes as the first communication link are simultaneously interrupted, and then a second communication link is re-established between the transceiver nodes, so that the data to be transmitted that originally required the first communication link to be transmitted can be reallocated and thus transmitted through the second communication link.

[0093] In some embodiments, re-establishing the second communication link between the transceiver nodes includes: re-planning the communication links between all nodes in the distributed cluster in a unified manner, and rebuilding the communication links between the nodes.

[0094] Specifically, when establishing the second communication link, the communication links between all nodes may be planned uniformly according to the network status of each node in the distributed cluster, and the most appropriate communication link may be determined as the communication link for data transmission.

[0095] For example, there are multiple nodes in the distributed cluster, node 1, node 2, node 3, node 4, node 5, node 6, if the first communication link a between node 1 and node 2 12 If a fault occurs, the network status of nodes 3, 4, 5, and 6 can be obtained. If the network status of node 3 is good, a second communication link a can be established between nodes 1, 2, and 3. 132 (Node 1-Node 3-Node 2), continue data transmission.

[0096] In the embodiment of the present application, by uniformly planning the communication links between all nodes in a distributed cluster, the reconstructed communication links are more appropriate communication links, thereby ensuring the efficiency of data transmission after a network failure.

[0097] In some embodiments, if the latest network status is consistent with the stored network status, a network normal flag is generated and announced in the distributed cluster.

[0098] Specifically, when a node's network status is detected for a fault, if the latest detected network status is consistent with the previously stored network status, a network normal flag is generated. After the network normal flag is generated, it is also notified to all nodes in the distributed cluster to inform other nodes of the normal network node, facilitating the connection of communication links between nodes.

[0099] In the embodiment of the present application, by notifying other nodes in the distributed cluster of the network normal identifier, other nodes know which nodes are normal, which facilitates the selection of nodes for data transmission.

[0100] In some embodiments, fault detection is performed on the network status of a node to obtain the latest network status of the node, including: determining the network type of the node; determining each network attribute to be detected based on the network type of the node; and performing fault detection on each network attribute to obtain the latest network status of the node at each network attribute.

[0101] Specifically, different network types have different network attributes, and each type of network corresponds to different network status indicators. Therefore, before obtaining network status, the network type must be determined. Currently, the most commonly used networks are InfiniBand networks (IB networks) and socket networks (Socket networks). After determining the node's network type, the network attributes of each node are determined, and fault detection is performed on each network attribute. Network attributes can include the status of the node's network card, the number of communication links between nodes, and the traffic rate of the communication links. After determining the network attributes, the network status corresponding to the network attributes is judged to be normal or abnormal based on experience or standards, thereby obtaining the latest network status.

[0102] In the embodiment of the present application, by determining the network type of each node and then detecting the network type, misjudgment caused by different items or indicators detected for different network types is avoided.

[0103] In order to further explain this application, the following describes in detail the fault detection method for the distributed cluster using node 1 and data block as specific implementation scenarios. Figure 3 As shown, the specific steps include:

[0104] Step 301: Determine the network type of node 1;

[0105] Step 302: Determine the network attributes that node 1 needs to detect based on the network type of node 1;

[0106] Step 303: Determine the network status of node 1 according to the network attributes of node 1;

[0107] Step 304: Determine whether the network status of node 1 is consistent with the last stored network status. If so, execute step 310; otherwise, execute step 305;

[0108] Step 305: Generate a network fault marker for node 1, where the network fault marker indicates a first communication link that has failed.

[0109] Step 306: Notify the network fault identifier of node 1 to all nodes in the distributed cluster;

[0110] Step 307: Determine the same transceiver node as the first communication link;

[0111] Step 308: interrupt all communication links with the same transmitting and receiving nodes;

[0112] Step 309: re-establishing a second communication link between the same transceiver nodes;

[0113] Step 310: Continue transmitting the data block via the second communication link;

[0114] Step 311: Generate a network normal flag for node 1, where the network normal flag indicates that data transmission can be performed according to the preset communication link.

[0115] Step 312: Notify the network normal flag of node 1 to all nodes in the distributed cluster;

[0116] Step 313: Transmit the data block via the preset communication link.

[0117] In an embodiment of the present application, by performing fault detection on the network status of nodes in a distributed cluster before communication, data transmission delays or even transmission failures due to network faults at the start of communication are avoided; by comparing the latest detected network status with the stored network status, it is possible to determine whether the latest network status has failed. The embodiment of the present application detects network status when each node in a distributed cluster is collaborating in parallel to process the same task, thereby avoiding the need for regular backups to prevent data loss between nodes after the task starts, which would result in excessive memory usage, and would not require additional equipment to back up the data of each node, which would increase costs.

[0118] Based on the same technical concept, the embodiment of the present application provides a fault detection device for a distributed cluster, such as Figure 4 As shown, the apparatus 400 includes:

[0119] The detection module 401 is used for any node to perform fault detection on the network status of the node before inter-node communication to obtain the latest network status of the node;

[0120] The comparison module 402 is configured to compare the latest network status with the stored network status, and if they are consistent, perform inter-node communication via the communication link established by the nodes.

[0121] Optionally, the apparatus 400 further includes:

[0122] A status notification module 403 is configured to generate a network fault marker if the latest network status is inconsistent with the stored network status, wherein the network fault marker is used to indicate a network fault of a node;

[0123] Notifying the network fault marker in the distributed cluster and using the latest network state as the stored network state;

[0124] The link avoidance module 404 is configured to perform inter-node communication by avoiding the communication link corresponding to the network fault represented by the network fault identifier.

[0125] Optionally, the link avoidance module 404 is specifically configured to:

[0126] Determining a second communication link having the same transceiver node as the first communication link;

[0127] The data to be transmitted corresponding to the first communication link is transmitted through the second communication link.

[0128] Optionally, the link avoidance module 404 is further configured to:

[0129] Before determining the second communication link having the same transceiver node as the first communication link, all communication links between the transceiver nodes associated with the first communication link are interrupted, and then the second communication link is re-established between the transceiver nodes.

[0130] Optionally, the link avoidance module 404 is specifically configured to:

[0131] The communication links between all nodes in the distributed cluster are re-planned in a unified manner to rebuild the communication links between the nodes.

[0132] Optionally, the status notification module 403 is further configured to:

[0133] If the latest network state is consistent with the initial network state of the node, a network normal marker is generated and announced in the distributed cluster.

[0134] Optionally, the detection module 401 is specifically configured to:

[0135] determining a network type of the node;

[0136] Determining each network attribute to be detected according to the network type of the node;

[0137] Fault detection is performed on each of the network attributes to obtain the latest network status of the node in each of the network attributes.

[0138] Based on the same technical concept, the embodiment of the present application provides a computer device, which can be a terminal or a server, such as Figure 5 As shown, it includes at least one processor 501 and a memory 502 connected to the at least one processor. The specific connection medium between the processor 501 and the memory 502 is not limited in the embodiment of the present application. Figure 5 For example, the processor 501 and the memory 502 are connected via a bus, which can be divided into an address bus, a data bus, a control bus, and the like.

[0139] In an embodiment of the present application, the memory 502 stores instructions that can be executed by at least one processor 501. The at least one processor 501 can execute the steps included in the above-mentioned fault detection method for a distributed cluster by executing the instructions stored in the memory 502.

[0140] The processor 501 is the control center of the computer device. It can connect various parts of the computer device using various interfaces and lines, and run or execute instructions stored in the memory 502 and call data stored in the memory 502. Optionally, the processor 501 may include one or more processing units. The processor 501 may integrate an application processor and a modem processor. The application processor mainly processes the operating system, user interface, and application programs, while the modem processor mainly handles wireless communications. It is understood that the modem processor may not be integrated into the processor 501. In some embodiments, the processor 501 and the memory 502 may be implemented on the same chip. In some embodiments, they may also be implemented on separate chips.

[0141] The processor 501 can be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, and can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of the present application can be performed using a combination of hardware and software modules in the processor.

[0142] The memory 502 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 502 may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (Random Access Memory, RAM), a static random access memory (Static Random Access Memory, SRAM), a programmable read-only memory (Programmable Read Only Memory, PROM), a read-only memory (Read Only Memory, ROM), an electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), a magnetic memory, a disk, an optical disk, etc. The memory 502 is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 502 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0143] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium storing a computer program executable by a computer device. When the program runs on the computer device, the computer device executes the steps of the above-mentioned fault detection method for a distributed cluster.

[0144] Based on the same inventive concept, an embodiment of the present application provides a computer program product, characterized in that the computer program product includes a computer program stored on a computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer device, the computer device executes the steps of the above-mentioned fault detection method for a distributed cluster.

[0145] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0146] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0147] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0148] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0149] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. A fault detection method for a distributed cluster, characterized in that: The nodes in the distributed cluster collaboratively process the same task in parallel, and the method includes: Before any node communicates with another node, it performs fault detection on the network status of the node to obtain the latest network status of the node; The latest network status is compared with the stored network status. If they are consistent, inter-node communication is performed through the communication link established by the nodes.

2. The method according to claim 1, wherein The method further comprises: If the latest network state is inconsistent with the stored network state, a network fault marker is generated, where the network fault marker is used to indicate a network fault of the node; Notifying the network fault marker in the distributed cluster and using the latest network state as the stored network state; Inter-node communication is performed by avoiding the communication link corresponding to the network fault represented by the network fault marker.

3. The method according to claim 2, wherein The network failure of the node includes a first communication link failure; The performing inter-node communication by avoiding the communication link indicated by the network fault marker includes: Determining a second communication link having the same transceiver node as the first communication link; The data to be transmitted corresponding to the first communication link is transmitted through the second communication link.

4. The method according to claim 3, wherein Before determining the second communication link having the same transceiver node as the first communication link, the method further includes: After interrupting all communication links between the transceiver nodes associated with the first communication link, a second communication link is re-established between the transceiver nodes.

5. The method according to claim 4, wherein The re-establishing of the second communication link between the transceiver nodes includes: The communication links between all nodes in the distributed cluster are re-planned in a unified manner to rebuild the communication links between the nodes.

6. The method according to any one of claims 1 to 5, wherein: Also includes: If the latest network state is consistent with the initial network state of the node, a network normal marker is generated and announced in the distributed cluster.

7. The method according to any one of claims 1 to 5, wherein: The performing fault detection on the network status of the node to obtain the latest network status of the node includes: determining a network type of the node; Determining each network attribute to be detected according to the network type of the node; Fault detection is performed on each of the network attributes to obtain the latest network status of the node in each of the network attributes.

8. A fault detection device for a distributed cluster, characterized in that: Each node in the distributed cluster collaboratively processes the same task in parallel, and the device includes: A detection module is used to perform fault detection on the network status of any node before any node performs inter-node communication to obtain the latest network status of the node; The comparison module is used to compare the latest network status with the stored network status, and if they are consistent, perform inter-node communication through the communication link established by the nodes.

9. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the methods of claims 1 to 7 when executing the program.

10. A computer-readable storage medium, characterized in that It stores a computer program that can be executed by a computer device. When the program is run on the computer device, the computer device executes the steps of any one of the methods according to claims 1 to 7.

Images