A method, apparatus and related equipment for fault handling in a distributed scenario
By monitoring the operational information of nodes and network interface cards (NICs), faults are identified and services are rescheduled. Combined with the sub-interface segmentation of NICs, the system instability caused by faults in distributed scenarios is resolved, achieving fault isolation and normalization of data transmission, thereby improving system stability and recovery efficiency.
Patent Information
- Application Number
- CN202611604455.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-10-16
- Publication Date
- 2026-11-17
AI Technical Summary
In distributed scenarios, the system stability of high-performance GPU clusters can be affected by faults, leading to low training efficiency. Existing technologies are unable to effectively handle these faults, making it a challenge to improve system stability.
By monitoring the operational and heartbeat information of nodes and network interface cards (NICs), faulty nodes and NICs are identified. A fault signal is broadcast to suspend services, a new service is created and scheduled to run on a healthy node. Fault isolation and data transmission are achieved by using the sub-interfaces of the NICs, ensuring normal data transmission.
It improves system stability and robustness, reduces the scope of failure impact, shortens task recovery time, and enhances network redundancy and robustness.
Smart Images

Figure CN123123355A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of deep learning technology, and in particular to a fault handling method, apparatus and related equipment in a distributed scenario. Background Technology
[0002] In current large-scale model training scenarios, distributed GPU clusters ranging from 1,000 to 10,000 GPUs have become the mainstream computing power carriers. Compared to the past, this has significantly improved computing power during large-scale model training. However, the expansion of GPU cluster scale has also significantly increased system vulnerability, leading to a higher probability of system failures. For example, when training a 405B parameter model with Meta, 419 failures occurred in a single run; during Llama 3.1 training, an average interruption occurred every 3 hours due to a failure. Various failures have affected the stability of 10,000-GPU cluster systems, severely impacting training efficiency. Therefore, how to handle failures and improve system stability in distributed scenarios has been a long-standing concern. Summary of the Invention
[0003] In view of this, this application provides a fault handling method, apparatus and related equipment in a distributed scenario to facilitate fault handling and improve system stability to a certain extent.
[0004] To achieve the above objectives, the following solution is proposed: A fault handling method in a distributed scenario includes: The system acquires the running information and heartbeat information of each node, as well as the status information of the network interface card (NIC) corresponding to each node. Each NIC is divided into N sub-interfaces, and the sub-interfaces of each NIC are numbered sequentially from 1 to N. Each sub-interface corresponds to a data plane, and sub-interfaces with the same number in different NICs correspond to the same data plane. Based on the heartbeat information of each node, determine whether there is a faulty node; If a fault exists, a fault signal is broadcast to other nodes so that they can suspend service operation. Based on the operational information of the faulty node, a new service is created and orchestrated, scheduled to run on the new node, and a restart signal is broadcast to other nodes so that they can restart the service, establish a connection with the new node, and keep all node services running synchronously. Based on the status information of the network interface card (NIC) for each node, determine whether there is a faulty NIC. If it exists, then identify the faulty sub-interface in the faulty network card; The data sent by the faulty sub-interface to the corresponding data plane is then sent to other nodes by the node corresponding to the faulty network card, so that the other nodes can send the data to the corresponding data plane through the corresponding network card, thus ensuring normal data transmission.
[0005] Optionally, based on the heartbeat information of each node, determine whether there is a faulty node, including: Based on the heartbeat information of each node, determine whether the heartbeat feedback of each node has timed out; If the timeout occurs, the corresponding node will be designated as the faulty node.
[0006] Optionally, after identifying the faulty sub-interface in the faulty network card, the following steps are also included: Determine whether the data sent by the faulty sub-interface involves cross-data plane transmission; If the data sent by the faulty sub-interface involves cross-data plane transmission, then determine the destination data plane; The data sent by the faulty sub-interface is sent to the corresponding sub-interface in the destination data plane of the same faulty network card, so that the sub-interface can send the data to the destination data plane.
[0007] Optionally, before scheduling to run on a new node, the following may also be included: Based on the creation and orchestration of new services, select a node from the candidate nodes that meets the operational requirements of the new service as the new node.
[0008] A fault handling device for a distributed scenario includes: The node information acquisition module is used to acquire the running information and heartbeat information of each node, as well as the status information of the network card corresponding to each node. Each network card is divided into N sub-interfaces. The sub-interfaces of each network card are numbered sequentially from 1 to N. Each sub-interface corresponds to a data plane, and sub-interfaces with the same number in different network cards correspond to the same data plane. The fault node processing module is used to determine whether there is a fault node based on the heartbeat information of each node; if there is, it broadcasts a fault signal to other nodes so that other nodes can suspend service operation; based on the operation information of the fault node, it creates and orchestrates a new service, schedules it to run on the new node, and broadcasts a restart signal to other nodes so that other nodes can restart the service, establish a connection with the new node, and keep all node services running synchronously. The faulty network interface card (NIC) processing module determines whether a faulty NIC exists based on the status information of the NICs corresponding to each node. If a faulty NIC exists, it identifies the faulty sub-interface within the faulty NIC. The data sent by the faulty sub-interface to the corresponding data plane is then sent by the node corresponding to the faulty NIC to other nodes, so that the other nodes can send the data to the corresponding data plane through the corresponding NIC, ensuring normal data transmission.
[0009] Optionally, the fault node processing module performs a process based on the heartbeat information of each node to determine whether a faulty node exists, including: Based on the heartbeat information of each node, determine whether the heartbeat feedback of each node has timed out; If the timeout occurs, the corresponding node will be designated as the faulty node.
[0010] Optionally, after performing the process of determining the faulty sub-interface in the faulty network interface card (NIC), the faulty NIC processing module further includes: Determine whether the data sent by the faulty sub-interface involves cross-data plane transmission; If the data sent by the faulty sub-interface involves cross-data plane transmission, then determine the destination data plane; The data sent by the faulty sub-interface is sent to the corresponding sub-interface in the destination data plane of the same faulty network card, so that the sub-interface can send the data to the destination data plane.
[0011] Optionally, the fault node handling module also includes the following steps before scheduling the process to run on a new node: Based on the creation and orchestration of new services, select a node from the candidate nodes that meets the operational requirements of the new service as the new node.
[0012] A fault handling device for a distributed scenario includes: a memory and a processor; The memory is used to store programs; The processor is used to execute the program to implement the various steps of the fault handling method in the distributed scenario as described above.
[0013] A readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the fault handling method in a distributed scenario as described in any of the preceding claims.
[0014] As can be seen from the above technical solutions, the fault handling method, apparatus, and related equipment provided in this application embodiment for a distributed scenario include: acquiring the operation information and heartbeat information of each node, as well as the status information of the network interface card (NIC) corresponding to each node. Each NIC is divided into N sub-interfaces, and the sub-interfaces of each NIC are numbered sequentially from 1 to N. Each sub-interface corresponds to a data plane, and sub-interfaces with the same number in different NICs correspond to the same data plane; determining whether there is a faulty node based on the heartbeat information of each node; if so, broadcasting a fault signal to other nodes so that other nodes can suspend service operation; creating and orchestrating new services based on the operation information of the faulty node, scheduling them to run on new nodes, and broadcasting a restart signal to other nodes so that other nodes can restart services, establish connections with new nodes, and keep all node services running synchronously; determining whether there is a faulty NIC based on the status information of the NIC corresponding to each node; if so, identifying the faulty sub-interface in the faulty NIC; and sending the data sent by the faulty sub-interface to the corresponding data plane by the node corresponding to the faulty NIC to other nodes so that other nodes can send the data to the corresponding data plane through the corresponding NIC, ensuring normal data transmission. This application monitors node failures and network interface card (NIC) failures. For node failures, it recreates and orchestrates the services of the failed node and schedules them to run on a new node to handle the failure. For NIC failures, it combines NIC segmentation to ensure the normal transmission of data from the failed sub-interface, thereby improving the stability of the system to a certain extent. Attached Figure Description
[0015] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0016] Figure 1 A flowchart of a fault handling method in a distributed scenario provided in an embodiment of this application; Figure 2 A schematic diagram of a network architecture provided for an embodiment of this application; Figure 3 This application provides a schematic diagram of the structure of a fault handling device in a distributed scenario. Figure 4 This is a hardware structure block diagram of a fault handling device in a distributed scenario provided in an embodiment of this application. Detailed Implementation
[0017] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0018] Figure 1 A flowchart of a fault handling method in a distributed scenario provided in this application embodiment is shown. The method may include the following steps: Step S100: Obtain the running information and heartbeat information of each node, as well as the status information of the network card corresponding to each node.
[0019] Specifically, each network card is divided into N sub-interfaces. The sub-interfaces of each network card are numbered sequentially from 1 to N. Each sub-interface corresponds to a data plane, and sub-interfaces with the same number in different network cards correspond to the same data plane.
[0020] Normally, one network interface card (NIC) corresponds to one uplink network interface. A very large cluster typically has 1,000 to 10,000 NICs, all interconnected in a full-mesh manner. Therefore, even if only one NIC fails, the failure can propagate throughout the entire network space, causing communication failures across the entire network. By dividing a physical NIC into multiple sub-interfaces, with each data plane corresponding to one sub-interface, network redundancy is improved through multiple sub-interfaces, and fault isolation is also achieved, preventing the spread of faults.
[0021] For example, an 800G network card can be divided into four sub-interfaces, each with 200G bandwidth. These four sub-interfaces correspond to four data planes. When one sub-interface fails, the other three sub-interfaces remain available, improving network redundancy and robustness, preventing the entire network card from becoming unusable, and limiting the failure to the network scope of a single sub-interface, thus achieving fault isolation.
[0022] refer to Figure 2 As shown, Figure 2 This is a schematic diagram of a network architecture provided in an embodiment of this application. Figure 2The diagram shows Node 1 and Node 2, as well as NIC 1 corresponding to Node 1 and NIC 2 corresponding to Node 2. NIC 1 and NIC 2 are each divided into 4 sub-interfaces. The sub-interfaces in the two NICs are numbered sequentially from 1 to 4. Each sub-interface corresponds to a data plane, and sub-interfaces with the same number in NIC 1 and NIC 2 all correspond to the same data plane. For example, sub-interface 1 of NIC 1 and sub-interface 1 of NIC 2 both correspond to data plane 1, and sub-interface 2 of NIC 1 and sub-interface 2 of NIC 2 both correspond to data plane 2.
[0023] Step S101: Based on the heartbeat information of each node, determine whether there is a faulty node.
[0024] Specifically, by acquiring the heartbeat information from each node in real time, it can be determined whether any node is faulty; a faulty node is then identified as a faulty node. If a faulty node is found, step S102 is executed.
[0025] Step S102: Broadcast the fault signal to other nodes.
[0026] Specifically, by broadcasting a fault signal to other healthy nodes, other nodes can suspend service operation. By waiting for the faulty node to handle the situation, other healthy nodes can ensure that the pace, operating status, and heartbeat information of each node are consistent.
[0027] Step S103: Based on the operating information of the faulty node, create and orchestrate new services, schedule them to run on the new node, and broadcast a restart signal to other nodes.
[0028] Specifically, for a faulty node, it is necessary to delete the services running on the faulty node. At this time, the service is not faulty, but the node running the service is faulty. Therefore, it is necessary to select a healthy node for the service running on the faulty node. Based on the running information of the faulty node, a new service is created and orchestrated. After it is scheduled to run on the new node, a restart signal can be broadcast to other nodes to notify them to restart the service and establish a connection with the new node to keep all node services running synchronously.
[0029] Furthermore, before scheduling to run on a new node, a node that meets the running requirements of the new service can be selected from the candidate nodes based on the newly created and orchestrated service.
[0030] In the above steps, by only processing the faulty node, creating and orchestrating new services based on the faulty node's operational information, and scheduling them to run on the new node, compared to re-orchestrending services on all nodes, the changes to the entire system's services caused by a single node failure are avoided, the impact of the failure is reduced, the scope of service changes is decreased, orchestration time can be saved to a certain extent, task recovery time can be shortened, and the robustness and stability of the entire cluster can be improved.
[0031] Step S104: Based on the status information of the network cards corresponding to each node, determine whether there are any faulty network cards.
[0032] Specifically, if a faulty network card is found, proceed to step S105.
[0033] Step S105: Identify the faulty sub-interface in the faulty network card.
[0034] Specifically, in the faulty network card, the faulty sub-interface is identified as the faulty sub-interface.
[0035] Step S106: The data sent by the faulty sub-interface to the corresponding data plane is sent to other nodes by the node corresponding to the faulty network card.
[0036] Specifically, after identifying the faulty sub-interface, the data sent by the faulty sub-interface to the corresponding data plane can be forwarded by the node corresponding to the faulty network interface card (NIC) to other nodes. These other nodes then forward the data to their respective data planes via their NICs, ensuring normal data transmission. For example, if sub-interface 1 in NIC 1 of node 1 fails, and sub-interface 1 cannot send data to the corresponding data plane 1, node 1 can forward the data that sub-interface 1 needs to send to node 2 via the switching network. Then, node 2's NIC 2 will forward the data to data plane 1 via sub-interface 1, thus ensuring normal data transmission.
[0037] The above embodiments provide a fault handling method in a distributed scenario, including: acquiring the running information and heartbeat information of each node, as well as the status information of the network interface card (NIC) corresponding to each node. Each NIC is divided into N sub-interfaces, and the sub-interfaces of each NIC are numbered sequentially from 1 to N. Each sub-interface corresponds to a data plane, and sub-interfaces with the same number in different NICs correspond to the same data plane; based on the heartbeat information of each node, determining whether there is a faulty node; if so, broadcasting a fault signal to other nodes so that other nodes can suspend service operation; based on the running information of the faulty node, creating and orchestrating a new service, scheduling it to run on a new node, and broadcasting a restart signal to other nodes so that other nodes can restart the service, establish a connection with the new node, and keep all node services running synchronously; based on the status information of the NIC corresponding to each node, determining whether there is a faulty NIC; if so, identifying the faulty sub-interface in the faulty NIC; and forwarding the data sent by the faulty sub-interface to the corresponding data plane to other nodes by the node corresponding to the faulty NIC, so that other nodes can send the data to the corresponding data plane through the corresponding NIC, ensuring normal data transmission. This application monitors node failures and network interface card (NIC) failures. For node failures, it recreates and orchestrates the services of the failed node and schedules them to run on a new node to handle the failure. For NIC failures, it combines NIC segmentation to ensure the normal transmission of data from the failed sub-interface, thereby improving the stability of the system to a certain extent.
[0038] In some embodiments of this application, step S101, determining whether a faulty node exists based on the heartbeat information of each node, may include: S11. Based on the heartbeat information of each node, determine whether the heartbeat feedback of each node has timed out.
[0039] Specifically, each node reports its current status via heartbeat feedback at a fixed frequency. Upon receiving heartbeat information from a node, it is generally assumed that the current node and network topology are in good condition. If a node experiences a heartbeat timeout, step S12 is executed.
[0040] S12. Designate the corresponding node as the faulty node.
[0041] Specifically, nodes that time out of heartbeat feedback are designated as fault nodes.
[0042] In some embodiments of this application, cross-data plane transmission may occur. Therefore, after determining the faulty sub-interface in the faulty network interface card, the following may also be included: S21. Determine whether the data sent by the faulty sub-interface involves cross-data plane transmission.
[0043] Specifically, if the data sent by the faulty sub-interface involves cross-data plane transmission, then step S22 is executed; if the data sent by the faulty sub-interface does not involve cross-data plane transmission, then step S106 is executed.
[0044] S22. Determine the target data plane.
[0045] Specifically, the destination data plane can be the data plane that the data sent by the faulty sub-interface ultimately needs to reach.
[0046] S23. Send the data sent by the faulty sub-interface to the sub-interface corresponding to the destination data plane in the same faulty network card.
[0047] Specifically, assuming the destination data plane is data plane 4 and the faulty sub-interface is sub-interface 3 of network card 1, sub-interface 3 originally needs to send data from data plane 3 to data plane 4 corresponding to sub-interface 4. However, sub-interface 3 has failed. At this time, the data that sub-interface 3 needs to send can be sent to sub-interface 4 of network card 1 through loopback, and sub-interface 4 will send the data to data plane 4 so that the data can be sent to the destination data plane 4 normally.
[0048] Although the above steps break the isolation between different sub-interfaces, they avoid service interruption due to network failures. By utilizing the characteristics of multiple sub-interfaces and multi-node full interconnection and multi-path, network robustness is improved, and fault handling at the network layer is achieved.
[0049] The following describes the fault handling device in a distributed scenario provided in the embodiments of this application. The fault handling device in a distributed scenario described below can be referred to in correspondence with the fault handling method in a distributed scenario described above.
[0050] Figure 3 This application provides a schematic diagram of a fault handling device structure in a distributed scenario, which may include: The node information acquisition module 10 is used to acquire the running information and heartbeat information of each node, as well as the status information of the network card corresponding to each node. The network card is divided into N sub-interfaces. The sub-interfaces of each network card are numbered sequentially from 1 to N. Each sub-interface corresponds to a data plane, and the sub-interfaces with the same number in different network cards correspond to the same data plane. The fault node processing module 20 is used to determine whether there is a fault node based on the heartbeat information of each node; if there is, it broadcasts a fault signal to other nodes so that other nodes can suspend service operation; based on the operation information of the fault node, it creates and orchestrates a new service, schedules it to run on the new node, and broadcasts a restart signal to other nodes so that other nodes can restart the service, establish a connection with the new node, and keep all node services running synchronously. The faulty network interface card (NIC) processing module 30 determines whether a faulty NIC exists based on the status information of the NICs corresponding to each node. If a faulty NIC exists, it identifies the faulty sub-interface within the faulty NIC. The data sent by the faulty sub-interface to the corresponding data plane is then sent by the node corresponding to the faulty NIC to other nodes, so that other nodes can send the data to the corresponding data plane through the corresponding NIC, ensuring normal data transmission.
[0051] The above embodiment provides a fault handling device in a distributed scenario, including: a node information acquisition module 10, used to acquire the running information and heartbeat information of each node, as well as the status information of the network interface card (NIC) corresponding to each node. Each NIC is divided into N sub-interfaces, and the sub-interfaces of each NIC are numbered sequentially from 1 to N. Each sub-interface corresponds to a data plane, and sub-interfaces with the same number in different NICs correspond to the same data plane; a fault node processing module 20, used to determine whether a fault node exists based on the heartbeat information of each node; if so, it broadcasts a fault signal to other nodes so that other nodes can suspend service operation; based on... The system monitors the operation information of faulty nodes, creates and orchestrates new services, schedules them to run on new nodes, and broadcasts restart signals to other nodes to restart their services. It also establishes connections with the new nodes to maintain synchronized service operation across all nodes. The faulty network interface card (NIC) processing module 30 determines whether a faulty NIC exists based on the status information of the NICs corresponding to each node. If a faulty NIC exists, it identifies the faulty sub-interface within the faulty NIC. Data sent by the faulty sub-interface to the corresponding data plane is then forwarded by the node corresponding to the faulty NIC to other nodes, allowing them to transmit data to the corresponding data plane via the corresponding NIC, ensuring normal data transmission. This application improves system stability to some extent by monitoring node and NIC failures. For node failures, it recreates and orchestrates services on faulty nodes and schedules them to run on new nodes. For NIC failures, it combines NIC segmentation to ensure normal transmission of data from faulty sub-interfaces.
[0052] Optionally, the fault node processing module 20 may perform a process of determining whether a fault node exists based on the heartbeat information of each node, which may include: Based on the heartbeat information of each node, determine whether the heartbeat feedback of each node has timed out; If the timeout occurs, the corresponding node will be designated as the faulty node.
[0053] Optionally, after performing the process of determining the faulty sub-interface in the faulty network interface card (NIC), the faulty NIC processing module 30 may further include: Determine whether the data sent by the faulty sub-interface involves cross-data plane transmission; If the data sent by the faulty sub-interface involves cross-data plane transmission, then determine the destination data plane; The data sent by the faulty sub-interface is sent to the corresponding sub-interface in the destination data plane of the same faulty network card, so that the sub-interface can send the data to the destination data plane.
[0054] Optionally, the fault node processing module 20 may further include the following before the process of scheduling to run on a new node: Based on the creation and orchestration of new services, select a node from the candidate nodes that meets the operational requirements of the new service as the new node.
[0055] This application also provides a fault handling device for a distributed scenario. Figure 4 The hardware structure block diagram of the fault handling device in a distributed scenario is shown. (Refer to...) Figure 4 The hardware structure of a fault handling device in a distributed scenario may include: at least one processor 1, at least one communication interface 2, at least one memory 3, and at least one communication bus 4. In this embodiment of the application, the number of processor 1, communication interface 2, memory 3, and communication bus 4 is at least one, and processor 1, communication interface 2, and memory 3 communicate with each other through communication bus 4; Processor 1 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. Memory 3 may include high-speed RAM, and may also include non-volatile memory, such as at least one disk storage device; The memory stores a program, and the processor can call the program stored in the memory. The program is used to implement each processing flow in the aforementioned fault handling method in a distributed scenario.
[0056] This application embodiment also provides a storage medium that can store a program suitable for execution by a processor, the program being used to implement various processing flows in the aforementioned fault handling method in a distributed scenario.
[0057] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.
[0058] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined with each other, and the same or similar parts can be referred to each other.
[0059] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A fault handling method in a distributed scenario, characterized in that, include: The system acquires the running information and heartbeat information of each node, as well as the status information of the network interface card (NIC) corresponding to each node. Each NIC is divided into N sub-interfaces, and the sub-interfaces of each NIC are numbered sequentially from 1 to N. Each sub-interface corresponds to a data plane, and sub-interfaces with the same number in different NICs correspond to the same data plane. Based on the heartbeat information of each node, determine whether there is a faulty node; If a fault exists, a fault signal is broadcast to other nodes so that they can suspend service operation. Based on the operational information of the faulty node, a new service is created and orchestrated, scheduled to run on the new node, and a restart signal is broadcast to other nodes so that they can restart the service, establish a connection with the new node, and keep all node services running synchronously. Based on the status information of the network interface card (NIC) for each node, determine whether there is a faulty NIC. If it exists, then identify the faulty sub-interface in the faulty network card; The data sent by the faulty sub-interface to the corresponding data plane is then sent to other nodes by the node corresponding to the faulty network card, so that the other nodes can send the data to the corresponding data plane through the corresponding network card, thus ensuring normal data transmission.
2. The method according to claim 1, characterized in that, Based on the heartbeat information of each node, determine whether there is a faulty node, including: Based on the heartbeat information of each node, determine whether the heartbeat feedback of each node has timed out; If the timeout occurs, the corresponding node will be designated as the faulty node.
3. The method according to claim 1 or 2, characterized in that, After identifying the faulty sub-interface within the faulty network card, the following steps are also included: Determine whether the data sent by the faulty sub-interface involves cross-data plane transmission; If the data sent by the faulty sub-interface involves cross-data plane transmission, then determine the destination data plane; The data sent by the faulty sub-interface is sent to the corresponding sub-interface in the destination data plane of the same faulty network card, so that the sub-interface can send the data to the destination data plane.
4. The method according to claim 1 or 2, characterized in that, Before scheduling to run on a new node, the following is also included: Based on the creation and orchestration of new services, select a node from the candidate nodes that meets the operational requirements of the new service as the new node.
5. A fault handling device for a distributed scenario, characterized in that, include: The node information acquisition module is used to acquire the running information and heartbeat information of each node, as well as the status information of the network card corresponding to each node. Each network card is divided into N sub-interfaces. The sub-interfaces of each network card are numbered sequentially from 1 to N. Each sub-interface corresponds to a data plane, and sub-interfaces with the same number in different network cards correspond to the same data plane. The fault node processing module is used to determine whether there is a fault node based on the heartbeat information of each node; if there is, it broadcasts a fault signal to other nodes so that other nodes can suspend service operation; based on the operation information of the fault node, it creates and orchestrates a new service, schedules it to run on the new node, and broadcasts a restart signal to other nodes so that other nodes can restart the service, establish a connection with the new node, and keep all node services running synchronously. The faulty network interface card (NIC) processing module determines whether a faulty NIC exists based on the status information of the NICs corresponding to each node; if so, it identifies the faulty sub-interface within the faulty NIC. The data sent by the faulty sub-interface to the corresponding data plane is then sent to other nodes by the node corresponding to the faulty network card, so that the other nodes can send the data to the corresponding data plane through the corresponding network card, thus ensuring normal data transmission.
6. The apparatus according to claim 5, characterized in that, The fault node handling module performs a process based on the heartbeat information of each node to determine whether a fault node exists, including: Based on the heartbeat information of each node, determine whether the heartbeat feedback of each node has timed out; If the timeout occurs, the corresponding node will be designated as the faulty node.
7. The apparatus according to claim 5 or 6, characterized in that, After executing the process of identifying the faulty sub-interface in the faulty network interface card (NIC), the faulty NIC processing module also includes: Determine whether the data sent by the faulty sub-interface involves cross-data plane transmission; If the data sent by the faulty sub-interface involves cross-data plane transmission, then determine the destination data plane; The data sent by the faulty sub-interface is sent to the corresponding sub-interface in the destination data plane of the same faulty network card, so that the sub-interface can send the data to the destination data plane.
8. The apparatus according to claim 5 or 6, characterized in that, Before the fault node handling module is scheduled to run on a new node, it also includes: Based on the creation and orchestration of new services, select a node from the candidate nodes that meets the operational requirements of the new service as the new node.
9. A fault handling device for a distributed scenario, characterized in that, include: Memory and processor; The memory is used to store programs; The processor is used to execute the program to implement the various steps of the fault handling method in a distributed scenario as described in any one of claims 1-4.
10. A readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the fault handling method in a distributed scenario as described in any one of claims 1-4.
