Network fault tolerance method, device, equipment, medium and product
By allocating primary and backup network interface cards (NICs) to each GPU in the GPU cluster and utilizing the primary NIC's own detection and switching capabilities, the resource consumption and time latency issues caused by external monitoring systems are resolved, thereby improving network fault tolerance and the stability of large-scale model tasks.
Patent Information
- Application Number
- CN202510819825.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-18
- Publication Date
- 2025-11-04
AI Technical Summary
In the event of a network failure in a GPU cluster, existing technologies require an external monitoring system to switch network cards, which results in significant resource consumption and a long switching time, affecting the stability of large model tasks.
By allocating a primary network interface card (NIC) and a backup NIC to each GPU in the GPU cluster, the primary NIC can detect its status and switch to the backup NIC in case of failure. The target communication library is used for data switching and recovery tasks, avoiding the resource overhead and time consumption of external monitoring systems.
It improves network fault tolerance, reduces network card switching time, lowers resource consumption, and enhances the stability and flexibility of large model tasks, making it suitable for inference and training tasks.
Smart Images
Figure CN120896837A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of artificial intelligence technology, specifically to the fields of network communication, cloud computing, computing power, and large models, and particularly to a network fault tolerance method, device, equipment, medium, and product. Background Technology
[0002] With the development of large-scale artificial intelligence (AI) models, there is a need for large-scale graphics processing unit (GPU) clusters.
[0003] A GPU cluster consists of multiple GPUs that communicate with each other over a network to complete large model tasks. To ensure the stable execution of these large model tasks, a certain level of network fault tolerance is required. Summary of the Invention
[0004] This disclosure provides a network fault tolerance method, apparatus, device, medium, and product.
[0005] According to one aspect of this disclosure, a network fault tolerance method is provided, comprising: obtaining the network interface card (NIC) status of a primary NIC in a GPU cluster; the GPU cluster being used to execute a large model task; the NIC status being obtained based on the primary NIC itself; suspending the execution of the large model task when the NIC status includes a fault state; after suspending the execution of the large model task, switching the data on the faulty NIC corresponding to the faulty state to a backup NIC corresponding to the faulty NIC; the backup NIC being allocated based on a target communication library, the target communication library being used to provide communication services for the GPU cluster; and resuming the execution of the large model task after determining, based on the target communication library, that the GPU cluster has completed the NIC switching.
[0006] According to another aspect of this disclosure, a network fault-tolerant device is provided, comprising: an acquisition module, configured to acquire the network interface card (NIC) status of a primary NIC in a GPU cluster; the GPU cluster is used to execute a large model task; the NIC status is obtained based on the primary NIC itself; an abort module, configured to abort the execution of the large model task when the NIC status includes a fault state; a switching module, configured to switch data on the faulty NIC corresponding to the fault state to a backup NIC corresponding to the faulty NIC after the execution of the large model task is aborted; the backup NIC is allocated based on a target communication library, the target communication library being used to provide communication services for the GPU cluster; and a recovery module, configured to resume the execution of the large model task after determining, based on the target communication library, that the GPU cluster has completed the NIC switching.
[0007] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to said at least one processor; wherein the memory stores instructions executable by said at least one processor, said instructions being executed by said at least one processor to enable said at least one processor to perform the method as described in any of the foregoing aspects.
[0008] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are configured to cause the computer to perform the method according to any of the preceding aspects.
[0009] According to another aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method according to any of the preceding aspects.
[0010] The technical solution disclosed herein can improve network fault tolerance performance.
[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description
[0012] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:
[0013] Figure 1 This is a schematic diagram based on the first embodiment of the present disclosure;
[0014] Figure 2 This is a schematic diagram according to the second embodiment of the present disclosure;
[0015] Figure 3 This is a schematic diagram according to the third embodiment of the present disclosure;
[0016] Figure 4 This is a schematic diagram of operator operations provided according to embodiments of this disclosure;
[0017] Figure 5 This is a schematic diagram of data transmission according to an embodiment of the present disclosure;
[0018] Figure 6 This is a schematic diagram according to the fourth embodiment of the present disclosure;
[0019] Figure 7 This is a schematic diagram according to the fifth embodiment of the present disclosure;
[0020] Figure 8This is a schematic diagram of an electronic device used to implement the network fault tolerance method of the embodiments of this disclosure. Detailed Implementation
[0021] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.
[0022] Figure 1 This is a schematic diagram based on the first embodiment of the present disclosure. This embodiment provides a network fault tolerance method, the method comprising:
[0023] 101. Obtain the network card status of the primary network card in the GPU cluster; the GPU cluster is used to execute large model tasks; the network card status is obtained based on the primary network card itself.
[0024] 102. If the network card is in a fault state, stop executing the large model task.
[0025] 103. After the execution of the large model task is terminated, the data on the faulty network card corresponding to the faulty state is switched to the backup network card corresponding to the faulty network card; the backup network card is allocated based on the target communication library, which is used to provide communication services for the GPU cluster.
[0026] 104. After determining that the GPU cluster has completed the network card switching based on the target communication library, resume the execution of the large model task.
[0027] In related technologies, in order to achieve network fault tolerance, the network status can be monitored by an external monitoring system, and network card switching can be performed when the network fails. However, this method requires the introduction of additional external resources, consumes more resources, and takes a long time to switch.
[0028] In this embodiment, the primary network interface card (NIC) learns its own NIC status, and when the NIC status includes a fault state, it performs task abort and NIC switching operations. After the NIC switching is complete, task execution resumes. This utilizes the GPU cluster's own resources for fault detection and handling. Compared to introducing an external monitoring system, this saves resource overhead, reduces NIC switching time, minimizes the impact on large model tasks, and improves their stability. Furthermore, a backup NIC is allocated based on the target communication library. Since the target communication library is a software resource, a backup NIC can be allocated based on software resources without changing hardware configuration, reducing implementation complexity. This improves network fault tolerance performance.
[0029] The method in this embodiment can be executed by any GPU in the GPU cluster.
[0030] GPU clusters are used to execute large model tasks.
[0031] Specifically, large models can be deployed on GPU clusters to complete large model tasks.
[0032] Large model tasks can specifically include: inference tasks, or training tasks.
[0033] In this context, the reasoning task refers to performing reasoning using a large model. For example, if the large model is a text-to-image model, then it can be used to generate images based on user-input text.
[0034] The training task refers to training a large model. For example, the text-to-image model mentioned above is obtained through a training process, which can be specifically completed through a training task.
[0035] Different GPUs in a GPU cluster can communicate with each other via a network.
[0036] To enable network communication, a network interface card (NIC) can be assigned to each GPU in the GPU cluster, and the GPUs communicate with each other via the NIC.
[0037] In related technologies, each GPU is typically assigned a network interface card (NIC), and different GPUs communicate with each other through their respective NICs.
[0038] During operation, the network interface card (NIC) may malfunction. Related technologies employ an external monitoring system to monitor NIC failures. If a failure occurs, a new NIC is reassigned to the GPU, and the NIC switchover is completed through operations such as restarting. However, this method is resource-intensive and time-consuming.
[0039] To improve network fault tolerance, in this embodiment, two network cards can be allocated to each GPU, referred to as the primary network card and the backup network card. Initially, data is transmitted through the primary network card. After the primary network card fails, the data on the primary network card is switched to the backup network card.
[0040] In related technologies, each GPU is assigned a network interface card (NIC). This can be implemented using existing communication libraries, which provide communication services for the GPU cluster. For example, the GPUs communicate based on these existing libraries. Furthermore, when initializing large model tasks, each GPU can be assigned a NIC based on an existing communication library.
[0041] In this embodiment, the existing communication library can be improved. The improved communication library is called the target communication library. The target communication library can provide communication services for the GPU cluster and has the ability to allocate two network cards (primary network card and backup network card) to each GPU.
[0042] Specifically, when an existing communication library allocates a network card to each GPU, it typically allocates the network card that is physically closest to the GPU to that GPU.
[0043] In this embodiment, a target communication library can be obtained by programming an existing communication library. This target communication library can allocate a primary network card and a backup network card to each GPU when allocating network cards. For example, the network card that is connected to the GPU and is physically closest to it can be designated as the primary network card of the GPU, and the network card that is connected to the GPU and is physically second closest to it can be designated as the backup network card of the GPU.
[0044] In this way, a primary and a backup network interface card (NIC) can be assigned to the GPU via the target communication library. This allows data to be switched to the backup NIC in the event of a failure of the primary NIC.
[0045] Figure 2 This is a schematic diagram according to the second embodiment of this disclosure. This embodiment provides an overall process of a network fault tolerance method.
[0046] like Figure 2 As shown, the overall process includes: network card status detection phase 201, task abort phase 202, network card switching phase 203, and task recovery phase 204.
[0047] Phase 201 of the network interface card (NIC) status detection mainly involves obtaining the NIC status of the primary NIC in the GPU cluster, such as normal or faulty status.
[0048] Phase 202, which involves suspending the execution of large model tasks, such as inference tasks, when the network interface card (NIC) is in a state including a fault state, i.e., when at least one primary NIC fails.
[0049] The network card switching phase 203 mainly involves performing the network card switching operation, that is, switching the data on the faulty primary network card to the backup network card.
[0050] Phase 204 of the recovery task mainly involves resuming the execution of large model tasks after the GPU cluster has completed the network card switching.
[0051] In related technologies, obtaining network card status and troubleshooting information through external monitoring systems requires significant resources and time.
[0052] In this embodiment, the network card status can be obtained based on the primary network card in the GPU cluster, and then the network card status perception and fault handling can be completed based on the GPU cluster's own resources.
[0053] Figure 3 This is a schematic diagram based on the third embodiment of the present disclosure, which provides an overall system corresponding to a GPU cluster.
[0054] like Figure 3 As shown, a GPU cluster can be composed of GPUs in multiple servers. Each server can include multiple GPUs, and each GPU can be assigned a primary network interface card (NIC) and a backup NIC.
[0055] For example, the first server includes a first GPU and a second GPU, the second server includes a third GPU and a fourth GPU, and the first GPU to the fourth GPU form a GPU cluster.
[0056] Each GPU will be assigned a primary network interface card (NIC) and a backup NIC. For example, the first GPU will be assigned a primary NIC and a backup NIC, and the primary and backup NICs of the other GPUs will be assigned in a similar manner.
[0057] GPUs on different servers can communicate over a network. For example, the first GPU can communicate with the second GPU via a network interface card (NIC).
[0058] Initially, the two communicate through the primary network interface card (NIC). For example, the first GPU sends data to the first primary NIC, which then sends the data to the third primary NIC via a network communication connection. The third primary NIC then sends the data to the third GPU.
[0059] If the primary network card fails, such as the first primary network card failing, the first GPU will send the data to the first backup network card. The first backup network card will then send the data to the third primary network card via network communication connection. The third primary network card will then send the data to the third GPU.
[0060] To address the issues associated with using external monitoring systems to monitor network interface card (NIC) status, this embodiment utilizes the primary NIC itself for detection to obtain the NIC status of each primary NIC in the GPU cluster.
[0061] Specifically, primary network interface cards (NICs) in different servers can form a probe group. Within a probe group, the NIC status of each primary NIC in the group can be obtained through active probing. Furthermore, different probe groups can be synchronized through aggregated communication, so that each primary NIC in the GPU cluster can know the NIC status of other primary NICs.
[0062] For example, refer to Figure 3The first server includes a first primary network interface card (NIC) and a second primary NIC, and the second server includes a third primary NIC and a fourth primary NIC. The first primary NIC and the third primary NIC can form a first detection group, and the second primary NIC and the fourth primary NIC can form a second detection group.
[0063] For a given probe group, the primary network interface card (NIC) within that probe group can obtain the NIC status of each primary NIC within that probe group through active probing.
[0064] Within different probe groups, network interface card (NIC) status synchronization can be performed based on set communication primitives. This synchronization process allows the NIC status of each primary NIC in one probe group to be synchronized to the primary NICs in other probe groups.
[0065] In the active detection method, network cards within the same detection group can communicate with each other, so that any network card in the same detection group can know the network card status of other network cards.
[0066] Taking the first detection group as an example, the first primary network card and the third primary network card can communicate with each other. Based on this communication process, the first primary network card can know the network card status of the third primary network card, and the third primary network card can know the status of the first primary network card.
[0067] Furthermore, communication between network interface cards (NICs) can employ the Remote Direct Memory Access (RDMA) protocol. For example, a primary NIC and a third primary NIC can communicate with each other using RDMA to obtain NIC status information. RDMA is a direct memory access technology that transfers data directly from the memory of one computer to another without the intervention of the operating systems of both computers. This reduces network transmission latency and is suitable for use in large-scale computing clusters.
[0068] In this way, within the same detection group, the network card status of each primary network card in the detection group can be obtained through active detection.
[0069] For different detection groups, network interface card (NIC) status synchronization can be performed based on set communication primitives.
[0070] For example, based on the set communication primitive, the first primary network interface card (NIC) can know the NIC status of the second and fourth primary NICs in the second probe group.
[0071] A collection communication primitive is, for example, the AllGather primitive. The AllGather primitive is a many-to-many communication primitive with multiple data senders and multiple data receivers. It can collect data from multiple nodes in a cluster onto a master node (Gather), and then distribute the collected data to other nodes (broadcast), that is, collect all the data in the cluster to all nodes.
[0072] Based on the AllGather primitive, the network card status of each primary network card in one probe group can be synchronized to the primary network cards in other probe groups.
[0073] In this way, the status of each primary network interface card (NIC) can be obtained based on the primary NIC itself.
[0074] The network interface card (NIC) status can be either normal or faulty. Based on this, the primary NIC corresponding to the faulty status is the faulty NIC, meaning that the primary NIC has failed.
[0075] When the network interface card (NIC) status includes a fault state, i.e., when at least one primary NIC in the GPU cluster fails, task abort and NIC switching operations can be performed, and the task execution process can be resumed after the GPU cluster completes the NIC switching.
[0076] In addition, the network card status of each primary network card can be written to shared memory for use in subsequent processes.
[0077] Shared memory refers to memory shared between the CPU and GPU.
[0078] In addition to the GPU, the server also includes a Central Processing Unit (CPU) to perform initialization and scheduling of related operations. For example, a new thread can be created in the CPU to control the primary network interface card (NIC) for active probing and NIC status synchronization.
[0079] On the CPU side, corresponding operations can also be performed based on the network card status. Therefore, the CPU also needs to know the network card status.
[0080] Therefore, the network card status mentioned above can be written into shared memory, so that the CPU and GPU can read the network card status in subsequent processes.
[0081] For the GPU, the network card status of each primary network interface card (NIC) can be obtained from this shared memory. After determining the network status, including fault conditions, operations such as task abort and NIC switching are performed.
[0082] In this embodiment, by obtaining the network card status from shared memory, the accuracy of the network card status can be guaranteed, and the network fault tolerance can be improved.
[0083] In some embodiments, for task abort: a normal exit operation can be performed on the operator corresponding to any GPU in the GPU cluster, and the resources occupied by the operator can be released after the operator exits normally; the operator is obtained based on the large model task.
[0084] In a GPU cluster scenario, after a large model task is created, it can be mapped to an operator. Then, each GPU in the cluster executes its corresponding operator. Each GPU can have one or more operators.
[0085] Operators are used to characterize data operations, such as the convolution operator, which performs convolution operations on data.
[0086] For current GPUs, during task termination, a normal exit operation can be performed on the operator, and the resources occupied by the operator can be released after the operator exits normally.
[0087] Normal exit for an operator means that the operator actively stops its execution process, which ensures the correctness of the overall process. Operator execution status can also include abnormal exit, which occurs when the operator is forced to stop its execution process, potentially leading to errors such as incorrect data transmission.
[0088] In this embodiment, performing a normal exit operation on the operator can ensure the correctness of data processing; otherwise, if a normal exit operation is not performed on the operator, the operator may exit abnormally, which may lead to incorrect data transmission and affect overall performance.
[0089] After an operator exits normally, its resources can be released. For example, the cache resources occupied by the operator can be released, and subsequent processes can reuse the cache resources.
[0090] Furthermore, different operation procedures can be executed for different operators.
[0091] Figure 4 This is a schematic diagram of operator operations provided according to embodiments of this disclosure.
[0092] During the execution of large model tasks, the CPU can map the large model task into operators and submit the operators to the GPU for processing.
[0093] like Figure 4 As shown, when the execution of a large model task is aborted, the CPU can stop submitting new operators.
[0094] For GPUs, the GPU performs exit operations on its operators.
[0095] Specifically, for operators that have been submitted but not yet executed, i.e., operators submitted by the CPU to the GPU but not yet processed by the GPU, a normal exit operation is performed directly on them.
[0096] For operators that are in execution—that is, operators submitted by the CPU to the GPU and executed by the GPU but not yet finished—a timeout exit is performed. This means waiting a set amount of time before performing a normal exit operation. Specifically, timeout exit logic code can be added to the target communication library, allowing the GPU to perform timeout exit operations on the corresponding operators.
[0097] In this embodiment, by performing a normal exit operation on the operator, uncontrollable errors can be avoided, and the network fault tolerance performance can be improved.
[0098] In some embodiments, for network interface card (NIC) switching: data transmitted based on the primary link can be switched to a backup link; the primary link includes: the link between the faulty NIC and the primary NIC of the peer GPU; the backup link includes: the link between the backup NIC corresponding to the faulty NIC and the primary NIC of the peer GPU, and the backup link is established after the fault status is obtained.
[0099] The peer GPU refers to the GPU that communicates with the current GPU. For example, if the current GPU is the sender, the peer GPU is the receiver; or if the current GPU is the receiver, the peer GPU is the sender.
[0100] Initially, the current GPU and the peer GPU communicate through the primary link. After the network card is switched, the current GPU and the peer GPU communicate through the backup link.
[0101] The primary link includes the link between the primary network interface card (NIC) corresponding to the current GPU and the primary NIC corresponding to the peer GPU, while the backup link includes the link between the backup NIC corresponding to the current GPU and the primary NIC corresponding to the peer GPU.
[0102] For example, refer to Figure 3 If the two communicating parties are the first GPU and the third GPU, then either one can be designated as the current GPU and the other as the peer GPU. For example, if the first GPU is designated as the current GPU, then the peer GPU is the third GPU.
[0103] When the primary network interface card (NIC) is functioning normally, communication is conducted through the primary link. For example, the primary link includes the link between the first primary NIC and the third primary NIC.
[0104] After the primary network interface card (NIC) fails, data will be switched to the backup link. For example, the backup link may include the link between the primary backup NIC and the third primary NIC.
[0105] In this way, the data transmitted on the first primary network card can be switched to the first backup network card.
[0106] Generally speaking, the link between primary network interface cards (NICs) is established during initialization. For example, when initializing a large model task, a link is established between the first primary NIC and the third primary NIC.
[0107] If a link is established between the backup network interface card (NIC) and the primary NIC during initialization, resource waste will occur. For example, if a link is pre-established between the first backup NIC and the third primary NIC, and between the second backup NIC and the third primary NIC, the link between the second backup NIC and the third primary NIC will be useless if the second primary NIC does not subsequently fail, resulting in resource waste.
[0108] Therefore, in this embodiment, a backup link can be established only after a failure is determined.
[0109] For example, during initialization, only links between primary network interface cards (NICs) are established. When a primary NIC (such as the first primary NIC) fails, links are established between the backup NIC corresponding to the failed primary NIC and other primary NICs, such as establishing a link between the first backup NIC and the third primary NIC, and establishing a link between the first backup NIC and the fourth primary NIC.
[0110] In this embodiment, by establishing a backup link after a failure occurs, a corresponding backup link can be created based on the actual faulty network card, allowing for targeted creation of backup links and avoiding resource waste.
[0111] In some embodiments, for task recovery: based on the set communication primitives provided by the target communication library, obtain the network card switching status of any GPU in the GPU cluster; if the network card switching status is complete, resume execution of the large model task.
[0112] The target communication library can provide set communication primitives, such as the AllGather primitive mentioned above. Based on this AllGather primitive, each GPU can learn the network interface card (NIC) switching status of other GPUs. The NIC switching status is used to indicate whether the NIC switching is complete. After any GPU in the GPU cluster has completed the NIC switching, it is determined that the GPU cluster has completed the NIC switching, and the execution of the large model task resumes after the NIC switching of the GPU cluster is completed.
[0113] For example, a GPU can re-receive and process new operators.
[0114] In this embodiment, the network card switching status of any GPU in the GPU cluster is obtained based on the set communication primitive, which can efficiently and accurately obtain the network card switching status, and thus accurately resume the execution of large model tasks.
[0115] In some embodiments, if all network card switching states are complete, the target data location corresponding to the large model task is determined, and the execution of the large model task is resumed based on the target data location;
[0116] The target data location is determined based on the receiving stop location and the sending stop location, which are recorded when the execution of the large model task is terminated.
[0117] Figure 5 This is a schematic diagram of data transmission according to an embodiment of the present disclosure.
[0118] For large model tasks, in some cases, GPUs can obtain data from CPUs. For example, the sending GPU (such as the first GPU) can obtain data from the CPU within its host server (such as the first server); this process can be called data transfer. The CPU can also send this data via a network interface card (NIC). Similarly, at the receiving end, the receiving GPU can also transfer data from its corresponding CPU.
[0119] When a large model task is terminated, the GPU stops transferring data from the CPU and records the corresponding stopping position. For example, the sending end records the sending stopping position and the receiving end records the receiving stopping position. This position can be the number of the data unit (such as the data packet) in the data.
[0120] When resuming a large model task, the sender and receiver can negotiate the target data location. For example, the sender and receiver can send their own send stop position or receive stop position to the other end. In this way, both the sender and receiver can know the above-mentioned send stop position and receive stop position. Then, based on these two positions, the target data location is determined. If the two are the data packet numbers, the position with the smaller number is taken as the target data location. After that, the large model task is resumed based on the target data location.
[0121] For example, the sender can resend data to the receiver from the target data location.
[0122] In this embodiment, the large model task is executed based on the target data location recovery, which can achieve data consistency between the sending end and the receiving end and ensure the continuity of communication data.
[0123] In conjunction with the above, this disclosure can also provide an overall process.
[0124] Figure 6This is a schematic diagram based on the fourth embodiment of the present disclosure, which provides a network fault tolerance method. For example... Figure 6 As shown, the method includes:
[0125] 601. Execute the service initialization process.
[0126] 602. Based on the probe group created in the above initialization process, obtain the network card status of the primary network card in the GPU cluster and write it to the shared memory.
[0127] 603. Obtain the network card status from shared memory.
[0128] 604. If the network card status includes a fault status, the execution of the large model task corresponding to the GPU cluster shall be suspended.
[0129] 605. After the execution of the large model task is terminated, the data on the faulty network card corresponding to the faulty state is switched to the backup network card corresponding to the faulty network card to complete the network card switching of the GPU.
[0130] 606. After confirming that all GPUs in the GPU cluster have completed network card switching, resume execution of the large model task.
[0131] The service initialization process can be executed by the CPU, and this process may include:
[0132] Create a GPU cluster, which includes multiple GPUs that need to communicate. For example, see... Figure 3 The first to fourth GPUs are used as a GPU cluster.
[0133] After creating the GPU cluster, a primary network interface card (NIC) and a backup NIC are assigned to each GPU in the cluster based on the target communication library. For example, a primary NIC and a backup NIC are assigned to the first GPU. The specific assignment can be based on physical distance, such as using the NIC that is closest to the first GPU in terms of physical distance as the primary NIC and the NIC that is the second closest to the first GPU in terms of physical distance as the backup NIC.
[0134] After allocating primary and backup network interface cards (NICs), the corresponding GPU memory region (MR) information is registered in both NICs. This allows for direct access to the GPU's video memory via the RDMA protocol. The MR information identifies the location of the GPU's video memory; by registering the MR, the NIC provides a secure and efficient way to access the GPU's video memory.
[0135] In addition, connections can be established between primary network interface cards (NICs). For example, a link can be established between the first primary NIC and the third primary NIC.
[0136] After establishing a link, multiple queue pairs (qp) can be created. In RDMA technology, qp is the logical channel for network interface cards (NICs) to achieve high-performance communication, enabling efficient communication between NICs.
[0137] In addition to creating GPU clusters, you can also create probe groups, which are used to detect and synchronize the network card status of each primary network card.
[0138] The probe groups consist of primary network interface cards (NICs) from different servers. For example, the first and third primary NICs form one probe group, and the second and fourth primary NICs form another. Primary NICs within the same probe group can communicate via the RDMA protocol, allowing the probe to obtain the NIC status of other primary NICs within the same probe group for a given primary NIC. Furthermore, different probe groups can synchronize their states using set communication primitives, enabling the probe to obtain the NIC status of each primary NIC in other probe groups for a given primary NIC.
[0139] Therefore, based on the detection and synchronization process of the probe group, the network card status of each network card in the GPU cluster can be obtained and written into the shared memory.
[0140] After that, both the CPU and GPU can obtain the network card status from the shared memory.
[0141] If the network interface card (NIC) is in a fault state, the execution of large model tasks will be aborted. For example, the CPU will stop submitting new operators, and the GPU will perform normal exit operations on its operators.
[0142] After the large model task is aborted, each GPU performs a network card switching operation.
[0143] Specifically, a link can be created between the backup network card corresponding to the failed primary network card and other primary network cards to obtain a backup link, and data can be switched from the primary link to the backup link.
[0144] For example, after the first primary network interface card (NIC) fails, a link is established between the first backup NIC and the third primary NIC, and then between the first backup NIC and the fourth primary NIC. Afterward, data destined for the third GPU is switched from the link between the first and third primary NICs to the link between the first backup NIC and the third primary NIC.
[0145] In addition, after creating a backup link, a corresponding QP can be created to achieve more efficient inter-NIC communication.
[0146] After the GPU completes the network card switching, the network card switching status of each GPU can be synchronized through the set communication primitive. After each GPU has completed the network card switching, it can be determined that the GPU cluster has completed the network card switching, thereby resuming the execution of large model tasks by the GPU cluster.
[0147] In this embodiment, network card status awareness and fault handling are completed through the GPU cluster's own resources, reducing task interruption time to the time of network card switching, which can usually be completed in seconds, thus reducing performance loss during network failures. It also avoids the significant resource overhead caused by external monitoring systems restarting large models, reducing resource consumption. Obtaining network card status based on probe groups has minimal impact on large model tasks, improving task execution stability. No additional hardware is required, allowing for flexible adaptation to various topologies, improving implementation flexibility and application scope. It is suitable for both inference and training tasks.
[0148] Figure 7 This is a schematic diagram according to the fifth embodiment of the present disclosure. This embodiment provides a network fault-tolerant device 700, which includes: an acquisition module 701, an abort module 702, a switching module 703, and a recovery module 704.
[0149] The acquisition module 701 is used to acquire the network card status of the primary network card in the GPU cluster; the GPU cluster is used to execute large model tasks; the network card status is obtained based on the primary network card itself; the abort module 702 is used to abort the execution of the large model task when the network card status includes a fault state; the switching module 703 is used to switch the data on the faulty network card corresponding to the faulty state to the backup network card corresponding to the faulty network card after the execution of the large model task is aborted; the backup network card is allocated based on the target communication library, and the target communication library is used to provide communication services for the GPU cluster; the recovery module 704 is used to resume the execution of the large model task after determining that the GPU cluster has completed the network card switching based on the target communication library.
[0150] In this embodiment, the primary network interface card (NIC) learns its own NIC status, and when the NIC is in a faulty state, it performs task abort and NIC switching operations. After the NIC switching is complete, task execution resumes. This utilizes the GPU cluster's own resources for fault detection and handling, saving resource overhead, reducing NIC switching time, minimizing impact on large model tasks, and improving their stability compared to introducing an external monitoring system. Furthermore, a backup NIC is allocated based on the target communication library. Since the target communication library is a software resource, a backup NIC can be allocated based on software resources without changing hardware configuration, reducing implementation complexity. This improves network fault tolerance performance.
[0151] In some embodiments, the acquisition module 701 is further configured to:
[0152] The network interface card (NIC) status is obtained from the shared memory; the NIC status is obtained by the probe group and written into the shared memory, and the probe group obtains it based on the primary NIC.
[0153] In this embodiment, by obtaining the network card status from shared memory, the accuracy of the network card status can be guaranteed, and the network fault tolerance can be improved.
[0154] In some embodiments, the abort module 702 is further configured to:
[0155] Perform a normal exit operation on the operator corresponding to any GPU in the GPU cluster, and release the resources occupied by the operator after the operator exits normally; the operator is obtained based on the large model task.
[0156] In this embodiment, by performing a normal exit operation on the operator, uncontrollable errors can be avoided, and the network fault tolerance performance can be improved.
[0157] In some embodiments, the switching module 703 is further configured to:
[0158] Data transmitted via the primary link will be switched to the backup link.
[0159] The primary link includes the link between the faulty network card and the primary network card of the peer GPU.
[0160] The backup link includes a link between the backup network card corresponding to the faulty network card and the primary network card of the peer GPU, and the backup link is established after the fault state is determined.
[0161] In this embodiment, by establishing a backup link after a failure occurs, a corresponding backup link can be created based on the actual faulty network card, allowing for targeted creation of backup links and avoiding resource waste.
[0162] In some embodiments, the recovery module 704 is further configured to:
[0163] Based on the set communication primitives provided by the target communication library, obtain the network card switching status of any GPU in the GPU cluster;
[0164] If all network card switching states are complete, resume execution of the large model task.
[0165] In this embodiment, the network card switching status of any GPU in the GPU cluster is obtained based on the set communication primitive, which can efficiently and accurately obtain the network card switching status, and thus accurately resume the execution of large model tasks.
[0166] In some embodiments, the recovery module 704 is further configured to:
[0167] If all network card switching states are complete, determine the target data location corresponding to the large model task, and resume execution of the large model task based on the target data location;
[0168] The target data location is determined based on the receiving stop location and the sending stop location, which are recorded when the execution of the large model task is terminated.
[0169] In this embodiment, the large model task is executed based on the target data location recovery, which can achieve data consistency between the sending end and the receiving end and ensure the continuity of communication data.
[0170] It is understood that the same or similar content in different embodiments of this disclosure can be referred to each other.
[0171] It is understood that the terms "first" and "second" in the embodiments of this disclosure are only used for distinction and do not indicate the degree of importance or the order of events.
[0172] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.
[0173] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.
[0174] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. Electronic device 800 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, servers, blade servers, mainframe computers, and other suitable computers. Electronic device 800 may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.
[0175] like Figure 8As shown, the electronic device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program transferred from a storage unit 808 to a random access memory (RAM) 803. The RAM 803 may also store various programs and data required for the operation of the electronic device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.
[0176] Multiple components in electronic device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of displays, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows electronic device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.
[0177] The computing unit 801 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as network fault-tolerant methods. For example, in some embodiments, the network fault-tolerant method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the network fault-tolerant method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform network fault-tolerant methods by any other suitable means (e.g., by means of firmware).
[0178] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.
[0179] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.
[0180] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
[0181] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).
[0182] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.
[0183] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service ecosystem, addressing the shortcomings of traditional physical hosts and VPS (Virtual Private Server, or simply "VPS") services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.
[0184] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.
[0185] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.
Claims
1. A network fault tolerance method, comprising: Get the network card status of the primary network card in the GPU cluster; The GPU cluster is used to execute large model tasks; The network interface card (NIC) status is obtained based on the primary NIC itself; If the network interface card (NIC) status includes a fault state, the execution of the large model task shall be aborted. After the execution of the large model task is terminated, the data on the faulty network card corresponding to the faulty state is switched to the backup network card corresponding to the faulty network card; the backup network card is allocated based on the target communication library, which is used to provide communication services for the GPU cluster; After determining that the GPU cluster has completed the network card switching based on the target communication library, the execution of the large model task resumes.
2. The method according to claim 1, wherein, The process of obtaining the network interface status of the primary network interface card in the GPU cluster includes: The network interface card (NIC) status is obtained from the shared memory; the NIC status is obtained by the probe group and written into the shared memory, and the probe group obtains it based on the primary NIC.
3. The method according to claim 1, wherein, The termination of the large model task includes: Perform a normal exit operation on the operator corresponding to any GPU in the GPU cluster, and release the resources occupied by the operator after the operator exits normally; the operator is obtained based on the large model task.
4. The method according to claim 1, wherein, The step of switching the data on the faulty network card corresponding to the faulty network card to the backup network card corresponding to the faulty network card includes: Data transmitted via the primary link will be switched to the backup link. The primary link includes the link between the faulty network card and the primary network card of the peer GPU. The backup link includes a link between the backup network card corresponding to the faulty network card and the primary network card of the peer GPU, and the backup link is established after the fault state is determined.
5. The method according to claim 1, wherein, The step of resuming execution of the large model task after determining that the GPU cluster has completed network card switching based on the target communication library includes: Based on the set communication primitives provided by the target communication library, obtain the network card switching status of any GPU in the GPU cluster; If all network card switching states are complete, resume execution of the large model task.
6. The method according to claim 5, wherein, If all network card switching states are complete, the large model task will resume execution, including: If all network card switching states are complete, determine the target data location corresponding to the large model task, and resume execution of the large model task based on the target data location; The target data location is determined based on the receiving stop location and the sending stop location, which are recorded when the execution of the large model task is terminated.
7. A network fault-tolerant device, comprising: The acquisition module is used to obtain the network card status of the primary network card in the GPU cluster; The GPU cluster is used to execute large model tasks; The network interface card (NIC) status is obtained based on the primary NIC itself; The abort module is used to abort the execution of the large model task when the network card status includes a fault status; The switching module is used to switch the data on the faulty network card corresponding to the faulty network card to the backup network card corresponding to the faulty network card after the execution of the large model task is stopped; the backup network card is allocated based on the target communication library, which is used to provide communication services for the GPU cluster; The recovery module is used to resume the execution of the large model task after determining, based on the target communication library, that the GPU cluster has completed the network card switching.
8. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-6.
9. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-6.
10. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-6.