Communication method and apparatus

Abstract

The application provides a communication method and device. The method comprises the following steps: in the case that the computing power server participates in model training and it is determined that a first network card in the computing power server has a communication fault, a replacement network card is selected for the first network card from network cards in the computing power server that do not have a communication fault; according to the replacement network card, the internal communication topology of the computing power server and the cluster communication topology are updated, and a first communication topology update message carrying the updated cluster communication topology is sent to other computing power servers in the communication domain in which the computing power server is located. Through the embodiment of the application, the communication interruption time caused by the network card communication fault can be effectively shortened, and the stability of the training environment is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to network communication technology, in particular to a communication method and device. BACKGROUND

[0002] With the model size getting larger and larger, the server cluster size required for training the model is getting larger and larger. In the case of constant single server hardware failure rate in the cluster, the probability of failure hardware in the entire cluster will rise exponentially.

[0003] In order to realize high-speed exchange of data, the current cluster computing generally uses RDMA (Remote Direct Memory Access) as a network protocol to carry data communication. This technology is that the network card directly accesses the address space of the computing power acceleration card or CPU (Center Process Unit), reads the data of the corresponding memory or main memory, and sends it to the opposite network card. In order to realize the direct read and write of the network card to the memory of the computing power acceleration card, the network design is to associate each computing power acceleration card with the network card under the PCIe (Peripheral Component Interconnect Express) Switch (switch). The data of this computing power acceleration card is communicated through the network card.

[0004] In the model training process, if the network card of any participating training computing power server has communication failure, the training task needs to be suspended, the computing power server is isolated, a new backup node is selected, the training task is redeployed, and the training process is recovered based on the last saved model training state (Checkpoint). SUMMARY

[0005] The present application provides a communication method and device to reduce the communication interruption time caused by network card communication failure.

[0006] The present application provides a communication method, applied to a computing power server, the method comprising:

[0007] In the case that the computing power server participates in model training and it is determined that the first network card in the computing power server has communication failure, a replacement network card is selected for the first network card from the network cards in the computing power server that do not have communication failure;

[0008] update the internal communication topology of the computing power server and the cluster communication topology according to the replacement network card, and send a first communication topology update message carrying the updated cluster communication topology to other computing power servers in a communication domain in which the computing power server is located; the updated internal communication topology is used to instruct the computing power server to perform communication data transmission and reception of a first computing power acceleration card through the replacement network card; and the updated cluster communication topology is used to instruct a receiving-end computing power server to communicate with the first computing power acceleration card through the replacement network card, the first computing power acceleration card being a computing power acceleration card associated with the first network card in the computing power server.

[0009] The embodiment of the application further provides a communication device deployed in a computing power server, the device comprising:

[0010] a selection unit configured to select a replacement network card for a first network card in the computing power server from network cards without communication faults in the computing power server in a case where the first network card has a communication fault and the computing power server participates in model training;

[0011] an update unit configured to update the internal communication topology of the computing power server and the cluster communication topology according to the replacement network card; the updated internal communication topology is used to instruct the computing power server to perform communication data transmission and reception of a first computing power acceleration card through the replacement network card; and the updated cluster communication topology is used to instruct a receiving-end computing power server to communicate with the first computing power acceleration card through the replacement network card, the first computing power acceleration card being a computing power acceleration card associated with the first network card in the computing power server;

[0012] a communication unit configured to send a first communication topology update message carrying the updated cluster communication topology to other computing power servers in a communication domain in which the computing power server is located.

[0013] As can be seen from the above technical solution, in the embodiment, for any computing power server participating in model training, a replacement network card is selected for a network card with a communication fault in the computing power server from network cards without communication faults in the computing power server in a case where the network card has a communication fault, the internal communication topology of the computing power server and the cluster communication topology are updated according to the selected replacement network card, and the updated cluster communication topology is notified to other computing power servers in the same communication domain, so that the communication between the server and the communication domain servers can be realized through the replacement network card to replace the faulty network card, the communication interruption time caused by the network card communication fault is effectively shortened at a low cost and complexity without the need to deploy two sets of networks on the network side to form a double plane, and the stability of the training environment is improved. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1A schematic diagram of an internal connection topology of a computing power server for model training;

[0015] Figure 2 A schematic diagram of a communication method provided by an embodiment of the present application;

[0016] Figure 3 A schematic diagram of a communication method implementation process provided by an embodiment of the present application;

[0017] Figure 4 A schematic diagram of a networking example provided by an embodiment of the present application;

[0018] Figure 5 A schematic diagram of a networking example provided by an embodiment of the present application;

[0019] Figure 6A A schematic diagram of an alternative network card selection provided by an embodiment of the present application;

[0020] Figure 6B A schematic diagram of another alternative network card selection provided by an embodiment of the present application;

[0021] Figure 7 A schematic diagram of a communication device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0022] In order for those skilled in the art to better understand the technical solutions provided by the embodiments of the present application, the internal connection topology of the computing power server will be briefly described below in conjunction with the drawings.

[0023] Please refer to Figure 1 , a schematic diagram of the internal connection topology of a typical computing power server for model training, as Figure 1 shown, in the computing power server for model training, each computing power acceleration card will exist two interconnection networks: one of which can be called northward connection network, that is, through PCIe Switch connection network card, and other computing power servers are connected in communication; the other can be called southward connection network, which is generally each computing power acceleration card manufacturer through private protocol to connect the computing power acceleration card inside the server.

[0024] Exemplarily, the southward network connection protocols and topologies of different computing power acceleration card manufacturers are different, but the bandwidth of the southward network connection is generally much higher than that of the northward network connection.

[0025] Taking Nvidia H100 as an example, the bandwidth of the southbound network connection of each computing power acceleration card is 900 GB / s, and the bandwidth of the northbound network connection is limited by PCIe 5.0, and the bandwidth is only 400 Gb / s, which is 1 / 18 of the bandwidth of the southbound network connection. Therefore, the traffic inside the server is exchanged through the southbound network, which can provide extremely high throughput.

[0026] For example, the computing power acceleration card can include, but is not limited to, a GPU (Graphics Processing Unit) or a DCU (Deep Computing Unit) and the like.

[0027] The computing power server can include, but is not limited to, a GPU server and the like.

[0028] In order to make the above-mentioned purposes, features and advantages of the embodiments of the present application more apparent and easy to understand, the technical solutions provided by the embodiments of the present application are described below with reference to the drawings.

[0029] The communication method provided by the embodiments of the present application is described below with reference to the drawings:

[0030] Referring to Figure 2 , Figure 2 The flow chart of the communication method provided by the embodiments of the present application is shown. The method can be applied to a computing power server. As shown in Figure 2 , the flow can include the following steps:

[0031] Step 201, in the case that the computing power server participates in model training and it is determined that a first network card in the computing power server has a communication fault, a replacement network card is selected for the first network card from the network cards in the computing power server that do not have communication faults.

[0032] In the present embodiment, in order to recover normal communication more quickly in the case that a network card communication fault occurs in the computing power server, the network card with a communication fault can be replaced by the network card without a communication fault, so as to reduce the communication interruption time.

[0033] Correspondingly, for any computing power server participating in model training, in the case that it is determined that any network card (which can be referred to as a first network card) has a communication fault, a replacement network card can be selected for the first network card from the network cards in the computing power server that do not have communication faults, so as to replace the first network card for communication, thereby improving the communication continuity in the case that a network card communication fault occurs.

[0034] In the present embodiment, the network card communication fault can include, but is not limited to, a physical layer fault, a network layer fault, or a driver / system layer fault and the like.

[0035] For example, the physical layer fault can include, but is not limited to, a damaged network card, a disconnected network cable, an abnormal switch port, and the like.

[0036] The network layer fault can include, but is not limited to, an inability to communicate with a gateway, a severe packet loss, and the like.

[0037] The driver / system layer fault can include, but is not limited to, a crashed network card driver, an abnormal kernel protocol stack, and the like.

[0038] Correspondingly, the detection of the communication fault of the network card can include, but is not limited to, physical layer detection (such as the physical state of the network card or the switch port), network layer detection (such as connectivity), driver / system layer detection (such as an abnormal network card driver or system kernel), or external notification (such as active alarm of the switch or management system).

[0039] In one example, the determination that the first network card in the computing power server has a communication fault can include:

[0040] In a case where the connectivity detection of the first network card with the gateway fails, it is determined that the first network card has a communication fault.

[0041] For example, the connectivity of the network card with the gateway can be determined by an ICMP (Internet Control Message Protocol) probe or a BFD (Bidirectional Forwarding Detection) detection method, or other customized methods.

[0042] As an example, the connectivity of the network card with the gateway can be detected in real time by the BFD detection method, so as to shorten the link fault discovery time, achieve millisecond-level link fault discovery, and thus more efficiently perform network card switching, shorten the time consumption of the entire process from fault discovery to data retransmission, and avoid training interruption caused by network card communication fault.

[0043] In one example, the determination that the first network card in the computing power server has a communication fault can include:

[0044] In a case where the first fault notification reported by the network card driver of the first network card is detected, it is determined that the first network card has a communication fault.

[0045] In one example, the determination that the first network card in the computing power server has a communication fault can include:

[0046] In a case where the second fault notification reported by the operation and maintenance monitoring module for the first network card is detected, it is determined that the first network card has a communication fault.

[0047] Exemplarily, the operation and maintenance monitoring module for network fault detection can be deployed in the computing power server. When the operation and maintenance monitoring module detects that the first network card has a network fault, for example, detects that the network card has abnormal traffic (such as continuous packet loss or zero traffic), the operation and maintenance monitoring module can report the network fault of the first network card and send a fault notification (which can be referred to as a second fault notification) for notifying that the first network card has a network fault.

[0048] In one example, the determination that the first network card in the computing power server has a communication fault can include:

[0049] In a case where the third fault notification sent by the first switch is detected, it is determined that the first network card has a communication fault; wherein the third fault notification is a fault notification sent in a case where a target port has a fault, and the target port is a port on the first switch that connects the first network card.

[0050] Exemplarily, the switch can notify the computing power server that the port (which can be referred to as a target port) on the switch that connects the first network card of the computing power server has a fault through a management protocol, such as the Simple Network Management Protocol (SNMP).

[0051] In one example, the determination that the first network card in the computing power server has a communication fault can include:

[0052] In a case where the port on the first network card that connects the switch is detected to have a fault, it is determined that the first network card has a communication fault.

[0053] Exemplarily, the computing power server can actively query the physical state of the network card to determine whether the network card port has a fault.

[0054] In this embodiment, the state of the network card of the computing power server and the state of the port of the switch can be actively detected, the fault discovery time can be effectively shortened, the network card switching can be more efficiently performed, the time consumption of the entire process from fault discovery to data retransmission can be shortened, and the training interruption caused by the network card communication fault can be avoided.

[0055] Step 202: updating, according to the replacement network card, the internal communication topology of the computing power server and the cluster communication topology, and sending a first communication topology update message carrying the updated cluster communication topology to other computing power servers in the communication domain in which the computing power server is located; wherein the updated internal communication topology is used to indicate that the computing power server performs communication data transmission and reception of the first computing power acceleration card through the replacement network card; and the updated cluster communication topology is used to indicate that the receiving end computing power server communicates with the first computing power acceleration card through the replacement network card, and the first computing power acceleration card is an algorithm acceleration card associated with the first network card in the computing power server.

[0056] In this embodiment, when a replacement network card for the first network card is determined, the computing server can update the communication topology in order to enable communication using the replacement network card instead of the first network card.

[0057] For example, updating the communication topology may include updating the communication topology within the computing server and updating the cluster communication topology.

[0058] Among them, updating the internal communication topology of the computing power server allows the computing power server to use a replacement network card to replace the first network card and send and receive communication data with the computing power acceleration card (which can be called the first computing power acceleration card) associated with the first network card.

[0059] The update of the cluster communication topology allows other computing servers within the same communication domain to communicate with the first computing accelerator card through the alternative network card of the computing server.

[0060] For example, in the case where the computing server is a GPU server, the first computing acceleration card can be the first GPU.

[0061] In this embodiment, the computing server can send the updated cluster communication topology in a communication topology update message (which may be called the first communication topology update message) to other computing servers in the communication domain where the computing server is located, so that the other computing servers can communicate with the first computing acceleration card through the alternative network card.

[0062] It can be seen that, in Figure 1 In the illustrated method, for any computing server participating in model training, if a network interface card (NIC) communication failure occurs within that server, a replacement NIC can be selected from among the NICs that are not experiencing communication failures. Based on the selected replacement NIC, the internal communication topology and cluster communication topology of the computing server are updated, and the updated cluster communication topology is notified to other computing servers within the same communication domain. Thus, communication within the server and between servers in the communication domain can be achieved by replacing the faulty NIC with the replacement NIC. Without needing to deploy two sets of networks on the network side to form a dual-plane architecture, this method effectively shortens the communication interruption time caused by NIC communication failures and improves the stability of the training environment with lower cost and complexity.

[0063] In one example, selecting a replacement network card for the first network card from among the network cards in the computing server that do not have communication failures may include:

[0064] If the network interface card (NIC) in the computing server that does not have communication faults includes at least one second NIC, a replacement NIC is selected for the first NIC from at least one second NIC; wherein the second NIC is a NIC that is connected to the same PCIe switch as the first NIC.

[0065] For example, in the process of selecting a replacement network card for the first network card from the network cards without communication failure in the computing power server, the network card connected to the same PCIe switch as the first network card can be preferentially selected, so that in the process of replacing the network card with communication failure for communication through the replacement network card, the communication data can not need to be forwarded through the internal network of the computing power server, but can be transmitted through the PCIe channel, improving the communication efficiency and reducing the communication delay.

[0066] Correspondingly, in the case where it is determined that the first network card has communication failure, the computing power server can query whether there is a network card (which can be referred to as a second network card) connected to the same PCIe server as the first network card and without communication failure in the server.

[0067] In the case where it is determined that there is a second network card, the computing power server can select a replacement network card for the first network card from the second network card.

[0068] For example, in the case where there are multiple second network cards in the computing power server, the replacement network card selected by the computing power server for the first network card can also not be limited to one, but multiple second network cards can be selected as the replacement network card for the first network card.

[0069] As an example, the above selecting a replacement network card for the first network card from the network cards without communication failure in the computing power server can further include:

[0070] In the case where the second network card is not included in the network cards without communication failure in the computing power server, but at least one third network card is included, a replacement network card is selected for the first network card from the at least one third network card; wherein the third network card is a network card connected to the same access switch as the first network card.

[0071] For example, in the case where there is no second network card in the computing power server, a network card (which can be referred to as a third network card) connected to the same access switch as the first network card can be preferentially selected to reduce the topology changes brought by the communication topology update.

[0072] For example, in the case where there is no second network card in the computing power server, a third network card connected to the same Leaf node as the first network card can be preferentially selected in the Spine-Leaf architecture networking.

[0073] In one example, the above selecting a replacement network card for the first network card from the network cards without communication failure in the computing power server can include:

[0074] In the case that the number of other computing power servers in the same communication domain is greater than the number of network cards without communication failure in the computing power server, the same substitute network card can be selected from the network cards without communication failure in the computing power server for communication between part of the first computing power acceleration cards and different other computing power servers in the communication domain.

[0075] It should be noted that in the present embodiment, not completely the same can include completely different, or partially the same, that is, for the above-mentioned plurality of other computing power servers, in the process of selecting a substitute network card for the plurality of other computing power servers, different network cards can be selected as substitute network cards for the plurality of other computing power servers respectively; or the same network card can be selected as a substitute network card for part of the other computing power servers, but at least two other computing power servers correspond to different substitute network cards, for example, the plurality of other computing power servers include computing power server A, computing power server B and computing power server C, the substitute network cards corresponding to the computing power server A and the computing power server B can be the same, and the substitute network card corresponding to the computing power server C is different from the substitute network cards corresponding to the computing power servers A and B.

[0076] For example, in order to reduce the probability of traffic congestion on the substitute network card, a plurality of substitute network cards can be selected in the process of selecting a substitute network card for the first network card, and the plurality of substitute network cards are used to replace the first network card for communication.

[0077] For example, in the case that the number of other computing power servers in the same communication domain is less than or equal to the number of network cards without communication failure in the computing power server, different substitute network cards can be selected from the network cards without communication failure in the computing power server for communication between the first computing power acceleration card and different other computing power servers in the communication domain.

[0078] In the case that the number of other computing power servers in the same communication domain is greater than the number of network cards without communication failure in the computing power server, the same substitute network card can be selected from the network cards without communication failure in the computing power server for communication between part of the first computing power acceleration cards and different other computing power servers in the communication domain.

[0079] For example, it is assumed that the communication domain includes computing power servers 1-5, the first network card (denoted as network card 1) in the computing power server has communication failure, and the network cards without communication failure include network cards 2-3, then for the computing power server 2, the substitute network card can be network card 2; for the computing power server 3, the substitute network card can be network card 3; for the computing power server 4, the substitute network card can be network card 2; for the computing power server 5, the substitute network card can be network card 3.

[0080] It should be noted that in the present embodiment, in the case where the replacement network card includes multiple network cards, the network card connected to the same PCIe switch as the first network card can also be preferentially selected; in the case where there is no network card connected to the same PCIe switch as the first network card, the network card connected to the same access switch as the first network card is preferentially selected.

[0081] In one example, selecting the replacement network card for the first network card from the network cards without communication failure in the hash power server can include:

[0082] According to the number of sessions of each network card without communication failure in the hash power server, the replacement network card for the first network card is selected in a manner that preferentially selects the network card with the least number of sessions;

[0083] Or,

[0084] According to the average rate of each network card without communication failure in the hash power server within a specified historical period, the replacement network card for the first network card is selected in a manner that preferentially selects the network card with the lowest average rate.

[0085] For example, in order to reduce the probability of traffic congestion of the replacement network card, in the process of selecting the replacement network card for the first network card, the network card with lower load can be preferentially selected.

[0086] For example, the load of the network card can be represented according to the number of sessions or the historical average rate.

[0087] As an example, in the process of selecting the replacement network card for the first network card, the network card with the least number of sessions can be preferentially selected as the replacement network card.

[0088] In the case where the replacement network card includes multiple network cards, the candidate network cards can be sorted in order from few to many according to the number of sessions, and a number of network cards with high ranking are selected as the replacement network card.

[0089] As an example, in the process of selecting the replacement network card for the first network card, the network card with the lowest average rate within a specified historical period can be preferentially selected as the replacement network card.

[0090] For example, the specified historical period can be set according to actual needs, such as the previous 24 hours with the current time as the end time point.

[0091] In the case where the replacement network card includes multiple network cards, the candidate network cards can be sorted in order from low to high according to the average rate within the specified historical period, and a number of network cards with high ranking are selected as the replacement network card.

[0092] In one example, the sending, by the computing power server, the first communication topology update message carrying the updated cluster communication topology to other computing power servers in the communication domain where the computing power server is located can include:

[0093] determining a remote computing power server that has established a communication session with the first network card;

[0094] sending the first communication topology update message carrying the updated cluster communication topology to the remote computing power server.

[0095] For example, in the process of updating the cluster communication topology, the remote computing power server that has established a communication session with the first network card can be determined as other computing power servers in the same communication domain as the computing power server (the computing power server to which the first network card belongs).

[0096] Correspondingly, in the case where the updated cluster communication topology is determined in the above manner, in the process of synchronizing the updated cluster communication topology, the computing power server that has established a communication session with the first network card (which can be referred to as a remote computing power server) can be determined.

[0097] For example, for any network card of a computing power server, the network card driver or kernel records the communication session established by the network card, and the address of the opposite end of the recorded communication session can be determined to synchronize the updated cluster communication topology.

[0098] The computing power server can determine the remote computing power server that has established a communication session with the first network card as other computing power servers in the same communication domain, and send the first communication topology update message carrying the updated cluster communication topology to the determined remote computing power server.

[0099] It should be noted that in the present embodiment, the communication domain can also be constructed by a static establishment method.

[0100] For example, the computing power servers belonging to the same communication domain in the cluster can be determined in advance, and the computing power servers can record the address information of other computing power servers in the same communication domain.

[0101] For any communication domain, in the case where there is a network card communication failure in any computing power server in the communication domain, the cluster communication topology can be updated in the above manner, and the updated cluster communication topology can be synchronized to other computing power servers in the same communication domain according to the recorded address information of other computing power servers in the same communication domain.

[0102] In one example, the communication method provided in the present embodiment can further include:

[0103] receive a second communication topology update message sent by the target computing power server; wherein the second communication topology update message is used to inform the receiving end computing power server to communicate with a second computing power acceleration card in the target computing power server through a replacement network card of the fourth network card; wherein the second computing power acceleration card is a computing power acceleration card associated with the fourth network card in the target computing power server; delete a communication session established through the fourth network card and the second computing power acceleration card, and reestablish a communication session with the second computing power acceleration card through the replacement network card of the fourth network card;

[0104] or,

[0105] In the case where it is determined that the communication through the fourth network card with the second computing power acceleration card fails, the communication session established through the fourth network card and the second computing power acceleration card is deleted, and a communication session is reestablished with the second computing power acceleration card through the replacement network card of the fourth network card.

[0106] For example, the specific implementation of the target computing power server sending the second communication topology update message can refer to the related description of the sending of the first communication topology update message in the above method flow.

[0107] For example, in the case where the computing power server receives the second communication topology update message sent by the target computing power server, it can be determined that the fourth network card of the target computing power server has a communication failure, and the communication topology needs to be updated for the fourth network card, and a communication session is established with the target computing power server through the replacement network card of the fourth network card.

[0108] In one example, the computing power server can actively delete the communication session established through the fourth network card and the second computing power acceleration card in the case where it receives the second communication topology update message sent by the target computing power server, and reestablish a communication session with the second computing power acceleration card through the replacement network card of the fourth network card, thereby improving the efficiency of communication session switching.

[0109] In another example, in order to avoid network congestion or session establishment failure caused by a large number of computing power servers collectively reestablishing communication sessions with the target computing power server, the computing power server can reestablish a communication session with the target computing power server on demand.

[0110] For example, the computing power server can reestablish a communication session with the second computing power acceleration card through the replacement network card of the fourth network card in the case where it is determined that the communication through the fourth network card with the second computing power acceleration card fails.

[0111] For example, the computing power server can reestablish a communication session with the second computing power acceleration card through the replacement network card of the fourth network card in the case where it is determined that the communication through the fourth network card with the second computing power acceleration card fails.

[0112] In one example, the communication data transceiving of the first computing power acceleration card through the replacement network card can include:

[0113] In the case where the replacement network card is connected to a different PCIe switch from the first network card, for the communication data received through the replacement network card, the replacement network card writes the communication data into the memory of the third computing power acceleration card, and the third computing power acceleration card sends the communication data to the first computing power acceleration card through the internal network of the computing power server.

[0114] For the communication data to be sent by the first computing power acceleration card, the first computing power acceleration card sends the communication data to the third computing power acceleration card through the internal network of the computing power server, the third computing power acceleration card writes the communication data into the memory of the third computing power acceleration card, and the replacement network card reads the communication data from the memory of the third computing power acceleration card and sends it.

[0115] The third computing power acceleration card is the computing power acceleration card associated with the replacement network card in the computing power server.

[0116] For example, in the process of communication data transceiving of the first computing power acceleration card through the replacement network card, for the communication data of the first computing power acceleration card, in the case where the replacement network card is connected to a different PCIe switch from the first network card, the replacement network card can read and write the communication data of the first computing power acceleration card to the memory of the third computing power acceleration card through PCIe, and the third computing power acceleration card and the first computing power acceleration card interact with the communication data of the first computing power acceleration card through the internal network of the computing power server (i.e. the southbound connection network mentioned above).

[0117] For the communication data of the first computing power acceleration card received through the replacement network card, the replacement network card can write the communication data into the memory of the third computing power acceleration card, and the third computing power acceleration card sends the communication data to the first computing power acceleration card through the internal network of the computing power server.

[0118] For the communication data to be sent by the first computing power acceleration card, the first computing power acceleration card can send the communication data to the third computing power acceleration card through the internal network of the computing power server, the third computing power acceleration card writes the communication data into the memory of the third computing power acceleration card, and then the replacement network card reads the communication data from the memory of the third computing power acceleration card and sends it.

[0119] In one example, the communication data transceiving of the first computing power acceleration card through the replacement network card can include:

[0120] In the case where the replacement network card is connected to the same PCIe switch as the first network card, the replacement network card writes the received communication data into the memory of the first computing power acceleration card.

[0121] For the communication data to be sent by the first computing power acceleration card, the first computing power acceleration card writes the communication data into the memory of the first computing power acceleration card, and the substitute network card reads the communication data from the memory of the first computing power acceleration card and sends it.

[0122] For example, in the case where the substitute network card is connected to the same PCIe switch as the first network card, the substitute network card can read and write communication data to the memory of the first computing power acceleration card through PCIe.

[0123] For the communication data of the first computing power acceleration card received through the substitute network card, the received communication data can be written into the memory of the first computing power acceleration card through the substitute network card, and then the first computing power acceleration card can read the communication data from the memory.

[0124] For the communication data to be sent by the first computing power acceleration card, the first computing power acceleration card writes the communication data into the memory of the first computing power acceleration card, and the substitute network card reads the communication data from the memory of the first computing power acceleration card and sends it.

[0125] In order for those skilled in the art to better understand the technical solutions provided by the embodiments of the present application, the technical solutions provided by the embodiments of the present application will be described below in conjunction with specific examples.

[0126] In this embodiment, in the process of model training, the computing power server can realize communication of the computing power acceleration card in the server and communication of the computing power acceleration card between servers through a collection communication library running on the distributed computing power acceleration card and CPU.

[0127] Referring to Figure 3 , the communication method implementation process in this embodiment can be as shown in Figure 3 , comprising:

[0128] S1, the collection communication library determines that the network card has a communication fault.

[0129] For example, the collection communication library can determine that the network card has a communication fault (also referred to as an access link fault) through active detection, or determine that the network card has a communication fault according to a received notification message.

[0130] S2, the collection communication library selects a substitute network card for the faulty network card according to the internal topology.

[0131] S3, the collection communication library updates the communication topology.

[0132] For example, updating the communication topology can include recalculating the internal communication topology of the server and recalculating the cluster communication topology.

[0133] S4, the collection communication library synchronizes the updated cluster communication topology to other nodes (computing power servers) in the same communication domain.

[0134] S5, the nodes in the communication domain update the communication topology and reestablish the session connection.

[0135] S6, all nodes in the communication domain resend the communication data that has not been sent through the old session using the new session.

[0136] For example, in the case of reestablishing the session connection as described above, the data that has not been sent due to communication interruption can be resent (sent completely). Since the communication session is reestablished, the memory occupied by the original session by the opposite end network card has been released, which is equivalent to a new read-write operation, so there will be no problem of abnormal occupation.

[0137] The following will describe some details in the above process in detail.

[0138] I. Fault discovery mechanism.

[0139] For example, fault discovery can include one or more of the following ways:

[0140] 1) In the case of failure of the connectivity detection of the network card and the gateway, it is determined that the network card has a communication fault;

[0141] 2) In the case of detecting the first fault notification reported by the network card driver of the network card, it is determined that the network card has a communication fault;

[0142] 3) In the case of detecting the network fault notification (such as the second fault notification described above) reported by the operation and maintenance monitoring module for the network card, it is determined that the network card has a communication fault;

[0143] 4) In the case of detecting the port fault notification (such as the third fault notification described above) sent by the first switch through the management channel, it is determined that the network card connected to the port has a communication fault;

[0144] 5) In the case of detecting that the port connected to the switch on the network card has a fault, it is determined that the first network card has a communication fault.

[0145] It should be noted that in the embodiments of the present application, in order to reduce the communication interruption caused by the communication fault of the network card, the fault discovery time can be shortened by the following ways:

[0146] a) actively detecting the state of the server network card / switch port, which can effectively shorten the fault discovery time. For example, the switch can discover that the port is physically down (closed) within 50ms;

[0147] b) Real-time detection of connectivity by BFD and other detection technologies can achieve millisecond-level link fault discovery.

[0148] II. Selection of alternative network cards.

[0149] For example, the collection communication library can select an alternative network card from network cards without communication failure for a network card with communication failure according to the internal topology.

[0150] For example, the collection communication library can preferentially select a network card without communication failure connected to the same PCIe switch as the network card with communication failure as the alternative network card.

[0151] For example, in the case of a network card without communication failure connected to the same PCIe switch as the network card with communication failure, a network card without communication failure connected to the same access switch as the network card with communication failure can be preferentially selected as the alternative network card.

[0152] For example, in order to avoid uneven and congested traffic on the alternative network card, multiple alternative network cards can be selected. Different communication pairs can use different alternative network cards.

[0153] For example, Figure 4 For example, assuming that the NIC (Network Interface Card, NIC for short) 0 of the computing power server 1 has communication failure, and the NIC 0 has communication sessions with the computing power server 2 and the computing power server 3, different alternative network cards can be selected for the NIC 0 for the computing power server 2 and the computing power server 3, respectively. For example, for the computing power server 2, the alternative network card can be NIC 1; for the computing power server 3, the alternative network card can be NIC 2, that is, NIC 1 and NIC 2 are used to communicate with the computing power acceleration card of server 1 and server 2, respectively.

[0154] For example, the way to select an alternative network card can include but is not limited to:

[0155] a) Preferentially selecting the network card with the least number of sessions;

[0156] b) Preferentially selecting the network card with the lowest average rate in the previous time period.

[0157] III. The collection communication library updates the communication topology.

[0158] For example, the collection communication library can recalculate the internal communication topology of the server and the cluster communication topology according to the selected alternative network card.

[0159] For example, for example, Figure 5As shown in the networking, it is assumed that the NIC (Network Interface Card, NIC for short) 0 of the computing power server 1 has a communication failure. The computing power server can select a replacement NIC for NIC0 from the NICs without communication failure and recalculate the communication topology.

[0160] Taking the replacement NIC NIC1 as an example, the recalculated communication topology can be as shown in Figure 6A , wherein, since NIC0 and NIC1 are connected to the same PCIe switch, the communication data sent to the computing power server of the computing power acceleration card 0 can be written into the memory of the computing power acceleration card 0 by NIC1 through PCIe.

[0161] Taking the replacement NIC NIC2 as an example, the recalculated communication topology can be as shown in Figure 6B , wherein, since NIC0 and NIC1 are connected to different PCIe switches, the communication data sent to the computing power server of the computing power acceleration card 0 can be written into the memory of the computing power acceleration card 2 by NIC2 through PCIe, and the communication data is sent to the computing power acceleration card 0 by the computing power acceleration card 2 through the internal network of the server.

[0162] Four, the collection communication library synchronizes the updated cluster communication topology to other nodes in the same communication domain.

[0163] For example, the collection communication library can synchronize the updated cluster communication topology to other nodes in the same communication domain.

[0164] Taking the example shown in Figure 6A , the computing power server 1 can synchronize the updated cluster communication topology to the computing power server 2, so that the computing power server 2 uses the NIC address of NIC2 to communicate with the computing power acceleration card 0 of the computing power server 1.

[0165] For example, the same communication domain can be determined in the following way:

[0166] a) Static: The computing power servers belonging to the same communication domain in the cluster can be determined in advance. The computing power servers can record the relevant information of other computing power servers in the same communication domain, such as address information.

[0167] b) Dynamic: The NIC driver or kernel of the NIC in the computing power server records the communication session established by the NIC. The remote computing power server that has established a communication session with the NIC can be determined as other computing power servers in the same communication domain.

[0168] Five, the nodes in the communication domain update the communication topology and reestablish the session connection.

[0169] Exemplarily, the method for updating a communication topology of a node in a communication domain and rebuilding a method for establishing a session connection can include but is not limited to (in the order as shown in the figure) Figure 6A Exemplarily, the method for updating a communication topology of a node in a communication domain and rebuilding a method for establishing a session connection can include but is not limited to (in the order as shown in the figure)

[0170] a) All nodes delete the communication session between NIC0 of the computing power server 1 and the computing power acceleration card 0 of the computing power server 1, and reestablish the communication session between NIC2 of the computing power server 1 and the computing power acceleration card 0 of the computing power server 1.

[0171] b) In the case of failure to send data through the current communication session with the computing power acceleration card 0 of the computing power server 1, the data sending can be re-performed, and in the case of determining that the current communication session is abnormal according to the retry logic, the communication session between NIC2 of the computing power server 1 and the computing power acceleration card 0 of the computing power server 1 is re-established.

[0172] The structure of the communication device provided by the embodiments of the present application is described below.

[0173] Referring to Figure 7 The structure of the communication device provided by the embodiments of the present application is described below. Figure 7 As shown in the figure, the communication device can include:

[0174] The selection unit is configured to, in a case where the computing power server participates in model training and it is determined that a first network card in the computing power server has a communication fault, select a replacement network card for the first network card from network cards in the computing power server that do not have a communication fault.

[0175] The updating unit is configured to update, according to the replacement network card, an internal communication topology of the computing power server and a cluster communication topology; wherein the updated internal communication topology is used to indicate that the computing power server performs communication data transmission and reception of a first computing power acceleration card through the replacement network card; and the updated cluster communication topology is used to indicate that a receiving-end computing power server communicates with the first computing power acceleration card through the replacement network card, the first computing power acceleration card being a computing power acceleration card associated with the first network card in the computing power server.

[0176] The communication unit is configured to send a first communication topology update message carrying the updated cluster communication topology to other computing power servers in a communication domain in which the computing power server is located.

[0177] Exemplarily, the selection unit, the updating unit, and the communication unit implement the specific implementation process of the communication method provided by the embodiments of the present application, which can be referred to the related description in the above embodiments.

Claims

1. A communication method characterized by comprising: The method is applied to a computing power server, and comprises the following steps: In a case where the computing power server participates in model training and it is determined that a first network card in the computing power server has a communication fault, a replacement network card is selected for the first network card from network cards in the computing power server that do not have a communication fault; According to the replacement network card, an internal communication topology of the computing power server and a cluster communication topology are updated, and a first communication topology update message carrying the updated cluster communication topology is sent to other computing power servers in a communication domain in which the computing power server is located; wherein the updated internal communication topology is used to indicate that the computing power server transmits and receives communication data of a first computing power acceleration card through the replacement network card; and the updated cluster communication topology is used to indicate that a receiving-end computing power server communicates with the first computing power acceleration card through the replacement network card, and the first computing power acceleration card is a computing power acceleration card associated with the first network card in the computing power server; The replacement network card is selected for the first network card from network cards in the computing power server that do not have a communication fault, comprising: In a case where at least one second network card is included in the network cards in the computing power server that do not have a communication fault, the replacement network card is selected for the first network card from the at least one second network card; wherein the second network card is a network card connected to a same Peripheral Component Interconnect Express (PCIe) switch as the first network card; For communication of the first computing power acceleration card with a plurality of other computing power servers in the communication domain in which the computing power server is located, different network cards are selected as corresponding replacement network cards for the plurality of other computing power servers from the network cards in the computing power server that do not have a communication fault; wherein at least two other computing power servers in the plurality of other computing power servers correspond to different replacement network cards.

2. The method of claim 1, wherein, The determination that the first network card in the computing power server has a communication fault comprises: In a case where connectivity detection of the first network card with a gateway fails, it is determined that the first network card has a communication fault; Or, In a case where a first fault notification reported by a network card driver of the first network card is detected, it is determined that the first network card has a communication fault; Or, In a case where a second fault notification reported by an operation and maintenance monitoring module for the first network card is detected, it is determined that the first network card has a communication fault; Or, In a case where a third fault notification sent by a first switch is detected, it is determined that the first network card has a communication fault; wherein the third fault notification is a fault notification sent in a case where a target port on the first switch has a fault, and the target port is a port on the first switch connected to the first network card; Or, In a case where a port connected to a switch on the first network card has a fault, it is determined that the first network card has a communication fault.

3. The method of claim 1, wherein, The replacement network card is selected for the first network card from network cards in the computing power server that do not have a communication fault, further comprising: In the case that the second network card is not included in the network cards without communication failure in the computing power server, but at least one third network card is included, a replacement network card is selected for the first network card from the at least one third network card; wherein the third network card is a network card connected to the same access switch as the first network card.

4. The method of claim 1, wherein, The method further comprises: According to the number of sessions of each network card without communication failure in the computing power server, a replacement network card is selected for the first network card in a manner of preferentially selecting the network card with the least number of sessions; Or, According to the average rate of each network card without communication failure in the computing power server within a specified historical period, a replacement network card is selected for the first network card in a manner of preferentially selecting the network card with the lowest average rate.

5. The method of claim 1, wherein, The method further comprises: Determining a remote computing power server that has established a communication session with the first network card; Sending a first communication topology update message carrying the updated cluster communication topology to the remote computing power server.

6. The method of claim 1, wherein, The method further comprises: Receiving a second communication topology update message sent by a target computing power server; wherein the second communication topology update message is used to notify the receiving end computing power server to communicate with a second computing power acceleration card in the target computing power server through a replacement network card of a fourth network card; wherein the second computing power acceleration card is a computing power acceleration card associated with the fourth network card in the target computing power server; Deleting the communication session established with the second computing power acceleration card through the fourth network card, and re-establishing a communication session with the second computing power acceleration card through the replacement network card of the fourth network card.

7. The method of claim 1, wherein, The method further comprises: In the case that the replacement network card and the first network card are connected to different Peripheral Component Interconnect Express (PCIe) switches, for the communication data received through the replacement network card, the communication data is written into the memory of a third computing power acceleration card by using the replacement network card, and the communication data is sent to the first computing power acceleration card by the third computing power acceleration card through the internal network of the computing power server; For the communication data to be sent by the first computing power acceleration card, the communication data is sent to the third computing power acceleration card by the first computing power acceleration card through the internal network of the computing power server, the communication data is written into the memory of the third computing power acceleration card by the third computing power acceleration card, and the communication data is read from the memory of the third computing power acceleration card by using the replacement network card and sent; Wherein, the third computing power acceleration card is a computing power acceleration card associated with the replacement network card in the computing power server.

8. The method of claim 1, wherein, The method further comprises: In a case where the replacement network card is connected to a same Peripheral Component Interconnect Express (PCIe) switch as the first network card, for communication data received through the replacement network card, the received communication data is written into the memory of the first computing power acceleration card through the replacement network card; For communication data to be sent by the first computing power acceleration card, the communication data is written into the memory of the first computing power acceleration card by the first computing power acceleration card, and the communication data is read from the memory of the first computing power acceleration card by the replacement network card and sent.

9. A communications device, characterized by The device is deployed in a computing power server, and the device comprises: A selection unit is configured to, in a case where the computing power server participates in model training and it is determined that a first network card in the computing power server has a communication fault, select a replacement network card for the first network card from network cards in the computing power server that do not have a communication fault; An updating unit is configured to update, according to the replacement network card, an internal communication topology of the computing power server and a cluster communication topology; wherein the updated internal communication topology is used to indicate that the computing power server performs communication data transceiving of a first computing power acceleration card through the replacement network card; and the updated cluster communication topology is used to indicate that a receiving end computing power server communicates with the first computing power acceleration card through the replacement network card, the first computing power acceleration card being an algorithm acceleration card associated with the first network card in the computing power server; A communication unit is configured to send a first communication topology updating message carrying the updated cluster communication topology to other computing power servers in a communication domain in which the computing power server is located. The selection unit selects a replacement network card for the first network card from network cards in the computing power server that do not have a communication fault, comprising: In a case where the network cards in the computing power server that do not have a communication fault include at least one second network card, the replacement network card is selected for the first network card from the at least one second network card; wherein the second network card is a network card connected to a same Peripheral Component Interconnect Express (PCIe) switch as the first network card; For communication of the first computing power acceleration card with a plurality of other computing power servers in a communication domain in which the computing power server is located, different replacement network cards are respectively selected for the first network card from the network cards in the computing power server that do not have a communication fault as corresponding replacement network cards of the plurality of other computing power servers; wherein at least two of the plurality of other computing power servers correspond to different replacement network cards.

Images