Method, apparatus and electronic device for data transmission

Abstract

The present disclosure relates to the technical field of communication, and specifically provides a data transmission method and device and electronic equipment. The method comprises: using a bound queue pair to send and receive training data of a target training model through at least one logical path; when any local network interface card or any opposite network interface card is monitored to be abnormal, the bound queue pair of the target logical path existing abnormality is re-bound to a standby logical path of the target logical path, so as to realize path migration of the training data. In this way, during large-scale model training, the training task interruption is reduced through queue pair re-binding between server internal network interface cards.

Description

Technical Field

[0001] This application relates to the field of communication technology, specifically to a method, apparatus, and electronic device for data transmission. Background Technology

[0002] In the field of artificial intelligence, since the training of large-scale models (such as large language models or computer vision models) usually requires the collaboration of multiple servers to complete distributed computing tasks, efficient network connectivity is crucial to ensuring data transmission and efficient utilization of computing resources.

[0003] During model training, if any network interface card in the server fails, the training task will usually fail. Therefore, how to reduce training task interruptions during large-scale model training is a technical problem that needs to be solved. Summary of the Invention

[0004] In view of this, embodiments of this application provide a method, apparatus, and electronic device for data transmission.

[0005] On one hand, this application provides a data transmission method applied to a server in a distributed training system. The server is interconnected with a peer server through multiple logical paths. Each logical path includes a local network interface card (NIC) of the server and a peer NIC of the peer server. At least one queue pair is bound to each logical path. The queue pair is used to transmit training data of the target training model. The method includes:

[0006] Training data for the target training model is sent and received via a bound queue through at least one logical path.

[0007] When any local or remote network interface card malfunction is detected, the queue pair bound to the malfunctioning target logical path is rebound to the backup logical path of the target logical path to achieve path migration of training data.

[0008] In one implementation, when any local network interface card or any peer network interface card malfunction is detected, the queue pair bound to the malfunctioning target logical path is rebound to the backup logical path of the target logical path, including:

[0009] When any local network interface card or any peer network interface card is detected to be abnormal, the path status of each logical path in the path index table is updated. The path index table includes the local network interface card, peer network interface card, path status, and running status of each logical path, as well as the primary and backup correspondence.

[0010] If, based on the path status, it is determined that the target logical path exists in at least one logical path, then the running status of the target logical path in the path index table is updated to disabled; the path status of the target logical path is set to unavailable.

[0011] Based on the primary and backup correspondence in the path index table, update the running status of the backup logical path corresponding to the target logical path in the path index table to enabled;

[0012] Based on the path index table and the queue pairs bound to the target logical path, a path switching operation is performed.

[0013] In one implementation, a path switching operation is performed based on a path index table and queue pairs bound to the target logical path, including:

[0014] Unbind the queue pairs corresponding to the target logical path;

[0015] Establish a binding relationship between the unbound queue and the backup logical path corresponding to the target logical path.

[0016] In one implementation, detecting an anomaly in either the local network interface card or the peer network interface card includes:

[0017] Use at least one of the following:

[0018] The status of each local network interface card is monitored by the network card monitoring component, and the abnormal local network interface cards are identified based on the monitoring results.

[0019] When a notification message is received from a switch included in the distributed training system, the system determines which peer network interface card is abnormal based on the notification message; the notification message is sent by the switch when it determines that a peer network interface card is abnormal.

[0020] In one implementation, the notification message includes a status field; the status field is used to indicate the network interface card's status information.

[0021] In one implementation, the method includes:

[0022] Based on the network card status information of each local network interface, a subscription announcement message is sent to the switch to subscribe to the network card status information of each server. The switch then announces the network card status information of each local network interface card to the peer server, enabling the peer server to switch paths when it detects any abnormality of any local or peer network interface card.

[0023] In one implementation, the subscription announcement message includes a subscription service field and a network interface card (NIC) status field;

[0024] The subscription service field indicates whether to subscribe to the network interface card (NIC) status information of each server; the NIC status field represents the NIC status information of each network interface card; the network interface card includes the local network interface card and the peer network interface card.

[0025] In one embodiment, the method further includes:

[0026] When the target logical path is determined to be restored, the queue pairs bound to the backup logical path will be rebound to the target logical path.

[0027] On one hand, this application provides a data transmission apparatus applied to a server in a distributed training system. The server is interconnected with a peer server through multiple logical paths. Each logical path includes a local network interface card (NIC) of the server and a peer NIC of the peer server. At least one queue pair is bound to each logical path. The queue pair is used to transmit training data of the target training model. The apparatus includes:

[0028] A transmission unit is used to send and receive training data of a target training model through at least one logical path using bound queues.

[0029] The migration unit is used to rebind the queue pairs bound to the abnormal target logical path to the backup logical path of the target logical path when any local network interface card or any peer network interface card is detected to be abnormal, so as to realize the path migration of training data.

[0030] In one implementation, the migration unit is used for:

[0031] When any local network interface card or any peer network interface card is detected to be abnormal, the path status of each logical path in the path index table is updated. The path index table includes the local network interface card, peer network interface card, path status, and running status of each logical path, as well as the primary and backup correspondence.

[0032] If, based on the path status, it is determined that the target logical path exists in at least one logical path, then the running status of the target logical path in the path index table is updated to disabled; the path status of the target logical path is set to unavailable.

[0033] Based on the primary and backup correspondence in the path index table, update the running status of the backup logical path corresponding to the target logical path in the path index table to enabled;

[0034] Based on the path index table and the queue pairs bound to the target logical path, a path switching operation is performed.

[0035] In one implementation, the migration unit is used for:

[0036] Unbind the queue pairs corresponding to the target logical path;

[0037] Establish a binding relationship between the unbound queue and the backup logical path corresponding to the target logical path.

[0038] In one implementation, the migration unit is used for:

[0039] Use at least one of the following:

[0040] The status of each local network interface card is monitored by the network card monitoring component, and the abnormal local network interface cards are identified based on the monitoring results.

[0041] When a notification message is received from a switch included in the distributed training system, the system determines which peer network interface card is abnormal based on the notification message; the notification message is sent by the switch when it determines that a peer network interface card is abnormal.

[0042] In one implementation, the notification message includes a status field; the status field is used to indicate the network interface card's status information.

[0043] In one embodiment, the migration unit is further configured to:

[0044] Based on the network card status information of each local network interface, a subscription announcement message is sent to the switch to subscribe to the network card status information of each server. The switch then announces the network card status information of each local network interface card to the peer server, enabling the peer server to switch paths when it detects any abnormality of any local or peer network interface card.

[0045] In one implementation, the subscription announcement message includes a subscription service field and a network interface card (NIC) status field;

[0046] The subscription service field indicates whether to subscribe to the network interface card (NIC) status information of each server; the NIC status field represents the NIC status information of each network interface card; the network interface card includes the local network interface card and the peer network interface card.

[0047] In one embodiment, the migration unit is further configured to:

[0048] When the target logical path is determined to be restored, the queue pairs bound to the backup logical path will be rebound to the target logical path.

[0049] On one hand, this application provides an electronic device, including:

[0050] Processor; and

[0051] The memory stores computer instructions that cause the processor to perform the steps of the methods provided in various alternative implementations of any of the data transfers described above.

[0052] On one hand, embodiments of this application provide a computer-readable storage medium storing computer instructions for causing a computer to perform steps of the methods provided in various optional implementations of any of the above-described data transmission methods.

[0053] On one hand, this application provides a computer program product including computer-readable code or a non-volatile computer-readable storage medium carrying computer-readable code. When the computer-readable code is run in the processor of an electronic device, the processor in the electronic device performs the steps of the method provided in various optional implementations of any of the above-described data transmission methods.

[0054] The data transmission method in this embodiment sends and receives training data for the target training model through at least one logical path using bound queue pairs. When any local or remote network interface card malfunction is detected, the queue pair bound to the malfunctioning target logical path is rebound to a backup logical path of the target logical path to achieve path migration of training data. In this way, during large-scale model training, the rebinding of queue pairs between network cards within the server reduces training task interruptions. Attached Figure Description

[0055] Figure 1 This is an architecture diagram of a distributed training system according to an embodiment of this application.

[0056] Figure 2 This is a flowchart of a method for network card status announcement in an embodiment of this application.

[0057] Figure 3 This is an example diagram of an LLDP message in an embodiment of this application.

[0058] Figure 4 This is an example diagram of a subscription announcement message format in an embodiment of this application.

[0059] Figure 5 This is an example diagram of an IPN notification message format in an embodiment of this application.

[0060] Figure 6 This is a flowchart of a data transmission method according to an embodiment of this application.

[0061] Figure 7 This is a structural block diagram of a data transmission device according to an embodiment of this application.

[0062] Figure 8 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0063] The technical solution of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. Furthermore, the technical features involved in the different embodiments of this application described below can be combined with each other as long as they do not conflict with each other.

[0064] In the field of artificial intelligence, large-scale model training typically relies on high-performance computing clusters, where multiple servers collaborate to complete distributed computing tasks. These training tasks often employ a multi-machine, multi-GPU parallel model. For example, NVIDIA's Collective Communications Library (NCCL) is used for ensemble communication operations such as All-Reduce and All-Gather to achieve gradient synchronization and parameter updates. Large-scale model training consumes significant computing resources, including CPU memory and GPU memory; therefore, efficient network connectivity is crucial for ensuring data transmission and efficient utilization of computing resources.

[0065] In high-speed network environments using Remote Direct Memory Access (RDMA) technology, a network interface card (NIC) failure can lead to interruption of queue pair (QP) communication, resulting in training task failure. Since training large models is extremely time-consuming (typically requiring days to weeks), maintaining the continuity of the training process is crucial for ensuring the consistency of results and the reproducibility of experiments. A single interruption not only wastes computational resources but also significantly extends the overall training time, while multiple interruptions can further compromise model convergence and increase additional parameter tuning costs.

[0066] Therefore, minimizing training interruptions during large model training is a technical problem that needs to be solved.

[0067] Based on the deficiencies of the aforementioned related technologies, this application provides a data transmission method, apparatus, and electronic device to address the problem of reducing training task interruptions.

[0068] This application provides a data transmission method that can be applied to electronic devices. This application does not limit the type of electronic device, which can be any suitable type of device, such as terminal devices and servers, etc. This application will not elaborate further.

[0069] In this embodiment, the application is applied to a distributed training system, which includes multiple servers. See also... Figure 1 The diagram shown is an architecture diagram of a distributed training system. Figure 1 In this example, which involves two servers, either server can be called the local server and the other server can be called the peer server.

[0070] Optionally, the distributed system may also include at least one switch. Figure 1 In this example, a distributed system consisting of a first switch, a second switch, and a third switch is used for illustration. The first switch is directly connected to the local server, the second switch is connected to both the first and third switches, and the third switch is directly connected to the peer server.

[0071] The server interconnects with the peer server through multiple logical paths. Each logical path includes one local network interface card (NIC) on the server and one peer NIC on the peer server. At least one queue pair is bound to each logical path, and these queue pairs are used to transmit training data for the target training model. In other words, different logical paths contain different combinations of NICs; each NIC combination includes one local network interface card and one peer network interface card. The local network interface card is the NIC in the local server, and the peer network interface card is the NIC in the peer server. The local server has multiple local network interface cards, and the peer server has multiple peer network interface cards. A network interface card can also be called a network card (NIC). It should be noted that if the NICs connected to both ends of two paths are the same, but the switches or other connections they pass through are different, then these two paths are considered the same logical path.

[0072] Both the local and remote servers are equipped with network interface card (NIC) monitoring components, which are used to collect the status of the local and remote NICs. For example, the NIC monitoring component can be an Automatic Path Detection (ASPD) component. Then, based on the status of the local and remote NICs, a path index table can be generated so that the local and remote servers can switch paths based on their respective path index tables.

[0073] In this embodiment of the application, before data transmission, each server first obtains its own network card status information during the initialization phase, and then notifies other servers of its own network card status information so as to establish multiple logical paths between servers in the future.

[0074] The following is combined with Figure 2 This example only illustrates the network interface card (NIC) status information notification on the local server during the initialization phase. See also... Figure 2 The diagram shown is a flowchart of a network card status announcement method according to an embodiment of this application. The following is a description of the method in conjunction with... Figure 1 The method is described below, and the specific implementation process is as follows:

[0075] Step 201: The local server detects the network interface card information of the local network interface card.

[0076] In one implementation, the ASPD in the local server collects network interface card (NIC) related information of the local network interface card and generates an initial path index table based on the NIC related information.

[0077] The network interface card (NIC) related information may include the NIC identifier, the GPU bound to the local network interface card, and the NIC status information (e.g., offline, online). It may also include the path status and running status of the logical path corresponding to the local network interface.

[0078] The path status indicates whether a logical path is available, and the running status indicates whether a logical path is enabled. At the current stage, only the network interface card information of the local network interface card has been obtained, and the network interface card information of the remote network interface card has not yet been obtained. Therefore, each logical path has not been successfully established. Thus, each logical path is currently in the idle state, i.e., the initial state. In this state, the logical path is unavailable, and the running status is disabled (i.e., not enabled).

[0079] Table 1

[0080]

[0081] Table 2

[0082]

[0083] For example, the local server contains 8 network interface cards (NICs), named NIC1, NIC2, ..., NIC8. A corresponding path index table is created for each NIC. In this path index table, the logical path corresponding to that NIC is designated as the primary logical path, and the other logical paths are designated as backup logical paths. The primary logical path can be represented by the character "Master," and the backup logical paths can be represented by the character "Backup."

[0084] Taking Tables 1 and 2 as examples, Table 1 is the path index table for NIC1. In path index table 1, the logical path corresponding to NIC1 is the primary logical path, and the logical paths corresponding to other network cards are backup logical paths. Table 2 is the path index table for NIC8. In path index table 8, the logical path corresponding to NIC8 is the primary logical path, i.e., Master, and the logical paths corresponding to other network cards are backup logical paths, i.e., Backup.

[0085] In other words, each logical path is a primary logical path and can also serve as a backup logical path for other logical paths. A logical path can have one or more backup logical paths, which is not limited here.

[0086] Traffic paths, or their running status, include enabled and disabled. Enabled can be represented as True, and disabled as False. True indicates the current traffic path, and False indicates that no current traffic is being transmitted. When a logical path with the role of master fails, its corresponding path status becomes inactive, and the corresponding traffic path becomes False. At this time, traffic is redirected to the logical path with the role of backup (i.e., the corresponding standby logical path), and the running status of the standby logical path is changed to selected, i.e., True. Inactive indicates that the path is in a listening state.

[0087] Step 202: The local server sends a subscription announcement message to the first switch.

[0088] In one implementation, the ASPD component in the local server sends a subscription announcement message to the first switch based on the Link Layer Discovery Protocol (LLDP) to subscribe to the network interface card status information of each server, and announces the network interface card status information of each local network interface card to the peer server through the switch.

[0089] Optionally, the subscription announcement message can be sent periodically or when a change in the status of the local network interface is detected. Network interface status changes can include network interface joining, network interface leaving, or access link failure.

[0090] The subscription notification message includes a subscription service field and a network interface card (NIC) status field. The subscription service field indicates whether to subscribe to the NIC status information of each server. The NIC status field indicates the NIC status information of each network interface card.

[0091] See Figure 3 The image shown is an example diagram of an LLDP message. An LLDP message includes the following fields: DA, SA, and Type.

[0092] Wherein, DA stands for Destination Physical Address (Media Access Control, MAC), which can be a fixed multicast MAC address such as 0x0180-C200-000E. SA stands for Source MAC Address, which can be a port MAC address or a device MAC address. If a port address is available, use the port MAC address; otherwise, use the device MAC address. Type stands for Frame Type, which is 0x88CC.

[0093] In this embodiment, the LLDP advertisement message is extended based on Intelligent Lossless Parameter Network (IPN) technology to obtain a subscription advertisement message. Specifically, an LLDP extended Type-Length-Value (TLV) field is added to carry the network interface card (NIC) IPN parameters, thereby enabling the NIC to join the advertisement function.

[0094] The following is combined with Figure 4 For an explanation of the IPN extended TLV format in the subscription announcement message, please refer to [link / reference]. Figure 4 The image shown is an example diagram of a subscription announcement message format. Figure 4 In the subscription announcement message, the extended fields include: OUI, subtype, IP address type, and IP address.

[0095] Wherein, OUI: Organizationally Unique Identifier (OUI), indicating the iPN specification followed by the host. subtype: indicates that this TLV is an IPN extended TLV, carrying IPN-related information. IP address type: a value of 1 indicates an IPv4 address; a value of 2 indicates an IPv6 address. Subscription service: indicates whether this server subscribes to network information change messages, i.e., whether it wants information on the joining and leaving status of other servers' network cards, i.e., network card change information of other servers. A value of 1 indicates subscription, and this server needs to be notified when other servers' network cards join or leave the network; a value of 0 indicates no subscription, and this server does not need to be notified when other servers' network cards join or leave the network. In this embodiment, each server subscribes to the above-mentioned network information change messages; IP address: the Internet Protocol (IP) address of this server. For IPv4 addresses, only the first 4 bytes are occupied, and the remaining bytes are filled with 0.

[0096] Step 203: When the first switch determines that there is a change in the network card status based on the received subscription announcement message, it sends a notification message to the local server.

[0097] In one implementation, when the first switch determines that there is a change in the network interface card (NIC) status based on the received subscription announcement message, it updates the NIC status information stored locally and sends the NIC change information to the switch.

[0098] In this way, when a network interface card (NIC) in one server changes its state, other servers can quickly detect the change, enabling the establishment and disconnection of connections between NICs and achieving rapid switching of logical paths.

[0099] Step 204: When the local server receives the notification message, it returns an acknowledgment message to the first switch.

[0100] Both the notification message and the acknowledgment message are IPN notification messages, i.e., status notification messages. The notification message is generated on the switch and sent to the server to notify the server of network information changes, i.e., network interface card (NIC) status changes. The acknowledgment message can be an acknowledgment (ACK) message, generated on the server and sent to the switch to acknowledge the received status change notification message. The notification message contains a status field; the status field indicates the NIC status information.

[0101] See Figure 5 The image shown is an example diagram of an IPN notification message format. Figure 5 In the IPN notification message, the following fields are mainly included: ETH type and TLV message.

[0102] Specifically, the value of the ETH type field is used to determine whether a message is an IPN notification message, and then the value of the TLV type field is used to determine whether the message is a notification message or an ACK message.

[0103] Table 3

[0104]

[0105] Table 4

[0106]

[0107] Table 3 is a description table for a TLV message. A TLV message contains multiple fields, namely TLV type, TLV message length, and notification message content. Table 4 is a description table for a subTLV type. Different TLVs carry different combinations of subTLVs; the definition of subTLVs is shown in Table 4. The subTLV type is the status field.

[0108] Step 205: The first switch notifies the other two switches of the network card status information.

[0109] In one implementation, after obtaining the network interface card (NIC) status information, the first switch encapsulates the NIC status information into a Border Gateway Protocol (BGP) update message using IPNi routing and sends the BGP update message to the second switch. The second switch then updates its locally stored NIC status information. Similarly, the second switch announces the NIC status information to the third switch, which then updates its locally stored NIC status information.

[0110] In this embodiment, only the first switch and the second switch are used as examples for illustration. In actual applications, after any switch in the system obtains the network card status information, it can encapsulate the network card status information into a BGP update message and notify its BGP neighbor, i.e., the adjacent switch, through the BGP update message.

[0111] Step 206: The third switch confirms that it has received the network card status information and sends a notification message to the peer server.

[0112] In this embodiment, the example of announcing the network card status information of other servers to the peer server through a third switch is used for illustration. In actual applications, any switch directly connected to the peer server can announce the network card status information to the peer server, and there is no restriction here.

[0113] Step 207: When the peer server receives the notification message, it obtains the network card status information of the local server and returns an acknowledgment message to the third switch.

[0114] When performing steps 206 and 207, please refer to steps 203 and 204, which will not be repeated here.

[0115] In this embodiment, the local server can be any server in the system. The example described is simply announcing the network interface card (NIC) status information of the local server to the peer server. Similarly, the NIC status information of the peer server can also be announced to the local server. After obtaining the NIC status information of the peer server, the local server can establish each logical path and update its local path index tables, which will not be elaborated further here.

[0116] Table 5

[0117]

[0118] Table 6

[0119]

[0120] For example, Table 5 is an updated path index table 1, and Table 6 is an updated path index table 8. "Active" indicates that the path connection was successfully established. In Tables 5 and 6, the path status is "Active" after the connection is successfully established.

[0121] After establishing multiple logical paths, data can be transmitted based on each logical path to achieve model training of the target training model. The following section will combine... Figure 6 For an explanation of the data transmission method, please refer to [link / reference]. Figure 6 The diagram shown is a flowchart of a data transmission method according to an embodiment of this application, applied to each server in a distributed training system, i.e., the local server. The following is a combination of... Figure 1 , Figure 2 and Figure 6 The method is described below, and the specific implementation process is as follows:

[0122] Step 601: Send and receive training data for the target training model through at least one logical path using the bound queue.

[0123] In other words, training data for queue pairs bound to each logical path is transmitted through each logical path between the two servers.

[0124] The training data refers to the data stream that needs to be transmitted between different servers during model training. In the local server, each logical path's local network interface card is bound to one or more queue pairs. Each queue pair includes a send queue and a receive queue; the send queue is used to send data, and the receive queue is used to receive data.

[0125] Step 602: When any local network interface card or any peer network interface card is detected to be abnormal, the queue pair bound to the abnormal target logical path is rebound to the backup logical path of the target logical path to realize the path migration of training data.

[0126] In one implementation, when performing step 602, the following steps may be used:

[0127] S6021: When any local network interface card or any peer network interface card is detected to be abnormal, the path status of each logical path in the path index table is updated. The path index table includes the local network interface card, peer network interface card, path status and running status of each logical path, as well as the primary and backup correspondence.

[0128] The primary / backup correspondence is determined by the roles.

[0129] In one implementation, at least one of the following is used:

[0130] Method 1: Use a network card monitoring component to monitor the status of each local network interface card, and identify any abnormal local network interface cards based on the monitoring results.

[0131] Method 2: When a notification message is received from a switch included in the distributed training system, the abnormal peer network interface card is determined based on the notification message; the notification message is sent by the switch when it determines that the peer network interface card is abnormal.

[0132] Furthermore, subscription announcement messages can be sent to the switch to announce subscriptions and network card status.

[0133] In one implementation, based on the network card status information of each local network interface, a subscription announcement message is sent to the switch to subscribe to the network card status information of each server, and the network card status information of each local network interface card is announced to the peer server through the switch, so that the peer server can switch paths when it detects any abnormality of any local network interface card or any abnormality of any peer network interface card.

[0134] The network interface card includes the local network interface card and the peer network interface card. The subscription announcement message includes a subscription service field and a network interface card status field; the subscription service field indicates whether to subscribe to the network interface card status information of each server; the network interface card status field indicates the network interface card status information of each network interface card.

[0135] S6022: If, based on the path status, it is determined that at least one logical path contains the target logical path, then the running status of the target logical path in the path index table is updated to disabled; the path status of the target logical path is set to unavailable.

[0136] S6023: Based on the primary / backup correspondence in the path index table, update the running status of the backup logical path corresponding to the target logical path in the path index table to enabled.

[0137] S6024: Execute a path switching operation based on the path index table and the queue pairs bound to the target logical path.

[0138] In one implementation, when executing S6024, the following steps may be taken:

[0139] S6024-1: Unbind the queue pairs corresponding to the target logical path.

[0140] S6024-2: Establish a binding relationship between the unbound queue pair and the backup logical path corresponding to the target logical path.

[0141] In this embodiment of the application, when any local network interface card or any peer network interface card is detected to be abnormal, the path index tables are updated. Specifically, in each path index table, the path status of the target logical path corresponding to the abnormal network card is updated to Inactive. In the path index table of the target logical path as the primary logical path, the running status of the target logical path is switched to False and the running status of the backup logical path in the path index table is switched to True. In addition, the training data of the target logical path is automatically switched to the corresponding backup logical path using the queue pair (QP) load balancing method.

[0142] Specifically, the target logical path is unbound from the locally bound QP, and the unbound QP is bound to a backup logical path to transmit the QP's training data. Using this method, viewing the training data on the switch side is uninterrupted.

[0143] Table 7

[0144]

[0145] For example, Table 7 shows a path index table 1 after a failover. If the local server detects an anomaly in either the local or remote network interface card (NIC1), the logical paths corresponding to NIC1 and NIC1 are considered the target logical paths with the anomaly. The target logical path's role in path index table 1 is Master. In path index table 1, the path status corresponding to either NIC1 or NIC1 is updated to Inactive, and the corresponding running status is switched to Disabled (False). The running status of the backup logical path in the path index table is also switched to Enabled (True). Subsequently, a queue pair (QP) load balancing method can be used to automatically switch the training data of the target logical path to the corresponding backup logical path.

[0146] Furthermore, it can be determined that when the target logical path is restored, the queue pairs bound to the backup logical path will be rebound to the target logical path.

[0147] Table 8

[0148]

[0149] Table 8 shows a path index table 1 after fault recovery. After the local server detects that either its local network interface card (NIC1) or the peer network interface card (NIC1) has returned to normal, and thus the target logical path has also returned to normal, it updates the path index tables again. In path index table 1, the path status corresponding to either local or peer NIC1 is updated to Active, and the corresponding running status is switched to True. The running status of the backup logical path in this path index table is also switched to False. Furthermore, a queue pair (QP) load balancing method can be used to automatically switch the training data from the backup logical path back to the target logical path. In this way, the training data can be switched back to the corresponding primary logical path.

[0150] In this embodiment, when the NIC fails, the ASPD QP load balancing mechanism automatically migrates traffic to a backup logical path. A single node failure does not affect the overall cluster topology, reducing training task failures caused by network interruptions, avoiding model parameter oscillations due to multiple interruptions, ensuring the integrity of the training process, and reducing resource consumption caused by failure interruptions. This is particularly suitable for large model training scenarios requiring continuous training for several weeks, meeting the needs of complex training architectures such as distributed data parallelism and model parallelism. Furthermore, seamless failover is achieved with minimal switching time and seamless communication. Fault detection, path switching, and traffic migration are all completed collaboratively by ASPD and IPN, eliminating the need for manual troubleshooting or reconfiguration, thus reducing operational costs.

[0151] Based on the same inventive concept, this application also provides a data transmission apparatus. Since the principle of the above-mentioned apparatus and device in solving the problem is similar to that of a data transmission method, the implementation of the above-mentioned apparatus can refer to the implementation of the method, and repeated details will not be elaborated further. This apparatus can be applied to electronic devices. This application does not limit the type of electronic device; it can be any suitable type of device, such as terminal devices and servers, etc., which will not be elaborated further in this application. The apparatus embodiment can be implemented by software, or by hardware, or a combination of software and hardware. Taking software implementation as an example, as a logically defined apparatus, it is formed by the processor of the electronic device reading the corresponding computer program instructions from the non-volatile memory into memory and running them.

[0152] See Figure 7 The diagram shown is a structural block diagram of a data transmission apparatus according to an embodiment of this application. In some embodiments, the data transmission apparatus exemplified in this application includes:

[0153] The transmission unit 701 is used to send and receive training data of the target training model through at least one logical path using a bound queue.

[0154] The migration unit 702 is used to rebind the queue pair bound to the target logical path with the abnormality to the backup logical path of the target logical path when any local network interface card or any peer network interface card is detected to be abnormal, so as to realize the path migration of training data.

[0155] In one embodiment, the migration unit 702 is used for:

[0156] When any local network interface card or any peer network interface card is detected to be abnormal, the path status of each logical path in the path index table is updated. The path index table includes the local network interface card, peer network interface card, path status, and running status of each logical path, as well as the primary and backup correspondence.

[0157] If, based on the path status, it is determined that the target logical path exists in at least one logical path, then the running status of the target logical path in the path index table is updated to disabled; the path status of the target logical path is set to unavailable.

[0158] Based on the primary and backup correspondence in the path index table, update the running status of the backup logical path corresponding to the target logical path in the path index table to enabled;

[0159] Based on the path index table and the queue pairs bound to the target logical path, a path switching operation is performed.

[0160] In one embodiment, the migration unit 702 is used for:

[0161] Unbind the queue pairs corresponding to the target logical path;

[0162] Establish a binding relationship between the unbound queue and the backup logical path corresponding to the target logical path.

[0163] In one embodiment, the migration unit 702 is used for:

[0164] Use at least one of the following:

[0165] The status of each local network interface card is monitored by the network card monitoring component, and the abnormal local network interface cards are identified based on the monitoring results.

[0166] When a notification message is received from a switch included in the distributed training system, the system determines which peer network interface card is abnormal based on the notification message; the notification message is sent by the switch when it determines that a peer network interface card is abnormal.

[0167] In one implementation, the notification message includes a status field; the status field is used to indicate the network interface card's status information.

[0168] In one embodiment, the migration unit 702 is further configured to:

[0169] Based on the network card status information of each local network interface, a subscription announcement message is sent to the switch to subscribe to the network card status information of each server. The switch then announces the network card status information of each local network interface card to the peer server, enabling the peer server to switch paths when it detects any abnormality of any local or peer network interface card.

[0170] In one implementation, the subscription announcement message includes a subscription service field and a network interface card (NIC) status field;

[0171] The subscription service field indicates whether to subscribe to the network interface card (NIC) status information of each server; the NIC status field represents the NIC status information of each network interface card; the network interface card includes the local network interface card and the peer network interface card.

[0172] In one embodiment, the migration unit 702 is further configured to:

[0173] When the target logical path is determined to be restored, the queue pairs bound to the backup logical path will be rebound to the target logical path.

[0174] On one hand, this application provides an electronic device, including:

[0175] Processor; and

[0176] The memory stores computer instructions that cause the processor to perform the steps of the methods provided in various alternative implementations of any of the data transfers described above.

[0177] On one hand, embodiments of this application provide a computer-readable storage medium storing computer instructions for causing a computer to perform steps of the methods provided in various optional implementations of any of the above-described data transmission methods.

[0178] On one hand, this application provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code. When the computer-readable code is run in the processor of an electronic device, the processor in the electronic device performs the steps of the method provided in various optional implementations of any of the above-described data transmission methods.

[0179] The data transmission method in this application includes sending and receiving training data of a target training model through at least one logical path using bound queue pairs. When any local or remote network interface card malfunction is detected, the queue pairs bound to the malfunctioning target logical path are rebound to a backup logical path of the target logical path to achieve path migration of training data. In this way, during large-scale model training, the rebinding of queue pairs between network cards within the server reduces training task interruptions.

[0180] In this embodiment of the application, an electronic device is also provided, including:

[0181] Processor; and

[0182] The memory stores computer instructions that cause the processor to execute the methods of any of the above-described embodiments.

[0183] In this application embodiment, a computer-readable storage medium is provided, storing computer instructions for causing a computer to perform the methods of any of the above embodiments.

[0184] This application also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in the processor of an electronic device, the processor in the electronic device performs the method of any of the above-described embodiments.

[0185] Figure 8 A schematic diagram of the structure of an electronic device 8000 is shown. (See also...) Figure 8 As shown, the electronic device 8000 includes a processor 8010 and a memory 8020, and optionally may also include a power supply 8030, a display unit 8040, and an input unit 8050.

[0186] The processor 8010 is the control center of the electronic device 8000. It connects various components through various interfaces and lines, and performs various functions of the electronic device 8000 by running or executing software programs and / or data stored in the memory 8020, thereby performing overall monitoring of the electronic device 8000.

[0187] In this embodiment, when the processor 8010 calls the computer program stored in the memory 8020, it executes the steps in the above embodiments.

[0188] Optionally, the processor 8010 may include one or more processing units; preferably, the processor 8010 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 8010. In some embodiments, the processor and memory may be implemented on a single chip; in some embodiments, they may also be implemented separately on independent chips.

[0189] The memory 8020 may primarily include a program storage area and a data storage area. The program storage area may store the operating system, various applications, etc.; the data storage area may store data created based on the use of the electronic device 8000, etc. In addition, the memory 8020 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0190] The electronic device 8000 also includes a power supply 8030 (such as a battery) that supplies power to various components. The power supply can be logically connected to the processor 8010 through a power management system, thereby enabling the management of charging, discharging, and power consumption.

[0191] The display unit 8040 can be used to display information input by the user or information provided to the user, as well as various menus of the electronic device 8000. In this embodiment, it is mainly used to display the display interface of various applications in the electronic device 8000, as well as text, images, and other objects displayed on the display interface. The display unit 8040 may include a display panel 8041. The display panel 8041 may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.

[0192] The input unit 8050 can be used to receive information such as numbers or characters input by the user. The input unit 8050 may include a touch panel 8051 and other input devices 8052. The touch panel 8051, also known as a touch screen, can collect touch operations on or near the user (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel 8051).

[0193] Specifically, the touch panel 8051 can detect user touch operations and the signals generated by these operations, convert them into touch point coordinates, send them to the processor 8010, and receive and execute commands from the processor 8010. Furthermore, the touch panel 8051 can be implemented using various types of sensors, including resistive, capacitive, infrared, and surface acoustic wave sensors. Other input devices 8052 can include, but are not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc.

[0194] Of course, the touch panel 8051 can cover the display panel 8041. When the touch panel 8051 detects a touch operation on or near it, it transmits the information to the processor 8010 to determine the type of touch event. Subsequently, the processor 8010 provides corresponding visual output on the display panel 8041 based on the type of touch event. Although in Figure 8 In this embodiment, the touch panel 8051 and the display panel 8041 are two separate components to realize the input and output functions of the electronic device 8000. However, in some embodiments, the touch panel 8051 and the display panel 8041 can be integrated to realize the input and output functions of the electronic device 8000.

[0195] The electronic device 8000 may also include one or more sensors, such as a pressure sensor, a gravity acceleration sensor, a proximity sensor, etc. Of course, depending on the specific application, the electronic device 8000 may also include other components such as a camera. Since these components are not the focus of this application's embodiments, therefore... Figure 8 It is not shown in the text and will not be described in detail here.

[0196] Those skilled in the art will understand that Figure 8 This is merely an example of an electronic device and does not constitute a limitation on the electronic device. It may include more or fewer components than shown, or a combination of certain components, or different components.

[0197] For ease of description, the above sections are divided into modules (or units) according to their functions and described separately. Of course, in implementing this application, the functions of each module (or unit) can be implemented in one or more software or hardware components.

Claims

1. A method for data transmission, characterized in that, A server applied in a distributed training system, wherein the server is interconnected with peer servers through multiple logical paths, each logical path including a local network interface card (NIC) of the server and a peer NIC of the peer server, and at least one queue pair is bound to each logical path, the queue pair being used to transmit training data of the target training model, the method comprising: Training data of the target training model is sent and received through at least one logical path using bound queues. When any local network interface card or any peer network interface card is detected to be abnormal, the queue pair bound to the abnormal target logical path will be rebound to the backup logical path of the target logical path to realize the path migration of training data.

2. The method according to claim 1, characterized in that, When any local network interface card or any peer network interface card malfunction is detected, the queue pair bound to the malfunctioning target logical path is rebound to the backup logical path of the target logical path, including: When any local network interface card or any peer network interface card is detected to be abnormal, the path status of each logical path in the path index table is updated. The path index table includes the local network interface card, peer network interface card, path status, and running status of each logical path, as well as the primary / backup correspondence. If, based on the path status, it is determined that the target logical path exists in at least one logical path, then the running status of the target logical path in the path index table is updated to disabled; the path status of the target logical path is unavailable. Based on the primary / backup correspondence in the path index table, the running status of the backup logical path corresponding to the target logical path is updated to enabled in the path index table; Based on the path index table and the queue pairs bound to the target logical path, a path switching operation is performed.

3. The method according to claim 2, characterized in that, The step of performing a path switching operation based on the path index table and the queue pairs bound to the target logical path includes: Unbind the queue pairs corresponding to the target logical path; The unbound queue is bound to the backup logical path corresponding to the target logical path.

4. The method according to claim 2, characterized in that, The detection of any local network interface card malfunction or any peer network interface card malfunction includes: Use at least one of the following: The status of each local network interface card is monitored by the network card monitoring component, and the abnormal local network interface cards are identified based on the monitoring results. When a notification message is received from a switch included in the distributed training system, an abnormal peer network interface card is determined based on the notification message; wherein, the notification message is sent by the switch when it determines that an abnormal peer network interface card exists.

5. The method according to claim 4, characterized in that, The notification message includes a status field; the status field is used to indicate the network interface card status information.

6. The method according to claim 4, characterized in that, The method includes: Based on the network card status information of each local network interface, a subscription announcement message is sent to the switch to subscribe to the network card status information of each server. The switch is then used to announce the network card status information of each local network interface card to the peer server, so that the peer server can switch paths when it detects any abnormality of any local network interface card or any abnormality of any peer network interface card.

7. The method according to claim 6, characterized in that, The subscription announcement message includes a subscription service field and a network interface card status field; The subscription service field is used to indicate whether to subscribe to the network card status information of each server; the network card status field represents the network card status information of each network interface card; the network interface card includes the local network interface card and the peer network interface card.

8. The method according to any one of claims 1-7, characterized in that, The method further includes: When the target logical path is determined to be restored, the queue pairs bound to the backup logical path are rebound to the target logical path.

9. A data transmission apparatus, characterized in that, A server used in a distributed training system, wherein the server is interconnected with peer servers through multiple logical paths, each logical path including a local network interface card (NIC) of the server and a peer NIC of the peer server, and at least one queue pair is bound to each logical path, the queue pair being used to transmit training data of the target training model, the device comprising: A transmission unit is used to send and receive training data of the target training model through at least one logical path using bound queues. The migration unit is used to rebind the queue pairs bound to the target logical path with the abnormality to the backup logical path of the target logical path when any local network interface card or any peer network interface card is detected to be abnormal, so as to realize the path migration of training data.

10. An electronic device, characterized in that, include: processor; as well as A memory storing computer instructions for causing the processor to perform the method according to any one of claims 1 to 8.

Images