A multi-device synchronous communication method, a computing device cluster and a chip
By pre-establishing a WQE queue that supports multiple transmissions in the computing device cluster and utilizing a memory bit counter, the problems of long synchronization time and difficulty in fault location for multiple devices are solved, achieving efficient synchronization and fault detection.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HUAWEI TECH CO LTD
- Filing Date
- 2024-08-08
- Publication Date
- 2026-08-04
AI Technical Summary
In large model training or inference scenarios, the synchronization time between computing devices increases, especially when the number of devices to be synchronized increases. Frequent generation of synchronization messages leads to excessively long synchronization processes, and existing methods are difficult to accurately locate faulty devices or communication links.
By pre-establishing multiple work queue elements (WQEs) in the computing device cluster, each WQE contains a synchronization message and supports multiple transmissions, reducing the need for temporary generation of synchronization messages. It also uses memory bits and counters to identify faulty devices or links and accurately locate the fault point.
It reduces the time spent synchronizing multiple devices, improves synchronization efficiency, and can accurately locate faulty devices or communication links, avoiding unnecessary consumption of computing resources.
Smart Images
Figure CN121509355B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of device synchronization, and more particularly to a multi-device synchronous communication method, a computing device cluster, and a chip. Background Technology
[0002] Currently, in applications such as large model training or inference, clusters consist of multiple servers, each containing at least one graphics processing unit (GPU) or neural network processing unit (NPU). Each computing device performs data processing operations in parallel, generating its own data to be synchronized, such as new model parameters and gradients. When a synchronization point is reached (e.g., the completion of a training step), the computing devices broadcast the data to be synchronized to the other devices. Furthermore, when other computing devices reach their synchronization points, they receive the data generated by the other devices, ensuring the overall effectiveness and correctness of the model training.
[0003] Specifically, when a computing device reaches a synchronization point and sends the acquired data to be synchronized to multiple computing devices, the computing device needs to temporarily initiate multiple synchronization messages to notify the multiple computing devices to be synchronized that they have received the data to be synchronized sent by the computing device. As the number of computing devices to be synchronized increases, the time it takes for the computing device to generate a synchronization message each time increases, and because the computing devices need to synchronize frequently, the time consumed by multi-device synchronization increases. Summary of the Invention
[0004] This application discloses a multi-device synchronous communication method, a computing device cluster, and a chip. After a first device in the computing device cluster sends data to be synchronized to multiple second devices, it sends multiple work queue elements (WQEs) in a queue that can be sent multiple times. Each WQE includes a first synchronization message, which notifies the multiple second devices that the data to be synchronized generated by the first device has been received. Since multiple WQEs in the queue can support multiple transmissions by the first device, the first device does not need to temporarily generate multiple WQEs after each transmission of data to be synchronized, thus reducing the time consumption of multi-device synchronization.
[0005] In a first aspect, this application provides a multi-device synchronous communication method, which is applied to a computing device cluster. The method includes: a first device in the computing device cluster obtains first data to be synchronized through service operation and sends the first data to be synchronized to multiple second devices. Then, the first device sends multiple WQEs in the queue to the multiple second devices to notify the second devices that the first data to be synchronized has been received. The multiple WQEs support multiple transmissions by the first device.
[0006] In the above process, multiple WQEs in the queue can be sent multiple times. When the first device sends the first data to be synchronized to multiple second devices, it sends multiple WQEs in the queue, including the first synchronization message, to the multiple second devices to be synchronized, so as to notify the second devices that the first data to be synchronized has been received. The first device does not need to temporarily generate WQEs including the first synchronization message, which can reduce the synchronization time of multiple devices and decouple the synchronization time from the number of devices in the cluster.
[0007] For example, before the first device runs the service, the multi-device synchronization communication method further includes: the first device establishing multiple WQEs in a queue according to the number of multiple second devices to be synchronized; generating a first synchronization message; and writing the first synchronization message and the address information of the multiple second devices into the multiple WQEs.
[0008] The above process only needs to be executed once in this application. Before the service starts running, the first device generates multiple WQEs according to the number of multiple second devices to be synchronized, and sends the multiple WQEs to the pre-built queue. Since WQEs support multiple transmissions, the first device does not need to execute the above process multiple times after sending new first synchronized data to the second devices multiple times. Therefore, it does not need to temporarily generate and send new WQEs to the queue, which can reduce the time for the device to generate WQEs multiple times and avoid unnecessary consumption of computing resources by the first device.
[0009] For example, after the first device sends a first synchronization message from a queue to a plurality of second devices, the method further includes: the first device receiving a second synchronization message from each of the plurality of second devices, wherein the first device determines, based on the received second synchronization data, that it has received the second synchronization data sent by the corresponding second device. Similarly, the second synchronization message is present in the WQE sent by the second device to the first device.
[0010] In the above process, the first device receives different second synchronization messages sent by multiple second devices. Each second synchronization message is used to notify the first device that it has received different second data to be synchronized. The first device can determine whether it has received data to be synchronized from all the devices to be synchronized based on the received second synchronization messages, so as to determine whether the first device has reached the synchronization point.
[0011] For example, the method further includes: when the number of second synchronization messages received is the same as the number of second devices, the first device determines that it has received second data to be synchronized sent by each of the second devices, and then obtains third data to be synchronized based on the first data to be synchronized and all the second data to be synchronized.
[0012] For example, after obtaining the third data to be synchronized, the method further includes: after the first device sends the newly generated third data to be synchronized to multiple second devices, the first device sends the first synchronization message in the queue to the multiple second devices again to notify the second devices that the third data to be synchronized has been received.
[0013] In the above process, after the first device reaches the synchronization point, obtains and sends the new third data to be synchronized, it sends the first synchronization message to multiple second devices again. The WQE to which the first synchronization message belongs is still the WQE that the first device issued to the queue before the service started. The first device does not need to generate a new WQE that includes the first synchronization message, which can reduce the time consumed in the device synchronization process.
[0014] For example, the first device includes a memory, wherein the memory includes a plurality of bits, each of which corresponds to a second device. The method further includes: if the number of received second synchronization messages differs from the number of second devices, the first device determines that at least one second device corresponding to a received second synchronization message has sent second data to be synchronized; the first device modifies the value of the bit corresponding to at least one second device in the memory based on the received second synchronization messages, thereby determining, based on the unmodified bits in the memory, one or more second devices that failed to properly send the second synchronization message to the first device, and generating a fault message.
[0015] Compared to the current method where computing devices determine whether they have received synchronization messages from all devices to be synchronized based on counters, the first device can determine whether it has not received synchronization messages from one or more second devices to be synchronized based on whether the values of the bits in the memory have been modified. This allows for precise location of faulty second devices or faulty communication links between the first and second devices. Generating fault messages enables timely recovery of faulty devices or links, avoiding adverse effects on the operation of the equipment.
[0016] For example, the method further includes: restoring multiple bits of the memory to their initial values when the number of second synchronization messages received by the first device is the same as the number of second devices, and the values of multiple bits included in the memory are modified.
[0017] In the above process, if the values of the bits indicating multiple second devices to be synchronized in the memory are all modified, the first device determines that it has received the synchronization data sent by all the second devices to be synchronized and has reached the synchronization point. It then restores all the bits in the memory to their initial values so that after the first device performs a new round of training or inference and receives the second synchronization message sent by multiple second devices again, it can accurately locate the faulty device or the faulty communication link.
[0018] For example, the specific process of the first device sending the first synchronization message in the queue to multiple second devices is as follows: The first device writes the producer index (PI) into the doorbell register and sends the first synchronization message in the queue to multiple second devices. The PI is used to indicate the number of WQEs in the queue sent by the first device. The PI is determined according to the number of multiple second devices.
[0019] In this application, the PI value written by the first device to the doorbell register each time is fixed. Compared with the current method where the first device calculates different PI values each time based on the number of multiple WQEs in the sent queue and the number of WQEs to be sent each time, the first device can reduce the consumption of computing resources, thereby avoiding the temporary generation of multiple new synchronization messages and reducing synchronization time.
[0020] For example, the first device further includes a counter for indicating the number of first synchronization messages sent by the first device to a plurality of second devices, and the method further includes: the first device setting the counter to zero when the number of first synchronization messages sent indicated by the counter is the number of a plurality of WQEs included in the queue.
[0021] In the above process, when the number accumulated by the counter reaches the number of multiple WQEs in the queue, the counter is set to zero, which can instruct the first device to resend the first WQE in the queue. This allows the first device to write a fixed PI to the doorbell register each time and send multiple WQEs in the queue without having to temporarily generate new WQEs, thus reducing the time consumed by multi-device synchronization.
[0022] Secondly, this application provides a computing device cluster, which includes a first device and a plurality of second devices. The first device is configured to obtain first data to be synchronized and send the first data to be synchronized to the plurality of second devices, and send a first synchronization message in a queue to the plurality of second devices. The queue includes a plurality of work queue elements (WQEs), each WQE including the address information of a second device and the first synchronization message, and each WQE is a WQE that supports multiple transmissions. Each of the plurality of second devices is configured to determine that the first data to be synchronized has been received upon receiving the first synchronization message.
[0023] For example, before running the service, the first device is also used to establish multiple WQEs in the queue according to the number of multiple second devices to be synchronized; generate a first synchronization message, and write the first synchronization message and the address information of the multiple second devices into the multiple WQEs.
[0024] For example, the first device is further configured to receive a second synchronization message sent by each of the multiple second devices after sending a first synchronization message in a queue to the multiple second devices, the second synchronization message being used to notify the first device that it has received the second data to be synchronized acquired and sent by each of the multiple second devices.
[0025] For example, the first device is further configured to obtain third data to be synchronized based on the first data to be synchronized and the second data to be synchronized, provided that the number of received second synchronization messages is the same as the number of second devices.
[0026] For example, after obtaining the third data to be synchronized, the first device is further configured to send the third data to be synchronized to multiple second devices; and send a first synchronization message in the queue to the multiple second devices, the first synchronization message being used to notify the second devices that the third data to be synchronized has been received.
[0027] For example, the first device includes a memory, wherein the memory includes a plurality of bits, each of the plurality of bits corresponding to a second device. The first device is further configured to: determine at least one second device corresponding to at least one received second synchronization message when the number of received second synchronization messages differs from the number of second devices; modify the value of the bit corresponding to at least one second device in the memory; and determine the second device that has not sent a second synchronization message based on the values of the plurality of bits in the memory, and generate a fault message.
[0028] For example, the first device is further configured to restore multiple bits of the memory to their initial values when the number of received second synchronization messages is the same as the number of second devices, and the values of multiple bits included in the memory are modified.
[0029] For example, the first device is specifically used to write the producer index PI into the doorbell register and send a first synchronization message in a queue to a plurality of second devices, wherein the PI is used to indicate the number of WQEs in the queue sent by the first device, and the PI is determined according to the number of the plurality of second devices.
[0030] For example, the first device further includes a counter for indicating the number of first synchronization messages sent by the first device to a plurality of second devices. The first device is also configured to set the counter to zero if the number of first synchronization messages sent, as indicated by the counter, is equal to the number of WQEs included in the queue.
[0031] Thirdly, this application provides a chip including a processor and a power supply circuit, the power supply circuit being used to supply power to the processor, causing the processor to perform the operation steps of the method described in the first aspect.
[0032] Fourthly, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium, and the instructions are executed by a computing device or a cluster of computing devices to implement the method described in the first aspect.
[0033] Fifthly, a computer program product comprising instructions is provided, the computer program product including instructions capable of running on a computing device or stored in any available medium, and when the computer program product is run on a computing device or a cluster of computing devices, causing the computing device or cluster of computing devices to perform the method described in the first aspect.
[0034] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description
[0035] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below.
[0036] Figure 1 This is a schematic diagram of the structure of a computing device cluster provided in an embodiment of this application;
[0037] Figure 2 This is a schematic diagram of the structure of a memory provided in an embodiment of this application;
[0038] Figure 3 This is a flowchart of a multi-device synchronous communication method provided in an embodiment of this application;
[0039] Figure 4 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application;
[0040] Figure 5 This is a schematic diagram of a computing device cluster provided in an embodiment of this application;
[0041] Figure 6 This is a schematic diagram of another computing device cluster provided in an embodiment of this application. Detailed Implementation
[0042] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.
[0043] First, the application scenarios involved in this application will be explained.
[0044] Currently, when multiple computing devices in a computing cluster are used in large model training or inference scenarios, multi-device synchronization is required multiple times. For example, in a large model training scenario, after each round of training, each computing device sends the new model gradient parameters it has obtained to other computing devices in the cluster. This ensures that each computing device, upon receiving all the new model gradient parameters, determines whether to synchronize and begin a new round of training, thus guaranteeing the effectiveness and correctness of the overall model training. In this process, each computing device needs to temporarily generate multiple new synchronization messages each time it sends new model gradient parameters. As the number of computing devices to be synchronized increases, the time for generating synchronization messages increases, and the synchronization time for multiple devices increases. The synchronization time for multiple devices is directly proportional to the number of computing devices.
[0045] Furthermore, each computing device in the above process includes a counter, which is used to indicate the number of synchronization messages received by the computing device from other computing devices to be synchronized. After the corresponding computing device receives the synchronization message sent by the node from other computing devices to be synchronized, it increments the current value of the counter by one. If the value of the counter reaches the number of other computing devices to be synchronized, it is determined that all computing devices to be synchronized have reached the synchronization point. However, if the value of the counter does not reach the number of other computing devices to be synchronized within a certain time threshold, it is determined that there is a faulty device among the computing devices to be synchronized that has not reached the synchronization point, or there is a faulty communication link, which prevents the synchronization message from being sent to the computing device. However, the counter alone cannot accurately locate the faulty device or the faulty communication link.
[0046] To address the aforementioned issues, this application provides a multi-device synchronization communication method. After sending acquired data to be synchronized to the target computing device each time, the computing device sends multiple pre-queueed synchronization messages to the target computing device. Based on the received synchronization messages from the target computing devices, the method locates faulty devices and faulty communication links. This method eliminates the need for the computing device to generate new synchronization messages each time, reducing synchronization time. It decouples the synchronization time of multiple computing devices from the number of computing devices in the computing device cluster and allows for precise location of faulty devices or faulty communication links.
[0047] The computing device cluster provided in this application will be explained below with reference to the accompanying drawings. The multi-device synchronous communication method provided in this application can be applied to the computing device cluster.
[0048] like Figure 1 As shown, Figure 1 This is a schematic diagram of a computing device cluster provided in an embodiment of this application. The computing device cluster includes multiple computing devices 100. Each computing device 100 includes at least one accelerator card 110, a processor 120, and a network card 130. The multiple computing devices 100 are connected to each other through a network.
[0049] Computing device 100 can be a bare metal server (BMS), a virtual machine, a container, or a storage device. A BMS refers to a general-purpose physical computing device, such as an ARM or x86 computing device; a virtual machine refers to a complete computer system simulated by software, possessing full hardware system functionality and running in a completely isolated environment. Any task that can be performed on a physical computer can also be performed in a virtual machine. When creating a virtual machine on an accelerator card, a portion of the physical machine's hard drive and memory capacity needs to be used as the virtual machine's hard drive and memory capacity. Each virtual machine has an independent basic input / output system (BIOS), hard drive, and operating system, and can be operated like a physical machine. A container is a portable software unit that can combine an application and all its dependencies into a single software package. This package is not limited by the underlying host operating system, thus eliminating the need to build complex environments and simplifying the application development and deployment process.
[0050] The accelerator card 110 can be a graphics processing unit (GPU), a neural network processing unit (NPU), a data processing unit (DPU), etc., and this application does not make any specific limitation. In some scenarios, the accelerator card is also referred to as an inference card.
[0051] Accelerator card 110 is used to handle large-scale parallel computing to facilitate tasks such as training or inference of artificial intelligence (AI) models. During the computation process, multiple accelerator cards need to synchronize multiple times, sending the generated synchronization data to each card. After confirming that it has received the synchronization data generated and sent by each of the multiple accelerator cards, it performs a new round of training or inference based on its own generated synchronization data and the received synchronization data to ensure the effectiveness and correctness of the overall model training or inference.
[0052] In the case of a network interface card (NIC) 111 that supports remote direct memory access (RDMA), the accelerator card 110 is used to create a queue, generate multiple work queue elements (WQEs), and distribute multiple WQEs to the queue before the accelerator card starts training or inference tasks. After sending the generated data to be synchronized to other accelerator cards to be synchronized, the NIC sends the multiple WQEs in the queue to the accelerator cards to be synchronized respectively, so as to notify the corresponding accelerator card to be synchronized that it has received the data to be synchronized sent by the accelerator card. Each WQE is a WQE that can be sent multiple times by the accelerator card.
[0053] In one possible implementation, each WQE includes a synchronization message and the address information of an accelerator card to be synchronized. The synchronization message can be a synchronization point identifier, which is determined by multiple accelerator cards in the accelerator card cluster during the initialization phase through a negotiation algorithm. The negotiation algorithm can be a timestamp negotiation algorithm, an election algorithm, an event-based negotiation algorithm, etc., and this application does not specifically limit its use. Specifically, the synchronization point identifier can be represented by a bitmap including multiple bits (e.g., 8 bits, 32 bits, 64 bits, etc.), with each bit corresponding to one accelerator card in the accelerator card cluster. For example, the synchronization point identifier of the first accelerator card includes 8 bits, with the first bit corresponding to the first accelerator card. The synchronization point identifier of the first accelerator card can be 10000000. In one possible implementation, the synchronization message can also be represented in other possible data formats, and this application does not specifically limit its use.
[0054] The accelerator card 110 includes a memory 112, which can be a synchronization register or other forms of memory, and this application does not specifically limit it.
[0055] like Figure 2 As shown, Figure 2 This is a schematic diagram of a memory structure provided in an embodiment of this application. The memory 112 includes multiple bits 20 (e.g., including 8 bits, 32 bits, or 64 bits, etc.). Each bit is used to indicate different accelerator cards in the accelerator card cluster. The value of each bit is used to indicate whether the accelerator card to which the memory belongs has received the data to be synchronized sent by the accelerator card corresponding to the bit. The initial value of each bit can be set to 0 or 1. This embodiment of the application describes the initial value of the bit as zero.
[0056] The accelerator card sends multiple pre-deployed WQEs (Wireless QEs) to multiple accelerator cards to be synchronized via the RDMA protocol. Similarly, it receives WQEs from other accelerator cards to be synchronized and modifies the value of the bit indicating the accelerator card to be synchronized in the synchronization register based on the synchronization message in the WQE. For example, suppose the initial value of the 8 bits in the memory of the first accelerator card is 0. The first accelerator card receives a WQE from a second accelerator card to be synchronized. This WQE includes the synchronization message 01000000, which notifies the first accelerator card that it has received data to be synchronized previously sent by the second accelerator card. Based on this synchronization message, the first accelerator card changes the value of the bit indicating the second accelerator card in its memory from 0 to 1. If the first accelerator card has not received any other WQEs from other accelerator cards to be synchronized, the value of the eight bits in its memory remains 01000000.
[0057] If the time taken to receive WQE data from an accelerator card to be synchronized reaches a certain time threshold, the accelerator card determines whether it has received all the data to be synchronized from the accelerator cards to be synchronized based on the values of the corresponding bits in the synchronization register. If the value of the bit corresponding to an accelerator card to be synchronized is still 0 in the synchronization register, the accelerator card determines that the accelerator card to be synchronized corresponding to the bit with a value of 0 is faulty, or determines that there is an abnormality in the communication link between the accelerator card to be synchronized corresponding to the bit with a value of 0 and the accelerator card itself, resulting in the accelerator card not receiving the data to be synchronized from the accelerator card to be synchronized.
[0058] Accelerator card 110 includes counter 113, which is used to determine the number of WQEs sent by the accelerator card to other accelerator cards to be synchronized by accumulation. The counter is also used to reset to zero if the number of synchronization messages sent determined by accumulation is the number of all synchronization messages stored in the communication queue.
[0059] Because the accelerator card writes the producer index (PI) to the doorbell register each time it sends data to be synchronized, driving the communication queue to send multiple stored WQEs, and the counter is reset to zero according to the aforementioned zeroing rule, the processor can write a fixed PI to the doorbell register each time. The PI indicates the number of multiple synchronization messages to be sent and is determined based on the number of accelerator cards to be synchronized. Compared to the current method where the accelerator card calculates different PIs based on the rounds of synchronization message sending in the communication queue and the number of multiple synchronization messages to be sent each time, thus driving the communication queue to send multiple synchronization messages newly generated by the first device, the accelerator card in this embodiment can reduce the consumption of computing resources, avoid temporarily generating new synchronization messages, and reduce synchronization time.
[0060] For example, suppose the counter is reset to zero only when it accumulates to a large value. The communication queue contains eight WQEs. After the accelerator card sends the data to be synchronized to multiple accelerator cards for the first time, it writes a PI of 8 to the doorbell register to drive the communication queue to send the eight WQEs. The counter determines the accumulated value to be 8 based on the eight WQEs sent. After the accelerator card sends the data to be synchronized to multiple accelerator cards for the second time, it calculates a new PI of 16 based on the round number of the synchronization message and the number of WQEs to be sent (8). The PI of 16 is then written to the doorbell register to drive the communication queue to send the eight newly generated synchronization messages. This process continues in the same manner. This will not be described in detail here.
[0061] Assume the counter is reset to zero when it reaches the total number of synchronization messages stored in the communication queue. The communication queue contains eight WQEs. After the accelerator card sends the data to be synchronized to multiple accelerator cards for the first time, it writes a PI value of 8 to the doorbell register to drive the communication queue to send the eight WQEs. The counter determines that the cumulative value has reached 8 based on the eight WQEs sent, and then resets the counter to zero. After the accelerator card sends the data to be synchronized to multiple accelerator cards for the second time, it again writes a PI value of 8 to the doorbell register to drive the communication queue to send the eight WQEs. There is no need to temporarily generate new WQEs. After the accelerator card sends the data to be synchronized to multiple accelerator cards for the second time, it again writes a PI value of 8 to the doorbell register. This will not be described in detail here.
[0062] The processor 120 can be any one or more of the following processors: central processing unit (CPU), graphics processing unit (GPU), microprocessor (MP), digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), complex programmable logic device (CPLD), etc. This application does not make any specific limitation.
[0063] Network interface card 130 can be an InfiniBand network interface card or a Remote Direct Memory Access over Converged Ethernet (RoCE) network interface card, etc. Network interface card 130 is connected to accelerator card 110 and is used to receive the data to be synchronized generated by accelerator card and to transmit the received WQE to accelerator card.
[0064] In one possible implementation, in the absence of a network interface card 111 that supports RDMA functionality, the processor 120 and network card 130 of the computing device replace the accelerator card in performing the steps of creating queues, generating WQE, and sending WQE to realize the multi-device synchronous communication method.
[0065] In summary, the computing device cluster provided in this application, after each time a computing device in the cluster sends the acquired data to be synchronized to another computing device, sends multiple synchronization messages pre-ordered to a queue to that computing device. Based on the received synchronization messages from the other computing devices, the faulty device and faulty communication link are located. This method eliminates the need for computing devices to generate new synchronization messages each time, reducing synchronization time. It decouples the synchronization time of multiple computing devices from the number of computing devices in the cluster and allows for precise location of faulty devices or faulty communication links.
[0066] The computing device cluster provided in this application has been explained above. The following section will combine... Figure 3 The multi-device synchronous communication method provided in this application is explained.
[0067] like Figure 3 As shown, Figure 3 This is a flowchart of a multi-device synchronous communication method provided in an embodiment of this application. The method is applied to... Figure 1 The multi-device synchronous communication method, which involves any server or accelerator card in the computing device cluster shown, includes the following steps.
[0068] S310: The first device sends multiple first synchronization messages to the queue.
[0069] Before operational execution, the first device and multiple second devices in the computing device cluster determine their respective synchronization messages through a negotiation algorithm and create a communication queue for sending the first synchronization message. This communication queue can be a jetty (JFS) queue. Then, based on the number of second devices to be synchronized in the computing device cluster, the first device sends multiple WQEs containing the first synchronization message to this communication queue. The number of WQEs is the same as the number of second devices to be synchronized, and each WQE supports multiple transmissions by the first device.
[0070] Specifically, if the first device has a Java runtime environment or Java development kit installed locally, it installs a Jetty server, which is a lightweight Java Web server. Then, a JFS queue is created in the Jetty server. The size of the JFS queue is configured according to the number of second devices to be synchronized. The JFS queue is used to store the issued WQEs. Each WQE supports multiple transmissions by the first device. Each WQE includes the first synchronization message of the first device and the address information of the second devices to be synchronized in the computing device cluster. The WQE may also include other types and quantities of data, which are not specifically limited in this application.
[0071] In the above process, the multiple WQEs sent by the first device to the communication queue can be sent multiple times after the first device sends the data to be synchronized multiple times. This eliminates the need for the first device to generate a new WQE each time, thus reducing the time consumed in the multi-device synchronization process.
[0072] S320: The first device obtains the first data to be synchronized and sends the first data to be synchronized to multiple second devices.
[0073] After the first device sends multiple WQEs to the communication queue, it performs model training or inference based on the current parameters and sample set, generates first data to be synchronized, and sends the first data to be synchronized to multiple second devices. In the case of the first device performing model training, the first data to be synchronized may be new model gradients, model parameters, etc. In other scenarios such as the first device being used for inference, the first data to be synchronized may be more types or quantities of data. This application does not make specific limitations on this.
[0074] In one possible implementation, the multiple second devices are determined in advance by each computing device in the computing device cluster through a negotiation algorithm. They can be one or more computing devices in the computing device cluster other than the first device. This application does not specifically limit the number of second devices.
[0075] S330: The first device sends multiple first synchronization messages to multiple second devices.
[0076] After the first device sends the first data to be synchronized to multiple second devices, it performs a synchronization message broadcasting task, that is, it sends multiple first synchronization messages to multiple second devices. The multiple first synchronization messages are respectively in multiple WQEs in the queue, and each WQE includes a first synchronization message and the address information of the second device.
[0077] Specifically, the first device rings the doorbell of the communication queue, that is, writes a fixed PI value into the doorbell register to drive the communication queue to send multiple WQEs to multiple second devices to be synchronized via the UB protocol or the RDMA protocol. The PI is used to indicate the number of WQEs in the queue sent by the first device, and is determined according to the number of multiple second devices.
[0078] When the communication queue is a JFS queue, since the JFS queue can support high-concurrency operations and is event-driven, after the first device sends the data to be synchronized to multiple second devices to be synchronized, it concurrently sends multiple first synchronization messages to the multiple second devices to be synchronized.
[0079] When a first device sends multiple first synchronization messages included in a communication queue to multiple second devices, a counter in the first device determines the number of multiple first synchronization messages sent by accumulating. When the number of first synchronization messages sent reaches the number of multiple second devices to be synchronized, the counter is reset to zero.
[0080] In the above process, after each transmission of data to be synchronized, the first device writes the same PI value into the doorbell register to drive the queue to send multiple WQEs concurrently. This avoids the first device generating multiple WQEs temporarily each time, improving synchronization efficiency and reducing the synchronization time of multiple devices. Furthermore, the counter in the first device is reset to zero according to the above rules, ensuring that the first device writes the same PI value into the doorbell register each time. This eliminates the need to calculate different PI values each time based on the number of WQEs in the queue and the number of times WQEs have been transmitted, saving computing resources.
[0081] S340: The first device receives the second synchronization messages sent by each of the multiple second devices.
[0082] In addition to the first device, each of the multiple second devices also sends its corresponding second synchronization message to the first device after sending its own trained second synchronization data, so as to notify the first device to confirm that it has received the second synchronization data sent by each second device to be synchronized.
[0083] Specifically, each of the multiple second devices sends multiple second WQEs to a pre-created queue based on the total number of other second devices in the computing device cluster (excluding itself) and the first device. Each second WQE includes a second synchronization message and the address information of the first device, or the address information of the second device. After obtaining the second data to be synchronized, each second device sends the second data to the first device and the other second devices, and then sends the second WQE. Correspondingly, after sending multiple first synchronization messages to the multiple second devices, the first device performs a synchronization message receiving task, that is, receives the second synchronization messages sent by each of the multiple second devices, and thus determines whether it has received the multiple second synchronization messages sent by all the multiple second devices to be synchronized.
[0084] Furthermore, the first device modifies the values of corresponding bits in its synchronization register based on the second synchronization messages received from each of the multiple second devices to be synchronized. For example, assuming the initial value of each bit in the synchronization register of the first device is 0, when the first device receives a second synchronization message from a second device to be synchronized, it modifies the value of the bit in the synchronization register indicating that second device to be synchronized to 1. Subsequently, after receiving more second synchronization messages, the first device modifies the value of the bits in the synchronization register indicating the aforementioned more second devices to 1.
[0085] In the above process, the first device determines the second synchronization messages sent by multiple second devices by modifying the bit values in the synchronization register. Thus, based on the unmodified bit values in the synchronization register, the second device corresponding to the unreceived second synchronization message can be determined. Compared to simply recording the number of received second synchronization messages, the above method can accurately locate the faulty computing device among multiple second devices, or determine the faulty communication link in the communication link between multiple second devices and the first device, based on the bit values in the synchronization register.
[0086] S350: The first device determines whether it has received all the second synchronization messages sent by multiple second devices.
[0087] If the timeout period for receiving the second synchronization message reaches a preset timeout period, the first device determines whether it has received all the second synchronization messages sent by the multiple second devices to be synchronized based on the value of the bit indicating the multiple second devices in the synchronization register.
[0088] If the first device determines that the values of the bits indicating multiple second devices in the synchronization register have been modified, it determines that it has received all the second synchronization messages sent by each of the multiple second devices to be synchronized, and determines that the multiple devices are synchronized. The first device then executes step S360. In addition to executing step S360, if the first device determines that the values of the bits indicating multiple second devices to be synchronized in the synchronization register have been modified, it resets all the bits in the synchronization register to their initial values. This is to ensure that when the first device receives second synchronization messages from each of the second devices again, it can accurately determine the second synchronization message sent by at least one second device in the new round of reception, thus preventing the values of the bits in the synchronization register from being repeatedly modified, which could affect the identification of faulty devices and faulty communication links.
[0089] If the first device determines that the values of the bits indicating multiple second devices in the synchronization register have not been modified, it executes step S370.
[0090] S360: The first device obtains the third data to be synchronized based on the first data to be synchronized and multiple second data to be synchronized.
[0091] After the first device completes the synchronization message receiving task and determines that multiple devices are synchronized, it performs subsequent tasks based on the first data to be synchronized and multiple second data to be synchronized, such as direct memory access (DMA), to conduct a new round of training or inference and obtain the third data to be synchronized.
[0092] After generating new data to be synchronized, the first device returns to step S320 to write a fixed PI value into the doorbell register after sending the new data to be synchronized to multiple second devices, so as to send multiple WQE messages including the first synchronization message in the queue to multiple second devices again.
[0093] Since the counter is reset to zero when the cumulative number of first synchronization messages sent by the first device is equal to the total number of first synchronization messages in the communication queue, the first device writes the same PI value into the doorbell register. This can drive the communication queue to send multiple WQEs including the first synchronization messages again. Compared to the first device calculating new doorbell parameters based on the round in which the multiple first synchronization messages in the communication queue are sent and the number of the multiple first synchronization messages in the communication queue, and then driving the communication queue to send new first synchronization messages, the method provided in this application embodiment can reduce the computational overhead of the first device during multi-device synchronization and ensure that each first synchronization message sent is a first synchronization message pre-delivered to the communication queue. The first device does not need to temporarily generate new synchronization messages when it reaches the synchronization point again, which can reduce the time consumption of multi-device synchronization.
[0094] S370: The first device identifies the faulty device or the faulty communication link and generates a fault message.
[0095] If the time to receive the second synchronization message reaches a preset timeout period, the first device determines that the bit in the synchronization register that still holds the initial value indicates the second device to be synchronized, thereby determining that the second synchronization message sent by the second device has not been received. Then, it determines that there is a faulty second device, or determines that there is a faulty communication link between the second device and the first device, and generates fault information for timely inspection, restart, and other operations on the faulty device and the faulty communication link. The fault information may include the identifier of the second device, etc., which is not specifically limited in this application.
[0096] In the above process, the first device determines whether it has received all the second synchronization messages sent by the second devices to be synchronized by checking whether the value of each bit indicating the second device in the synchronization register, which includes multiple bits, has been modified, thereby determining whether multi-device synchronization is complete. Compared to the current method where the first device can only determine the number of received synchronization messages through a counter, in this embodiment, the first device can determine the second devices corresponding to the multiple received second synchronization messages by checking the value of the bits indicating the multiple second devices to be synchronized in the synchronization register. In addition, the first device can determine whether the multiple second devices corresponding to the aforementioned bits are faulty, or whether the communication link between the multiple second devices corresponding to the aforementioned bits and the first device is faulty, based on the unmodified value of the bits indicating the second devices to be synchronized, thereby enabling timely fault recovery operations.
[0097] In summary, in the multi-device synchronization communication method provided in this application embodiment, the computing device pre-sends synchronization messages to a queue. Upon reaching a synchronization point, it sends multiple pre-sent synchronization messages to the communication queue. This eliminates the need to temporarily generate a WQE containing synchronization messages each time a synchronization point is reached. This decouples the multi-device synchronization time from the number of computing devices requiring synchronization, reducing the synchronization time. Furthermore, the computing device can accurately determine a faulty computing device or a faulty communication link based on the values of unmodified bits in the synchronization register.
[0098] In one specific implementation, in a fully interconnected topology network comprising eight nodes, the nodes are interconnected. Each of the eight compute nodes has a synchronization register containing eight bits, each bit indicating a node. Each of the eight compute nodes creates a Jetty server, creates a JFS queue within the Jetty server, and issues seven WQEs to the JFS queue. For example, the first node of the eight nodes issues seven WQEs to the JFS queue. Each WQE includes a synchronization message pre-negotiated by all nodes, and the address information of one of the nodes from the second to the eighth node.
[0099] The first node sequentially issues a synchronous message broadcast task, a synchronous message receiving task, and a DMA task in the business flow. Since the business flow is a queue executed in sequence, it can be determined that the first node executes the synchronous message receiving task after the synchronous message broadcast task is completed, and then executes the DMA task after the synchronous message receiving task is completed.
[0100] When the first node sends the training data to be synchronized to the second to eighth nodes, it performs a synchronization message broadcasting task, that is, writes the PI value of 7 into the doorbell register, and drives the JFS queue to send seven first synchronization messages to the second to eighth nodes. Similarly, any node from the second to the eighth node also performs a synchronization message broadcasting task when it sends the training data to be synchronized to the other seven nodes besides itself.
[0101] After sending the seven first synchronization messages in the JFS queue, the first node executes a synchronization message receiving task to determine whether it has received synchronization messages sent by the second to eighth nodes, and whether it has received the data to be synchronized sent by the second to eighth nodes respectively. When the first node receives a synchronization message from a node, it modifies the bit in the synchronization register indicating that node from its initial value to its expected value according to the synchronization point identifier included in the synchronization message, until the first node determines that the bits in the synchronization register indicating the second to eighth nodes are all set to their expected values, and determines that it has received the data to be synchronized sent by each of the second to eighth nodes. After that, the first node resets the synchronization register to its initial value.
[0102] Once the first node confirms that it has received all synchronization messages sent by the second to eighth nodes, it determines that the synchronization message receiving task is complete. Similarly, once the synchronization message receiving tasks executed by the second to eighth nodes are completed, and it is confirmed that any of the first to eighth nodes includes the synchronization data generated by each of the first to eighth nodes, the multi-device synchronization between the first and eighth nodes is complete. After multi-device synchronization, each node generates new synchronization data based on its own generated synchronization data and the synchronization data received from the other seven nodes, and repeats the steps of sending synchronization data and synchronization messages.
[0103] In summary, the computing device cluster and multi-device synchronization communication method provided in this application, before training, involve the computing devices pre-building a communication queue and sending synchronization messages to the queue. Each time a computing device reaches a synchronization point, it sends the synchronization messages in the queue to the computing devices to be synchronized. Based on the received synchronization messages from the computing devices to be synchronized, the faulty device and faulty communication link are located. This method eliminates the need for the computing devices to generate synchronization messages temporarily each time, reducing synchronization time. It decouples the synchronization time of multiple computing devices from the number of computing devices in the cluster and accurately locates the faulty device or faulty communication link based on the values of each bit in the synchronization register of the computing device.
[0104] The multi-device synchronous communication method provided in this application has been described in detail above. The following section will combine... Figures 4-6 The structure of the computing device provided in this application will be explained.
[0105] like Figure 4 As shown, Figure 4 This is a schematic diagram of a computing device 100 provided in an embodiment of this application. The computing device 100 includes a processor 120, a bus 140, a memory 150, and a communication interface 160. The processor 120, memory 150, and communication interface 160 communicate via the bus 140. The computing device 100 can be a server. It should be understood that this application does not limit the number of processors, memories, and accelerator cards in the computing device 100. The memory 150 is used to store instructions, and the processor 120 is used to execute the instructions stored in the memory 150. The memory 150 stores program code, and the processor 120 can call the program code stored in the memory 150 to perform the following operations:
[0106] Obtain the first piece of data to be synchronized and send it to multiple computing devices;
[0107] Send a first synchronization message from a queue to multiple computing devices. The queue includes multiple work queue elements (WQEs). Each WQE includes the address information of a second device and the first synchronization message. Each WQE is a WQE that supports multiple transmissions. The first synchronization message is used to notify the second devices that they have received the first data to be synchronized.
[0108] It should be understood that the memory 150 stores executable program code, and the processor 120 executes the executable program code to achieve the above. Figure 3 The operation steps of the multi-device synchronization method are shown. The program code may include one or more software units, such as the generation unit 151, acquisition unit 152, processing unit 153, and sending unit 154 shown in the above-mentioned one or more software unit diagrams. The generation unit is used to implement... Figure 3 S310, S360 and their optional steps, the acquisition unit is used to implement Figure 3 S320, S340, and their optional steps are used by the processing unit to implement Figure 3 S350, S370 and their optional steps, the transmitting unit is used to implement Figure 3 The S320, S330 and their optional steps are not described in detail here.
[0109] Bus 140 can be a Peripheral Component Interconnect Express (PCIe) bus, an Extended Industry Standard Architecture (EISA) bus, a Unified Bus (Ubus or UB), a Compute Express Link (CXL), a Cache Coherent Interconnect for Accelerators (CCIX), etc. The Unified Bus is also known as the Lingqu Bus. Buses can be divided into address buses, data buses, control buses, etc. For ease of representation, Figure 4 The bus is represented by only one line, but this does not mean that there is only one bus or one type of bus. Bus 140 may include a path for transmitting information between various components of computing device 100 (e.g., processor 120, memory 150, communication interface 160). The unified bus may also be referred to as the Lingqu bus.
[0110] The processor 120 may include any one or more of the following computing devices: central processing unit (CPU), graphics processing unit (GPU), microprocessor (MP) or digital signal processor (DSP), ASIC, FPGA, CPLD, NPU, SoC, offload card, accelerator card, etc.
[0111] Memory 150 may include volatile memory, such as random access memory (RAM). Memory 150 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD). Furthermore, memory 150 may also be implemented using storage class memory (SCM), phase change memory (PCM), or other types of storage media.
[0112] It is worth noting that the same type of storage medium can be configured in the same computing device to realize the function of memory 150, or two or more types of storage media can be configured to realize the function of memory 150. This application does not limit this.
[0113] The communication interface 160 includes, but is not limited to, a transceiver module such as a network card, to enable communication between the computing device 100 and other devices or communication networks.
[0114] As one possible implementation method, Figure 3 The method described herein can be implemented either by the processor in computing device 100 executing program code in memory, or by the accelerator card 110 in computing device 100. The accelerator card 110 can be implemented by computing devices such as DPU, NPU, GPU, XPU, SoC, or AI chips; this application does not specifically limit its implementation. The accelerator card 110 includes a processing unit and a power supply circuit. The power supply circuit supplies power to the processing unit, enabling the processing unit to execute the above-described... Figure 3 The operational steps of the multi-device synchronization method are described. For example, the processing unit can implement the functions of the generation unit, acquisition unit, processing unit, and sending unit mentioned above. The functional descriptions of the above-mentioned units can be found in the relevant content above, and will not be repeated here.
[0115] As one possible implementation method, Figure 3 The method described above can be implemented either by the processor in computing device 100 executing program code in memory, or by using accelerator card 110, or by a combination of processor 120 and accelerator card 110 in computing device 100. Figure 3 The operation steps performed by the corresponding entity in the multi-device synchronous communication method are described above. Specifically, the program code stored in the memory 150 includes some unit modules, which enable the processor 120 to execute the program code to implement some operation steps of the multi-device synchronous communication method. The power supply circuit in the accelerator card 110 supplies power to the processing unit, enabling the processing unit to execute another part of the operation.
[0116] like Figure 5 As shown, Figure 5 This is a schematic diagram of a computing device cluster provided in this application, which includes at least one computing device. The memory 150 of one or more computing devices 100 in the computing device cluster may store the same implementation. Figure 3 The instructions for the operation steps of the multi-device synchronous communication method. Alternatively, different implementations may be stored in the memory 150 of one or more computing devices 100. Figure 3The instructions for the operation steps of the multi-device synchronous communication method, in other words, a combination of one or more computing devices 100 can jointly execute the instructions for the operation steps required by the multi-device synchronous communication method.
[0117] As one possible implementation, the same implementation can be stored in the accelerator cards 110 of one or more computing devices 100 in the computing device cluster. Figure 3 The instructions for the operation steps of the multi-device synchronous communication method. Alternatively, different implementations may be stored in the accelerator cards 110 of one or more computing devices 100. Figure 3 The instructions for the operation steps of the multi-device synchronous communication method, in other words, a combination of one or more computing devices 100 can jointly execute the instructions for the operation steps required to implement the multi-device synchronous communication method.
[0118] As one possible implementation, the same implementation is stored in the memory 150 of one or more computing devices 100 in the computing device cluster. Figure 3 In the multi-device synchronous communication method described above, instructions for a portion of the operation steps are stored in the accelerator card 110, which also contains instructions for implementing another portion of the operation steps. Alternatively, different implementations may be stored in the memory 150 of one or more computing devices 100. Figure 3 The multi-device synchronous communication method contains instructions for a portion of the operation steps, and the accelerator card 110 stores different instructions for implementing another portion of the operation steps. In other words, a combination of one or more computing devices 100 can jointly execute the instructions for the operation steps required to implement the multi-device synchronous communication method.
[0119] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN), a local area network (LAN), or similar. Figure 6 One possible implementation is shown. For example... Figure 6 As shown, Figure 6 This is a schematic diagram of another computing device cluster provided in this application. Two computing devices, 100A and 100B, are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device.
[0120] It should be understood that Figure 6 The functions of computing device 100A shown can also be performed by multiple computing devices. Similarly, the functions of computing device 100B can also be performed by multiple computing devices.
[0121] This application embodiment also provides a chip, which includes a processor and a power supply circuit. The power supply circuit is used to supply power to the processor, and the processor is used to perform... Figure 3The operational steps corresponding to the multi-device synchronous communication method shown are omitted here for brevity. The processor can be implemented using a GPU, or a computing device such as a DPU, NPU, XPU, SoC, offloading card, or accelerator card.
[0122] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this application.
Claims
1. A method for multi-device synchronous communication, characterized in that, The method is applied to a cluster of computing devices, and the method includes: The first device in the computing device cluster obtains the first data to be synchronized and sends the first data to be synchronized to multiple second devices; The first device sends a first synchronization message from a queue to the plurality of second devices. The queue includes a plurality of work queue elements (WQEs). Each WQE includes the address information of a second device and the first synchronization message. Each WQE is a WQE that supports multiple transmissions. The first synchronization message is used to notify the second devices that they have received the first data to be synchronized.
2. The method according to claim 1, characterized in that, Before the first device in the computing device cluster obtains the first data to be synchronized, the method further includes: The first device establishes multiple WQEs in the queue based on the number of multiple second devices to be synchronized; The first device generates the first synchronization message and writes the first synchronization message and the address information of the plurality of second devices into the plurality of WQEs.
3. The method according to claim 1 or 2, characterized in that, After the first device sends the first synchronization message in the queue to the plurality of second devices, the method further includes: The first device receives a second synchronization message sent by each of the plurality of second devices. The second synchronization message is used to notify the first device that it has received the second data to be synchronized acquired and sent by each of the second devices.
4. The method according to claim 3, characterized in that, The method further includes: If the number of second synchronization messages received by the first device is the same as the number received by the second device, the third synchronization data is obtained based on the first synchronization data and the second synchronization data.
5. The method according to claim 4, characterized in that, After obtaining the third data to be synchronized, the method further includes: The first device sends the third data to be synchronized to the plurality of second devices; The first device sends the first synchronization message in the queue to the plurality of second devices. The first synchronization message is used to notify the second devices that the third data to be synchronized has been received.
6. The method according to any one of claims 3-5, characterized in that, The first device includes a memory, the memory including a plurality of bits, each of the plurality of bits corresponding to a second device; The method further includes: If the number of second synchronization messages received by the first device is different from the number of second devices, the first device determines at least one second device corresponding to at least one second synchronization message received. The first device modifies the value of the bit corresponding to at least one second device in the memory; The first device determines the second device that has not sent the second synchronization message based on the values of the plurality of bits in the memory, and generates a fault message.
7. The method according to claim 6, characterized in that, The method further includes: If the number of second synchronization messages received by the first device is the same as the number of second devices, and the values of the plurality of bits included in the memory are modified, the plurality of bits in the memory shall be restored to their initial values.
8. The method according to any one of claims 1-7, characterized in that, The first device sends a first synchronization message from the queue to the plurality of second devices, including: The first device writes the producer index (PI) into the doorbell register and sends the first synchronization message in the queue to the plurality of second devices. The PI is used to indicate the number of WQEs in the queue sent by the first device, and the PI is determined based on the number of the plurality of second devices.
9. The method according to claim 8, characterized in that, The first device further includes a counter for indicating the number of the first synchronization messages sent by the first device to the plurality of second devices, and the method further includes: The first device sets the counter to zero if the number of the first synchronization messages sent, as indicated by the counter, is equal to the number of multiple WQEs included in the queue.
10. A computing device cluster, characterized in that, Includes a first device and multiple second devices; The first device is configured to obtain a first piece of data to be synchronized and send the first piece of data to be synchronized to the plurality of second devices; The first device is configured to send a first synchronization message in a queue to the plurality of second devices, wherein the queue includes a plurality of work queue elements (WQEs), each WQE includes the address information of a second device and the first synchronization message, and each WQE is a WQE that supports multiple transmissions. Each of the plurality of second devices is configured to determine, upon receiving the first synchronization message, that the first data to be synchronized has been received.
11. A chip, characterized in that, The chip includes a processor and a power supply circuit, the power supply circuit being used to supply power to the processor so that the processor performs the method according to any one of claims 1-9.
12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions, which, when executed by a computing device or a cluster of computing devices, perform the method as described in any one of claims 1-9.
13. A computer program product containing instructions, characterized in that, When the instruction is executed by a computing device or a cluster of computing devices, the cluster of computing devices causes the computing device cluster to perform the method as described in any one of claims 1-9.