Method, apparatus, device and product for communication

Abstract

The invention relates to a method, a device, equipment and a program product for communication. The method includes receiving first data from a first node, the first data being sent in response to a trigger of a work queue element chain. The method includes sending an indication message to the second node in response to completion of receiving the first data, the indication message being used for indicating the second node to send data. Further, the method includes receiving second data from the second node, the second data being transmitted in response to the reception of the indication message.

Description

Technical Field

[0001] This disclosure relates to the field of computers, and more specifically to methods, apparatus, devices and products for communication. Background Technology

[0002] As the scale of large model parameters expands from tens of billions to trillions and the amount of training data grows exponentially, a single node or single GPU can no longer meet the needs of efficient training. Distributed parallel training has become the key to realizing the deployment of large models. As the underlying technology in distributed training, communication undertakes the task of coordinating data exchange between multiple nodes and multiple GPUs, and its performance directly affects the training efficiency of large models. Summary of the Invention

[0003] In a first aspect of the embodiments of this disclosure, a method for communication is provided. The method includes receiving first data from a first node, the first data being sent in response to the triggering of a work queue element chain. The method further includes sending an indication message to a second node in response to completion of receiving the first data, the indication message instructing the second node to send data. Furthermore, the method includes receiving second data from the second node, the second data being sent in response to receiving the indication message.

[0004] In a second aspect of embodiments of this disclosure, a method for communication is provided. The method includes receiving an indication message from a first node, the indication message indicating that data is to be sent to the first node, and the indication message being generated in response to the first node completing the reception of first data from a second node. The method also includes sending second data to the first node in response to triggering a work queue element chain.

[0005] In a third aspect of the embodiments of this disclosure, an apparatus for communication is provided, the apparatus including a first data receiving module configured to receive first data from a first node, the first data being transmitted in response to the triggering of a work queue element chain. The apparatus also includes an indication message sending module configured to send an indication message to a second node in response to completion of receiving the first data, the indication message instructing the second node to transmit data. Furthermore, the apparatus includes a second data receiving module configured to receive second data from the second node, the second data being transmitted in response to receiving the indication message.

[0006] In a fourth aspect of embodiments of this disclosure, an apparatus for communication is provided. The apparatus includes an indication message receiving module configured to receive an indication message from a first node, the indication message indicating the transmission of data to the first node, and the indication message being generated in response to the first node completing the reception of first data from a second node. The apparatus also includes a second data transmitting module configured to transmit second data to the first node, the second data being transmitted in response to the triggering of a work queue element chain.

[0007] In a fifth aspect of the embodiments of this disclosure, an electronic device is provided. The electronic device includes one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement the method of the first aspect.

[0008] In a sixth aspect of the embodiments of this disclosure, a computer program product is provided. The computer program product is tangibly stored on a non-transitory computer-readable medium and includes machine-executable instructions that, when executed, cause a machine to implement the method of the first aspect.

[0009] The summary section is provided to present the chosen concepts in a simplified form, which will be further described in the detailed description below. The summary section is not intended to identify key or principal features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. Attached Figure Description

[0010] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0011] Figure 1 A schematic diagram of an example environment in which several embodiments of the present disclosure may be implemented is shown;

[0012] Figure 2 A flowchart of a method for communication according to some embodiments of the present disclosure is shown;

[0013] Figure 3 A flowchart is shown for another method of communication according to some embodiments of the present disclosure;

[0014] Figure 4A A schematic diagram illustrating an example of all-to-all communication according to some embodiments of this disclosure is shown;

[0015] Figure 4B A schematic diagram of a multi-to-multi communication scenario based on some embodiments of this disclosure is shown;

[0016] Figure 5 A schematic diagram of a step-by-step interleaved transmission scenario for all-to-all communication according to some embodiments of the present disclosure is shown;

[0017] Figure 6A A schematic diagram of a method for communication based on instruction messages and chained primitives, according to some embodiments of the present disclosure, is shown.

[0018] Figure 6B A schematic diagram illustrating examples of communication based on chained primitives, representing some embodiments of this disclosure, is shown.

[0019] Figure 7 A schematic diagram of a packet-based method for communication according to some embodiments of the present disclosure is shown;

[0020] Figure 8 A schematic diagram is shown of a method for packet communication based on instruction messages and chained primitives that can implement some embodiments of the present disclosure;

[0021] Figure 9 A block diagram of an apparatus for communication according to some embodiments of the present disclosure is shown;

[0022] Figure 10 A block diagram of another apparatus for communication according to some embodiments of the present disclosure is shown; and

[0023] Figure 11 A block diagram of a device capable of implementing several embodiments of the present disclosure is shown. Detailed Implementation

[0024] It is understood that all user-related data involved in this technical solution should be obtained and used only after authorization from the user. This means that if it is necessary to use a user's personal information in this technical solution, the user's explicit consent and authorization are required before obtaining this data; otherwise, no related data collection and use will be carried out. It should also be understood that when implementing this technical solution, relevant laws and regulations should be strictly followed in the process of data collection, use, and storage, and necessary technical measures should be taken to protect user data security and ensure the secure use of data.

[0025] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0026] For example, upon receiving a user's proactive request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0027] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0028] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0029] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0030] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc., may refer to different or the same objects unless explicitly stated. Other explicit and implicit definitions may also be included below.

[0031] As mentioned above, the communication clusters of large models have now reached the kilocalorie level or even larger. Communication volume increases quadratically with the number of nodes, and each network interface card (NIC) needs to receive data many times its own bandwidth, leading to severe network congestion, increased latency, and decreased throughput. Furthermore, the strong dependency between computation and communication causes GPU waiting, resulting in decreased GPU computing power utilization. In addition, traditional congestion control methods are difficult to effectively perform in complex congested scenarios, especially in scenarios with small messages. Therefore, improving communication performance has become a key issue for the training and inference performance of large models.

[0032] Therefore, embodiments of this disclosure provide a method for communication. In embodiments of this disclosure, first data is received from a first node, which is sent in response to the triggering of a work queue element chain. Subsequently, in response to the completion of receiving the first data, an indication message is sent to a second node, which instructs the second node to send data. Subsequently, second data is received from the second node, which is sent in response to the receipt of the indication message. By sending the indication message, embodiments of this disclosure ensure that data is sent only after the indication message is received, thereby avoiding communication congestion and improving communication performance.

[0033] Figure 1 A schematic diagram of an example environment 100 in which various embodiments of the present disclosure may be implemented is shown. Reference Figure 1 As shown, example environment 100 may include a sender and a receiver, each of which may include one or more nodes. A node may be a GPU, a CPU, or any other computing device capable of participating in communication. The sender in environment 100 includes sending node 102-1, sending node 102-2, and receiving node 104. Sending node 102-1 (also called the first node) includes data 106 (also called first data), and sending node 102-2 (also called the second node) includes data 108 (also called second data). In some embodiments, data 106 and data 108 may be tagged fragments, or other data involved in communication and computation. Receiving node 104 includes indication information 110, which is used to indicate the transmission of data. This indication information may be in the form of instructions, such as a transmission timing instruction.

[0034] like Figure 1As shown, receiving node 104 can receive data 106 from sending node 102-1. In some embodiments, data 106 is sent when the work queue element chain is triggered. In some embodiments, the work queue element chain consists of several work queue elements (WQEs) linked sequentially, scheduled from front to back according to the linking order of the work queue elements. After receiving node 104 finishes receiving data 106, receiving node 104 can send indication information 110 to sending node 102-2, and receiving node 104 can then receive data 108 from sending node 102-2. In this way, receiving node 104 can only receive data 108 after sending indication information 110, thereby avoiding communication congestion and improving communication performance. It should be understood that the architecture and functionality in example environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure. Embodiments of this disclosure can also be applied to other environments with different structures and / or functions. For example, in some embodiments, environment 100 may also include three or more sending nodes or receiving nodes.

[0035] The following will combine Figures 2 to 11 The method according to embodiments of this disclosure is described in detail. For ease of understanding, the specific data mentioned in the following description are exemplary and not intended to limit the scope of this disclosure. It is understood that the embodiments described below may also include additional actions not shown and / or actions shown may be omitted, and the scope of this disclosure is not limited in this respect.

[0036] Figure 2 A flowchart of a method 200 for communication according to some embodiments of the present disclosure is shown. Method 200 can be performed by a means for communication, which may be, for example, a standalone device or system. This means can be implemented in software and / or hardware. The method 200 will now be illustrated schematically using this means as an example. Method 200 includes blocks 202, 204, and 206.

[0037] refer to Figure 2 As shown in block 202, method 200 is able to receive first data from a first node, which is sent in response to a triggering of a work queue element chain. In some embodiments, the work queue element chain consists of a plurality of work queue elements (WQEs) linked sequentially and scheduled from front to back according to the linking order of the work queue elements. In some embodiments, the first data may be a tagged fragment, or it may be any other data involved in communication and computation. For example, in Figure 1 In the environment 100 shown, receiving node 104 can receive data 106 from sending node 102-1.

[0038] In block 204, method 200 can send an indication message to the second node in response to completing the reception of the first data. This indication message instructs the second node to send data. In some embodiments, the indication information is used to indicate data transmission, and it can be data in the form of instructions, such as a transmission timing instruction. For example, in... Figure 1 In the environment 100 shown, after receiving node 104 has finished receiving data 106, receiving node 104 can send indication information 110 to sending node 102-2.

[0039] In box 206, method 200 is able to receive second data from the second node, which is sent in response to the receipt of an indication message. For example, in Figure 1 In the environment 100 shown, after receiving node 104 finishes receiving data 106, receiving node 104 can send indication information 110 to sending node 102-2. Receiving node 104 can then receive data 108 from sending node 102-2. In this way, method 200 can only receive data after sending the indication information, thereby avoiding communication congestion and improving communication performance.

[0040] Figure 3 A flowchart of another method 300 for communication according to some embodiments of the present disclosure is shown. Method 300 can be performed by a means for communication, which may be, for example, a standalone device or system. The means can be implemented by software and / or hardware. The method 300 will now be illustrated schematically using this means as an example. Method 300 includes blocks 302 and 304.

[0041] refer to Figure 3 As shown, in block 302, method 300 is able to receive an indication message from a first node, the indication message being used to indicate sending data to the first node, and the indication message being generated in response to the first node completing the reception of first data from a second node. In some embodiments, the indication information is used to indicate the sending of data, and it can be data in the form of instructions, such as a sending timing instruction. For example, in... Figure 1 In the environment 100 shown, the sending node 102-2 can receive indication information 110 from the receiving node 104.

[0042] In block 304, method 300 can send second data to the first node, which is sent in response to a triggering of a work queue element chain. In some embodiments, the work queue element chain consists of a plurality of work queue elements (WQEs) linked sequentially and scheduled from front to back according to the linking order of the work queue elements. In some embodiments, the second data may be a tagged fragment, or it may be any other data involved in communication and computation. For example, in Figure 1In the environment 100 shown, the receiving node 104 can receive data 108 from the sending node 102-2. In this way, method 300 can only send data after receiving the instruction information, thereby avoiding communication congestion and improving communication performance.

[0043] In some embodiments, the communication in methods 200 and 300 can be All-to-All communication (All2All). All2All is a key communication mechanism that supports parallel processing across nodes in a distributed system and is also an important set communication primitive in large models. The following will use two simple examples to describe some embodiments of All-to-All communication in this disclosure and many-to-many scenarios of All-to-All communication. Figure 4A A schematic diagram of an example 400 of all-to-all communication according to some embodiments of this disclosure is shown. Figure 4B A schematic diagram of a multi-to-multi communication scenario 410 with all-to-all communication according to some embodiments of the present disclosure is shown.

[0044] refer to Figure 4A As shown, P0, P1, P2, and P3 are four nodes in All-to-All communication. Initially, node P0 includes data A0, B0, C0, and D0; node P1 includes data A1, B1, C1, and D1; node P2 includes data A2, B2, C2, and D2; and node P3 includes data A3, B3, C3, and D3. In All-to-All communication, each node sends a specific series of data to all other nodes and receives corresponding data, such as... Figure 4A As shown, for data series A, nodes P1, P2, and P3 send data A1, A2, and A3 to node P0; for data series B, nodes P0, P2, and P3 send data B0, B2, and B3 to node P1; for data series C, nodes P0, P1, and P3 send data C0, C1, and C3 to node P2; and for data series D, nodes P0, P1, and P2 send data D0, D1, and D2 to node P3. Ultimately, node P0 holds data A0, A1, A2, and A3; node P1 holds data B0, B1, B2, and B3; node P2 holds data C0, C1, C2, and C3; and node P3 holds data D0, D1, D2, and D3. Figure 4A The example of all-to-all communication shown enables full-dimensional exchange of data from all nodes in the same series, where each node acts as both a sender distributing data and a receiver aggregating data in the same series.

[0045] In some embodiments, All2All can be used in large model expert parallel scenarios (MoE). The basic process of the MoE layer is to first perform dynamic routing, then expert computation, and finally result aggregation. This requires two All2All operations: All2All Dispatch (distribution phase): Based on the routing decision of the gating network, the input token is distributed to the corresponding top K experts through All2All operation; All2All Combine (aggregation phase): After each expert completes local computation, the computation result is sent back to the source GPU for weighted computation through reverse All2All according to the original token sending path.

[0046] refer to Figure 4B As shown, the All2All operation is a multi-sender multi-transmission communication mode. In scenario 410, nodes 0, 1, and 2 act as both senders distributing data and receivers aggregating data of the same series. Node 0 sends data to nodes 1 and 2, node 1 sends data to nodes 0 and 2, and node 2 sends data to nodes 0 and 1.

[0047] Figure 5 A schematic diagram of a step-by-step interleaved transmission scenario 500 for all-to-all communication according to some embodiments of this disclosure is shown. (See reference...) Figure 5 Scenario 500 illustrates an ideal All2All communication mode, with Figure 4B The multi-to-multi scenario shown is not compatible. Scenario 500 uses a step-by-step interleaved sending method to strictly synchronize the sending behavior of the nodes in each step, ensuring that each node only performs "one send and one receive" communication at any given time.

[0048] Figure 4A Example 400 and Figure 4B In scenario 410, as the scale of communication continues to increase, the performance bottleneck of All2All operations can lead to network congestion and idle computing resources. Figure 5 Scenario 500 represents an ideal All-to-All communication mode, where no blocking occurs. However, in reality, due to the asynchronous startup times of various nodes and differences in execution times between nodes, the strictly synchronous interleaved transmission method in Scenario 500 cannot be achieved. This inevitably leads to multiple-to-one communication scenarios, resulting in performance degradation. Therefore, the following section discusses... Figure 6A and Figure 6B To describe two methods for communication according to some embodiments of this disclosure. Figure 6A The diagram illustrates a method 600 for communication based on instruction messages and chained primitives, representing some embodiments of this disclosure. Figure 6BA schematic diagram of example 610 for communication based on chained primitives, representing some embodiments of this disclosure, is shown.

[0049] refer to Figure 6A As shown, nodes 1, 2, 3, and 4 are sending nodes, and nodes a, b, c, and d are receiving nodes. The specific process of method 600 is as follows: 601: Node 1 sends data to node a (WQE1a, WQExy represents the operation from node x to node y); 602: After completing 601 through the first WQE, node a is triggered by the chaining operation to send an indication message to node 4 (the indication message is also called Credit, and the process in 602 is WQEa4); 603: After receiving the indication message from node a, node 4 sends data to node a through the triggered WQE (WQE4a); 604: After receiving the data sent by node 4, node a is triggered by the chaining operation to send an indication message to node 3 (WQEa3); 605: After receiving the indication message from node a, node 3 sends data to node a through the corresponding WQE (WQE3a); 606: Node a... After receiving data from node 3, the Chaining operation triggers the sending of an indication message (WQEa2) to node 2; 607: After receiving the indication message from node a, node 2 sends data to node a via the corresponding WQE (WQE2a). Figure 6A As shown, the execution order of 601-607 strictly corresponds to the triggering logic of the WQE chain. The next operation is triggered only after the previous operation is completed. Each WQE is blocked by the Chaining operation and is only triggered to execute after the previous WQE is completed. In this way, the above process is automatically completed by the network card without the participation of the CPU or GPU, thereby realizing the automated execution of a WQE chain composed of multiple WQEs.

[0050] refer to Figure 6BAs shown, Example 610 includes nodes a 612, b 614, and c 616. Node a 612 includes network interface card a 618, and node b 614 includes CPU 624, QPx 626, QPy 628, and network interface card b 620. In some embodiments, a QP (Queue Pair) can consist of an SQ (Send Queue) and an RQ (Receive Queue). For example, QPy 628 represents the send queue, and a Chaining operation is set at the head of the send queue to block the execution of subsequent WQE. QPx 626 represents the receive queue, and node c 616 includes network interface card c 622. The specific process includes: 631: CPU 624 pre-submits work queue elements to QPy 628; 632: NIC a 618 sends a message to NIC b 620; 633: After receiving the message, NIC b 620 sends the result back to QPx 626; 634: QPx 626 triggers the execution of the pre-submitted work queue elements in QPy 628, releasing QPy 628 from its blocking state; 635: NIC b 620 sends a message to NIC c 622, completing the entire interaction process. In this way, a series of WQEs distributed across one or more nodes can be pre-submitted, their execution blocked by the Chaining primitive. After the NIC processes a specific message, it automatically triggers the execution of the blocked WQEs, enabling data transmission without CPU or GPU involvement, automated execution by the NIC, and thus achieving automated execution of a WQE chain composed of multiple WQEs.

[0051] Figure 6A Method 600 and Figure 6B Method 610 in the text can achieve complete zero-blocking, with each receiving node receiving data from at most one sending node at any given time, thus achieving completely congestion-free data communication. However, it has the following drawbacks in small message scenarios: First, the sender needs to receive an indication message before triggering the next WQE, resulting in a time bubble (RTT) between two data transmissions while waiting for the indication message, which increases the completion time of All2All operations in small message scenarios; Second, each receiving node receives data from at most one sending node at any given time, and in large-scale All2All scenarios, it is usually impossible to set too many concurrent QPs between two GPUs (for example, in NCCL, the default number of QPs is 1 in scenarios with more than 32 GPUs), so it is difficult to fully utilize the bandwidth in small message scenarios. Therefore, the following combines... Figure 7 and Figure 8 To describe two methods for communication according to some embodiments of this disclosure. Figure 7 A schematic diagram of a packet-based method 700 for communication according to some embodiments of the present disclosure is shown. Figure 8 A schematic diagram is shown of a method 800 for packet communication based on instruction messages and chain primitives, which can implement some embodiments of the present disclosure.

[0052] refer to Figure 7 As shown, method 700 evenly divides the sending nodes 0, 1, 2, 3, 4, 5 and the receiving nodes a, b, c, d, e, f into multiple groups: a first group of sending nodes 702, a second group of sending nodes 704, a first group of receiving nodes 706, and a second group of receiving nodes 708. In some embodiments, any pair of sending and receiving groups can be... Figure 6A Method 600 shown, for example, refers to Figure 8 The specific process illustrated is as follows: the first group of sending nodes 804 includes nodes 1 and 2; the second group of sending nodes 806 includes nodes 3 and 4; and the receiving nodes include nodes a, b, c, and d. The specific process includes: 801: Nodes 1 and 4 simultaneously send data to receiving node a; 802: Node a simultaneously sends indication messages to nodes 2 and 3; 803: Nodes 2 and 3 send data to node 1 in parallel. This process is executed concurrently through two WQE chains. This method increases data transmission concurrency and improves network throughput. Furthermore, with multiple senders simultaneously transmitting data to the receiver, communication bubbles within a single group can be masked, thereby improving network bandwidth utilization.

[0053] In some embodiments, sending nodes can be divided into multiple groups according to certain rules, and receiving nodes can also be divided into multiple groups according to the same rules. For sending group x and receiving group y (assuming that the number of nodes in each group is m, and the node sequence numbers are 0, 2, ..., m-1), WQEs are submitted according to the following rules: For sending node i: submit the data WQE to receiving node i+1 (or node i in M2N); the next step is divided into m-2 (m-1 in M2N) steps. For step s (s∈{2,3,⋯,m-1}): for the receiving node with sequence number k=(i+s)% m (k=(i+s-1)% m in M2N), submit a blocking WQE and a data WQE to that node in sequence; the blocking WQE is bound to the completion queue between nodes i and k. For receiving node j: there are m-2 steps (m-1 in M2N). For step s (s∈{1,2,⋯,m-2}): for the node with sequence number k=(js-1+N)% m (k=(j-s+N)% m in M2N), a blocking WQE and an indication message WQE (i.e., the WQE used to send the indication message) are submitted to that node in sequence. The blocking WQE is bound to the completion queue between node (k+1)% m and j.

[0054] Figure 9 A block diagram of a communication apparatus 900 according to some embodiments of the present disclosure is shown. Figure 9 As shown, the device 900 includes a first data receiving module 902 configured to receive first data from a first node, the first data being sent in response to triggering a work queue element chain. The device 900 also includes an instruction message sending module 904 configured to send an instruction message to a second node in response to completion of receiving the first data, the instruction message instructing the second node to send data. Furthermore, the device 900 includes a second data receiving module 906 configured to receive second data from the second node, the second data being sent in response to receiving the instruction message.

[0055] In some embodiments, the device 900 further includes a second indication message sending module configured to send a second indication message to a third node in response to completion of receiving second data. The second indication message instructs the third node to send data. The device 900 also includes a third data receiving module configured to receive third data from the third node, the third data being sent in response to receipt of the second indication message.

[0056] In some embodiments, where the first node and the second node form a first sending group, and the third node and the fourth node form a second sending group, the apparatus 900 further includes a third data module configured to receive third data from the third node, the third data being sent in response to triggering a second work queue element chain. The apparatus 900 also includes a second indication message module configured to send a second indication message to the fourth node in response to completion of receiving the third data, the second indication message instructing the fourth node to send data. The apparatus 900 further includes a fourth data receiving module configured to receive fourth data from the fourth node, the fourth data being sent in response to receiving the second indication message.

[0057] In some embodiments, the work queue element chain sequentially links a first work queue element, a second work queue element, and a third work queue element. The first work queue element is used to receive first data from a first node, the second work queue element is used to send an indication message to a second node, and the third work queue element is used to receive second data from the second node.

[0058] In some embodiments, the second and third work queue elements are pre-blocked using a chained primitive.

[0059] In some embodiments, the apparatus 900 further includes a scheduling module configured to schedule the chain of work queue elements via a network interface card (NIC). The apparatus 900 also includes a communication module configured to communicate based on the NIC-based scheduling.

[0060] In some embodiments, communication includes all-to-all communication.

[0061] It is understood that by utilizing the apparatus 900 of this disclosure, at least one of the many advantages achievable by the methods or processes described above can be realized. For example, the apparatus 900 can avoid communication congestion and improve communication performance.

[0062] Figure 10 A block diagram of a communication apparatus 1000 according to some embodiments of the present disclosure is shown. Figure 10 As shown, the apparatus 1000 includes an instruction message receiving module 1002 configured to receive an instruction message from a first node, the instruction message being used to instruct the sending of data to the first node, and the instruction message being generated in response to the first node completing the reception of first data from a second node. The apparatus 1000 also includes a second data sending module 1004 configured to send second data to the first node, the second data being sent in response to the triggering of a work queue element chain.

[0063] In some embodiments, the apparatus 1000 further includes a second indication message receiving module configured to receive a second indication message from a third node, the second indication message being used to indicate sending data to the third node, and the second indication message being generated in response to the third node completing the reception of third data from the fourth node. The apparatus 1000 also includes a third data sending module configured to receive third data from the third node, the third data being sent in response to the reception of the second indication message.

[0064] In some embodiments, where the first node and the third node form a first receiving group, the second node and the fourth node form a first sending group, the work queue element chain is a first work queue element chain, and the indication message is a first indication message, the apparatus 1000 further includes a second indication information module configured to receive a second indication message from the third node. The indication message is used to indicate sending data to the third node, and the second indication message is generated in response to the third node completing the reception of third data from the fourth node. The apparatus 1000 also includes a fourth data sending module configured to send fourth data to the third node, the fourth data being sent in response to the triggering of the second work queue element chain.

[0065] In some embodiments, the work queue element chain sequentially links a first work queue element and a second work queue element. The first work queue element is used to receive an indication message from a first node, and the second work queue element is used to send second data to the first node.

[0066] In some embodiments, the second work queue elements are pre-blocked via a chaining primitive.

[0067] In some embodiments, the apparatus 1000 further includes a scheduling module configured to schedule the chain of work queue elements via a network interface card (NIC). The apparatus 1000 also includes a communication module configured to communicate based on the NIC-based scheduling.

[0068] In some embodiments, communication includes all-to-all communication.

[0069] It is understood that by utilizing the apparatus 1000 of this disclosure, at least one of the many advantages achievable by the methods or processes described above can be realized. For example, the apparatus 1000 can avoid communication congestion and improve communication performance.

[0070] Figure 11 A block diagram of a device 1100 capable of implementing several embodiments of the present disclosure is shown. (See reference...) Figure 11As shown, device 1100 includes a central processing unit (CPU) and / or a graphics processing unit (GPU) 1101, which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) 1102 or loaded from storage unit 1108 into random access memory (RAM) 1103. Various programs and data required for the operation of device 1100 can also be stored in RAM 1103. The CPU / GPU 1101, ROM 1102, and RAM 1103 are interconnected via bus 1104. Input / output (I / O) interface 1105 is also connected to bus 1104. Although not shown in... Figure 11 As shown, device 1100 may also include a coprocessor.

[0071] Multiple components in device 1100 are connected to I / O interface 1105, including: input unit 1106, such as keyboard, mouse, etc.; output unit 1107, such as various types of monitors, speakers, etc.; storage unit 1108, such as disk, optical disk, etc.; and communication unit 1109, such as network card, modem, wireless transceiver, etc. Communication unit 1109 allows device 1100 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0072] The various methods or processes described above can be executed by CPU / GPU 1101. For example, in some embodiments, the methods may be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 1108. In some embodiments, part or all of the computer program may be loaded and / or installed on device 1100 via ROM 1102 and / or communication unit 1109. When the computer program is loaded into RAM 1103 and executed by CPU / GPU 1101, one or more steps or actions in the methods or processes described above can be performed.

[0073] In some embodiments, the methods and processes described above can be implemented as a computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of this disclosure.

[0074] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0075] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, a local area network (LAN), a wide area network (WAN), and / or a wireless network, to an external computer or external storage device. The network may include copper cables, fiber optic cables, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0076] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​and conventional procedural programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to execute the computer-readable program instructions, thereby implementing various aspects of this disclosure.

[0077] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0078] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0079] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of devices, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0080] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or technical improvements to the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for communication, comprising: Receive first data from the first node, the first data being sent in response to a triggering of the work queue element chain; In response to completing the reception of the first data, an indication message is sent to the second node, the indication message being used to instruct the second node to send data; as well as Second data is received from the second node, and the second data is sent in response to the receipt of the indication message.

2. The method according to claim 1, wherein the indication message is a first indication message, further comprising: In response to the completion of receiving the second data, a second indication message is sent to the third node, the second indication message being used to instruct the third node to send data; as well as Third data is received from the third node, which is sent in response to the receipt of the second indication message.

3. The method according to claim 1, wherein the first node and the second node constitute a first sending group, the third node and the fourth node constitute a second sending group, the work queue element chain is a first work queue element chain, and the indication message is a first indication message, further comprising: Receive third data from a third node, the third data being sent in response to a triggering of the second work queue element chain; In response to the completion of receiving the third data, a second indication message is sent to the fourth node, the second indication message being used to instruct the fourth node to send data; as well as Fourth data is received from the fourth node, which is sent in response to the receipt of the second indication message.

4. The method according to claim 1, wherein the work queue element chain sequentially links a first work queue element, a second work queue element, and a third work queue element, the first work queue element is used to receive the first data from the first node, the second work queue element is used to send the indication message to the second node, and the third work queue element is used to receive the second data from the second node.

5. The method according to claim 4, further comprising: The second work queue element and the third work queue element are blocked in advance using a chained primitive.

6. The method of claim 5, further comprising: The work queue element chain is scheduled via the network interface card; as well as Communication is based on the scheduling of the network interface card.

7. The method of claim 1, wherein the communication comprises all-to-all communication.

8. A method for communication, comprising: A first node receives an indication message, the indication message being used to indicate sending data to the first node, and the indication message being generated in response to the first node completing the reception of first data from the second node; as well as Send second data to the first node, the second data being sent in response to a triggering of the work queue element chain.

9. The method according to claim 8, wherein the indication message is a first indication message, further comprising: A second indication message is received from the third node, the second indication message being used to indicate sending data to the third node, and the second indication message being generated in response to the third node completing the reception of third data from the fourth node; and Send the fourth data to the third node.

10. The method according to claim 8, wherein the first node and the third node constitute a first receiving group, the second node and the fourth node constitute a first sending group, the work queue element chain is a first work queue element chain, and the indication message is a first indication message, further comprising: A second indication message is received from the third node, the indication message being used to indicate sending data to the third node, and the second indication message is generated in response to the third node completing the reception of third data from the fourth node; as well as A fourth data is sent to the third node in response to a triggering of the second work queue element chain.

11. The method according to claim 8, wherein the work queue element chain sequentially links a first work queue element and a second work queue element, the first work queue element is used to receive the indication message from the first node, and the second work queue element is used to send the second data to the first node.

12. The method of claim 11, further comprising: The second work queue element is blocked in advance using a chained primitive.

13. The method of claim 12, further comprising: The work queue element chain is scheduled via the network interface card; as well as Communication is based on the scheduling of the network interface card.

14. The method of claim 8, wherein the communication comprises all-to-all communication.

15. An apparatus for communication, comprising: The first data receiving module is configured to receive first data from the first node, the first data being sent in response to the triggering of the work queue element chain; An instruction message sending module is configured to send an instruction message to a second node in response to the completion of receiving the first data, the instruction message being used to instruct the second node to send data; as well as The second data receiving module is configured to receive second data from the second node, the second data being sent in response to the receipt of the indication message.

16. An apparatus for communication, comprising: An instruction message receiving module is configured to receive an instruction message from a first node, the instruction message being used to instruct the sending of data to the first node, and the instruction message being generated in response to the first node completing the reception of first data from a second node; as well as The second data sending module is configured to send second data to the first node, the second data being sent in response to a triggering of the work queue element chain.

17. An electronic device comprising: processor; as well as A memory coupled to the processor, the memory having instructions stored therein, which, when executed by the processor, cause the electronic device to perform the method according to any one of claims 1 to 14.

18. A computer program product tangibly stored on a non-transitory computer-readable medium and comprising machine-executable instructions that, when executed, cause a machine to perform the method according to any one of claims 1 to 14.

Images