RDMA-based multicast method and node device for distributed collective communication

Abstract

Disclosed are an RDMA-based multicast method and a corresponding node device, which are applicable to distributed collective communication involving a source end and N destination ends. On the basis of an example of the method, a source end sends a multicast packet on the basis of a first UD-type QP, wherein the multicast packet carries a first virtual endpoint ID and a first multicast ID; and each destination end can send a unicast packet to the source end on the basis of a second UD-type QP, wherein the unicast packet carries a second virtual endpoint ID and a second multicast ID. The first multicast ID of the source end and the second multicast IDs of the N destination ends are all the same, and the first virtual endpoint ID of the source end is different from the second virtual endpoint IDs of the N destination ends. In this way, the multicast packet sent by the source end can be forwarded to all destination ends by means of a multicast function of a network switch, and the reliability of distributed collective communication can be improved by each destination end replying with the unicast packet to the source end.

Description

RDMA-based multicast method and distributed collective communication node device Technical Field

[0001] The present disclosure relates to distributed collective communication, and more particularly to a multicast solution applicable to distributed collective communication based on Remote Direct Memory Access (RDMA). Background Art

[0002] With the deep integration of artificial intelligence generated content (AIGC) across numerous industries, the number of parameters in large models continues to grow, and computing tasks are becoming increasingly large and complex. This means that distributed training systems will have to handle large-scale computing power and tasks, and the communication efficiency of distributed training will seriously affect its performance.

[0003] On the other hand, RDMA, as the current mainstream high-performance transmission method, has been widely used in the communication of distributed training systems. For example, in Artificial Intelligence (AI) or High Performance Computing (HPC) applications, distributed collective communications such as allreduce and allgather often use RDMA, an efficient communication method. However, collective communications in distributed systems are generally based on point-to-point unicast communication at the transport layer to achieve many-to-many or one-to-many communications. Specifically, a collective communication call at the application layer is completed by multiple point-to-point communications at the transport layer. This is because in distributed training, when communication is initialized, point-to-point connection establishment begins at the transport layer, and subsequent collective communications are carried out based on the point-to-point connection at the transport layer. This distributed collective communication method implemented by RDMA-based point-to-point unicast communication may incur significant system overhead and may also waste bandwidth resources.

[0004] Summary of the Invention

[0005] The present disclosure provides a multicast solution based on RDMA for distributed collective communication, which can provide more powerful support for multi-computing power collaboration within large-scale data centers.

[0006] In a first aspect, the present disclosure provides an RDMA-based multicast method, which is applied to a source end in a distributed collective communication, the method comprising: sending a multicast message based on a first UD type QP, wherein the multicast message carries a first virtual endpoint ID and a first multicast ID, so that the multicast message is forwarded to N destination ends in the distributed collective communication via the multicast function of the network switch, N is an integer greater than or equal to 2, and each destination end processes the multicast message after determining that the first multicast ID carried by the multicast message is the same as its own second multicast ID and identifying that the multicast message comes from the source end based on the first endpoint ID; and, when a unicast message sent by the destination end is received based on a second UD type QP, the unicast message is processed after determining that the second multicast ID carried by the unicast message is the same as the first multicast ID and identifying that the unicast message comes from the destination end based on the second endpoint ID carried by the unicast message.

[0007] In a second aspect, the present disclosure provides an RDMA-based multicast method, which is applied to one of the N destination ends in a distributed collective communication, where N is an integer greater than or equal to 2, and the method includes: receiving a multicast message based on a first UD type QP, wherein the multicast message carries the first virtual endpoint ID and the first multicast ID of the source end in the distributed collective communication; processing the multicast message after determining that the first multicast ID carried by the multicast message is the same as the second multicast ID of the destination end itself, and identifying that the multicast message comes from the source end based on the first endpoint ID; and, when replying the processing result of the multicast message to the source end, sending a unicast message to the source end based on a second UD type QP, wherein the unicast message carries the second virtual endpoint ID and the second multicast ID, so that the source end processes the unicast message after determining that the second multicast ID carried by the unicast message is the same as the first multicast ID, and identifying that the unicast message comes from the destination end based on the second endpoint ID.

[0008] In a third aspect, the present disclosure provides a node device for distributed collective communication, which serves as a source end in the distributed collective communication and includes a processor and a network card supporting RDMA. The processor is configured to: send a multicast message via the network card based on a first UD type QP, wherein the multicast message carries a first virtual endpoint ID and a first multicast ID, so that the multicast message is forwarded to N destination ends in the distributed collective communication via the multicast function of the network switch, where N is an integer greater than or equal to 2, and each destination end processes the multicast message after determining that the first multicast ID carried by the multicast data message is the same as its own second multicast ID and identifying that the multicast message comes from the source end based on the first endpoint ID; and, when a unicast message sent by the destination end is received via the network card based on a second UD type QP, the processor processes the unicast message after determining that the second multicast ID carried by the unicast message is the same as the first multicast ID and identifying that the unicast message comes from the destination end based on the second endpoint ID carried by the unicast message.

[0009] In a fourth aspect, the present disclosure provides a node device for distributed collective communication, as one of the N destination ends in the distributed collective communication, comprising a processor and a network card supporting RDMA. The processor is configured to: based on a first UD type QP, receive a multicast message via the network card, wherein the multicast message carries the first virtual endpoint ID and the first multicast ID of the source end in the distributed collective communication; after determining that the first multicast ID carried by the multicast message is the same as the second multicast ID of the destination end itself and identifying that the multicast message comes from the source end based on the first endpoint ID, process the multicast message; and, when replying to the source end with the processing result of the multicast message, based on a second UD type QP, send a unicast message to the source end via the network card, wherein the unicast message carries the second virtual endpoint ID and the second multicast ID, so that the source end processes the unicast message after determining that the second multicast ID carried by the unicast message is the same as the first multicast ID and identifying that the unicast message comes from the destination end based on the second endpoint ID.

[0010] At least one of the above technical solutions adopted in the present disclosure can achieve the following beneficial effects:

[0011] By creating a first UD type QP and a second UD type QP that support RDMA at the source end and all destination ends in the distributed collective communication, on the one hand, the source end can use the multicast function of the network switch based on the first UD type QP to send multicast messages to the network layer of multiple destination ends. Compared with conventional RDMA-based point-to-point communication, this avoids the repeated sending of the same data, not only saving bandwidth resources from the source end to the network switch, but also effectively reducing transmission delay; on the other hand, each destination end can send unicast messages to the source end based on the second UD type QP, including feeding back packet reception status to the source end, thereby effectively improving the reliability of the distributed collective communication.

[0012] In particular, before the source and all destinations enter the data multicast process of distributed collective communication, the source notifies the data information of the data to be transmitted in a multicast manner based on the first UD type QP, and each destination responds to the notification confirmation information in a unicast manner based on the second UD type QP, so that the source and all destinations can update and maintain their respective data packet sending and receiving records in real time during the data multicast process, and the source can confirm whether all destinations have correctly received the sent data packets based on its own packet sending record and the packet receiving record fed back by each destination after the multicast process for a specific data block is completed, and trigger the retransmission of data packets that have not been correctly received by all destinations as appropriate. In this way, compared with the method in which the receiving end responds to the packet receiving confirmation in real time, not only can the number of confirmation messages be significantly reduced, but also the waiting time caused by the real-time feedback of packet loss information can be avoided, thereby effectively improving the efficiency of distributed collective communication. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings described herein are used to provide a further understanding of the present disclosure and constitute a part of the present disclosure. The illustrative embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation of the present disclosure.

[0014] FIG1 is a schematic diagram of a method for implementing one-to-many multicast in distributed collective communication according to an embodiment of the present disclosure.

[0015] FIG2 is a schematic diagram of another method for implementing one-to-many multicast in distributed collective communication according to an embodiment of the present disclosure.

[0016] 3A and 3B are schematic diagrams of a node architecture for performing RDMA-based distributed training using a unified communication interface (UCX) according to an embodiment of the present disclosure.

[0017] FIG4 is a schematic diagram of another method for implementing one-to-many multicast in distributed collective communication according to an embodiment of the present disclosure.

[0018] FIG5A is a flowchart of a method for establishing an RDMA-based multicast connection suitable for distributed collective communication according to an embodiment of the present disclosure.

[0019] FIG5B and FIG5B(continued) are flowcharts of an RDMA-based multicast method applicable to distributed collective communication according to an embodiment of the present disclosure.

[0020] FIG6 is a schematic diagram of a packet sending status record of a source end established according to an embodiment of the present disclosure.

[0021] FIG7 is a schematic diagram of a packet receiving status record of a destination end established according to an embodiment of the present disclosure.

[0022] FIG8 is a flow chart of an RDMA-based multicast method according to an embodiment of the present disclosure, which is applied to a source end in distributed collective communication.

[0023] FIG9 is a flow chart of an RDMA-based multicast method according to an embodiment of the present disclosure, which is applied to one of N destinations in distributed collective communication.

[0024] FIG10 is a functional module diagram of a node device for distributed collective communication according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] To make the objectives, technical solutions, and advantages of the present disclosure more clear, the technical solutions of the present disclosure will be clearly and completely described below in conjunction with the specific embodiments of the present disclosure and the corresponding drawings. The described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present disclosure.

[0026] The technical solutions provided by various embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0027] In current distributed systems, in addition to point-to-point communication, collective communication is more common in one-to-many and many-to-many scenarios. Because some current mainstream transport layer protocols only support point-to-point transmission, one-to-many and many-to-many data transmission in collective communication is also mostly implemented using point-to-point data transmission. This involves multiple calls to point-to-point transmission to complete one-to-many and many-to-many data forwarding.

[0028] Taking one-to-many broadcast (Broadcast, one data sender, multiple data receivers) as an example, it is necessary to broadcast the data on one node to multiple other nodes in the cluster. Currently, a method for implementing one-to-many broadcast can be shown in Figure 1. The source node (hereinafter also referred to as the source end) 110 forwards the same message to different destination nodes (hereinafter also referred to as the destination end) 130-1, 130-2, ... 130-N via the network switch 120 in sequence. Among them, although N is greater than 2 in the figure, those skilled in the art should understand that N can also be equal to 2. In other words, N represents the number of destination nodes, and as long as N is an integer greater than or equal to 2, the so-called one-to-many broadcast can be achieved. If the amount of data is M, the bandwidth is B, and n is the number of nodes, the total time required to implement one-to-many broadcast in this way can be calculated as follows:

[0029] Among them, t s Indicates the startup time of a single transmission, including the total time spent on network layer protocol stack processing, CPU interruption application layer protocol stack processing, and data forwarding to the network port.

[0030] Obviously, this approach introduces additional overhead that is linearly proportional to the communication scale, resulting in increased latency. Furthermore, when the bandwidth from source 110 to network switch 120 is large, this approach can easily waste bandwidth resources. Furthermore, when the amount of distributed training data is large, this approach can easily create a bandwidth bottleneck link.

[0031] Currently, another mainstream method for implementing one-to-many broadcasting in distributed collective communication can be shown in Figure 2. During the communication process between a source node 210 and multiple destination nodes 230-1, 230-2, ...230-N, destination nodes 230-1, 230-2, ...230-N-1 are used to assist in intermediate data forwarding. Specifically, the source end 210 splits the message to be sent into multiple data blocks and sends one data block at a time. After receiving the complete data block, the first destination node 230-1 forwards it to the next destination node 230-2. This process continues, and the data block is forwarded to the last destination node 230-N in sequence. Because the source end 210 can send the next data block to the first destination node while the data block is being transmitted between destination nodes 230-1, 230-2, ...230-N, this method can achieve pipelined and parallel transmission of data blocks, effectively improving the efficiency of inter-node communication in the distributed system.

[0032] Specifically, if the data volume is M, the bandwidth is B, and n is the number of nodes, and the source divides the M data into s blocks, the total time required to implement the one-to-many broadcast method can be calculated as follows:

[0033] Among them, t sIndicates the startup time of a single transmission, including the total time spent on network layer protocol stack processing, CPU interruption application layer protocol stack processing, and data forwarding to the network port.

[0034] While the above implementation improves communication efficiency and reduces bandwidth bottlenecks to a certain extent, it still incurs additional overhead that scales linearly with the communication scale. Furthermore, the destination nodes 230-1, 230-2, ..., 230-N-1, which assist in intermediate data forwarding, need to copy and forward the received message, which also increases CPU interrupt and processing latency.

[0035] In view of this, the present invention aims to propose an RDMA-based multicast method suitable for distributed collective communication. By utilizing network switches to complete the replication and distribution of multicast messages, network layer multicast replaces application layer multicast to avoid repeated sending of the same data, it can effectively save the bandwidth from the source to the network switching node, and by reducing the processing time of intermediate assisting forwarding data, it can also effectively reduce transmission latency.

[0036] Unified Communication X (UCX) aims to provide a unified abstract communication interface to optimize RDMA-based communications and support the needs of various applications. In the current collective communication of distributed systems, in order to optimize RDMA-based data transmission, most communication libraries (such as the NVIDIA Collective Communications Library (NCCL), the Open Message Passing Interface (OpenMPI), etc.) are compatible with UCX, as shown in Figures 3A and 3B. Among them, UCX is based on endpoints for inter-node communication, that is, when the source end uses UCX, it needs to create an endpoint and connect to the endpoint created by the destination end. In addition, each endpoint is uniquely identified by a corresponding endpoint ID. In addition, in AI or HPC applications, for example, when performing an AI distributed training, multiple processes are usually started, and communication libraries such as NCCL and OMPI will assign a ranking value rank to each process to uniquely identify the corresponding process.

[0037] Therefore, the present invention mainly implements network layer multicast (hereinafter referred to as network RDMA multicast) suitable for distributed collective communication based on UCX and RDMA, so as to realize distributed collective communication by utilizing the multicast function of the network switch. Specifically, when the distributed training process of the source end wants to call the broadcast communication primitive to broadcast data, a multicast connection can be established based on the underlying UCX (hereinafter referred to as multicast connection establishment), and then the established multicast connection is used to send a multicast data message, so as to utilize the multicast function of the network switch to forward the multicast data message to all destination ends. Accordingly, when the destination end receives the multicast data message, it can reply to the source end ACK / NACK. In this way, if packet loss occurs, the source end will retransmit, thereby effectively ensuring the reliability of the multicast.

[0038] The framework of the aforementioned network RDMA multicast communication can be shown in Figure 4. Here, a source end 410 establishes an RDMA-based multicast connection with multiple destination ends 430-1, 430-2, ..., 430-N using UCX, thereby enabling the multicast function of the network switch 420 to implement RDMA-based multicast suitable for distributed collective communication.

[0039] If the data volume is M, the maximum transmission unit (MTU) is U, the bandwidth is B, and n is the number of nodes, the time required to implement one-to-many broadcast in the above scheme can be calculated as follows:

[0040] Among them, T0 is the additional delay caused by multicast sending one data. Multicast needs to add some additional header information in the data payload, so data parsing and encapsulation introduce some additional delay compared with the above two methods. s Indicates the startup time of a single transmission, including the total time spent on network layer protocol stack processing, CPU interruption application layer protocol stack processing, and data forwarding to the network port.

[0041] Since UCX is based on a point-to-point communication framework, in order to implement a reliable network RDMA multicast solution, according to an embodiment of the present invention, the UCX communication framework is first modified to implement one-to-many multicast connection establishment.

[0042] 1. Multicast connection establishment based on UCX for distributed collective communication

[0043] First, the source node and the destination node participating in the distributed collective communication need to establish a multicast connection based on RDMA (for example, RDMA compliant with the RoCEV2 protocol).

[0044] Specifically, since RDMA's unreliable datagram (UD) type queue pair (QP) cannot guarantee that the destination will receive the data packet, nor can it guarantee the order in which the data packets are received, in order to implement a reliable multicast connection suitable for distributed collective communication based on RDMA, in the technical solution according to an embodiment of the present invention, two UD type QPs are created at the source end and all the destination ends, wherein: the first UD type QP is used to establish a multicast connection between the source end and all the destination ends, even if the source end and all the destination ends are constructed as a multicast group, so that the multicast message sent by the source end can be forwarded to each destination end using the multicast function of the network switch, thereby realizing distributed collective communication; and the second UD type QP is used to establish a unicast mapping (hereinafter also referred to as a virtual unicast link) between the source end and each destination end, so that each destination end can send a unicast message to the source end, including replying an ACK / NACK message, thereby effectively ensuring the reliability of the distributed collective communication.

[0045] Secondly, the UCX point-to-point communication framework needs to be modified. That is, the limitations of the UCX point-to-point communication framework must be broken through to expand the unicast link to implement a UCX-based point-to-multipoint connection solution, thereby ensuring that the destination can correctly identify the multicast data packets sent by the source and that the destination can reliably unicast ACK / NACK replies to the source.

[0046] The connection establishment scheme may be shown in Figure 5A. As shown in Figure 5A, the connection establishment scheme may be generally divided into two stages, namely, establishing a multicast connection and establishing a virtual unicast link.

[0047] As shown above the dotted line in FIG5A , the stages of establishing a multicast connection may specifically include:

[0048] ① The source end 510 creates two UD-type QPs, and ② each destination end 530-1, ..., 530-N also creates two UD-type QPs. The first UD-type QP is used to build and join a multicast group, and the second UD-type QP is used to establish a virtual unicast link.

[0049] It should be noted that although FIG5A shows step ① as being executed before step ②, in fact, the execution order of these two steps is not limited, that is, step ① can also be executed simultaneously with step ②, or after step ②.

[0050] ③ The source end 510 generates a unique multicast Internet Protocol (IP) address.

[0051] Accordingly, all destination terminals 530 - 1 , . . . , 530 -N can obtain the multicast IP address through the original point-to-point transmission solution in the distributed collective communication.

[0052] ④ The source end 510 and all destination ends 530-1, ...530-N generate the same 3-byte multicast ID (groupid) according to the same rules based on the multicast IP address to identify the multicast to which they belong, that is, to identify the multicast group created by the source end and all destination ends using the aforementioned first UD type QP.

[0053] At this point, a multicast connection is successfully established between source 510 and all destinations 530-1, ..., 530-N. Specifically, if source 510 sends a multicast data packet carrying the multicast ID groupid, upon receiving the packet, destinations 530-1, ..., 530-N can correctly identify and process the multicast data packet by matching their stored multicast ID groupid with the multicast ID groupid carried in the packet. The multicast ID groupid only needs to have its most significant bit set to 1 (e.g., 0x800001).

[0054] In addition, as shown below the dotted line in FIG5A , the stage of establishing a virtual unicast link may specifically include:

[0055] ⑤ The source end 510 randomly generates a 3-byte virtual endpoint ID v_epid_0_0, and all destination ends generate different 3-byte virtual endpoint IDs v_epid_1_0, ....v_epid_N_0 according to their own ranking value rank according to certain rules to identify the messages sent and received by each node. As mentioned above, UCX is based on endpoints for inter-node communication, and uses endpoint IDs to uniquely identify corresponding endpoints. In an embodiment of the present invention, in order to implement point-to-multipoint multicast connection based on UCX, a virtual endpoint ID is extended on the basis of the endpoint ID of UCX, which is mainly used to identify each node that sends and receives messages. Among them, the virtual endpoint ID v_epid_0_0 generated by the source end 510 and the virtual endpoint IDs v_epid_1_0, ....v_epid_N_0 generated by each destination end all meet the requirement that the highest bit is 0 (such as 0x000001).

[0056] ⑥ The source 510 initiates a connection request, i.e. generates a CREQ message, and sends the CREQ message via multicast. The CREQ message carries information such as the source's own virtual endpoint ID v_epid_0_0, multicast ID, and qualified provider network (QPN).

[0057] ⑦ After receiving the CREQ message from source 510, all destinations 530-1, ..., 530-N unicast a connection request response, or CREP message, to the source. This CREP message carries the destination's self-generated virtual endpoint IDs (v_epid_1_0, ..., v_epid_N_0), the multicast ID, and the QPN. It also carries an acknowledgment (ACK) flag to ensure that source 510 can respond with an ACK message to the corresponding destination after receiving the CREP message.

[0058] ⑧ After the source end 510 successively receives CREP messages from each destination end 530-1, ...530-N, it saves the information carried in these CREP messages. It should be noted that the order in which the source end 510 receives the CREP messages from each destination end 530-1, ...530-N is uncertain and is not necessarily the same as the order in which each destination end 530-1, ...530-N receives the CREQ messages from the source end 510. Therefore, the source end 510 counts the received CREP messages until it receives CREP messages carrying ACK information from all destination ends 530-1, ...530-N. Furthermore, after confirming that it has received CREP messages carrying ACK information from all destination ends 530-1, ...530-N, the source end 510 can reply with ACK messages to all destination ends 530-1, ...530-N via multicast based on the ACK information in the received CREP messages.

[0059] In particular, if the received CREP message carries a NACK instead of an ACK, the source end 510 will resend the CREQ message via multicast. In this case, the destination end that previously replied to the CREP message with an ACK will ignore the duplicate CREQ message, while the destination end that previously replied to the CREP message with a NACK will process the duplicate CREQ message and, after correctly processing it, reply to the source end 510 with a CREP message with an ACK.

[0060] Through the above processing, a virtual unicast link is successfully established for each destination end 530-1, ...530-N to send unicast reply messages (which can carry ACK / NACK information for packet receipt confirmation and / or packet loss report) to the source end 510, specifically including the unicast mappings {v_epid_0_0, v_epid_1_0}, ...{v_epid_0_0, v_epid_N_0} established based on the virtual endpoint ID.

[0061] At this point, the source end can send multicast messages to multiple destination ends based on the aforementioned first UD type QP, and each destination end can send unicast messages to the source end based on the aforementioned second UD type QP. The multicast message carries the virtual endpoint ID of the source end and the multicast ID, so that the multicast message can be forwarded to each destination end via the multicast function of the network switch, and each destination end identifies and processes the multicast message by comparing the multicast ID carried by the multicast message with its own multicast ID; the unicast message carries the multicast ID and the virtual endpoint ID of the destination end itself, so that the source end can correctly identify and process the unicast message. In other words, through the above operations, the multicast connection based on UCX is completed.

[0062] 2. Network RDMA multicast implementation for distributed collective communication

[0063] To ensure the reliability of distributed collective communication data transmitted via RDMA multicast over the network, a guarantee mechanism is necessary. Generally, this can be achieved by requiring the receiving end of each data packet to reply with an acknowledgment message to ensure correct receipt. However, this approach clearly wastes network bandwidth. Furthermore, due to network instability, the acknowledgment message sent by the destination end may be lost, further causing correctly received data packets to be retransmitted. This, in turn, wastes network resources and compromises the efficiency of distributed collective communication.

[0064] Therefore, according to an embodiment of the present invention, based on the multicast connection suitable for distributed collective communication established based on UCX in the early stage, a network RDMA multicast method suitable for distributed collective communication is designed.

[0065] As shown in Figures 5B and 5B (continued), this network RDMA multicast method can be roughly divided into three phases: information notification, data multicast, and packet receipt confirmation. Information notification primarily provides initialization preparation conditions for the subsequent two phases, including synchronizing data information about the data to be transmitted between the source and all destinations, and establishing corresponding packet sending and receiving status records. Data multicast primarily involves the source sending the data packets contained in the data to be transmitted via multicast. The source and all destinations will update their own packet sending and receiving status records in real time during this process. Packet receipt confirmation primarily confirms packet reception status based on the corresponding packet sending and receiving status records after the transmission of a specific data block has completed.

[0066] As shown in FIG5B , the information notification stage may specifically include the following steps ① to ④.

[0067] ① The source end 510 in the distributed collective communication divides the data to be transmitted into K data blocks according to the data size and a certain threshold. Due to the unreliability of using UD type QP to transmit data, it is recommended to use MTU as the maximum supported message. Therefore, each data block can send data according to the MTU size, and a total of Data packets. Among them, represents the ceiling operator, assuming If s is equal to s, each data block is further divided into s data packets.

[0068] The source end 510 in the distributed collective communication establishes a packet sending status record in the order of s data packets according to the number s of data packets contained in each data block. The packet sending status record can use a packet sending status map send_pkt_map that uses the serial number of the data packet as a position index to record the status identifier of the data packet at the corresponding position, as shown in Figure 6. N represents the number of destination ends in the distributed collective communication; the binary status identifiers pkt1, pkt2, ..., pkti, ..., pkts respectively identify the status of s data packets, and can take values ​​0 or 1. For example, if the status identifier pkti takes the value 0, it means that the i-th data packet has not been sent; and if the status identifier pkti takes the value 1, it means that the i-th data packet has been sent. I represents the serial number of the data packet, which is an integer greater than 0 and less than s.

[0069] ② When preparing to send a data packet via network RDMA multicast, the source 510 first constructs a multicast information announcement message containing the data information. The data information includes the source's virtual endpoint ID v_epid_0_0, the multicast ID groupid, and the information announcement tag tag0 at the beginning of the data field. The data information may include at least the amount of data to be transmitted (M), the K data blocks into which it is divided, the s data packets into which each data block is further divided, and the sequence number of each data packet.

[0070] Then, source 510 sends the multicast information announcement message based on the first UD-type QP created previously and starts the first timer. Simultaneously, source 510 initializes the status flags pkt1, pkt2, ..., pkts of the s packets in the packet transmission status record to 0. Subsequently, whenever source 510 sends a packet with the corresponding sequence number, it changes the corresponding status flag to 1.

[0071] ③ When the destination end 530-1, ... 530-N receives the multicast information notification message from the source end 510, the virtual endpoint ID v_epid_0_0, the multicast ID groupid and the information notification identifier tag0 carried in the multicast information notification message can be used to determine the data information that the source end 510 of the current multicast group needs to send, and establish a corresponding packet receiving status record. Among them, the packet receiving status record can use the packet receiving status map recv_pkt_map that uses the serial number of the data packet as the position index and records the status identifier of the data packet at the corresponding position, as shown in Figure 7. Among them, the status identifiers pkt1, pkt2...pkts of the s data packets in the packet receiving status record are initialized to 0. In this way, each time the destination end receives a data packet with the corresponding serial number, the corresponding status identifier can be changed to 1.

[0072] At the same time, each destination 530-1, ..., 530-N unicasts an information notification acknowledgment message (hereinafter referred to as an information notification ACK message) to the source 510 using the previously created second UD-type QP, allowing the source to confirm that the multicast information notification message has been correctly received by the destination. Specifically, the information notification ACK message sent by the destination 530-1, ..., 530-N carries the destination's own virtual endpoint ID v_epid_1_0, ..., v_epid_N_0 and the multicast ID groupid, allowing the source 510 to confirm packet receipt.

[0073] ④ If all the destination terminals 530-1, ..., 530-N reply information announcement ACK messages are collected within the timing range of the first timer (hereinafter also referred to as the first predetermined duration), the source terminal 510 begins to multicast the data message. If, at the end of the first timer (i.e., the first predetermined duration has elapsed since the multicast information announcement message was sent) all the destination terminals' reply information announcement ACK messages have not been collected (including no reply information announcement ACK messages from any destination terminal), the source terminal 510 will resend the multicast information announcement message and restart the first timer until all the destination terminals' reply information announcement ACK messages are collected.

[0074] It can be seen that collecting the information notification ACK messages replied by all destination ends 530-1, ... 530-N is a prerequisite for the source end 510 to start data multicast transmission. In particular, in order to avoid endless collection waiting due to possible loss of information notification ACK messages, the first timer can be used to constrain the waiting time. In other words, if the information notification ACK messages replied by all destination ends 530-1, ... 530-N are still not collected when the first timer expires, the source end 510 can infer that at least one destination end is unable to normally receive the multicast information notification message, and / or the information notification ACK message replied by at least one destination end is lost, and resend the multicast information notification message, and restart the first timer at the same time.

[0075] As shown in FIG. 5B and FIG. 5B (continued), the data multicast process mainly includes the following steps ⑤ and ⑥.

[0076] ⑤ Source 510 begins the multicast datagram transmission process. Specifically, source 510 constructs a multicast datagram based on the data packet to be sent. The multicast datagram begins with the source's virtual endpoint ID v_epid_0_0, the multicast ID groupid, and the datagram tag tag1. Source 510 then uses the previously created first UD-type QP to send the multicast datagram, ensuring that the multicast datagram can be forwarded to the destinations 530-1, ..., 530-N via the multicast function of the network switches.

[0077] At the same time, the source end 510 sets the status flag corresponding to the sent data packet in the packet sending status record to 1.

[0078] ⑥ When the destination terminals 530-1, ... 530-N receive the multicast data packet, they can correctly identify the packet through the virtual endpoint ID v_epid_0_0, multicast ID groupid, and data packet identifier tag1 carried by the packet, and set the corresponding position in their own packet reception status record according to the sequence number psn of the carried data packet.

[0079] Specifically, destinations 530-1, ..., 530-N set the status flags at the corresponding positions in their own packet reception status records to 1 based on the packet sequence numbers (PSNs) carried in the multicast data packets sent by source 510. For example, assuming that source 510 sends multiple packets with PSNs starting from 1 and increasing sequentially to s, and the multicast data packets received by destinations 530-1, ..., 530-N carry sequence number i, then destinations 530-1, ..., 530-N will set the status flag pkti at the corresponding position i in their own packet reception status records to 1.

[0080] The source end 510 and the destination ends 530-1, ... 530-N may repeat the above steps ⑤ and ⑥ respectively until all s data packets of the same data block are multicasted and sent.

[0081] Then, as described below, source 510 initiates packet receipt confirmation for this data block. After confirming that all destinations 530-1, ..., 530-N have correctly received all s packets of this data block, it terminates the multicast transmission process for this data block and initiates the multicast transmission process for the next data block. As shown in Figure 5B (continued), packet receipt confirmation for this data block primarily includes the following steps (7) through (9).

[0082] ⑦ When source 510 has sent all s packets of a data block, it needs to obtain packet receipt confirmation from the destination. At this time, source 510 constructs a multicast status announcement message, which carries the source's virtual endpoint ID v_epid_0_0, the multicast ID groupid, and the status announcement tag tag2 at the beginning of the data column, and starts the second timer.

[0083] ⑧ When destinations 530-1, ..., 530-N receive the multicast status notification message, they construct a status response message based on their own packet reception status records and unicast this status response message to source 510 using the previously created second UD-type QP. Specifically, this status response message also carries the destination's own virtual endpoint IDs v_epid_1_0, ..., v_epid_N_0 and the multicast ID groupid, allowing source 510 to confirm the message.

[0084] It should be noted that, when receiving the multicast status announcement message, destinations 530-1, ..., 530-N may not have received a packet of the data block, that is, may not have received all s packets of the data block. In this case, destinations 530-1, ..., 530-N construct a status response message based on their current packet reception status records and unicast it to source 510. This allows source 510 to accurately understand the packet reception status at the destination and trigger the retransmission of the packets that the destination has not yet received.

[0085] For example, assume that, upon receiving the multicast status announcement message, destination 530-t has already received the remaining s-1 packets in the data block, excluding the second packet. Meanwhile, upon receiving the multicast status announcement message, the remaining N-1 destinations 530-1, ..., 530-t-1, 530-t+1, ..., 530-N have successfully received all s packets in the data block. In other words, upon receiving the multicast status announcement message, the status flag pkt2 in the packet reception status record on destination 530-t takes on a value of 0, while the remaining s-1 status flags all take on a value of 1. Furthermore, all s status flags in the packet reception status records on destinations 530-1, ..., 530-t-1, 530-t+1, ..., 530-N take on a value of 1. In this case, the destinations 530-1, ..., 530-t-1, 530-t, 530-t+1, ..., 530-N will faithfully feed back the aforementioned data packet reception status information to the source 510 via status response messages. This will ensure that after the source 510 performs the operation in step 9 below, the status identifier corresponding to the second data packet in its packet transmission status record will be (1, 0), rather than (0, 1) indicating that all destinations correctly received the data packet. Accordingly, the source 510 will confirm that one destination did not receive the second data packet and will return to step 5 to retransmit the second data packet.

[0086] ⑨ If the source end 510 does not receive any status response message from the destination end within the timing range of the second timer (hereinafter also referred to as the second predetermined duration), it retransmits the multicast status announcement message. Otherwise, the source end 510 updates the packet sending status record based on the virtual endpoint IDs v_epid_1_0, ....v_epid_N_0 and the packet receiving status record carried in the received status response message, including performing an AND operation (hereinafter represented by the operator "^") on the corresponding position of the packet sending status record based on the packet receiving status record in the status response message.

[0087] Specifically, within the timing range of the second timer, each time a status response message is received from a destination, the source end 510 performs the following processing on each status identifier pkti in the packet receiving status record carried by the status response message: If the value of pkti is 1, then (J, pkti) in the packet sending status record is updated to (J-1, pkti^1); if the value of pkti is 0, then (J, pkti) in the packet sending status record is updated to (J, pkti^0). Here, i represents the sequence number of the data packet, which is an integer less than or equal to s and greater than or equal to 1; J is an integer less than or equal to N and greater than or equal to 1, which can be used to represent the number of destinations that did not receive the i-th data packet. That is, NJ can represent the count of destinations that correctly received the i-th data packet.

[0088] For example, assuming there is 1 source end and 3 destination ends (N=3), and each data block is divided into 5 data packets (s=5), after the source end 510 sends out all 5 data packets in a data block, the packet sending status is recorded as [(3,1), (3,1), (3,1), (3,1), (3,1)].

[0089] Next, suppose that the destination 530-1 has received all five data packets, and its packet receiving status record is [1,1,1,1,1]. After the source 510 receives the status response message from the destination 530-1, it updates the packet sending status record from [(3,1),(3,1),(3,1),(3,1),(3,1)] to [(2,1),(2,1),(2,1),(2,1),(2,1)] based on the packet receiving status record [1,1,1,1,1].

[0090] Assume that destination 530-2 receives four data packets except the second one, and the packet receiving status record is [1, 0, 1, 1, 1]. After receiving the status response message from destination 530-2, source 510 updates its packet sending status record from [(2, 1), (2, 1), (2, 1), (2, 1), (2, 1)] to [(1, 1), (2, 0), (1, 1), (1, 1), (1, 1)] based on the packet receiving status record [1, 0, 1, 1, 1].

[0091] If the destination 530-3 receives three packets in addition to the second and fifth packets, and the packet receiving status record is [1, 0, 1, 1, 0], then after the source 510 receives the status response message from the destination 530-3, it will further update the packet sending status record from [(1, 1), (2, 0), (1, 1), (1, 1), (1, 1)] to [(0, 1), (2, 0), (0, 1), (0, 1), (1, 0)] based on the packet receiving status record [1, 0, 1, 1, 0].

[0092] When the second timer expires (i.e., the second predetermined time has passed since the multicast status announcement message was sent), the source end 510 checks the status identifiers of each data packet in the packet sending status record in turn, and when it confirms that the status identifiers of all data packets are (0, 1), it ends the sending of the data block and clears the cache of the data block on itself. If there is a data packet with a status identifier other than (0, 1) in the packet sending status record, the source end 510 will again perform the operation of sending the multicast data packet using the first UD type QP in step ⑤ to retransmit the data packet. Accordingly, after the destination end receives the retransmitted data packet, if it has not received the data packet before, it will perform the reception processing as described above and update the corresponding position in its own packet receiving status record accordingly. At the same time, based on the updated packet receiving status record, it constructs the aforementioned status response message and unicasts it back to the source end 510, so that the source end 510 can update the packet sending status record accordingly; and if the data packet has been correctly received before, it can delete the duplicate message and does not need to reply the status response message to the source end again.

[0093] Continuing with the previous example, source 510 checks the packet status record [(0,1),(2,0),(0,1),(0,1),(1,0)] and finds that the first, third, and fourth data packets are correctly received by all destinations, but two destinations did not receive the second data packet, and one destination did not receive the fifth data packet. This triggers the multicast retransmission of the second and fifth data packets.

[0094] Accordingly, after receiving the multicast data packet of the retransmitted data packet, destination 530-2 will find that it has already received the fifth data packet and will ignore it. However, since it has not received the second data packet before, it will receive and process it and update its packet receiving status record from [1,0,1,1,1] to [1,1,1,1,1]. Destination 530-2 will then construct a new status response message based on its updated packet receiving status record and unicast it back to source 510. In response to this new status response message, source 510 can update its packet sending status record from [(0,1),(2,0),(0,1),(0,1),(1,0)] to [(0,1),(1,0),(0,1),(0,1),(1,0)]. It should be noted that when the source 510 updates the packet sending status record based on the updated packet receiving status record [1,1,1,1,1], it will only update the status flags other than (0,1) in the packet sending status record. This is because the status flag (0,1) indicates that the corresponding data packet has been correctly received by all destinations and therefore does not need to be updated.

[0095] Furthermore, after receiving the multicast data packet containing the retransmitted data packet, destination 530-3 discovers that it has not previously received the second and fifth data packets. It then processes the received data packets and updates its own packet receiving status record from [1, 0, 1, 1, 0] to [1, 1, 1, 1, 1]. Destination 530-3 then constructs a new status response message based on its updated packet receiving status record and unicasts it back to source 510. In response to this new status response message, source 510 updates its packet sending status record from [(0, 1), (1, 0), (0, 1), (0, 1), (1, 0)] to [(0, 1), (0, 1), (0, 1), (0, 1), (0, 1)]. Similarly, when source 510 updates its packet sending status record, it only updates the status flags that are not (0, 1).

[0096] At this point, source 510 can confirm, based on the packet transmission status record [(0,1),(0,1),(0,1),(0,1),(0,1)], that all five data packets belonging to the same data block have been correctly received by all destinations, thereby ending the multicast transmission process for that data block and initiating the multicast transmission process for the next data block. For example, source 510 can return to executing the operations in step 5 above until all the data to be sent, as indicated by the multicast information announcement message in step 2, has been sent and correctly received by all destinations.

[0097] Finally, in combination with FIG5B and FIG5B (continued), it can be seen that according to the network RDMA multicast scheme applicable to distributed collective communication according to an embodiment of the present invention, in the data multicast phase, the source sends a multicast data message carrying the virtual endpoint ID and multicast ID of the source via the first UD type QP, so as to use the network switch to complete the replication and distribution of the multicast message, thereby avoiding the repeated transmission of the same data, not only saving the bandwidth from the source to the network switch, but also effectively reducing the transmission delay. In addition, by performing information synchronization in the information notification phase in which the data to be transmitted is divided into multiple data blocks, and each data block is further divided into multiple data packets, and in the packet confirmation phase, based on the packet sending status record and the packet receiving status record updated in real time in the data multicast phase, the transmission confirmation of the data packet is performed in units of data blocks, not only significantly reducing the number of confirmed messages, but also avoiding the waiting feedback time caused by the real-time feedback of packet loss information, thereby effectively improving the efficiency of distributed collective communication.

[0098] Based on the detailed introduction of the above specific examples, according to an embodiment of the present disclosure, a multicast method based on RDMA is provided, which can be applied to a source end in distributed collective communication. As shown in Figure 8, the multicast method may include: step S810, sending a multicast message based on a first UD type QP, wherein the multicast message carries a first virtual endpoint ID and a first multicast ID, so that the multicast message is forwarded to N destination ends in the distributed collective communication via the multicast function of the network switch, N is an integer greater than or equal to 2, and each destination end processes the multicast message after determining that the first multicast ID carried by the multicast message is the same as its own second multicast ID and identifying that the multicast message comes from the source end based on the first endpoint ID; and step S820, when receiving a unicast message sent by the destination end based on a second UD type QP, after determining that the second multicast ID carried by the unicast message is the same as the first multicast ID and identifying that the unicast message comes from the destination end based on the second endpoint ID carried by the unicast message, processing the unicast message.

[0099] Optionally, as shown in FIG8 , before step S810 , the method further includes: pre-creating a first UD type QP and a second UD type QP at the source end (step S810p1 ); and pre-generating a first multicast ID and a first virtual endpoint ID at the source end (step S810p2 ).

[0100] Optionally, step S810p2 specifically includes: randomly generating a first virtual endpoint ID; generating the first multicast ID according to a predetermined rule based on the multicast IP address; and sending the multicast IP address to each destination end, so that each destination end generates a second multicast ID according to the predetermined rule based on the multicast IP address. In this way, the first multicast ID of the source end and the second multicast ID of each destination end are exactly the same, and can be used to identify the message within the multicast group constructed by the source end and all the destination ends. In addition, the first virtual endpoint ID of the source end and the second virtual endpoint ID of each destination end are different from each other, and can be used to identify the specific source of the message within the multicast group.

[0101] Optionally, referring to Figure 5B, before sending a multicast data message carrying a data message identifier and a data packet to be sent based on the first UD type QP (step ⑤), the method also includes: sending a multicast information announcement message based on the first UD type QP (step ②), wherein the multicast information announcement message carries an information announcement identifier and data information related to the data to be transmitted, the data information includes the data volume M of the data to be transmitted, the data to be transmitted is divided into K data blocks, each data block is further divided into s data packets, and the sequence number of each data packet, K is an integer greater than or equal to 1, and s is an integer greater than or equal to 1; and, receiving information announcement confirmation messages returned by N destination ends based on the second UD type QP (step ③), wherein each information announcement confirmation message carries a confirmation identifier indicating that the destination end has received the multicast information announcement message.

[0102] Further, as mentioned above, in order to constrain the waiting time for receiving N information notification confirmation messages, referring to Figure 5B, receiving information notification confirmation messages returned by N destination ends based on the second UD type QP may specifically include: judging whether the number of received information notification confirmation messages is less than N at the end of the first predetermined time period (step ④), wherein the first predetermined time period starts from the time when the multicast information notification message is sent; if the result of the above judgment is yes, resend the multicast information notification message, and restart the timing of the first predetermined time period to return to the above judgment step ④.

[0103] In addition, by maintaining and updating the packet receiving status record on each destination end, the source end can perform batch packet receiving confirmation in units of data blocks based on its own packet sending record and the packet receiving status record on each destination end. Specifically, referring to FIG5B (continued), for each data block in the K data blocks, after sending all the to-be-sent data packets contained in the data block, the method further includes: sending a multicast status notification message for the data block based on the first UD type QP, wherein the multicast status notification message carries a status notification identifier (step ⑦); receiving a status response message based on the second UD type QP, wherein the status response message carries the packet receiving status record of the destination end itself (step ⑧); and determining whether the N destination ends have received the sent data packets under the data block based on the sending records of each data packet under the data block and the packet receiving status record in the received status response message (step ⑨).

[0104] Optionally, determining whether N destination ends have received the sent data packets under the data block based on the sending records of each data packet under the data block and the packet receiving status records in the received status response message may specifically include: judging whether each sent data packet under the data block is received by all N destination ends based on the sending records of each data packet under the data block and the packet receiving status records carried by each status response message received within a second predetermined time period, wherein the second predetermined time period starts from the sending of the multicast status announcement message; if it is determined that there is a data packet that has not been received by all N destination ends, sending a multicast data message carrying the data packet based on the first UD type QP, and restarting the timing of the second predetermined time period; returning to the above-mentioned judgment process until it is determined that all data packets included in the data block have been received by the N destination ends.

[0105] Further, as mentioned above, in order to constrain the waiting time for receiving the response status message, as shown in Figure 5B (continued), if no status response message is received at the end of the second predetermined time period, the method also includes: resending the multicast status announcement message and restarting the timing of the second predetermined time period.

[0106] Accordingly, according to an embodiment of the present disclosure, a multicast method based on RDMA is provided, which can be applied to one of N destination ends in distributed collective communication. As shown in Figure 9, the multicast method may include: step S910, receiving a multicast message based on a first UD type QP, wherein the multicast message carries the first virtual endpoint ID and the first multicast ID of the source end in the distributed collective communication; step S920, after determining that the first multicast ID carried by the multicast message is the same as its own second multicast ID and identifying that the multicast message comes from the source end based on the first endpoint ID, processing the multicast message; step S930, when replying to the source end with the processing result of the multicast message, sending a unicast message to the source end based on the second UD type QP, wherein the unicast message carries the second virtual endpoint ID and the second multicast ID.

[0107] Optionally, as shown in FIG9 , before step S910 , the method further includes: pre-creating a first UD type QP and a second UD type QP at the destination end (step S910p1 ); and pre-generating a second multicast ID and a second virtual endpoint ID at the destination end (step S910p2 ).

[0108] Optionally, step S910p2 specifically includes: generating a second virtual endpoint ID according to a predetermined rule based on the destination's own ranking value; and receiving a multicast IP address sent by the source, and generating a second multicast ID based on the multicast IP address according to the same rule used to generate the first multicast ID by the source. In this way, the source's first multicast ID and the second multicast IDs of each destination are identical and can be used to identify messages within the multicast group established by the source and all destinations. Furthermore, the source's first virtual endpoint ID and the second virtual endpoint IDs of each destination are different and can be used to identify the specific source of messages within the multicast group.

[0109] Optionally, referring to Figure 5B, before receiving a multicast data message carrying a data message identifier and a data packet to be sent based on the first UD type QP (step ⑤), the method also includes: receiving a multicast information announcement message based on the first UD type QP (step ②), wherein the multicast information announcement message carries an information announcement identifier and data information related to the data to be transmitted, the data information includes the data volume M of the data to be transmitted, the data to be transmitted is divided into K data blocks, each data block is further divided into s data packets, and the sequence number of each data packet, K is an integer greater than or equal to 1, and s is an integer greater than or equal to 1; sending an information announcement confirmation message to the source end based on the second UD type QP (step ③), wherein the information announcement confirmation message carries a confirmation identifier indicating that the destination end has received the multicast information announcement message.

[0110] As shown in FIG5B , processing the multicast information announcement message may specifically include: establishing a packet receiving status record based on the information announcement identifier carried by the multicast information announcement message and the data information (step ③), wherein the packet receiving status record includes a status identifier corresponding to each data packet, and the status identifier is recorded at a position corresponding to the sequence number of the data packet. Furthermore, processing the multicast data packet may specifically include: receiving and processing the data packet carried by the multicast data packet based on the data packet identifier carried by the multicast data packet, and updating the status identifier corresponding to the data packet in the destination end's own packet receiving status record (step ⑥).

[0111] By maintaining and updating the packet receiving status record on each destination end, the source end can perform batch packet receiving confirmation in units of data blocks based on its own packet sending record and the packet receiving status record on each destination end. Specifically, as shown in Figure 5B (continued), after receiving the multicast data packet based on the first UD type QP (step ⑥), the method may further include: receiving a multicast status notification message based on the first UD type QP, wherein the multicast status notification message carries a status notification identifier (step ⑦); and generating a status response message based on the multicast status notification message, and sending the status response message to the source end based on the second UD type QP (step ⑧), wherein the status response message carries the packet receiving status record of the destination end itself.

[0112] The above are one or more implementation methods of the present disclosure. Based on the same idea, the present disclosure also provides a corresponding distributed collective communication node device. Referring to Figure 4, the node device mainly includes a processor such as a CPU and a GPU and a network card that supports RDMA.

[0113] Specifically, when the node device acts as the source end 410 in the distributed collective communication, the processor is configured to: send a multicast message via the network card based on the first UD type QP, wherein the multicast message carries a first virtual endpoint ID and a first multicast ID, so that the multicast message is forwarded to N destination ends in the distributed collective communication via the multicast function of the network switch, N is an integer greater than or equal to 2, and each destination end processes the multicast message after determining that the first multicast ID carried by the multicast data message is the same as its own second multicast ID and identifying that the multicast message comes from the source end based on the first endpoint ID; and, when a unicast message sent by the destination end is received via the network card based on the second UD type QP, the unicast message is processed after determining that the second multicast ID carried by the unicast message is the same as the first multicast ID and identifying that the unicast message comes from the destination end based on the second endpoint ID carried by the unicast message.

[0114] In addition, when the node device serves as the destination end 430-1, 430-2, ..., 430-N in the distributed collective communication, the processor is configured to: based on the first UD type QP, receive a multicast message via the network card, wherein the multicast message carries the first virtual endpoint ID and the first multicast ID of the source end in the distributed collective communication; after determining that the first multicast ID carried by the multicast message is the same as the second multicast ID of the destination end itself, and identifying that the multicast message comes from the source end based on the first endpoint ID, process the multicast message; and, when replying the processing result of the multicast message to the source end, based on the second UD type QP, send a unicast message to the source end via the network card, wherein the unicast message carries the second virtual endpoint ID and the second multicast ID, so that the source end processes the unicast message after determining that the second multicast ID carried by the unicast message is the same as the first multicast ID, and identifying that the unicast message comes from the destination end based on the second endpoint ID.

[0115] According to an embodiment of the present disclosure, another distributed collective communication node device is provided. As shown in FIG10 , the node device 1000 mainly includes a transceiver module 1010 and a processing module 1020 .

[0116] Specifically, when the node device 1000 acts as the source end 410 in the distributed collective communication, the processing module 1020 is configured to: send a multicast message via the transceiver module 1010 based on the first UD type QP, wherein the multicast message carries a first virtual endpoint ID and a first multicast ID, so that the multicast message is forwarded to N destination ends in the distributed collective communication via the multicast function of the network switch, N is an integer greater than or equal to 2, and each destination end processes the multicast message after determining that the first multicast ID carried by the multicast data message is the same as its own second multicast ID and identifying that the multicast message comes from the source end based on the first endpoint ID; and, when a unicast message sent by the destination end is received via the transceiver module 1010 based on the second UD type QP, the unicast message is processed after determining that the second multicast ID carried by the unicast message is the same as the first multicast ID and identifying that the unicast message comes from the destination end based on the second endpoint ID carried by the unicast message.

[0117] In addition, when the node device 1000 serves as the destination end 430-1, 430-2, ..., 430-N in the distributed collective communication, the processing module 1020 is configured to: based on the first UD type QP, receive the multicast message via the transceiver module 1010, wherein the multicast message carries the first virtual endpoint ID and the first multicast ID of the source end in the distributed collective communication; after determining that the first multicast ID carried by the multicast message is the same as the second multicast ID of the destination end itself, and based on the first endpoint I After D identifies that the multicast message comes from the source end, it processes the multicast message; and when it is to reply to the source end with the processing result of the multicast message, based on the second UD type QP, it sends a unicast message to the source end via the transceiver module 1010, wherein the unicast message carries the second virtual endpoint ID and the second multicast ID, so that the source end processes the unicast message after determining that the second multicast ID carried by the unicast message is the same as the first multicast ID and identifying that the unicast message comes from the destination end based on the second endpoint ID.

[0118] For example, the node devices shown in Figures 4 and 10 can be terminal devices such as desktop computers and laptops, or servers or other specialized devices. The modules shown in Figure 10 can be hardware-implemented modules, software-implemented modules, or a combination of hardware and software to achieve the described functions. These modules are merely a division of logical functions; in actual implementation, they can be fully or partially integrated into a single physical entity or physically separated. Furthermore, these modules can be implemented entirely as software invoked by a processing element, entirely as hardware, or partially as software invoked by a processing element, while others can be implemented in hardware. For example, the determination module can be a separate processing element, integrated into a chip within the aforementioned device, or stored as program code in the memory of the aforementioned device, invoked by a processing element within the device to perform the functions of the determination module. The implementation of the other modules is similar. Furthermore, these modules can be fully or partially integrated or implemented independently. The processing element described herein can be an integrated circuit with signal processing capabilities. During implementation, each step of the above method or each module above may be completed by an integrated logic circuit of hardware in a processor element or by instructions in the form of software.

[0119] For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASICs), one or more digital signal processors (DSPs), or one or more field programmable gate arrays (FPGAs). For another example, when a module is implemented by scheduling program code on a processing element, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0120] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present disclosure is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)).

[0121] The present disclosure further provides a computer-readable storage medium having a computer program stored thereon, which can be used to execute the RDMA-based multicast method provided in Figure 8 or Figure 9. The storage medium can be a non-volatile storage medium.

[0122] Furthermore, at the hardware level, the node device shown in Figure 4 may include not only a processor and network interface card, but also memory and non-volatile storage. It may also include other hardware required for services, such as an internal bus. The processor reads the corresponding computer program from the non-volatile storage into the memory and then runs it, implementing the RDMA-based multicast method described in Figures 8 and 9.

[0123] Of course, in addition to software implementation, this disclosure does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0124] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0125] The various embodiments of this disclosure are described in a progressive manner. Similar portions between the various embodiments can be referenced to each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiments are generally similar to the method embodiments, so their description is relatively simple. For relevant portions, refer to the description of the method embodiments.

[0126] The foregoing is merely an embodiment of the present disclosure and is not intended to limit the present disclosure. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure are intended to be included within the scope of the claims of the present disclosure.

Claims

1. A multicast method based on Remote Direct Data Access (RDMA), applied to a source end in distributed collective communication, the method comprising: Sending a multicast message based on a first unreliable datagram UD type queue pair QP, wherein the multicast message carries a first virtual endpoint ID and a first multicast ID, so that the multicast message is forwarded to N destination ends in the distributed collective communication via a multicast function of a network switch, where N is an integer greater than or equal to 2, and each destination end processes the multicast message after determining that the first multicast ID carried by the multicast message is the same as its own second multicast ID and identifying that the multicast message comes from the source end based on the first endpoint ID; and When a unicast message sent by the destination end is received based on the second UD type QP, after determining that the second multicast ID carried by the unicast message is the same as the first multicast ID, and identifying that the unicast message comes from the destination end based on the second endpoint ID carried by the unicast message, the unicast message is processed.

2. The method according to claim 1, characterized in that Before sending the multicast message based on the first UD type QP, the method further includes: Generate multicast Internet Protocol IP addresses; generating the first multicast ID according to the multicast IP address and a predetermined rule; and The multicast IP address is sent to each destination end, so that each destination end generates the second multicast ID according to the multicast IP address and the predetermined rule.

3. The method according to claim 1 or 2, characterized in that Before sending the multicast message based on the first UD type QP, the method further includes: The first virtual endpoint ID is randomly generated.

4. The method according to any one of claims 1 to 3, characterized in that The sending of the multicast message based on the first UD type QP specifically includes: A multicast data message is sent based on the first UD type QP, wherein the multicast data message carries a data message identifier and a data packet to be sent.

5. The method according to claim 4, characterized in that Before sending the multicast data packet based on the first UD type QP, the method further includes: Sending a multicast information announcement message based on the first UD type QP, wherein the multicast information announcement message carries an information announcement identifier and data information related to the data to be transmitted; the data information includes a data volume M of the data to be transmitted, K data blocks into which the data to be transmitted is divided, each of the data blocks is further divided into s data packets, and a sequence number of each data packet, where K is an integer greater than or equal to 1, and s is an integer greater than or equal to 1; The information announcement confirmation messages returned by the N destination ends are received based on the second UD type QP, wherein each of the information announcement confirmation messages carries a confirmation identifier indicating that the destination end has received the multicast information announcement message.

6. The method according to claim 5, characterized in that The receiving, based on the second UD type QP, the information notification confirmation messages returned by the N destination ends specifically includes: At the end of a first predetermined time period, determining whether the number of received information notification acknowledgment messages is less than N, wherein the first predetermined time period is calculated from the time when the multicast information notification message is sent; If the result of the above judgment is yes, the multicast information notification message is resent and the timing of the first predetermined time length is restarted to return to the step of judging whether the number of the received information notification acknowledgment messages is less than N.

7. The method according to claim 5 or 6, characterized in that The sending of the multicast data packet based on the first UD type QP specifically includes: For each data packet to be sent included in each data block in the K data blocks, a multicast data message carrying the data packet is sent based on the first UD type QP.

8. The method according to claim 7, characterized in that For each data block in the K data blocks, after all to-be-sent data packets contained in the data block are sent, the method further includes: Sending a multicast status notification message for the data block based on the first UD type QP, wherein the multicast status notification message carries a status notification identifier; receiving a status response message based on the second UD type QP, wherein the status response message carries a packet receiving status record of the destination end itself; According to the sending record of each data packet under the data block and the packet receiving status record in the received status response message, it is determined whether the N destination ends have received the sent data packets under the data block.

9. The method according to claim 8, characterized in that The determining, based on the sending record of each data packet under the data block and the packet receiving status record in the received status response message, whether the N destination ends have received the sent data packets under the data block specifically includes: Determining whether each sent data packet under the data block is received by all N destination terminals based on the sending record of each data packet under the data block and the packet reception status record carried by each status response message received within a second predetermined time period, wherein the second predetermined time period is calculated from the sending of the multicast status announcement message; If it is determined that there is a data packet that has not been received by all N destination ends, sending a multicast data packet carrying the data packet based on the first UD type QP, and restarting the timing of the second predetermined time length; Return to the process of determining whether the sent data packet is received by all N destination ends according to the packet receiving status record carried in the received status response message, until it is determined that all data packets included in the data block have been received by the N destination ends.

10. The method according to claim 9, characterized in that If no status response message is received before the second predetermined time period ends, the method further includes: Resend the multicast status notification message, and Restart the timing of the second predetermined time period.

11. A multicast method based on Remote Direct Data Access (RDMA), applied to one of N destinations in distributed collective communication, where N is an integer greater than or equal to 2, comprising: Receiving a multicast message based on a first unreliable datagram UD type queue pair QP, wherein the multicast message carries a first virtual endpoint ID and a first multicast ID of a source end in the distributed collective communication; After determining that the first multicast ID carried by the multicast message is the same as the second multicast ID of the destination end, and identifying that the multicast message comes from the source end based on the first endpoint ID, processing the multicast message; and When the processing result of the multicast message is to be replied to the source end, a unicast message is sent to the source end based on the second UD type QP, wherein the unicast message carries the second virtual endpoint ID and the second multicast ID, so that the source end processes the unicast message after determining that the second multicast ID carried by the unicast message is the same as the first multicast ID and identifying that the unicast message comes from the destination end based on the second endpoint ID.

12. The method according to claim 11, characterized in that Before receiving the multicast message based on the first UD type QP, the method further includes: Receiving a multicast Internet Protocol (IP) address sent by the source end; The second multicast ID is generated according to the multicast IP address and in accordance with the same rule as that used by the source end to generate the first multicast ID.

13. The method according to claim 11 or 12, characterized in that Before receiving the multicast message based on the first UD type QP, the method further includes: The second virtual endpoint ID is generated according to the ranking value of the destination end itself and in accordance with a predetermined rule.

14. The method according to any one of claims 11 to 13, characterized in that The receiving a multicast message based on the first UD type QP specifically includes: A multicast data message is received based on the first UD type QP, wherein the multicast data message carries a data message identifier and a data packet to be sent.

15. The method according to claim 14, characterized in that Before receiving the multicast data packet based on the first UD type QP, the method further includes: Receiving a multicast information announcement message based on the first UD type QP, wherein the multicast information announcement message carries an information announcement identifier and data information related to data to be transmitted; the data information includes a data volume M of the data to be transmitted, K data blocks into which the data to be transmitted is divided, each of the data blocks is further divided into s data packets, and a sequence number of each of the data packets, where K is an integer greater than or equal to 1, and s is an integer greater than or equal to 1; An information announcement confirmation message is sent to the source end based on the second UD type QP, wherein the information announcement confirmation message carries an confirmation flag indicating that the destination end has received the multicast information announcement message.

16. The method according to claim 15, characterized in that Processing the multicast information announcement message includes: establishing a packet receiving status record according to the information announcement identifier and the data information carried by the multicast information announcement message, wherein the packet receiving status record includes a status identifier corresponding to each data packet, and the status identifier is recorded at a position corresponding to the sequence number of the data packet; Processing the multicast data message includes: receiving and processing the data packet carried by the multicast data message according to the data message identifier carried by the multicast data message, and updating the status identifier corresponding to the data packet in the packet receiving status record of the destination end itself.

17. The method according to claim 16, characterized in that After receiving the multicast data packet based on the first UD type QP, the method further includes: receiving a multicast status notification message based on the first UD type QP, wherein the multicast status notification message carries a status notification identifier; Generate a status response message according to the multicast status notification message, wherein the status response message carries the packet receiving status record of the destination end itself; and Send the status response message to the source end based on the second UD type QP.

18. A node device for distributed collective communication, serving as a source end in the distributed collective communication, comprising a processor and a network card supporting remote direct data access (RDMA), characterized in that: The processor is configured to: Based on a first unreliable datagram UD type queue pair QP, sending a multicast message via the network card, wherein the multicast message carries a first virtual endpoint ID and a first multicast ID, so that the multicast message is forwarded to N destination ends in the distributed collective communication via the multicast function of the network switch, where N is an integer greater than or equal to 2, and each destination end processes the multicast message after determining that the first multicast ID carried by the multicast data message is the same as its own second multicast ID and identifying that the multicast message comes from the source end based on the first endpoint ID; and When a unicast message sent by the destination end is received via the network card based on the second UD type QP, after determining that the second multicast ID carried by the unicast message is the same as the first multicast ID, and identifying that the unicast message comes from the destination end based on the second endpoint ID carried by the unicast message, the unicast message is processed.

19. A node device for distributed collective communication, serving as one of N destinations in the distributed collective communication, comprising a processor and a network card supporting remote direct data access (RDMA), characterized in that: The processor is configured to: Based on a first unreliable datagram UD type queue pair QP, receiving a multicast message via the network card, wherein the multicast message carries a first virtual endpoint ID and a first multicast ID of a source end in the distributed collective communication; After determining that the first multicast ID carried by the multicast message is the same as the second multicast ID of the destination end, and identifying that the multicast message comes from the source end based on the first endpoint ID, processing the multicast message; and When a processing result of the multicast message is to be replied to the source end, a unicast message is sent to the source end via the network card based on the second UD type QP, wherein the unicast message carries a second virtual endpoint ID and the second multicast ID, so that the source end processes the unicast message after determining that the second multicast ID carried by the unicast message is the same as the first multicast ID and identifying that the unicast message comes from the destination end based on the second endpoint ID.

Images