Ensemble communication method and device, computer equipment, storage medium and program product

Abstract

The invention relates to the technical field of communication, and discloses a set communication method and device, computer equipment, a storage medium and a program product, and the set communication method comprises the steps: obtaining a sending request transmitted by a receiving end; wherein the sending request is generated based on a plurality of available queue pairs obtained by the receiving end based on the RECV request; determining a plurality of available queue pairs based on the sending request, and transmitting the to-be-transmitted data to a receiving end in parallel through the plurality of available queue pairs by using an adaptive traffic affinity scheduling strategy; and after the parallel transmission of the to-be-transmitted data is completed, checking the transmission operation of the sending end, and if the transmission operation of the sending end fails, retransmitting the to-be-transmitted data by using a retransmission strategy. According to the method, the availability of the link is guaranteed to the maximum extent, the function of avoiding the fault link is realized, and the training efficiency and reliability of the large model are greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of communication technology, and in particular to a collective communication method, apparatus, computer equipment, storage medium and program product. Background Art

[0002] With the explosive growth of computing power and data volume, large-scale model training has ushered in a new stage in the development of artificial intelligence. Large-scale model training often requires the collaborative work of thousands, tens of thousands, or even more than 100,000 GPUs (Graphics Processing Units). As the communication bridge between GPUs, the collective communication library has become a key factor affecting the efficiency of large-scale model training, and its communication efficiency and reliability face huge challenges.

[0003] Since the implementation of the relevant collective communication library cannot guarantee bandwidth maximization, the training time of large models is long, and the training efficiency and reliability of large models cannot be guaranteed. Summary of the Invention

[0004] In view of this, the present invention provides a collective communication method, apparatus, computer equipment, storage medium and program product to solve the problem of being unable to guarantee the training efficiency and reliability of large models.

[0005] In a first aspect, the present invention provides a collective communication method, applied to a sending end, the method comprising:

[0006] Obtaining a sending request transmitted by a receiving end; wherein the sending request is generated by the receiving end based on the multiple available queue pairs obtained by the receiving end based on the RECV request;

[0007] Based on the sending request, multiple available queue pairs are determined, and the data to be transmitted is transmitted to the receiving end in parallel through multiple available queue pairs using an adaptive traffic affinity scheduling strategy;

[0008] After the parallel transmission of the data to be transmitted is completed, the transmission operation of the sending end is checked. If the transmission operation of the sending end fails, the data to be transmitted is retransmitted using a retransmission strategy.

[0009] This embodiment provides a collective communication method that utilizes an adaptive traffic affinity scheduling strategy to transmit the data to be transmitted in parallel to the receiving end through multiple available queue pairs. After the parallel transmission of the data to be transmitted is completed, the transmission operation of the sending end is checked. If the transmission operation of the sending end fails, the data to be transmitted is retransmitted using a retransmission strategy. By using the adaptive traffic affinity scheduling strategy and the retransmission strategy, multiple available queue pairs are used for parallel transmission, thereby maximizing the link availability, realizing the function of avoiding faulty links, and greatly improving the efficiency and reliability of large model training.

[0010] In an optional implementation, determining multiple available queue pairs based on a sending request and using an adaptive traffic affinity scheduling strategy to transmit the data to be transmitted to the receiving end in parallel through the multiple available queue pairs includes:

[0011] Obtain the amount of data allocated to multiple available queue pairs in the last transmission cycle, and the data transmission time of each available queue pair in the last transmission cycle;

[0012] Calculate a score value for each available queue pair based on the amount of data allocated to the multiple available queue pairs in the previous transmission cycle and the data transmission time of each available queue pair in the previous transmission cycle;

[0013] The data to be transmitted is distributed to a plurality of available queue pairs based on the score value of each available queue pair, so that the data to be transmitted is transmitted to the receiving end in parallel by using the plurality of available queue pairs.

[0014] This embodiment provides a collective communication method that calculates the score value of each available queue pair and then distributes the data to be transmitted to multiple available queue pairs based on the score value of each available queue pair. When a hash collision or other reasons cause congestion on some links, the data volume of the available queue pairs is reasonably distributed through an adaptive traffic affinity scheduling strategy, actively adapting to the bandwidth and load of each link to maximize the final bandwidth, avoiding traffic hash collisions, improving bandwidth utilization, saving training time, and significantly improving the efficiency and reliability of large model training.

[0015] In a second aspect, the present invention provides a collective communication method, applied to a receiving end, the method comprising:

[0016] Create a RECV request and obtain multiple available queue pairs based on the RECV request;

[0017] generating a send request based on a plurality of available queue pairs, and transmitting the send request to a sender;

[0018] Receive feedback data transmitted by the sender; the feedback data is generated by the sender determining multiple available queue pairs based on the send request, and using the adaptive traffic affinity scheduling strategy to transmit the data to be transmitted in parallel through multiple available queue pairs.

[0019] This embodiment provides a collective communication method, in which the receiving end obtains multiple available queue pairs through RECV requests, and the sending end uses an adaptive traffic affinity scheduling strategy to transmit the data to be transmitted in parallel through multiple available queue pairs. Then, the receiving end receives the feedback data transmitted by the sending end and uses the available queue pairs for parallel transmission, thereby maximizing link availability and improving overall transmission efficiency.

[0020] In an optional implementation, creating a RECV request and obtaining multiple available queue pairs based on the RECV request include:

[0021] Determining a plurality of queue pairs based on the RECV request;

[0022] Obtain the status identifier of each queue pair, and detect queue pairs in abnormal states based on the status identifier of each queue pair;

[0023] A standby queue pair is obtained, and a queue pair in an abnormal state among the multiple queue pairs is replaced with the standby queue pair, thereby obtaining multiple available queue pairs.

[0024] This embodiment provides a collective communication method. When a link fails, an abnormal queue pair among multiple queue pairs is replaced with a standby queue pair, and multiple available queue pairs are used for parallel transmission. During the transmission process, the transmission quality of the queue pairs is sensed, and interrupted queue pairs are eliminated, thereby avoiding single point failures, reducing or eliminating the superposition effect of single point failures, improving overall reliability, reducing the waste of computing resources caused by link failures, saving computing resources and time costs, and improving overall transmission efficiency.

[0025] In an optional embodiment, the method further includes:

[0026] Check the transmission operation of the receiving end. If the transmission operation of the receiving end fails, retransmit the sending request using the retransmission strategy.

[0027] This embodiment provides a collective communication method that checks the transmission operation of the receiving end and then uses a retransmission strategy to retransmit the sending request. This allows the receiving end to resend lost or damaged data when a problem with data transmission is found, ensuring that the sending end can correctly receive complete information, thereby improving the reliability of data transmission.

[0028] In a third aspect, the present invention provides a collective communication device, applied to a transmitting end, the device comprising:

[0029] An acquisition module, configured to acquire a sending request transmitted by a receiving end; wherein the sending request is generated by the receiving end based on a RECV request to obtain multiple available queue pairs and based on the multiple available queue pairs;

[0030] A transmission module is used to determine multiple available queue pairs based on the sending request, and use an adaptive traffic affinity scheduling strategy to transmit the data to be transmitted in parallel to the receiving end through the multiple available queue pairs;

[0031] The checking module is used to check the transmission operation of the sending end after the parallel transmission of the data to be transmitted is completed. If the transmission operation of the sending end fails, the data to be transmitted is retransmitted using a retransmission strategy.

[0032] In a fourth aspect, the present invention provides a collective communication device, applied to a receiving end, the device comprising:

[0033] Create a module and create a RECV request, and obtain multiple available queue pairs based on the RECV request;

[0034] a generating module, configured to generate a sending request based on a plurality of available queue pairs, and transmit the sending request to a sending end;

[0035] The receiving module is used to receive feedback data transmitted by the sending end; the feedback data is generated by the sending end by determining multiple available queue pairs based on the sending request, and using the adaptive traffic affinity scheduling strategy to transmit the data to be transmitted in parallel through multiple available queue pairs.

[0036] In a fifth aspect, the present invention provides a computer device comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, computer instructions being stored in the memory, and the processor executing the collective communication method of the above-mentioned first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0037] In a sixth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the collective communication method of the first aspect or any corresponding embodiment thereof.

[0038] In a seventh aspect, the present invention provides a computer program product comprising computer instructions, wherein the computer instructions are used to enable a computer to execute the collective communication method of the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0040] Figure 1 This is a flow chart of a collective communication method applied to a sending end according to an embodiment of the present invention;

[0041] Figure 2 is a schematic diagram of the physical structure of data transmission between nodes corresponding to the inter-machine collective communication operation according to an embodiment of the present invention;

[0042] Figure 3 is a schematic diagram of the logical structure of data transmission between nodes corresponding to the inter-machine collective communication operation according to an embodiment of the present invention;

[0043] Figure 4 is a flow chart of another collective communication method applied to a sending end according to an embodiment of the present invention;

[0044] Figure 5 1 is a flow chart of a collective communication method applied to a receiving end according to an embodiment of the present invention;

[0045] Figure 6 is a flow chart of another collective communication method applied to a receiving end according to an embodiment of the present invention;

[0046] Figure 7 is a schematic diagram of data transmission with automatic avoidance of failed links and adaptive traffic affinity scheduling according to an embodiment of the present invention;

[0047] Figure 8 This is a schematic diagram of automatically removing a faulty link when a switch experiences a pseudo-dead state or an imperceptible fault occurs on a network link according to an embodiment of the present invention;

[0048] Figure 9 is a schematic diagram of automatically performing traffic affinity scheduling when a hash collision or other reasons cause congestion on some links according to an embodiment of the present invention;

[0049] Figure 10 This is a structural block diagram of a collective communication device applied to a transmitting end according to an embodiment of the present invention;

[0050] Figure 11 This is a structural block diagram of a collective communication device applied to a receiving end according to an embodiment of the present invention;

[0051] Figure 12 Schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0052] To make the purpose, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative efforts shall fall within the scope of protection of the present invention.

[0053] Collective communication is the communication between nodes in parallel computing. Nodes often form a specific topology, and communication efficiency is improved through carefully designed communication timing. The Collective Communication Library (CCL) is the software implementation of collective communication, provided as a software module to upper-level applications. As a communication bridge between GPUs, the CCL has become a key factor affecting the training efficiency of large models, and its communication efficiency and reliability face significant challenges.

[0054] On the one hand, training large models means a larger amount of communication data, and the communication rate between GPUs is directly related to the model training time. In addition to the huge amount of data, collective communication also has the characteristic of extremely small number of data streams. Once a hash collision occurs, it will inevitably lead to bandwidth contention in the local link, and then cause the local data transmission time to increase exponentially. During a round of collective communication, the overall task must wait until all local communications are completed. Therefore, local bandwidth contention will cause the overall training time to increase exponentially.

[0055] Assuming the hash algorithm is optimal, given N links and k flows, the approximate formula for hash collision probability (there is no exact elementary function formula) is as follows:

[0056]

[0057] It has been calculated that in the case of 64 links, when the number of data streams reaches 25, the probability of hash collision is close to 100%. It can be seen that hash collision is the norm rather than a low-probability event. Therefore, maximizing communication bandwidth utilization is one of the key technologies to improve the efficiency of large model training.

[0058] In order to solve the hash collision problem, methods such as stream orchestration and micro-granularization have been adopted. However, the above methods have obvious shortcomings, namely, the implementation scheme is complex and requires special hardware support.

[0059] On the other hand, large-scale model training requires very high stability. Local communication failures will directly lead to training interruptions. Once training is interrupted, it takes a very long time to restart the training task, which in turn brings very great economic pressure and wastes time. Therefore, minimizing the number of training interruptions is also one of the key technologies to improve the efficiency of large-scale model training.

[0060] Assuming that the probability of a single link failure in the network environment that the large model training relies on is p, the probability P of n links failing is:

[0061] P=1-(1-p) n (2)

[0062] Calculations show that in a 64-link scenario, even if a single link fails at 1%, the overall failure rate will reach 47.44%. Currently, the most widely used collective communication library, NCCL (NVIDIA Collective Communication Library, implemented by NVIDIA), will exit training if it encounters any single point of failure. This shows the importance of reducing the superposition of single-link failures.

[0063] In summary, there are two key issues in collective communication: avoiding hash collisions in traffic to improve bandwidth utilization to save training time; and reducing or eliminating the superposition effect of single point failures to improve overall reliability.

[0064] Under the influence of gravity, fluids can automatically select the optimal path to achieve maximum falling speed. Therefore, based on the concept of fluid gravity, the embodiments of the present invention provide a collective communication method. At the bottom layer of collective communication, namely RDMA (Remote Direct Memory Access) communication, multiple QP (Queue Pair) transmission technology is adopted, and adaptive traffic affinity scheduling and retransmission strategies are implemented. Specifically, to maximize link availability and automatically avoid faulty links, multiple QPs are used for parallel transmission. During the transmission process, the transmission quality of the QP is perceived, interrupted QPs are eliminated, and task allocation is adjusted in real time based on bandwidth. Ultimately, single point failures and hash collisions are avoided, and overall transmission efficiency is improved. This significantly improves the efficiency and reliability of large model training and provides a foundation for fault location and automatic repair.

[0065] An embodiment of the present invention provides a collective communication method. It should be noted that the execution subject of the collective communication method provided by the embodiment of the present invention can be a terminal, and the terminal includes a sending end and a receiving end. The sending end and the receiving end communicate with each other through a switch, wherein the terminal can be but is not limited to various personal computers, laptops, smart phones, tablet computers, Internet of Things devices, etc.

[0066] According to an embodiment of the present invention, an embodiment of a collective communication method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0067] In this embodiment, a collective communication method is provided, which can be used for the above-mentioned sending end. Figure 1 is a flow chart of a collective communication method according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0068] Step S101 , obtaining a sending request transmitted by a receiving end; wherein the sending request is generated by the receiving end based on a RECV request to obtain multiple available queue pairs and based on the multiple available queue pairs.

[0069] Specifically, RECV is a programming language function used to receive data.

[0070] Furthermore, the sender obtains a send request from fiforeqs (first-in-first-out requests), waits for all send requests to arrive, and then replies with an ACK (Acknowledge character).

[0071] Step S102 : determining a plurality of available queue pairs based on the sending request, and using an adaptive traffic affinity scheduling strategy to transmit the data to be transmitted in parallel to the receiving end through the plurality of available queue pairs.

[0072] Specifically, the sender (Sender) and the receiver (Receiver) are set as nodes (NodeA, NodeB, ..., NodeN). The collective communication operation can be split into a series of node-to-node data transmissions. Logically, multiple pairs of QPs are used for parallel data transmission, such as Figure 2 As shown in , a node is connected to multiple switches (such as Leaf and Spine), and data can be transmitted through multiple links; Figure 3 As shown, each pair of QPs (QP1, QP2, ..., QPn) may correspond to a certain physical link, and two pairs of QPs may be mapped to the same link.

[0073] Furthermore, the QP group specified by the receiving end is used to transmit data in parallel (multisend), and the data volume distribution of the available queue pairs is adaptively adjusted.

[0074] Step S103: After the parallel transmission of the data to be transmitted is completed, the transmission operation of the sending end is checked. If the transmission operation of the sending end fails, the data to be transmitted is retransmitted using a retransmission strategy.

[0075] Specifically, after the sending end completes sending data, a WC (Work Completion) is generated, and the corresponding request count (i.e., the statistics of the number of request messages during the communication process) is decremented.

[0076] Furthermore, after the sender sends the data, a poll CQ check operation is executed to check whether it is successful. The poll CQ check operation refers to the operation of polling the completion queue (CQ). If the poll CQ check operation result is that all operations are completed, the corresponding information is cleared and the occupied space is released. If the poll CQ check operation result is that the transmission fails, it is determined whether there is work done. If not, if it is a SEND request (i.e., the sender request), an available QP is reselected to retransmit the data. If it is other requests, it is determined whether it has timed out. If so, the request is retransmitted. When all QPs are unavailable, the task will fail and exit.

[0077] Furthermore, after each round of transmission is completed, the receiver decides whether to retransmit the data. Regardless of whether retransmission is required, the receiver notifies the sender which QPs (excluding faulty QPs) should be used for the next round of data transmission. During each round of transmission, the sender adjusts the task allocation ratio between QPs in real time, and repeats the above steps until the transmission task is completed.

[0078] This embodiment provides a collective communication method that utilizes an adaptive traffic affinity scheduling strategy to transmit the data to be transmitted in parallel to the receiving end through multiple available queue pairs. After the parallel transmission of the data to be transmitted is completed, the transmission operation of the sending end is checked. If the transmission operation of the sending end fails, the data to be transmitted is retransmitted using a retransmission strategy. By using the adaptive traffic affinity scheduling strategy and the retransmission strategy, multiple available queue pairs are used for parallel transmission, thereby maximizing the link availability, realizing the function of avoiding faulty links, and greatly improving the efficiency and reliability of large model training.

[0079] In this embodiment, a collective communication method is provided, which can be used for the above-mentioned sending end. Figure 4 is a flow chart of a collective communication method according to an embodiment of the present invention. Figure 4 As shown, the process includes the following steps:

[0080] Step S401: Obtain a sending request transmitted by a receiving end; wherein the sending request is generated by the receiving end based on a plurality of available queue pairs obtained by the RECV request and based on the plurality of available queue pairs. Figure 1 Step S101 of the illustrated embodiment will not be described in detail here.

[0081] Step S402 : determining a plurality of available queue pairs based on the sending request, and using an adaptive traffic affinity scheduling strategy to transmit the data to be transmitted in parallel to the receiving end through the plurality of available queue pairs.

[0082] Specifically, the above step S402 includes:

[0083] Step S4021: Acquire the amount of data allocated to multiple available queue pairs in the last transmission cycle, and the data transmission time of each available queue pair in the last transmission cycle.

[0084] Step S4022 : Calculate the score of each available queue pair based on the amount of data allocated to the multiple available queue pairs in the last transmission cycle and the data transmission time of each available queue pair in the last transmission cycle.

[0085] Specifically, the QP group specified by the receiving end is used to transmit data in parallel, and the distribution of data volume is adaptively adjusted according to the QP score value. Each pair of QPs has a corresponding score value, which represents the ability to transmit data.

[0086] Furthermore, let the amount of data D allocated to z available queue pairs in the last transmission cycle be D = {d1, d2, K, d z}, the data transmission time of each available queue pair in the last transmission cycle is t1, t2, ..., t z , the fixed cost of the software is Δt, then the score of the i-th QP is score i The calculation formula is:

[0087]

[0088] Where Z represents the total number of available queue pairs.

[0089] Step S4023 : Allocate the data to be transmitted to multiple available queue pairs based on the score value of each available queue pair, so as to transmit the data to be transmitted to the receiving end in parallel by using the multiple available queue pairs.

[0090] Specifically, the data to be transmitted is allocated to different available queue pairs according to the score value of each available queue pair. The calculation formula for the data volume of each available queue pair after data allocation is:

[0091] D i ′=data_size*score i (4)

[0092] Among them, D i ′ represents the amount of data in the i-th available queue pair, and data_size represents the size of the data to be transmitted.

[0093] Step S403: After the parallel transmission of the data to be transmitted is completed, the transmission operation of the sending end is checked. If the transmission operation of the sending end fails, the data to be transmitted is retransmitted using the retransmission strategy. Figure 1 Step S103 of the illustrated embodiment will not be described in detail here.

[0094] This embodiment provides a collective communication method that calculates the score value of each available queue pair and then distributes the data to be transmitted to multiple available queue pairs based on the score value of each available queue pair. When a hash collision or other reasons cause congestion on some links, the data volume of the available queue pairs is reasonably distributed through an adaptive traffic affinity scheduling strategy, actively adapting to the bandwidth and load of each link to maximize the final bandwidth, avoiding traffic hash collisions, improving bandwidth utilization, saving training time, and significantly improving the efficiency and reliability of large model training.

[0095] In this embodiment, a collective communication method is provided, which can be used for the above-mentioned receiving end. Figure 5 is a flow chart of a collective communication method according to an embodiment of the present invention. Figure 5 As shown, the process includes the following steps:

[0096] Step S501: Create a RECV request, and obtain multiple available queue pairs based on the RECV request.

[0097] Step S502: Generate a sending request based on multiple available queue pairs, and transmit the sending request to the sending end.

[0098] Specifically, the receiving end executes postfifo (puts data into a buffer in a first-in, first-out manner) and transmits the receiving end's information (including the available queue pair information that the sending end hopes to use for sending) to the sending end.

[0099] Step S503, receiving feedback data transmitted by the sending end; the feedback data is generated by the sending end by determining multiple available queue pairs based on the sending request, and using the adaptive traffic affinity scheduling strategy to transmit the data to be transmitted in parallel through multiple available queue pairs.

[0100] Specifically, the feedback data transmitted by the sending end carries Immediate Data, notifying the receiving end that data has arrived. The receiving end generates a WC, and the corresponding request count decreases.

[0101] Furthermore, the transmission operation of the receiving end is checked, and if the transmission operation of the receiving end fails, the sending request is retransmitted using a retransmission strategy.

[0102] Furthermore, the receiving end determines whether the poll CQ check operation is successful based on wc_status (work completion status, used to indicate relevant status information after a work request is executed). If it is a RECV request, it determines whether the postfifo is successful. If the transmission fails, the data transmission is performed again.

[0103] This embodiment provides a collective communication method, in which the receiving end obtains multiple available queue pairs through RECV requests, and the sending end uses an adaptive traffic affinity scheduling strategy to transmit the data to be transmitted in parallel through multiple available queue pairs. Then, the receiving end receives the feedback data transmitted by the sending end and uses the available queue pairs for parallel transmission, thereby maximizing link availability and improving overall transmission efficiency.

[0104] In this embodiment, a collective communication method is provided, which can be used for the above-mentioned receiving end. Figure 6 is a flow chart of a collective communication method according to an embodiment of the present invention. Figure 6 As shown, the process includes the following steps:

[0105] Step S601: Create a RECV request and obtain multiple available queue pairs based on the RECV request. Figure 5 Step S501 of the illustrated embodiment will not be described in detail here.

[0106] Specifically, the above step S601 includes:

[0107] Step S6011: Determine multiple queue pairs based on the RECV request.

[0108] Specifically, a RECV request is used to read the buffer and multiple queue pairs are found.

[0109] Step S6012: Obtain a status identifier of each queue pair, and detect a queue pair in an abnormal state based on the status identifier of each queue pair.

[0110] Step S6013: Obtain a standby queue pair, and replace the abnormal queue pair among the multiple queue pairs with the standby queue pair to obtain multiple available queue pairs.

[0111] Specifically, each pair of QPs will have a status identifier. If a QP in an abnormal state is detected, it will be replaced with a backup QP to ensure that the currently selected n QPs are available, thereby achieving the fault link avoidance function.

[0112] Step S602: Generate a send request based on multiple available queue pairs and transmit the send request to the sender. Figure 5 Step S502 of the illustrated embodiment will not be described in detail here.

[0113] Step S603: Receive feedback data transmitted by the sender. The sender determines multiple available queue pairs based on the send request, and uses the adaptive traffic affinity scheduling strategy to transmit the data to be transmitted through multiple available queue pairs in parallel. Figure 5 Step S503 of the illustrated embodiment will not be described in detail here.

[0114] This embodiment provides a collective communication method. When a link fails, an abnormal queue pair among multiple queue pairs is replaced with a standby queue pair, and multiple available queue pairs are used for parallel transmission. During the transmission process, the transmission quality of the queue pairs is sensed, and interrupted queue pairs are eliminated, thereby avoiding single point failures, reducing or eliminating the superposition effect of single point failures, improving overall reliability, reducing the waste of computing resources caused by link failures, saving computing resources and time costs, and improving overall transmission efficiency.

[0115] The following describes the specific steps of a collective communication method and the advantages of the method through specific embodiments.

[0116] Example 1:

[0117] like Figure 7 As shown, the point-to-point communication at the bottom layer of collective communication adopts a multi-channel transmission mechanism. The specific steps of a collective communication method include:

[0118] 1) The receiver creates a RECV request and finds n available QPs. Each pair of QPs has a status flag. If an abnormal QP is detected, it is replaced with a backup QP to ensure that all n currently selected QPs are available, thus achieving fault link avoidance.

[0119] 2) The receiver executes postfifo and passes the receiver's information (including the QP group information that the sender hopes the sender will use for sending) to the sender.

[0120] 3) The sender receives the requests, waits for all receiver information to arrive, and responds with an ACK.

[0121] 4) Use the QP group specified by the Receiver to transmit data in parallel, and adaptively adjust the data volume distribution according to the QP score value.

[0122] 5) After sending the data, the sender generates a WC and the corresponding request count decreases.

[0123] 6) The sender's message carries Immediate Data, notifying the receiver that data has arrived. The receiver generates a WC, and the corresponding request count decreases.

[0124] 7) After the sender sends the data and the receiver sends the postfifo, a poll CQ is executed to check whether the operation is successful. If the transmission fails, an available QP is reselected to retransmit the data. When all QPs are unavailable, the task will fail and exit.

[0125] Example 2:

[0126] The Fluid-Gravity algorithm mainly includes fault link avoidance, adaptive traffic affinity scheduling, and transmission failure retransmission mechanism; among them, fault link avoidance: facing scenarios where network links are interrupted or very serious congestion occurs, the link is actively switched on the end side to avoid directly terminating the entire large model training task due to a link interruption; Adaptive traffic affinity scheduling: facing link congestion problems caused by hash collisions, the transmission traffic size allocated on each link is adaptively adjusted in real time according to the current bandwidth capacity, so that the collective communication is close to the optimal bandwidth in any scenario; Transmission failure retransmission mechanism: facing scenarios where network problems cause inter-machine RDMA data transmission to fail, a retransmission mechanism is set up, and when transmission fails, an available link will be reselected to retransmit the data.

[0127] From the perspective of fault avoidance, the implementation of the related collective communication library does not have the function of avoiding faulty links. Once a link fails, the task training will be terminated. Assuming that the single link failure rate is p, the overall failure rate of the related collective communication library is:

[0128] P=1-(1-p) n (5)

[0129] Using the above "fluid-gravity" algorithm, the failure rate is:

[0130] P′=p n (6)

[0131] In order to intuitively reflect the effect of this embodiment, assuming that the failure rate is 1%, combined with different numbers of links, the following Table 1 can be obtained.

[0132] Table 1:

[0133]

[0134] If the number of links is fixed at 64 and the single-link failure rate is used as a variable, the following Table 2 can be obtained.

[0135] Table 2:

[0136]

[0137] It can be seen that the failure rate of the native collective communication library increases sharply with the expansion of the network scale, and also increases sharply with the increase of the single point failure rate, while the overall failure rate of the "fluid-gravity" algorithm tends to 0.

[0138] Then, we analyze from the perspective of bandwidth utilization: suppose the bandwidths of different links are: b1, b2, ..., b n , in particular, let the link fixed bandwidth and maximum bandwidth be b max , let the minimum bandwidth among n links be bmin Assume that the amount of data transferred between GPUs is S. The time it takes for the related collective communication library to execute a task depends on the slowest link, and the time it takes is T:

[0139]

[0140] After adopting the “fluid-gravity” algorithm, the local link bandwidth is the sum of multiple QP bandwidths, and the maximum value b is taken. max , then the time consumed T' is:

[0141]

[0142] Similarly, to more intuitively demonstrate the effect of the "fluid-gravity" algorithm, we set the link fixed bandwidth to 400G, set the worst bandwidth as a variable, and set the transmitted data volume to 1024G, then we can obtain the following Table 3.

[0143] Table 3:

[0144] Scenario NCCL "Fluid-Gravity" Worst link 400G, total time 2.56s 2.56s Worst link 200G, total time 5.12s 2.56s Worst link 100G, total time 10.24s 2.56s Worst link 50G, total time 20.48s 2.56s Worst link 50G, total time 40.96s 2.56s Worst link 10G, total time 102.4s 2.56s

[0145] It can be seen that the "Fluid-Gravity" algorithm can always maintain good latency, while the time consumption of the related collective communication library will deteriorate sharply as the bandwidth of the worst link (the link with the most serious hash collision) decreases.

[0146] Therefore, compared with related collective communication libraries, the "Fluid-Gravity" algorithm can automatically avoid link failures that occur before and during the task. When the network is abnormal, the training task can still persist until the next checkpoint, saving computing resources and time costs to the greatest extent. The proposed adaptive traffic affinity scheduling maximizes the bandwidth of the collective communication network, ensuring that collective communication maintains good latency even when hash collisions occur.

[0147] Example 3:

[0148] like Figure 8 As shown in the figure, each node in the intelligent computing network has 8 GPU cards, and the corresponding 8 network cards are connected to 8 Leaf switches. The 8 Leaf switches can connect to a total of 32 nodes (256 GPU cards), which is 1 Block. The 32 uplink ports of each Leaf switch are connected to different Spine switches. Each Spine switch has 64 downlink ports, which can connect to a total of 8 Blocks, with a total of 2048 GPU cards. Each GPU card can communicate with the GPU of another Block through 32 different paths.

[0149] When a switch experiences a pseudo-death experience or an imperceptible failure occurs in a network link, the collective communication method in this embodiment can automatically eliminate the faulty link to ensure maximum communication bandwidth; when switch Spine1 fails, all QPs communicating through the link are marked as unavailable and replaced by backup QPs, and data transmission is performed via paths passing through other Spine switches, so that the training task is not interrupted and the waste of computing resources caused by link failure is reduced.

[0150] When a hash collision or other reasons cause congestion on some links, the collective communication method in this embodiment can automatically perform traffic affinity scheduling, actively adapt to the bandwidth and load of each link, and maximize the final bandwidth; Figure 9 As shown in the figure, when congestion occurs on the link passing through Spine 1, traffic affinity scheduling can be adaptively performed to assign fewer tasks to the QP passing through this link and more tasks to links with higher bandwidth, thereby maximizing the current bandwidth utilization.

[0151] This embodiment also provides a collective communication device for implementing the above-mentioned embodiments and preferred implementations. Details already described will not be repeated. As used below, the term "module" may refer to a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation using hardware, or a combination of software and hardware, is also possible and contemplated.

[0152] This embodiment provides a collective communication device, which is applied to a sending end, such as Figure 10 Shown, including:

[0153] The acquisition module 1001 is configured to acquire a sending request transmitted by a receiving end; wherein the sending request is generated by the receiving end based on a RECV request to obtain multiple available queue pairs and based on the multiple available queue pairs;

[0154] The transmission module 1002 is configured to determine a plurality of available queue pairs based on the sending request, and transmit the data to be transmitted to the receiving end in parallel through the plurality of available queue pairs using an adaptive traffic affinity scheduling strategy;

[0155] The checking module 1003 is configured to check the transmission operation of the sending end after the parallel transmission of the data to be transmitted is completed. If the transmission operation of the sending end fails, the data to be transmitted is retransmitted using a retransmission strategy.

[0156] In some optional implementations, the transmission module 1002 includes:

[0157] an acquisition unit, configured to acquire the amount of data allocated to the plurality of available queue pairs in the previous transmission cycle, and the data transmission time of each available queue pair in the previous transmission cycle;

[0158] a calculation unit, configured to calculate a score value for each available queue pair based on an amount of data allocated to the plurality of available queue pairs in a previous transmission cycle and a data transmission time of each available queue pair in the previous transmission cycle;

[0159] The allocating unit is configured to allocate the data to be transmitted to a plurality of available queue pairs based on the score value of each available queue pair, so as to transmit the data to be transmitted to the receiving end in parallel by using the plurality of available queue pairs.

[0160] This embodiment also provides a collective communication device, which is applied to a receiving end, such as Figure 11 Shown, including:

[0161] A creation module 1101 is used to create a RECV request and obtain multiple available queue pairs based on the RECV request;

[0162] A generating module 1102 is configured to generate a sending request based on a plurality of available queue pairs and transmit the sending request to a sending end;

[0163] The receiving module 1103 is used to receive feedback data transmitted by the sending end; the feedback data is generated by the sending end by determining multiple available queue pairs based on the sending request, and using the adaptive traffic affinity scheduling strategy to transmit the data to be transmitted in parallel through multiple available queue pairs.

[0164] In some optional implementations, the creation module 1101 includes:

[0165] a determining unit, configured to determine a plurality of queue pairs based on a RECV request;

[0166] a detection unit, configured to obtain a status identifier of each queue pair and detect a queue pair in an abnormal state based on the status identifier of each queue pair;

[0167] The replacement unit is used to obtain a standby queue pair and replace an abnormal queue pair among the multiple queue pairs with the standby queue pair to obtain multiple available queue pairs.

[0168] In some optional embodiments, the method further includes:

[0169] The retransmission module is used to check the transmission operation of the receiving end. If the transmission operation of the receiving end fails, the retransmission strategy is used to retransmit the sending request.

[0170] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0171] The collective communication device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0172] The embodiment of the present invention also provides a computer device having the above Figure 10 or Figure 11 The collective communication device shown.

[0173] See also Figure 12 , Figure 12 is a structural diagram of a computer device provided by an optional embodiment of the present invention, such as Figure 12 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components utilize different buses to communicate with each other and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of a GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories. Equally, multiple computer devices can be connected, and each device provides part of the necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 12 A processor 10 is taken as an example.

[0174] The processor 10 may be a central processing unit, a network processor, or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit, a programmable logic device, or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic, or any combination thereof.

[0175] The memory 20 stores instructions that can be executed by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0176] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and application programs required for at least one function; the data storage area may store data created based on the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0177] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0178] The computer device further includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Figure 12 The bus connection is taken as an example.

[0179] The input device 30 can receive input digital or character information and generate key signal input related to user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a trackpad, a touch pad, an indicator stick, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 can include a display device, an auxiliary lighting device (e.g., an LED), and a tactile feedback device (e.g., a vibration motor). The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display, and a plasma display. In some optional embodiments, the display device can be a touch screen.

[0180] The embodiment of the present invention also provides a computer-readable storage medium. The above-mentioned method according to the embodiment of the present invention can be implemented in hardware, firmware, or implemented as a computer code that can be recorded in a storage medium, or implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and downloaded through a network and will be stored in a local storage medium, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state drive, etc.; further, the storage medium can also include a combination of the above-mentioned types of memory. It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor or hardware, the method shown in the above embodiment is implemented.

[0181] A portion of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the form in which the computer program instruction exists in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc. Accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium that can be accessed by the computer.

[0182] Although the embodiments of the present invention have been described with reference to the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention. Such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A collective communication method, applied to a sending end, characterized in that: The method comprises: Obtaining a sending request transmitted by a receiving end; wherein the sending request is generated by the receiving end based on a RECV request to obtain multiple available queue pairs and based on the multiple available queue pairs; Determine a plurality of available queue pairs based on the sending request, and transmit the data to be transmitted to the receiving end in parallel through the plurality of available queue pairs using an adaptive traffic affinity scheduling strategy; After the parallel transmission of the data to be transmitted is completed, the transmission operation of the sending end is checked. If the transmission operation of the sending end fails, the data to be transmitted is retransmitted using a retransmission strategy.

2. The method according to claim 1, characterized in that The determining of the plurality of available queue pairs based on the sending request, and using an adaptive traffic affinity scheduling strategy to transmit the to-be-transmitted data to the receiving end in parallel through the plurality of available queue pairs, includes: Obtaining the amount of data allocated to the plurality of available queue pairs in the last transmission cycle and the data transmission time of each of the available queue pairs in the last transmission cycle; Calculating a score value for each of the available queue pairs based on the amount of data allocated to the plurality of available queue pairs in the last transmission cycle and the data transmission time of each of the available queue pairs in the last transmission cycle; The data to be transmitted is distributed to a plurality of the available queue pairs based on the score value of each of the available queue pairs, so as to transmit the data to be transmitted to the receiving end in parallel by using the plurality of the available queue pairs.

3. A collective communication method, applied to a receiving end, characterized in that: The method comprises: Creating a RECV request, and obtaining a plurality of available queue pairs based on the RECV request; generating a sending request based on the plurality of available queue pairs, and transmitting the sending request to a sending end; Receive feedback data transmitted by the sending end; the feedback data is generated by the sending end determining multiple available queue pairs based on the sending request, and using an adaptive traffic affinity scheduling strategy to transmit the data to be transmitted in parallel through multiple available queue pairs.

4. The method according to claim 3, characterized in that The creating a RECV request and obtaining a plurality of available queue pairs based on the RECV request include: determining a plurality of queue pairs based on the RECV request; Obtaining a status identifier of each queue pair, and detecting a queue pair in an abnormal state based on the status identifier of each queue pair; A standby queue pair is obtained, and the queue pair in the abnormal state among the multiple queue pairs is replaced with the standby queue pair to obtain a plurality of available queue pairs.

5. The method according to claim 3, characterized in that Also includes: The transmission operation of the receiving end is checked, and if the transmission operation of the receiving end fails, the sending request is retransmitted using a retransmission strategy.

6. A collective communication device, applied to a transmitting end, characterized in that: The device comprises: An acquisition module, configured to acquire a sending request transmitted by a receiving end; wherein the sending request is generated by the receiving end based on a RECV request to obtain a plurality of available queue pairs and based on the plurality of available queue pairs; a transmission module, configured to determine a plurality of available queue pairs based on the sending request, and transmit the data to be transmitted in parallel to the receiving end through the plurality of available queue pairs using an adaptive traffic affinity scheduling strategy; The checking module is used to check the transmission operation of the sending end after the parallel transmission of the data to be transmitted is completed, and if the transmission operation of the sending end fails, retransmit the data to be transmitted using a retransmission strategy.

7. A collective communication device, applied to a receiving end, characterized in that: The device comprises: A creation module is configured to create a RECV request and obtain a plurality of available queue pairs based on the RECV request; a generating module, configured to generate a sending request based on the plurality of available queue pairs, and transmit the sending request to a sending end; A receiving module is used to receive feedback data transmitted by the sending end; the feedback data is generated by the sending end determining multiple available queue pairs based on the sending request, and using an adaptive traffic affinity scheduling strategy to transmit the data to be transmitted in parallel through multiple available queue pairs.

8. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the collective communication method according to any one of claims 1 to 5 by executing the computer instructions.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the collective communication method according to any one of claims 1 to 5.

10. A computer program product, characterized in that The method comprises computer instructions for causing a computer to execute the collective communication method according to any one of claims 1 to 5.

Images