Communication method, device, electronic device and storage medium for computing power cluster

Abstract

The present disclosure provides a communication method, device, electronic device and storage medium for a computing power cluster, and relates to technical fields such as graphics processors, computing power clusters, model training, and generative large language models. The method includes: in the process of communicating with a communication receiving end using the RDMA protocol, obtaining a first packet loss rate corresponding to the RDMA protocol; in response to the first packet loss rate being higher than a first preset packet loss rate, initiating a first handshake request to the communication receiving end to switch to TCP for communication; receiving a first handshake response returned by the communication receiving end for the first handshake request, and determining the first starting transmission position of the data according to the last data receiving position in the first handshake response; and communicating with the communication receiving end using TCP starting from the data corresponding to the first starting transmission position. The application of this method can improve the overall communication efficiency and data transmission volume, thereby improving the training efficiency of the generative large language model and shortening the time required for training.

Description

Technical Field

[0001] The present disclosure relates to the field of communication technology, specifically to technical fields such as graphics processors, computing power clusters, model training, generative large language models, and more particularly to a communication method, device, electronic device, computer-readable storage medium, and computer program product for a computing power cluster. Background Art

[0002] As the demand for computing power in the era of large models gradually increases, the number of chips in the cluster is increasingly required. A single computing cluster of 1,000 or even 10,000 cards can no longer meet the demand. As a result, the training of larger computing clusters in the future will have to rely on multiple computing clusters. These computing clusters are usually distributed in different physical locations (for example, in different computer rooms) or even across regions.

[0003] How to efficiently utilize these computing clusters and minimize the impact of cross-computer room communication time on training performance is an urgent problem to be solved by technical personnel in this field. Summary of the invention

[0004] The embodiments of the present disclosure provide a communication method, device, electronic device, computer-readable storage medium, and computer program product for a computing power cluster.

[0005] In the first aspect, an embodiment of the present disclosure proposes a communication method for a computing power cluster, which is applied to a communication initiator, including: in the process of communicating with a communication receiving end using a remote direct data access (RDMA) protocol, obtaining a first packet loss rate corresponding to the RDMA protocol; in response to the first packet loss rate being higher than a first preset packet loss rate, initiating a first handshake request to the communication receiving end to switch to a transmission control protocol (TCP) for communication; receiving a first handshake response returned by the communication receiving end for the first handshake request, and determining a first starting transmission position of the data based on the last data receiving position in the first handshake response; and communicating with the communication receiving end using TCP starting from the data corresponding to the first starting transmission position.

[0006] In the second aspect, an embodiment of the present disclosure proposes a communication device for a computing power cluster, which is applied to a communication initiator, including: a first packet loss rate acquisition unit, configured to acquire a first packet loss rate corresponding to the remote direct data access RDMA protocol during communication with a communication receiving end using the RDMA protocol; a TCP protocol switching handshake request initiating unit, configured to initiate a first handshake request to the communication receiving end to switch to the transmission control protocol TCP for communication in response to the first packet loss rate being higher than a first preset packet loss rate; a first handshake response receiving and first starting transmission position determining unit, configured to receive a first handshake response returned by the communication receiving end for the first handshake request, and determine the first starting transmission position of the data according to the last data receiving position in the first handshake response; a TCP protocol communication unit, configured to communicate with the communication receiving end using TCP starting from the data corresponding to the first starting transmission position.

[0007] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that when the at least one processor executes, the communication method for a computing power cluster as described in the first aspect can be implemented.

[0008] In a fourth aspect, an embodiment of the present disclosure provides a non-transitory computer-readable storage medium storing computer instructions, which are used to enable a computer to implement the communication method for a computing power cluster as described in the first aspect when executed.

[0009] In a fifth aspect, an embodiment of the present disclosure provides a computer program product including a computer program, which, when executed by a processor, can implement the communication method for a computing power cluster as described in the first aspect.

[0010] The communication solution for computing power clusters provided by the present disclosure provides a solution for switching the transmission protocol between the remote direct data access RDMA protocol and the transmission control protocol TCP according to the packet loss rate for the communication initiator, so that when the packet loss rate is relatively high, the TCP protocol that can achieve more stable data transmission is selected, and when the packet loss rate is relatively low, the RDMA protocol that can increase the data transmission volume is selected, so as to better balance the network environment of cross-cluster communication and intra-cluster communication, thereby improving the overall communication efficiency and data transmission volume, and further improving the training efficiency of the generative large language model and shortening the time required for training.

[0011] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Other features, objects and advantages of the present disclosure will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0013] Figure 1 is an exemplary system architecture in which the present disclosure may be applied;

[0014] Figure 2 A flow chart of a communication method for switching from an RDMA protocol to a TCP protocol provided in an embodiment of the present disclosure;

[0015] Figure 3 A flow chart of a communication method for switching from a TCP protocol back to an RDMA protocol provided in an embodiment of the present disclosure;

[0016] Figure 4 A flowchart of a method for determining whether a communication receiving end belongs to a different computing power cluster according to actual delay and processing provided in an embodiment of the present disclosure;

[0017] Figure 5-1 A schematic diagram of the structure of a dedicated line interconnection and DCN interconnection across data centers provided in an embodiment of the present disclosure;

[0018] Figure 5-2 A structural diagram of a method for obtaining packet loss rates corresponding to RDMA and TCP respectively through different shared memories provided in an embodiment of the present disclosure;

[0019] Figure 5-3 A flowchart of a method for obtaining packet loss rate using RDMA and TCP provided in an embodiment of the present disclosure;

[0020] Figure 5-4 A schematic diagram of a flow chart of switching transmission protocols between a sending side and a receiving side when performing cross-machine room transmission provided by an embodiment of the present disclosure;

[0021] Figure 6 A structural block diagram of a communication device provided in an embodiment of the present disclosure;

[0022] Figure 7 A schematic diagram of the structure of an electronic device suitable for executing a communication method provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0023] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description. It should be noted that the embodiments in the present disclosure and the features in the embodiments may be combined with each other without conflict.

[0024] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0025] Figure 1 An exemplary system architecture 100 is shown to which embodiments of the communication method, apparatus, electronic device, and computer-readable storage medium for a computing power cluster of the present disclosure can be applied.

[0026] like Figure 1 As shown, the system architecture 100 may include a first computing cluster 110 and a second computing cluster 120. The first computing cluster 110 includes multiple graphics processors, such as 111, 112, 113, 114, and 115 in the form of graphics cards (not just 5, Figure 1 Only a few are used as examples), the second computing cluster 120 also includes multiple graphics processors, such as 121, 122, 123, 124 and 125, which are also in the form of graphics cards.

[0027] In different computing power clusters, any device that initiates communication can be regarded as a communication initiator. It can be that only a graphics processor that constitutes a computing power cluster independently serves as a communication initiator. When all graphics processors in the overall computing power cluster initiate communication and transmit data to the graphics processors in another computing power cluster, the computing power cluster that initiates communication can also be regarded as a communication initiator as a whole.

[0028] The graphics processors that make up the computing power cluster can communicate with other graphics processors in the same computing power cluster or with other graphics processors in other computing power clusters independently or through communication components combined with pre-installed communication control applications. It should be noted that there can be many reasons and purposes for initiating communication, which are not specifically limited here. Only corresponding solutions are given for the communication method and the determination of transmission parameters.

[0029] A graphics processor is usually manifested as hardware. Of course, in special scenarios (such as simulation scenarios), it can also be manifested as software or a software running product, which is not specifically limited here.

[0030] The communication initiator can communicate with the communication receiver through the built-in communication control application. Figure 1 Taking the graphics card 112 as the communication initiator and the graphics card 121 as the communication receiver as an example, the graphics card 112 can achieve cross-cluster communication in the following ways when running the communication control application: first, in the process of communicating with the communication receiver using the RDMA protocol, a first packet loss rate corresponding to the RDMA protocol is obtained; then, when the first packet loss rate is higher than the first preset packet loss rate, a first handshake request is initiated to the communication receiver to switch to the transmission control protocol TCP for communication; then, a first handshake response returned by the communication receiver for the first handshake request is received, and a first starting transmission position of the data is determined according to the last data receiving position in the first handshake response; finally, the TCP is used to communicate with the communication receiver starting from the data corresponding to the first starting transmission position.

[0031] In addition to the cross-cluster communication presented in the above example, Figure 1 It is also shown that the graphics card 111 as the communication initiator can communicate with the graphics card 113 as the communication receiver in the cluster. This communication process can also be implemented under the control of the communication control application. However, since the intra-cluster communication is not affected by the fluctuation of the external network environment, it usually does not cause the first packet loss rate higher than the first preset packet loss rate to appear when the RDMA protocol is used first. Therefore, it actually shows that the protocol switching strategy provided above is no longer effective in the intra-cluster communication process. Of course, the above switching strategy can also be applied only to the cross-cluster communication scenario, and the intra-cluster communication scenario can be configured to use only the RDMA protocol.

[0032] Furthermore, it is not possible to clearly determine at the communication receiving end whether the communication receiving end belongs to a different computing power cluster from itself. This can be confirmed in a variety of ways, such as by using communication delay, packet loss rate, etc. to confirm whether the transmission environment is within a cluster or across clusters.

[0033] It should be understood that Figure 1 The number of graphics cards, graphics processors, and computing power clusters in the example is only for reference. Any number of graphics cards, graphics processors, and computing power clusters may be used as required.

[0034] Please refer to Figure 2 , Figure 2 A flow chart of a communication method for switching from an RDMA protocol to a TCP protocol provided in an embodiment of the present disclosure, wherein process 200 includes the following steps:

[0035] Step 201: in the process of communicating with a communication receiving end using an RDMA protocol, obtaining a first packet loss rate corresponding to the RDMA protocol;

[0036] This step is intended to be performed by the execution subject of the communication method (e.g. Figure 1 The graphics card 112 and the graphics card 111 shown in the figure, both of which belong to the first computing power cluster) use the RDMA protocol to communicate with the communication receiving end (for example Figure 1 When the graphics card 121 and the graphics card 113 shown in the figure, which belong to the second computing power cluster and the first computing power cluster respectively, communicate, a first packet loss rate corresponding to the RDMA protocol is obtained during the communication process.

[0037] Among them, the RDMA (Remote Direct Memory Access) protocol is a network protocol that allows computers to directly access the memory of remote computers. It is often used in scenarios such as high-performance computing (HPC), data centers, and storage networks. RDMA can achieve high-speed data transmission between hosts without involving the intervention of the operating system kernel, so it has extremely low latency and high bandwidth. When using the RDMA protocol, the sender (i.e., the communication initiator) directly writes data to the memory of the receiver (usually called the communication receiver) without the intervention of the operating system. This direct memory access reduces the intermediate layer in the data transmission process, thereby improving data transmission efficiency.

[0038] Among them, the packet loss rate refers to the proportion of data packets that fail to successfully reach the receiver during the data transmission process due to various reasons (such as network congestion, transmission errors, hardware failures, etc.). The packet loss rate is an important indicator to measure the quality of network communication, usually expressed as a percentage. A poor network environment will lead to a higher packet loss rate, that is, a high packet loss rate can reflect the current poor network communication quality, and may even affect the reliability and performance of the application. The packet loss rate can be determined in the following ways:

[0039] 1) Real-time monitoring: The packet loss situation may be tracked in real time by deploying network monitoring modules at the communication initiator and the communication receiver. Specifically, the communication initiator and the communication receiver will record the sending and receiving status of each data packet, and monitor whether there is packet loss through the protocol stack or hardware-assisted mechanism;

[0040] 2) Packet loss statistics supported by the protocol: The RDMA protocol itself may also support packet loss statistics in the underlying transmission. The communication initiator and the communication receiver can obtain packet loss information based on the feedback information in the protocol (such as error reports, ACK packets, or confirmation packets, etc.);

[0041] 3) Packet loss rate at the network layer or application layer: In some implementations, the acquisition of packet loss rate is not limited to packet loss at the physical layer, but may also include packet loss at higher layers (such as the transport layer or application layer), depending on the design of the network monitoring solution. For example, the packet loss rate can be stored in a place that is easy to obtain by building a corresponding packet loss rate statistics process, such as shared memory.

[0042] The first packet loss rate described in this step refers to the packet loss rate counted by the communication initiator as the execution subject during data communication with the communication receiving end using the RDMA protocol, and it is expected that the first packet loss rate can be used to reflect the current network communication quality.

[0043] An implementation method including but not limited to may be:

[0044] The first packet loss rate is obtained from the first shared memory corresponding to the RDMA network card. Specifically, the first packet loss rate is obtained from the register of the RDMA network card by the first packet loss rate acquisition process started by the above-mentioned execution subject when communicating using the RDMA protocol and stored in the first shared memory. That is, when the RDMA protocol is used for data communication, the RDMA protocol will store the statistically obtained packet loss related information in the register of its RDMA network card by default, and the collective communication library usually does not provide corresponding operation instructions for obtaining the packet loss rate of the RDMA protocol. Therefore, this implementation method moves the packet loss rate information from the register to the first shared memory by means of a self-built packet loss rate acquisition process, so as to facilitate the acquisition.

[0045] Step 202: In response to the first packet loss rate being higher than the first preset packet loss rate, initiating a first handshake request to the communication receiving end to switch to TCP for communication;

[0046] This step is established when the first packet loss rate obtained in step 201 is higher than the first preset packet loss rate, and is intended for the execution subject to initiate a first handshake request to the communication receiving end to switch to TCP for communication. Among them, the first preset packet loss rate is a demarcation value or critical value determined based on historical experience to measure whether the current network communication quality under the RDMA protocol is good or bad, that is, if the actual first packet loss rate is higher than the first preset packet loss rate, it can be considered that the current network communication quality is poor, conversely, if the actual first packet loss rate is not higher than the first preset packet loss rate, it can be considered that the current network communication quality is good. Specifically, the first preset packet loss rate can be set by an expert with relevant experience in combination with the actual situation in a specific application scenario, or it can be obtained by analyzing the input samples representing the current application scenario through a pre-trained model, which is not specifically limited here. More specifically, the first preset packet loss rate can also be set to 4%.

[0047] Since the first packet loss rate is higher than the first preset packet loss rate, it means that the execution subject is in a network environment with poor network communication quality during the current period when communicating with the communication receiving end using the RDMA protocol. Therefore, in order to avoid the high packet loss rate caused by the poor network communication quality and thus affect the overall training task, the above-mentioned execution subject will initiate a first handshake request to the communication receiving end in this step, wanting to switch the current RDMA protocol to the TCP protocol that can bring a relatively lower packet loss rate in a poor network environment. That is, the first handshake request is used to inform the communication receiving end that the communication initiator wants to switch the communication protocol and wants to switch to the TCP protocol.

[0048] The reason why TCP protocol can bring lower packet loss rate than RDMA protocol in poor network environment is mainly related to the working mechanism, error recovery strategy and network adaptability of these two protocols. The specific reasons are listed below:

[0049] 1) Error recovery mechanism

[0050] TCP is a connection-oriented protocol with a built-in powerful error detection and recovery mechanism. Specifically, TCP uses a retransmission mechanism to ensure reliable data transmission. When a data packet is lost or damaged, TCP detects packet loss through sequence numbers and acknowledgments (ACKs) and resends the lost data packet. Even in an unstable or delayed network environment, TCP will continue to try to retransmit the lost packet until it is successfully transmitted, thereby effectively reducing the packet loss rate;

[0051] The RDMA protocol transfers data directly in memory. It is usually used in low-latency, high-bandwidth network environments and improves data transmission efficiency by bypassing the operating system kernel. RDMA does not have built-in retransmission and error recovery mechanisms like TCP. In a poor network environment, if packet loss occurs, RDMA may not be able to effectively handle this situation, resulting in the inability to recover from packet loss in a timely manner.

[0052] 2) Flow control and congestion control

[0053] TCP has flow control and congestion control mechanisms that can dynamically adjust the data transmission rate according to the network conditions. TCP will continuously adjust the sending window size according to the network bandwidth, delay and packet loss to avoid network overload and excessive packet loss. In poor network environments, TCP congestion control (such as slow start, congestion avoidance, fast retransmission, etc.) can effectively avoid network crashes or further packet loss;

[0054] RDMA directly operates memory during transmission, avoiding the overhead of traditional network protocol stacks, but it usually relies on the underlying network hardware (such as InfiniBand, RoCE, etc.) to implement flow control. If the network environment is poor, RDMA may not have sufficient adaptability and is prone to data loss when network congestion or errors occur.

[0055] 3) Network quality and robustness

[0056] TCP was designed with unstable network quality in mind, so its mechanisms include retransmission, confirmation, flow control, congestion control, etc., which can cope with various changes in network quality. In poor network environments (such as high packet loss rate, network jitter, etc.), TCP can ensure the reliability of data transmission through adaptive mechanisms;

[0057] RDMA is usually used in data center networks and has high requirements for the network. If the network is lost or congested, RDMA may not be able to compensate effectively because it lacks protocol-level error recovery and control like TCP. The RDMA protocol is more dependent on the performance of the underlying network hardware. Once the network quality deteriorates, data loss will be more serious.

[0058] 4) Differences in application scenarios

[0059] TCP is usually used in reliability and error recovery scenarios. It can ensure that data arrives at the receiver in order and reliably. It is suitable for various network environments, including unstable networks.

[0060] RDMA is usually used in high-performance computing and storage scenarios with extremely low latency requirements, and is suitable for high-quality, low-packet-loss network environments, such as InfiniBand or RoCE (RDMA over Converged Ethernet, a technology that enables data transmission by attaching the RDMA protocol to Ethernet frames). In these high-quality networks, RDMA can give full play to its advantages of low latency and high throughput, but in poor network environments, it lacks an effective packet loss recovery mechanism and may not be able to guarantee reliable data transmission.

[0061] 5) Interaction and adaptability of the network layer

[0062] TCP can adjust to different network environments with the support of the operating system kernel. For example, TCP can use congestion control algorithms (such as TCP Reno, Cubic, etc.) to cope with different network conditions and adjust the data transmission rate appropriately according to the network load to avoid further packet loss;

[0063] RDMA interacts directly with network hardware, bypassing the traditional protocol stack of the operating system. Although this can reduce latency, it also means that when the network status is poor, RDMA's adaptability is weak, and it is prone to packet loss or failure to adjust the transmission rate in time.

[0064] Therefore, it is precisely because the TCP protocol has powerful error recovery mechanisms, congestion control, flow control and other functions, and can better adapt to network instability, thereby achieving a lower packet loss rate than the RDMA protocol, that this step chooses to switch the RDMA protocol to the TCP protocol in time when the current communication environment is found to be poor through the first packet loss rate.

[0065] Step 203: receiving a first handshake response returned by the communication receiving end in response to the first handshake request, and determining a first starting transmission position of the data according to the last data receiving position in the first handshake response;

[0066] On the basis of step 202, this step is intended to cause the execution subject to receive the first handshake response returned by the communication receiving end in response to the first handshake request, and determine the first starting transmission position of the data according to the last data receiving position in the first handshake response. That is, after receiving the first handshake request sent by the execution subject, the communication receiving end learns through parsing the first handshake request that the communication initiating end wants to change the data transmission protocol used for subsequent data transmission, and specifically change it to use the TCP protocol. Accordingly, the communication receiving end will return a first handshake response containing the location of the last data received through the RDMA protocol (i.e., the last data receiving position) to the communication initiating end if it also supports the TCP protocol, so that the communication initiating end knows from the received first handshake response that the communication receiving end agrees to change the transmission protocol to the TCP protocol and clearly defines the data position where the transmission should start using the TCP protocol.

[0067] Step 204: Using TCP to start communicating with the communication receiving end from the data corresponding to the first starting transmission position.

[0068] On the basis of step 203, this step aims at the execution subject using TCP to communicate with the communication receiving end starting from the data corresponding to the first starting transmission position.

[0069] The communication method applied to the communication initiator provided by the embodiment of the present disclosure provides a solution for switching the transmission protocol between the remote direct data access RDMA protocol and the transmission control protocol TCP according to the packet loss rate for the communication initiator, so that when the packet loss rate is relatively high, the TCP protocol that can better achieve stable data transmission is selected, and when the packet loss rate is relatively low, the RDMA protocol that can better increase the data transmission volume is selected, so as to better take into account the network environment of cross-cluster communication and intra-cluster communication, thereby improving the overall communication efficiency and data transmission volume, and further improving the training efficiency of the generative large language model and shortening the time required for training.

[0070] Based on the solution provided in the previous embodiment on how to switch from the RDMA protocol to the TCP protocol to cope with a poor network environment, please refer to Figure 3 , Figure 3 A flow chart of a communication method for switching from a TCP protocol back to an RDMA protocol provided in an embodiment of the present disclosure, wherein process 300 includes the following steps:

[0071] Step 301: in the process of communicating with a communication receiving end using TCP, obtaining a second packet loss rate corresponding to the TCP protocol;

[0072] This step is intended to enable the above-mentioned execution subject to obtain a second packet loss rate corresponding to the TCP protocol during the process of communicating with the communication receiving end using TCP.

[0073] Among them, TCP (Transmission Control Protocol) is a connection-oriented, reliable, byte-stream-based transport layer communication protocol. The TCP protocol uses flow control, confirmation mechanism, and retransmission mechanism to ensure that data is correctly and completely transmitted from the sender to the receiver.

[0074] Based on the concept of packet loss rate that has been clarified in step 201, the second packet loss rate described in this step specifically refers to the packet loss rate calculated by the mechanism of the TCP protocol when the communication initiator uses the TCP protocol to communicate data with the communication receiver. This packet loss rate may be related to factors such as retransmission events, window size, flow control, and delay in the TCP protocol. The purpose of obtaining this packet loss rate is to measure the packet loss phenomenon that occurs during TCP communication in order to evaluate the transmission quality of the network or adjust the corresponding network parameters. Although the TCP protocol has a better ability to resist poor network environments than the RDMA protocol, the quality of the current network environment can also be reflected by the size of the second packet loss rate.

[0075] An implementation method including but not limited to may be:

[0076] The second packet loss rate is obtained from the second shared memory. Specifically, the second packet loss rate is obtained by calling the Socket interface by the collective communication library when using TCP and stored in the second shared memory. The second shared memory is a memory used to obtain information related to TCP.

[0077] Step 302: In response to the second packet loss rate meeting the preset network quality requirement, initiating a second handshake request to the communication receiving end to switch back to the RDMA protocol for communication;

[0078] This step is based on the fact that the packet loss rate determined in step 301 meets the preset network quality requirement, and is intended for the execution subject to initiate a second handshake request to the communication receiving end to switch back to the RDMA protocol for communication. Among them, the preset network quality requirement is used to require that the network communication quality should be good. Therefore, all parameter requirements that can characterize the current network environment as having good network communication quality can be used as the preset network quality requirement, such as at least one of the packet loss rate within a first preset time period being less than the second preset packet loss rate and the packet loss rate within a plurality of consecutive training steps (for example, within 3 consecutive training steps) being less than the second preset packet loss rate, the second preset packet loss rate is not greater than the first preset packet loss rate, and the training step refers to the specific training step in which the communication initiator initiates the communication to the communication receiving end for the purpose of training a generative large language model.

[0079] Among them, the second preset packet loss rate is a demarcation value or critical value determined based on historical experience to measure whether the current network communication quality is good or bad under the TCP protocol, that is, if the actual second packet loss rate is higher than the second preset packet loss rate, it can be considered that the current network communication quality is poor. On the contrary, if the actual second packet loss rate is not higher than the second preset packet loss rate, it can be considered that the current network communication quality is good. Specifically, the second preset packet loss rate can be set by experts with relevant experience in combination with the actual situation in a specific application scenario, or it can be obtained by analyzing the input samples representing the current application scenario through a pre-trained model, which is not specifically limited here. More specifically, the first preset packet loss rate can also be set to 4%.

[0080] Since the second packet loss rate meets the preset network quality requirement, it means that the execution subject is in a network environment with good network communication quality during the current period when communicating with the communication receiving end using the TCP protocol. Therefore, in order to make full use of the good network communication quality to maximize the amount of data transmitted per unit time and thereby improve the overall training task efficiency, the above-mentioned execution subject will initiate a second handshake request to the communication receiving end in this step, wanting to switch the current TCP protocol to the RDMA protocol that can bring a larger data transmission throughput per unit time under a better network environment. That is, the second handshake request is used to inform the communication receiving end that the communication initiator wants to switch the communication protocol and wants to switch to the RDMA protocol.

[0081] Step 303: receiving a second handshake response returned by the communication receiving end in response to the second handshake request, and determining a second starting transmission position of the data according to the last data receiving position in the second handshake response;

[0082] This step is intended to cause the above-mentioned execution subject to receive the second handshake response returned by the communication receiving end in response to the second handshake request, and determine the second starting transmission position of the data according to the last data reception position in the second handshake response. That is, after receiving the second handshake request sent by the above-mentioned execution subject, the communication receiving end learns through parsing the second handshake request that the communication initiating end wants to change the data transmission protocol used for subsequent data transmission, and specifically change it to use the RDMA protocol. Accordingly, if the communication receiving end also supports the RDMA protocol, it will return a second handshake response containing the location of the last data received through the TCP protocol (i.e., the last data reception position) to the communication initiating end, so that the communication initiating end knows from the received second handshake response that the communication receiving end agrees to change the transmission protocol to the RDMA protocol and clearly defines the data position where the RDMA protocol should be used to start transmission.

[0083] Step 304: Communicate with the communication receiving end starting from the data corresponding to the second starting transmission position using the RDMA protocol.

[0084] On the basis of step 303, this step aims at enabling the execution subject to communicate with the communication receiving end by using the RDMA protocol starting from the data corresponding to the second starting transmission position.

[0085] That is, this embodiment provides an implementation scheme of how to switch from the TCP protocol back to the RDMA protocol through steps 301 to 304, so as to confirm that the current network environment is better by finding that the second packet loss rate meets the preset network quality requirement, and thus choose to switch back to the RDMA protocol to make fuller use of the better network environment to transmit more data per unit time.

[0086] The transmission protocol switching strategy provided in the above embodiment is not actually selectively applied according to the communication type (i.e., intra-cluster communication or cross-cluster communication). In fact, both communication types can be adopted, that is, the switching strategy can be configured for each communication initiator in the computing power cluster, and there is no need to choose to use different switching strategies according to the communication type. However, considering that intra-cluster communication rarely encounters a poor network environment, which causes the RDMA protocol to correspond to a large packet loss rate, it is actually possible to configure the above switching strategy only for cross-cluster communication.

[0087] On this basis, in order to deepen your understanding of how to determine whether the communication initiator and the communication receiver belong to different computing power clusters, please refer to Figure 4 , Figure 4 A flowchart of a processing method for determining whether a communication receiving end belongs to a different computing power cluster according to actual delay provided in an embodiment of the present disclosure, wherein process 400 includes the following steps:

[0088] Step 401: Determine the actual delay with the communication receiving end;

[0089] This step is intended to determine the actual delay between the execution subject and the communication receiving end. An implementation method including but not limited to may be:

[0090] First, the time when data is sent to the communication receiving end is determined as the starting time; then, the time when the confirmation notification returned by the communication receiving end after receiving the data is received is determined as the ending time; finally, based on the starting time and the ending time, the actual delay is calculated.

[0091] Furthermore, in order to ensure the accuracy of the actual delay calculated, a preset logarithm of start and end time pairs consisting of a start time and a matching end time can be selected, that is, the delay values ​​corresponding to each start and end time pair are calculated respectively to obtain a delay set, and then the actual delay represented by the average delay is obtained by taking the average of each delay value in the delay set, so as to eliminate accidental or fluctuating errors by taking the average value.

[0092] Step 402: Determine whether the actual delay exceeds the preset delay, if so, execute step 403, otherwise execute step 405;

[0093] This step is intended for the execution subject to compare the actual delay with the preset delay, and then execute different subsequent processing branches according to the two comparison results. In this embodiment, the preset delay exists as a delay boundary value between a higher delay and a lower delay, and the higher delay corresponds to the delay during cross-cluster communication, and the lower delay corresponds to the delay during intra-cluster communication.

[0094] Step 403: Determine that the communication receiving end belongs to a different computing power cluster;

[0095] This step is based on the judgment result of step 402 that the actual delay exceeds the preset delay, and is intended for the above-mentioned execution subject to determine that the communication receiving end belongs to a different computing power cluster.

[0096] Specifically, the different computing power clusters described in this embodiment mainly refer to computing power clusters for training generative large language models, which are composed of graphics processors in different computer rooms located in different locations. If different computer rooms are directly divided into different computing power clusters, different computing power clusters can also be simply understood as different computer rooms, that is, cross-cluster communication can also be adaptively changed to cross-computer room communication.

[0097] Step 404: Control the default to use the RDMA protocol to communicate with the communication receiving end belonging to different computing power clusters;

[0098] On the basis of step 403, the execution subject is intended to control the default use of the RDMA protocol to communicate with the communication receiving end belonging to different computing power clusters, that is, the switching strategy provided by the above embodiment is adopted for the cross-cluster communication configuration, and the RDMA protocol is set to be used by default first, and the TCP protocol is switched only when a high packet loss rate occurs during the use of the RDMA protocol.

[0099] Step 405: Determine that the communication receiving end and the communication receiving end belong to the same computing power cluster;

[0100] This step is based on the judgment result of step 402 that the actual delay does not exceed the preset delay, and is intended for the above-mentioned execution subject to determine that the communication receiving end belongs to the same computing power cluster.

[0101] In contrast to different computing power clusters, which mainly refer to computing power clusters for training generative large language models that are composed of graphics processors in different computer rooms at different locations, the same computing power cluster can refer to computing power clusters for training generative large language models that are composed of graphics processors in the same computer room at the same location. Similarly, if different computing power clusters can be simplified to different computer rooms, the same computing power cluster can also be simplified to the same computer room.

[0102] Step 406: Control to use only the RDMA protocol to communicate with the communication receiving end belonging to the same computing power cluster.

[0103] On the basis of step 405, this step aims to control the communication initiator for intra-cluster communication to use only the RDMA protocol to communicate with the communication receiver belonging to the same computing power cluster. That is, the above switching strategy is not configured for the intra-cluster communication device, so that it can only use the RDMA protocol for data communication. This is a solution given based on the actual situation that intra-cluster communication rarely encounters a poor internal network environment, which affects the intra-cluster communication effect.

[0104] Of course, in the case of intra-cluster communication, you can also directly use the transmission protocol provided by the existing collective communication library for data communication. NCCL (NVIDIA Collective Communications Library) is an efficient communication library developed by NVIDIA, mainly used to accelerate collective communication operations in deep learning frameworks. It is widely used in distributed training, especially in multi-GPU or multi-node environments. NCCL focuses on providing high-performance communication protocols and algorithms, aiming to significantly improve data transmission speed and parallel computing efficiency by leveraging GPU hardware acceleration.

[0105] NCCL provides a variety of common collective communication operations (CollectiveCommunications) for distributed training. These operations are usually used to synchronize and share data between multiple computing nodes. In deep learning training, common collective communication operations include:

[0106] 1) AllReduce: multiple nodes exchange data and perform some kind of aggregation on the data (such as summation, averaging, etc.). In deep learning, it is often used to synchronize gradients in distributed training;

[0107] 2) Broadcast: Broadcast data from one node to all other nodes. Often used to synchronize model parameters or other shared data to various nodes;

[0108] 3) AllGather: Collects and aggregates data from multiple nodes to all nodes. Used for input aggregation during data distributed training;

[0109] 4) Reduce: Exchange data between multiple nodes and aggregate the data, usually aggregating data from multiple nodes to one node;

[0110] 5) ReduceScatter: Aggregates data from multiple nodes and distributes it to each node. Contrast with AllGather.

[0111] The following advantages can be achieved by using NCCL:

[0112] 1) Hardware acceleration: NCCL takes full advantage of NVIDIA’s hardware features, such as NVLink (for high-bandwidth communication between GPUs) and NVIDIA RDMA, providing extremely high-bandwidth and low-latency communication;

[0113] 2) Multi-GPU support: NCCL is designed specifically for GPUs and supports single-node and multi-GPU configurations. It also supports distributed training across nodes, which can effectively improve training efficiency.

[0114] 3) Efficient algorithms: NCCL implements a variety of optimized communication algorithms that are optimized for different network topologies and hardware characteristics to minimize communication bottlenecks;

[0115] 4) Ease of use and compatibility: NCCL provides an easy-to-integrate API interface that can be seamlessly integrated with popular deep learning frameworks. Its API design is concise and clear, allowing users to focus on application layer logic without having to worry about the underlying communication implementation;

[0116] 5) Cross-platform support: NCCL supports a variety of Linux systems and GPU hardware in the CUDA environment, and is suitable for various multi-GPU configurations.

[0117] In addition, NCCL is used as the default communication library by multiple deep learning frameworks (such as TensorFlow, PyTorch, MXNet, etc.) to accelerate communication during multi-GPU or distributed training. Specifically, these frameworks call the NCCL library to handle tasks such as gradient synchronization and model parameter broadcasting. In this way, the framework can achieve efficient distributed training, especially in scenarios where multiple GPUs are used. For example, the torch.distributed module in PyTorch provides integration with the NCCL library, and the communication backend can be specified as NCCL through configuration to perform efficient communication operations in a multi-GPU environment.

[0118] This embodiment provides a method for determining whether it is intra-cluster communication or cross-cluster communication based on the actual delay between the two ends of communication through steps 401-406, and provides corresponding methods for determining matching transmission protocol switching strategies and configurations for the two different communication types, thereby performing different types of communications through different protocol switching strategies, thereby ultimately improving the overall communication efficiency and data throughput.

[0119] In addition to determining whether it is intra-cluster communication or cross-cluster communication based on the actual delay in the previous embodiment, it can also be determined based on the packet loss rate. For example, if the first packet loss rate is lower than the first preset packet loss rate within the second preset time period, then it can be determined that the communication receiving end belongs to the same computing power cluster, and then the above-mentioned execution subject can also control the use of the RDMA protocol to communicate with the communication receiving end belonging to the same computing power cluster.

[0120] To deepen understanding, the present disclosure also combines specific application scenarios, and gives detailed explanations of the problems existing in the application scenarios, the ideas for solving the problems, and the specific implementation solutions:

[0121] As the demand for computing power in the era of large models gradually increases, the number of chips in the cluster is increasingly required. A single cluster of 1,000 or even 10,000 chips can no longer meet the demand. As a result, the training of larger clusters in the future will have to rely on multiple computing clusters. These clusters are usually distributed in different physical locations, even across regions. How to efficiently use these clusters and minimize the impact of cross-computer room communication time on training performance is a hot topic in the industry.

[0122] Usually a GPU server will be equipped with multiple GPU network cards (Graphics Processing Unit NetworkInterface Card, which is a hardware device that combines GPU and network interface card. It combines traditional network communication functions with powerful parallel computing capabilities. It is usually used in high-performance computing, artificial intelligence and big data analysis. Compared with traditional network cards, GPU network cards can perform parallel computing while transmitting data, significantly improving the efficiency and speed of data processing, especially when processing large amounts of data and complex calculations). These GPU network cards will be responsible for the transmission of cross-machine traffic between GPUs. At the same time, a main network card will be configured to meet basic network needs such as server management, access to cloud services, storage, etc. There are two possible ways to interconnect data center clusters, such as Figure 5-1 As shown in the figure, one way is to use dedicated lines to connect the HPN (High Performance Networking) AI networks of different clusters. This method can build a larger interconnection bandwidth to meet the actual needs, and can achieve better performance after optimizing the efficiency of RDMA long-distance transmission across computer rooms, but the construction cost is relatively high; another way is to directly reuse DCN (Data Center Network). Using DCN does not require additional construction costs, but the DCN bandwidth across computer rooms is not guaranteed. These bandwidths are shared and cannot be guaranteed to be exclusively used by model training, so the network quality is also not guaranteed.

[0123] For most R&D personnel related to model training, they usually prefer to reuse the existing infrastructure. Therefore, without introducing additional construction costs, they usually choose to reuse the DCN network solution to connect multiple computing clusters. In this scenario, the model usually chooses pipeline parallelism, that is, pp (Point The strategy of dividing the data flow (point-to-point) into different computer rooms is because pp transmission can be overlapped (data streams or signals can be transmitted on the same transmission medium in an overlapping manner, and clever technical means are used to ensure that they do not interfere with each other and can effectively utilize bandwidth and transmission resources). If the pp transmission performance is not too bad, it can be hidden by the computing process. In this case, there may be two scenarios for pp transmission, one is pp transmission within the computing power cluster, and the other is pp transmission across computer rooms. The communication of large model training will use the RDMA of the collective communication library NCCL for transmission, but because the DCN network across computer rooms is shared and the bandwidth is not guaranteed, there may be a lot of packet loss during peak hours, resulting in poor RDMA transmission. At this time, choosing RDMA for transmission is no longer a suitable choice, and it needs to be degraded to TCP protocol for transmission. However, for the two scenarios within the computer room and across computer rooms, RDMA is usually better for the former, and cross-computer room involves the question of whether TCP protocol is better or RDMA protocol is better. The current collective communication library NCCL cannot automatically select, and can only configure a fixed parameter in advance, which cannot adapt to both scenarios at the same time.

[0124] Therefore, this embodiment needs to implement the following key functions to overcome the defects of the prior art:

[0125] 1) The packet loss rate can be obtained in RDMA transmission and TCP transmission scenarios respectively;

[0126] 2) How to automatically switch between two transport protocols in the collective communication library.

[0127] That is, this embodiment intends to provide a solution for adaptively adjusting the transmission protocol. The communication library can automatically configure whether it is RDMA transmission or TCP transmission based on information such as the bandwidth and delay of the link and the packet loss rate. It is particularly suitable for scenarios such as remote mixed training where there is pp transmission in the same computer room and across computer rooms, and the bandwidth and network quality across computer rooms cannot be guaranteed, thereby solving the technical defects that the current related solutions only support static configuration, can only adapt to a single scenario, and are not flexible enough. The specific implementation means will be described in detail below:

[0128] 1. Packet loss rate detection

[0129] The result of packet loss rate detection is very important for determining whether RDMA is needed for transmission, because generally speaking, the DCN network across computer rooms will not be configured with PFC (Priority Flow Control), etc., and RDMA transmission cannot be guaranteed to be in a lossless environment without packet loss. The existing collective communication library does not consider the link quality issue whether using TCP or RDMA, so it does not have the function of link packet loss rate detection. This embodiment designs a packet loss rate detection module for RDMA transmission and TCP transmission respectively. The collective communication library NCCL can obtain the packet loss rate of RDMA and TCP through the packet loss rate detection module. For specific acquisition methods, please refer to Figure 5-2 shown.

[0130] That is, for RDMA, because the collective communication library uses the verbs interface internally to perform data transmission and reception operations of the RDMA network card, there is no special verbs interface to obtain packet loss information. Therefore, this embodiment will create a special RDMA retransmission rate statistics process to obtain the retransmission statistics information of the network card, which can be obtained through the register of the network card. The RDMA retransmission rate statistics process will periodically obtain this data from the register and store it in the shared memory of the server. The collective communication library NCCL in the large model training process will also periodically access the shared memory and read the RDMA network card retransmission rate data in the shared memory.

[0131] For TCP, the NCCL collective communication library can directly use the TCP-related Socket interface to obtain information about the number of retransmitted packets and the total number of sent data packets, so no additional process is required to obtain them.

[0132] In this embodiment, the process of the collective communication library obtaining the RDMA and TCP retransmission rates is as follows: Figure 5-3 As shown in the figure, because there is a switch between RDMA connection and TCP connection, the collective communication library needs to establish both RDMA connection and TCP connection at the initialization stage, instead of just one of them. After the connection establishment process is completed, the RDMA retransmission rate detection process and the TCP connection retransmission rate detection thread need to be started respectively. They will continuously obtain the retransmission rate information by reading the network card register and the related function interface of the Socket, which will serve as the basis for the subsequent transmission protocol switching.

[0133] Reading the retransmission rate information of RDMA or TCP depends on which transport protocol is currently used, so the shared memory areas of the two transport protocols will not be read at the same time.

[0134] 2. Switch the RDMA / TCP transport protocol based on the packet loss rate information

[0135] Cross-computer room transmission is usually pp transmission in large model training. In this process, the sender and receiver will give priority to using the RDMA protocol for transmission. That is, assuming that the DCN network bandwidth across computer rooms is sufficient and the network quality is good, try to use the RDMA transmission protocol first; the subsequent process is mainly divided into two steps:

[0136] 1) Switching from RDMA protocol to TCP protocol

[0137] The sender of pp transmission will constantly detect the retransmission rate of RDMA. If it exceeds 5%, it will usually have a significant impact on the transmission performance of RDMA, and the protocol switching needs to be triggered. Since the protocol switching process requires both parties to perceive, there is a handshake process: the sender will send a control signal to the receiver to inform the receiver that it can prepare to switch the protocol. The receiver will record the location of the received data at this time and inform the sender of this location through the second handshake. After receiving it, the sender will know where to start transmitting data after switching to the TCP protocol, and will use the TCP protocol to transmit data thereafter.

[0138] 2) TCP protocol switches back to RDMA protocol transmission

[0139] The purpose of switching to TCP protocol is to deal with the poor quality of DCN network and avoid a large performance drop that will have a great impact on model training. This process usually occurs during peak hours. If it is off-peak, there is not much business traffic sharing, the network bandwidth is restored, and the packet loss rate decreases, then RDMA transmission can be restored. At this time, in order to avoid oscillation, the packet loss rate of the three model training steps will be continuously observed. If the TCP packet loss rate does not exceed 5% in the three training steps, it means that the network quality has been restored, then prepare to notify the receiving end to switch back to RDMA transmission. The process is the same as the process of switching to TCP. A handshake process will be initiated to negotiate the location of data transmission. After the negotiation is completed, RDMA can be switched back to transmission.

[0140] The complete switching strategy provided by the above complete solution can be found in Figure 5-4 The flow chart shown presents both the sending side and the receiving side, which correspond to the communication initiating end and the communication receiving end in the above-mentioned embodiment respectively.

[0141] Further references Figure 6 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a communication device for a computing power cluster. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0142] like Figure 6As shown, the communication device 600 for the computing power cluster of this embodiment may include: a first packet loss rate acquisition unit 601, a TCP protocol switching handshake request initiating unit 602, a first handshake response receiving and first starting transmission position determining unit 603, and a TCP protocol communication unit 604. Among them, the first packet loss rate acquisition unit 601 is configured to acquire the first packet loss rate corresponding to the RDMA protocol in the process of communicating with the communication receiving end using the remote direct data access RDMA protocol; the TCP protocol switching handshake request initiating unit 602 is configured to initiate a first handshake request to the communication receiving end to switch to the transmission control protocol TCP for communication in response to the first packet loss rate being higher than the first preset packet loss rate; the first handshake response receiving and first starting transmission position determining unit 603 is configured to receive the first handshake response returned by the communication receiving end for the first handshake request, and determine the first starting transmission position of the data according to the last data receiving position in the first handshake response; the TCP protocol communication unit 604 is configured to communicate with the communication receiving end using TCP starting from the data corresponding to the first starting transmission position.

[0143] In this embodiment, for the communication device 600 of the computing power cluster: the specific processing of the first packet loss rate acquisition unit 601, the TCP protocol switching handshake request initiating unit 602, the first handshake response receiving and the first starting transmission position determining unit 603, and the TCP protocol communication unit 604 and the technical effects thereof can be referred to respectively. Figure 2 The relevant descriptions of steps 201 - 204 in the corresponding embodiment are not repeated here.

[0144] In some optional implementations of this embodiment, the first packet loss rate acquiring unit 601 may be further configured to:

[0145] A first packet loss rate is obtained from a first shared memory corresponding to the RDMA network card; wherein the first packet loss rate is obtained from a register of the RDMA network card by a first packet loss rate acquisition process started when communicating using the RDMA protocol and stored in the first shared memory.

[0146] In some optional implementations of this embodiment, the communication device 600 for the computing power cluster may also include:

[0147] A second packet loss rate acquisition unit is configured to acquire a second packet loss rate corresponding to the TCP protocol during the process of communicating with the communication receiving end using TCP;

[0148] An RDMA protocol switching handshake request initiating unit is configured to initiate a second handshake request to switch back to the RDMA protocol for communication to the communication receiving end in response to the second packet loss rate meeting the preset network quality requirement;

[0149] A second handshake response receiving and second start transmission position determining unit is configured to receive a second handshake response returned by the communication receiving end in response to the second handshake request, and determine a second start transmission position of the data according to the last data receiving position in the second handshake response;

[0150] The RDMA protocol communication unit is configured to communicate with the communication receiving end by using the RDMA protocol starting from the data corresponding to the second starting transmission position.

[0151] In some optional implementations of this embodiment, the second packet loss rate acquiring unit may be further configured to:

[0152] The second packet loss rate is obtained from the second shared memory; wherein the second packet loss rate is obtained by calling the socket interface by the collective communication library when using TCP and is stored in the second shared memory.

[0153] In some optional implementations of this embodiment, the preset network quality requirement includes at least one of the following:

[0154] The packet loss rate within a first preset time period is less than a second preset packet loss rate, and the packet loss rate within a plurality of consecutive training steps is less than the second preset packet loss rate; wherein the second preset packet loss rate is not greater than the first preset packet loss rate.

[0155] In some optional implementations of this embodiment, the communication device 600 for the computing power cluster may also include:

[0156] The different computing power cluster belonging determination unit is configured to determine that the communication receiving end belongs to different computing power clusters respectively in response to the actual delay between the communication receiving end and the communication receiving end exceeding the preset delay;

[0157] The first control unit is configured to control the default use of the RDMA protocol to communicate with the communication receiving end belonging to different computing power clusters.

[0158] In some optional implementations of this embodiment, the communication device 600 may further include: an actual delay calculation unit, which may include:

[0159] A starting time determination subunit, configured to determine the time of sending data to the communication receiving end as the starting time;

[0160] The end time determination subunit is configured to determine the time of receiving the confirmation notification returned by the communication receiving end after receiving the data as the end time;

[0161] The actual delay calculation subunit is configured to calculate the actual delay based on the start time and the end time.

[0162] In some optional implementations of this embodiment, the actual delay calculation subunit may be further configured as follows:

[0163] Calculate the delay values ​​corresponding to the start and end time pairs consisting of the start time and the matching end time of the preset number of logarithms, and obtain the delay set;

[0164] The average of the delay values ​​in the delay set is calculated to obtain the actual delay.

[0165] In some optional implementations of this embodiment, the communication device 600 may further include:

[0166] A first same computing power cluster belonging determination unit is configured to determine that both the communication receiving end and the communication receiving end belong to the same computing power cluster in response to the actual delay between the communication receiving end and the communication receiving end not exceeding the preset delay;

[0167] The second control unit is configured to control communication to a communication receiving end belonging to the same computing power cluster using only the RDMA protocol.

[0168] In some optional implementations of this embodiment, the communication device 600 may further include:

[0169] A second same computing power cluster belonging determination unit is configured to determine that both the communication receiving end and the communication receiving end belong to the same computing power cluster in response to the first packet loss rate being lower than the first preset packet loss rate within a second preset time period;

[0170] The third control unit is configured to control communication to a communication receiving end belonging to the same computing power cluster using only the RDMA protocol.

[0171] In some optional implementations of this embodiment, different computing power clusters include: computing power clusters for training generative large language models composed of graphics processors in different computer rooms located at different locations; correspondingly, the same computing power cluster includes: computing power clusters for training generative large language models composed of graphics processors in the same computer room located at the same location.

[0172] This embodiment exists as an apparatus embodiment corresponding to the above-mentioned method embodiment. The communication device for a computing power cluster provided by this embodiment provides a scheme for switching the transmission protocol between the remote direct data access RDMA protocol and the transmission control protocol TCP according to the packet loss rate for the communication initiator, so that when the packet loss rate is relatively high, the TCP protocol that can better achieve stable data transmission is selected, and when the packet loss rate is relatively low, the RDMA protocol that can better increase the data transmission volume is selected, so as to better take into account the network environment of cross-cluster communication and intra-cluster communication, thereby improving the overall communication efficiency and data transmission volume, and then improving the training efficiency of the generative large language model and shortening the time required for training.

[0173] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can implement the communication method for a computing power cluster described in any of the above embodiments when executing.

[0174] According to an embodiment of the present disclosure, the present disclosure further provides a readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to implement the communication method for a computing power cluster described in any of the above embodiments when executed.

[0175] According to an embodiment of the present disclosure, the present disclosure also provides a computer program product, which, when executed by a processor, can implement the steps of the communication method for a computing power cluster described in any of the above embodiments.

[0176] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0177] like Figure 7As shown, the device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0178] A number of components in the device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0179] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above, such as the communication method. For example, in some embodiments, the communication method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the communication method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform the communication method in any other appropriate manner (e.g., by means of firmware).

[0180] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0181] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0182] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0183] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0184] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0185] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client-server relationship is generated by computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and virtual private server (VPS) services.

[0186] According to the technical solution of the embodiment of the present disclosure, a solution is provided for the communication initiator to switch the transmission protocol between the remote direct data access RDMA protocol and the transmission control protocol TCP according to the packet loss rate, so that when the packet loss rate is relatively high, the TCP protocol that can better achieve stable data transmission is selected, and when the packet loss rate is relatively low, the RDMA protocol that can better increase the data transmission volume is selected, so as to better take into account the network environment of cross-cluster communication and intra-cluster communication, thereby improving the overall communication efficiency and data transmission volume, and further improving the training efficiency of the generative large language model and shortening the time required for training.

[0187] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0188] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A communication method for a computing power cluster, applied to a communication initiator, comprising: In a process of communicating with a communication receiving end using a remote direct data access (RDMA) protocol, obtaining a first packet loss rate corresponding to the RDMA protocol; In response to the first packet loss rate being higher than a first preset packet loss rate, initiating a first handshake request to the communication receiving end to switch to the transmission control protocol TCP for communication; receiving a first handshake response returned by the communication receiving end in response to the first handshake request, and determining a first starting transmission position of data according to a last data receiving position in the first handshake response; Using the TCP to communicate with the communication receiving end starting from the data corresponding to the first starting transmission position; In response to the actual delay between the communication receiving end and the communication receiving end exceeding the preset delay, determining that the communication receiving end and the communication receiving end belong to different computing power clusters, respectively, and the different computing power clusters include: computing power clusters for training a generative large language model composed of graphics processors in different computer rooms located at different locations; The control defaults to using the RDMA protocol to communicate with the communication receiving end belonging to different computing power clusters.

2. The method according to claim 1, wherein: The obtaining a first packet loss rate corresponding to the RDMA protocol includes: The first packet loss rate is obtained from a first shared memory corresponding to the RDMA network card; wherein the first packet loss rate is obtained from a register of the RDMA network card by a first packet loss rate acquisition process started when communicating using the RDMA protocol and stored in the first shared memory.

3. The method according to claim 1, further comprising: In the process of communicating with the communication receiving end using the TCP, obtaining a second packet loss rate corresponding to the TCP protocol; In response to the second packet loss rate meeting the preset network quality requirement, initiating a second handshake request to the communication receiving end to switch back to the RDMA protocol for communication; receiving a second handshake response returned by the communication receiving end in response to the second handshake request, and determining a second starting transmission position of the data according to the last data receiving position in the second handshake response; The RDMA protocol is used to communicate with the communication receiving end starting from the data corresponding to the second starting transmission position.

4. The method according to claim 3, wherein: The obtaining of a second packet loss rate corresponding to the TCP protocol includes: The second packet loss rate is obtained from the second shared memory; wherein the second packet loss rate is obtained by calling the socket interface by the collective communication library when using the TCP and is stored in the second shared memory.

5. The method according to claim 3, wherein: The preset network quality requirement includes at least one of the following: The packet loss rate within a first preset time period is less than a second preset packet loss rate, and the packet loss rate within a plurality of consecutive training steps is less than the second preset packet loss rate; wherein the second preset packet loss rate is not greater than the first preset packet loss rate.

6. The method according to claim 1, wherein: The actual delay is calculated by the following steps: Determine the time when data is sent to the communication receiving end as the starting time; Determine the time when the confirmation notification returned by the communication receiving end after receiving the data is received as the end time; The actual delay is obtained by calculation based on the start time and the end time.

7. The method according to claim 6, wherein: The calculating the actual delay based on the start time and the end time includes: Calculate the delay values ​​corresponding to the start and end time pairs consisting of the start time and the matching end time of the preset number of logarithms, and obtain the delay set; An average of the delay values ​​in the delay set is calculated to obtain the actual delay.

8. The method according to claim 1, further comprising: In response to the actual delay between the communication receiving end and the communication receiving end not exceeding the preset delay, determining that the communication receiving end and the communication receiving end belong to the same computing power cluster; Control to communicate only with the communication receiving end belonging to the same computing power cluster using the RDMA protocol.

9. The method according to claim 1, further comprising: In response to the first packet loss rate being lower than the first preset packet loss rate within a second preset time period, determining that the communication receiving end and the communication receiving end belong to the same computing power cluster; Control to communicate only with the communication receiving end belonging to the same computing power cluster using the RDMA protocol.

10. The method according to any one of claims 1 to 9, wherein: The same computing power cluster includes: a computing power cluster for training a generative large language model, which is composed of graphics processors located in the same computer room at the same location.

11. A communication device for a computing power cluster, applied to a communication initiator, comprising: A first packet loss rate acquisition unit is configured to acquire a first packet loss rate corresponding to a remote direct data access RDMA protocol during communication with a communication receiving end using the RDMA protocol; A TCP protocol switching handshake request initiating unit, configured to initiate a first handshake request to switch to a transmission control protocol TCP for communication to the communication receiving end in response to the first packet loss rate being higher than a first preset packet loss rate; a first handshake response receiving and first start transmission position determining unit, configured to receive a first handshake response returned by the communication receiving end in response to the first handshake request, and determine a first start transmission position of data according to a last data receiving position in the first handshake response; A TCP protocol communication unit, configured to communicate with the communication receiving end using the TCP starting from the data corresponding to the first starting transmission position; The different computing power clusters belonging determination unit is configured to determine that the communication receiving end belongs to different computing power clusters in response to the actual delay between the communication receiving end and the communication receiving end exceeding the preset delay, and the different computing power clusters include: computing power clusters for training generative large language models composed of graphics processors in different computer rooms located at different locations; The first control unit is configured to control the default use of the RDMA protocol to communicate with the communication receiving end belonging to different computing power clusters.

12. The device according to claim 11, wherein The first packet loss rate acquiring unit is further configured to: The first packet loss rate is obtained from a first shared memory corresponding to the RDMA network card; wherein the first packet loss rate is obtained from a register of the RDMA network card by a first packet loss rate acquisition process started when communicating using the RDMA protocol and stored in the first shared memory.

13. The apparatus according to claim 11, further comprising: A second packet loss rate acquisition unit is configured to acquire a second packet loss rate corresponding to the TCP protocol during the process of communicating with the communication receiving end using the TCP protocol; An RDMA protocol switching handshake request initiating unit, configured to initiate a second handshake request to the communication receiving end to switch back to the RDMA protocol for communication in response to the second packet loss rate meeting a preset network quality requirement; A second handshake response receiving and second starting transmission position determining unit is configured to receive a second handshake response returned by the communication receiving end in response to the second handshake request, and determine a second starting transmission position of the data according to the last data receiving position in the second handshake response; The RDMA protocol communication unit is configured to communicate with the communication receiving end by using the RDMA protocol starting from the data corresponding to the second starting transmission position.

14. The device according to claim 13, wherein: The second packet loss rate acquiring unit is further configured to: The second packet loss rate is obtained from the second shared memory; wherein the second packet loss rate is obtained by calling the socket interface by the collective communication library when using the TCP and is stored in the second shared memory.

15. The device according to claim 13, wherein: The preset network quality requirement includes at least one of the following: The packet loss rate within a first preset time period is less than a second preset packet loss rate, and the packet loss rate within a plurality of consecutive training steps is less than the second preset packet loss rate; wherein the second preset packet loss rate is not greater than the first preset packet loss rate.

16. The apparatus according to claim 11, further comprising: An actual delay calculation unit, the actual delay calculation unit comprising: A starting time determination subunit, configured to determine the time when data is sent to the communication receiving end as the starting time; The end time determination subunit is configured to determine the time of receiving the confirmation notification returned by the communication receiving end after receiving the data as the end time; The actual delay calculation subunit is configured to calculate the actual delay based on the start time and the end time.

17. The device according to claim 16, wherein: The actual delay calculation subunit is further configured to: Calculate the delay values ​​corresponding to the start and end time pairs consisting of the start time and the matching end time of the preset number of logarithms, and obtain the delay set; An average of the delay values ​​in the delay set is calculated to obtain the actual delay.

18. The apparatus according to claim 11, further comprising: A first same computing power cluster belonging determination unit is configured to determine that both the communication receiving end and the communication receiving end belong to the same computing power cluster in response to the actual delay between the communication receiving end and the communication receiving end not exceeding the preset delay; The second control unit is configured to control the communication to the communication receiving end belonging to the same computing power cluster only using the RDMA protocol.

19. The apparatus according to claim 11, further comprising: A second same computing power cluster belonging determination unit is configured to determine that both the communication receiving end and the communication receiving end belong to the same computing power cluster in response to the first packet loss rate being lower than the first preset packet loss rate within a second preset time period; The third control unit is configured to control the use of only the RDMA protocol to communicate with the communication receiving end belonging to the same computing power cluster.

20. The device according to any one of claims 11 to 19, wherein: The same computing power cluster includes: a computing power cluster for training a generative large language model, which is composed of graphics processors located in the same computer room at the same location.

21. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the communication method for a computing power cluster described in any one of claims 1-10.

22. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute the communication method for a computing power cluster according to any one of claims 1-10.

23. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of the communication method for a computing power cluster according to any one of claims 1 to 10 are implemented.

Images