Cross-computing power cluster communication method, device, electronic device and storage medium

Abstract

The present disclosure provides a cross-computing power cluster communication method, device, electronic device and storage medium, which relate to the technical fields of graphics processor, computing power cluster, model training, generative large language model, etc. The method includes: in response to the communication receiving end belonging to different computing power clusters, the number of RDMA connections is increased on the basis of the initial number of connections until the detected actual bandwidth value no longer continues to increase, and the target number of connections is obtained; while maintaining the target number of connections, the size of the buffer is increased on the basis of the initial size until the detected actual bandwidth value no longer continues to increase, and the target size is obtained; based on the target number of connections and the target size, the cross-cluster transmission parameters are determined; and the communication receiving end belonging to different computing power clusters is communicated according to the cross-cluster transmission parameters. The application of this method can improve the overall communication efficiency and data transmission volume, and also improve the training efficiency of the generative large language model and shorten the time required for training.

Description

Technical Field

[0001] The present disclosure relates to the field of communication technology, specifically to technical fields such as graphics processors, computing power clusters, model training, generative large language models, and more particularly to a cross-computing power cluster communication method, device, electronic device, computer-readable storage medium, and computer program product. Background Art

[0002] As the amount of training computations gradually expands in the era of large models, more and more chips and larger computing clusters are needed. Building a single computing cluster of 100,000 cards can achieve extreme computing efficiency, but it has extremely high requirements for infrastructure and may not be able to be placed in a single computer room.

[0003] At the same time, for scenarios that require Wanka training, if multiple Qianka clusters can be utilized to reduce the construction cost of the Wanka cluster, as well as to efficiently utilize fragmented Qianka clusters and activate computing power supply, these are issues that technical personnel in this field need to solve urgently. Summary of the invention

[0004] The embodiments of the present disclosure provide a method, device, electronic device, computer-readable storage medium, and computer program product for cross-computing power cluster communication.

[0005] In a first aspect, an embodiment of the present disclosure proposes a cross-computing power cluster communication method, which is applied to a communication initiator, including: in response to belonging to different computing power clusters respectively from the communication receiving end, increasing the number of remote direct data access RDMA connections on the basis of the initial number of connections until the detected actual bandwidth value no longer continues to increase, thereby obtaining a target number of connections; while the number of RDMA connections remains at the target number of connections, increasing the size of the buffer buffer on the basis of the initial size until the detected actual bandwidth value no longer continues to increase, thereby obtaining a target size; determining cross-cluster transmission parameters based on the RDMA connection number set as the target number of connections and the buffer size set as the target size; and communicating with the communication receiving end belonging to the different computing power clusters according to the cross-cluster transmission parameters.

[0006] In a second aspect, an embodiment of the present disclosure proposes a cross-computing power cluster communication device, which is applied to a communication initiator, and includes: a connection number increase processing unit, which is configured to increase the number of remote direct data access (RDMA) connections on the basis of the initial number of connections in response to the communication receiving end belonging to different computing power clusters respectively, until the detected actual bandwidth value no longer continues to increase, thereby obtaining a target number of connections; a buffer increase processing unit, which is configured to increase the size of the buffer buffer on the basis of the initial size while the RDMA connection number remains at the target connection number, until the detected actual bandwidth value no longer continues to increase, thereby obtaining a target size; a cross-cluster transmission parameter determination unit, which is configured to determine the cross-cluster transmission parameters based on the RDMA connection number set as the target connection number and the buffer size set as the target size; a first cross-cluster communication unit, which is configured to communicate with the communication receiving end belonging to different computing power clusters according to the cross-cluster transmission parameters.

[0007] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that when the at least one processor executes, the cross-computing power cluster communication method described in the first aspect can be implemented.

[0008] In a fourth aspect, an embodiment of the present disclosure provides a non-transitory computer-readable storage medium storing computer instructions, which are used to enable a computer to implement the cross-computing power cluster communication method described in the first aspect when executed.

[0009] In a fifth aspect, an embodiment of the present disclosure provides a computer program product comprising a computer program, which, when executed by a processor, can implement the steps of the cross-computing power cluster communication method described in the first aspect.

[0010] The cross-computing power cluster communication solution provided by the present invention provides a method for re-determining cross-cluster transmission parameters that match the cross-cluster communication environment for the situation where the communication initiator and the communication receiver belong to different computing power clusters respectively. That is, since cross-cluster communication has a significantly increased latency compared to intra-cluster communication, a method for successively increasing the number of remote direct data access (RDMA) connections and the size of the buffer zone BUFFER is provided for cross-cluster communication to determine the target number of connections and the target size that match the actual bandwidth value, and then communicating with the communication receivers belonging to different computing power clusters according to the cross-cluster transmission parameters determined based on the target number of connections and the target size, so that cross-cluster communication and intra-cluster communication respectively use transmission parameters that match the actual communication environment, thereby improving the overall communication efficiency and data transmission volume, thereby improving the training efficiency of the generative large language model and shortening the time required for training.

[0011] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Other features, objects and advantages of the present disclosure will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0013] Figure 1 is an exemplary system architecture in which the present disclosure may be applied;

[0014] Figure 2 A flow chart of a cross-computing power cluster communication method provided in an embodiment of the present disclosure;

[0015] Figure 3 A flowchart of a method for determining whether a communication receiving end belongs to a different computing power cluster according to actual delay provided in an embodiment of the present disclosure;

[0016] Figure 4 A flowchart of a method for increasing the number of RDMA connections and determining a target number of connections provided by an embodiment of the present disclosure;

[0017] Figure 5 A flowchart of a method for increasing the buffer size and determining the target size provided by an embodiment of the present disclosure;

[0018] Figure 6 A flowchart of a method for adjusting cross-cluster transmission parameters according to a degree of bandwidth reduction provided by an embodiment of the present disclosure;

[0019] Figure 7-1 A schematic diagram of a structure of cross-data center transmission provided by an embodiment of the present disclosure;

[0020] Figure 7-2 A flow chart of a cross-computing power cluster communication method combined with a specific application scenario provided in an embodiment of the present disclosure;

[0021] Figure 8 A structural block diagram of a cross-computing power cluster communication device provided in an embodiment of the present disclosure;

[0022] Fig. 9 A schematic diagram of the structure of an electronic device suitable for executing a cross-computing power cluster communication method provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0023] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description. It should be noted that the embodiments in the present disclosure and the features in the embodiments may be combined with each other without conflict.

[0024] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0025] Figure 1 An exemplary system architecture 100 is shown to which embodiments of the cross-computing power cluster communication method, apparatus, electronic device, and computer-readable storage medium disclosed in the present invention can be applied.

[0026] like Figure 1 As shown, the system architecture 100 may include a first computing cluster 110 and a second computing cluster 120. The first computing cluster 110 includes multiple graphics processors, such as 111, 112, 113, 114, and 115 in the form of graphics cards (not just 5, Figure 1 Only a few are used as examples), the second computing cluster 120 also includes multiple graphics processors, such as 121, 122, 123, 124 and 125, which are also in the form of graphics cards.

[0027] In different computing power clusters, any device that initiates communication can be regarded as a communication initiator. It can be that only a graphics processor that constitutes a computing power cluster independently serves as a communication initiator. When all graphics processors in the overall computing power cluster initiate communication and transmit data to the graphics processors in another computing power cluster, the computing power cluster that initiates communication can also be regarded as a communication initiator as a whole.

[0028] The graphics processors that make up the computing power cluster can communicate with other graphics processors in the same computing power cluster or with other graphics processors in other computing power clusters independently or through communication components in combination with pre-installed communication control applications. It should be noted that there can be many reasons and purposes for initiating communication, which are not specifically limited here. Only corresponding solutions are given for the communication method and the determination of transmission parameters.

[0029] A graphics processor is usually manifested as hardware. Of course, in special scenarios (such as simulation scenarios), it can also be manifested as software or a software running product, which is not specifically limited here.

[0030] The communication initiator can communicate with the communication receiver through the built-in communication control application. Figure 1 Taking the graphics card 112 as the communication initiator and the graphics card 121 as the communication receiver as an example, the graphics card 112 can achieve cross-cluster communication in the following ways when running the communication control application: first, the remote direct data access RDMA connection number is increased on the basis of the initial connection number until the detected actual bandwidth value no longer continues to increase, thereby obtaining the target connection number; then, while the RDMA connection number remains at the target connection number, the buffer size is increased on the basis of the initial size until the detected actual bandwidth value no longer continues to increase, thereby obtaining the target size; then, based on the RDMA connection number set to the target connection number and the Buffer size set to the target size, the cross-cluster transmission parameters are determined; finally, communication is performed with the graphics card 121 as the communication receiver according to the cross-cluster transmission parameters.

[0031] In addition to the cross-cluster communication presented in the above example, Figure 1 It is also shown in the figure that the graphics card 111 as the communication initiator can communicate with the graphics card 113 as the communication receiver within the cluster. This communication process can also be implemented under the control of the communication control application.

[0032] Furthermore, it is not possible to clearly determine at the communication receiving end whether the communication receiving end belongs to a different computing power cluster from itself. This can be confirmed in a variety of ways, such as by using communication delay, packet loss rate, etc. to confirm whether the transmission environment is within a cluster or across clusters.

[0033] It should be understood that Figure 1 The number of graphics cards, graphics processors, and computing power clusters in the example is only for reference. Any number of graphics cards, graphics processors, and computing power clusters may be used as required.

[0034] Please refer to Figure 2 , Figure 2A flowchart of a cross-computing power cluster communication method provided by an embodiment of the present disclosure, wherein process 200 includes the following steps:

[0035] Step 201: In response to the communication receiving end belonging to different computing power clusters, the number of RDMA connections is increased based on the initial number of connections until the detected actual bandwidth value no longer increases, thereby obtaining a target number of connections;

[0036] This step is intended to be performed by the execution subject of the cross-computing cluster communication method (e.g. Figure 1 The graphics card 112 shown as the communication initiator, which belongs to the first computing power cluster, communicates with the communication receiving ends (eg, Figure 1 When the graphics card 121 shown as the communication receiving end belongs to the second computing power cluster) communicates, the remote direct data access RDMA connection number is increased based on the initial connection number until the detected actual bandwidth value no longer increases, thereby obtaining the target connection number.

[0037] Specifically, the different computing power clusters described in the present disclosure mainly refer to computing power clusters for training generative large language models composed of graphics processors in different computer rooms located at different locations. If different computer rooms are directly divided into different computing power clusters, different computing power clusters can also be simply understood as different computer rooms, that is, cross-cluster communication can also be adaptively changed to cross-computer room communication. How to clarify whether the communication receiving end belongs to a different computing power cluster from itself at the communication receiving end can also be confirmed in a variety of ways, such as by using communication delay, packet loss rate, etc. to confirm whether the transmission environment is within the cluster or across clusters. It should be noted that compared with intra-cluster communication, cross-cluster communication usually has a significantly higher delay than intra-cluster communication and is more susceptible to network environment fluctuations when using the same RDMA technology to communicate, resulting in a larger packet loss rate. Therefore, these characteristics can be used to determine whether the communication initiator and the communication receiver belong to different computing power clusters, that is, whether the communication is intra-cluster communication or cross-cluster communication.

[0038] Among them, RDMA (Remote Direct Memory Access) technology is a data acquisition technology that allows computers to directly access each other's memory without the intervention of the operating system kernel. The present disclosure uses RDMA technology to perform low-latency and high-throughput cross-cluster communication. The number of RDMA connections refers to the number of connections that can be established between the communication initiator and the communication receiver when using RDMA for communication, that is, the number of communication channels established between the two ends of communication. Each connection usually represents a session, in which data can be directly accessed from the memory of one end to the memory of the other end. The number of RDMA connections is usually affected by hardware limitations, network topology, and specific system configuration parameters, and the number of RDMA connections also has the following relationship with performance:

[0039] 1) Concurrency: Increasing the number of RDMA connections can improve the system's concurrency capability, allowing multiple data transfer tasks to proceed simultaneously. However, too many connections may also lead to resource competition (such as bandwidth, memory, or processor), thus affecting the overall system performance;

[0040] 2) Latency: Usually each connection has a fixed connection establishment and teardown time. Too many connections may lead to higher system overhead and longer connection initialization time.

[0041] 3) Throughput: By properly managing the number of connections, the throughput of data transmission can be improved. For example, by using multiple connections, more data can be transmitted in parallel, reducing the bottleneck caused by a single connection.

[0042] Therefore, when the communication initiator uses RDMA technology to communicate with the communication receiver, the throughput of a single data transmission can be improved by increasing the number of RDMA connections. However, the specific increase method and increase range need to consider not only various influencing factors but also the overall impact on performance.

[0043] In this step, after each increase in the number of connections, the current bandwidth value is re-detected, in order to determine whether the detected bandwidth value increases with the increase in the number of connections through the bandwidth value obtained by each detection. If the detected bandwidth value increases, it means that the increase in the number of connections does achieve the purpose of improving the throughput of a single data transmission, that is, increasing the number of connections is effective and there is still room for the number of connections to continue to increase. Once the bandwidth values ​​detected twice before and after the increase in the number of connections do not increase, it can be considered that there is no room for the number of connections to continue to increase, that is, it is no longer possible to continue to improve the data transmission throughput by adjusting the number of connections, that is, the number of connections before the last increase in the number of connections can be considered as the target number of connections. For example, assuming that the number of connections is increased by 2 each time on the basis of the initial number of connections, and when the number of connections is increased to 20, a larger bandwidth value can be detected than before the increase. However, when the number of connections is further increased from 20 to 22, it is found that the bandwidth value re-detected after the increase is consistent with the bandwidth value before the increase (that is, the bandwidth value does not continue to increase). In this case, it should be considered that the increase in the number of connections is invalid, and the number of connections after the last increase - 20 should be used as the target number of connections. Of course, if each time the number of connections is increased by 2, the bandwidth value obtained by re-detection after the increase can increase by the same or greater magnitude than the bandwidth value detected before the increase, it can be seen that the number of connections can actually be increased with a greater increase. Similarly, if the number of connections continues to increase to 22, it is found that the bandwidth value re-detected after the increase is larger than the bandwidth value before the increase, but the increase is lower than the increase in the number of connections, then it can be seen that although the number of connections can continue to increase on the basis of 20, it can no longer be increased with a step size of "2". At this time, in order to determine a more accurate target number of connections, a smaller increase step size can be selected on the basis of 20 to continue to increase until the number of connections before the increase, which is completely consistent with the bandwidth values ​​detected before and after the last increase, is determined as the target number of connections.

[0044] A method of detecting the bandwidth value including but not limited to may include the following steps:

[0045] Obtain and calculate the actual data transmission size at the time corresponding to the actual delay;

[0046] The detected bandwidth value is determined based on the actual data transmission size and the actual delay.

[0047] That is, this implementation combines the actual delay of communication between the communication initiator and the communication receiver and the actual data transmission size at the corresponding moment, so as to calculate the bandwidth value representing the size of data that can be transmitted in seconds, provided that the actual data transmission size can be transmitted under the actual delay. The actual data transmission size can be determined based on the actual Buffer size at the actual moment, and can be determined based on the default size of the Buffer if the Buffer size has not been adjusted.

[0048] Step 202: When the number of RDMA connections remains at the target number of connections, the buffer size is increased based on the initial size until the detected actual bandwidth value stops increasing, thereby obtaining the target size.

[0049] On the basis of step 201, this step aims to increase the size of the buffer based on the initial size while the number of RDMA connections remains at the target number of connections, until the detected actual bandwidth value stops increasing, thereby obtaining the target size.

[0050] In the field of data transmission, Buffer is a small memory area used to store and buffer data. During the data transmission process, when data arrives from the source (such as hard disk, network, sensor, etc.) to the target (such as display, receiving end, application, etc.), there is often a difference between the transmission rate and the processing rate. At this time, Buffer plays a bridging role. It temporarily stores data in the buffer to ensure that the data flow between the source and the target can proceed smoothly. Its main role is to help improve the performance and efficiency of the system, especially when dealing with unstable data flow or communication between devices with different rates.

[0051] That is, this step aims to further try to improve the data transmission throughput by adjusting the Buffer size when the data transmission throughput cannot be adjusted by adjusting the number of RDMA connections. That is, it is expected that the bandwidth value obtained by detection can be further improved by adjusting the Buffer size.

[0052] Similar to step 201, in this step, after each increase in the buffer size, the current bandwidth value is re-detected, in order to determine whether the detected bandwidth value increases with the increase in the buffer size through the bandwidth value obtained by each detection. If the detected bandwidth value increases, it means that the increase in the buffer size does achieve the purpose of improving the throughput of a single data transmission, that is, the increase in the buffer size is effective and there is still room for the buffer size to continue to increase. Once the bandwidth values ​​detected twice before and after the increase in the buffer size do not increase, it can be considered that there is no room for the number of connections to continue to increase, that is, it is no longer possible to continue to improve the data transmission throughput by adjusting the buffer size, that is, it can be considered that the buffer size before the last increase in the buffer size is determined as the target size. For example, assuming that each time the initial size is increased by 2MB, and the buffer size is increased to 8MB, a larger bandwidth value can be detected than before the increase. However, when the number of connections is increased from 8MB to 10MB, it is found that the bandwidth value re-detected after the increase is the same as the bandwidth value before the increase (that is, the bandwidth value does not continue to increase). In this case, the increase in the buffer size should be considered invalid, and the buffer size after the last increase - 8MB should be used as the target size. Of course, if the bandwidth value detected after the increase is increased by 2MB each time the buffer size is increased by the same or greater amount than the bandwidth value detected before the increase, it can be seen that the buffer size can actually be increased with a greater increase. Similarly, if the number of connections continues to increase to 10MB, it is found that the bandwidth value detected after the increase is larger than the bandwidth value before the increase, but the increase is lower than the increase in the buffer size. In this case, it can be seen that although the buffer size can continue to increase on the basis of 8MB, it can no longer be increased with a step size of "2MB". At this time, in order to determine a more accurate target size, a smaller step size can be selected on the basis of 8MB to continue to increase until the buffer size before the increase, which is completely consistent with the bandwidth value detected before and after the last increase, is determined as the target size.

[0053] Step 203: Determine cross-cluster transmission parameters based on the number of RDMA connections set as the target number of connections and the buffer size set as the target size;

[0054] On the basis of step 202, this step aims to determine the cross-cluster transmission parameters by the above-mentioned execution subject based on the number of RDMA connections set as the target number of connections and the buffer size set as the target size. That is, the number of RDMA connections set as the target number of connections and the buffer size set as the target size exist as the main components of the cross-cluster transmission parameters, but they are not necessarily all components, and other parameters that can also be used to affect data transmission can also be included, which are not specifically limited here.

[0055] Step 204: Communicate with communication receiving terminals belonging to different computing power clusters according to the cross-cluster transmission parameters.

[0056] On the basis of step 203, this step aims at enabling the execution subject to communicate with the communication receiving end belonging to different computing power clusters according to the cross-cluster transmission parameters.

[0057] The cross-computing power cluster communication method applied to the communication initiator provided by the embodiment of the present disclosure provides a method for re-determining the cross-cluster transmission parameters that match the cross-cluster communication environment for the situation that the communication initiator and the communication receiving end belong to different computing power clusters respectively. That is, for the cross-cluster communication having a significantly increased latency compared to the intra-cluster communication, a method for successively increasing the number of remote direct data access (RDMA) connections and the size of the buffer BUFFER is provided for the cross-cluster communication to determine the target number of connections and the target size that match the actual bandwidth value, and then communicating with the communication receiving ends belonging to different computing power clusters according to the cross-cluster transmission parameters determined based on the target number of connections and the target size, so that the cross-cluster communication and the intra-cluster communication respectively use the transmission parameters that match the actual communication environment, thereby improving the overall communication efficiency and data transmission volume, thereby improving the training efficiency of the generative large language model and shortening the time required for training.

[0058] To deepen your understanding of how to determine whether the communication initiator and the communication receiver belong to different computing power clusters, please refer to Figure 3 , Figure 3 A flowchart of a method for determining whether a communication receiving end belongs to a different computing power cluster according to actual latency provided by an embodiment of the present disclosure, wherein process 300 includes the following steps:

[0059] Step 301: Determine the actual delay with the communication receiving end;

[0060] This step is intended to determine the actual delay between the execution subject and the communication receiving end. An implementation method including but not limited to may be:

[0061] First, the time when data is sent to the communication receiving end is determined as the starting time; then, the time when the confirmation notification returned by the communication receiving end after receiving the data is received is determined as the ending time; finally, based on the starting time and the ending time, the actual delay is calculated.

[0062] Furthermore, in order to ensure the accuracy of the actual delay calculated, a preset logarithm of start and end time pairs consisting of a start time and a matching end time can be selected, that is, the delay values ​​corresponding to each start and end time pair are calculated respectively to obtain a delay set, and then the actual delay represented by the average delay is obtained by taking the average of each delay value in the delay set, so as to eliminate accidental or fluctuating errors by taking the average value.

[0063] Step 302: Determine whether the actual delay exceeds the preset delay, if so, execute step 303, otherwise execute step 304;

[0064] This step is intended for the execution subject to compare the actual delay with the preset delay, and then execute different subsequent processing branches according to the two comparison results. In this embodiment, the preset delay exists as a delay boundary value between a higher delay and a lower delay, and the higher delay corresponds to the delay during cross-cluster communication, and the lower delay corresponds to the delay during intra-cluster communication.

[0065] Step 303: Determine that the communication receiving end belongs to a different computing power cluster;

[0066] This step is based on the judgment result of step 302 that the actual delay exceeds the preset delay, and is intended for the above-mentioned execution subject to determine that the communication receiving end belongs to a different computing power cluster.

[0067] Since this situation corresponds to the execution triggering condition of step 201 , the subsequent execution can be continued according to the solution provided by process 200 .

[0068] Step 304: Determine that the communication receiving end and the communication receiving end belong to the same computing power cluster;

[0069] This step is based on the judgment result of step 302 that the actual delay does not exceed the preset delay, and is intended for the above-mentioned execution subject to determine that the communication receiving end belongs to the same computing power cluster.

[0070] In contrast to different computing power clusters, which mainly refer to computing power clusters for training generative large language models that are composed of graphics processors in different computer rooms at different locations, the same computing power cluster can refer to computing power clusters for training generative large language models that are composed of graphics processors in the same computer room at the same location. Similarly, if different computing power clusters can be simplified to different computer rooms, the same computing power cluster can also be simplified to the same computer room.

[0071] Step 305: Communicate with the communication receiving end belonging to the same computing power cluster according to the intra-cluster transmission parameters determined by the collective communication library.

[0072] On the basis of step 304, this step aims at enabling the execution subject to communicate with the communication receiving end belonging to the same computing power cluster according to the intra-cluster transmission parameters determined by the collective communication library.

[0073] NCCL (NVIDIA Collective Communications Library) is an efficient communication library developed by NVIDIA, mainly used to accelerate collective communication operations in deep learning frameworks. It is widely used in distributed training, especially in multi-GPU or multi-node environments. NCCL focuses on providing high-performance communication protocols and algorithms, aiming to significantly improve data transmission speed and parallel computing efficiency by leveraging GPU hardware acceleration.

[0074] NCCL provides a variety of common collective communication operations (CollectiveCommunications) for distributed training. These operations are usually used to synchronize and share data between multiple computing nodes. In deep learning training, common collective communication operations include:

[0075] 1) AllReduce: multiple nodes exchange data and perform some kind of aggregation on the data (such as summation, averaging, etc.). In deep learning, it is often used to synchronize gradients in distributed training;

[0076] 2) Broadcast: Broadcast data from one node to all other nodes. Often used to synchronize model parameters or other shared data to various nodes;

[0077] 3) AllGather: Collects and aggregates data from multiple nodes to all nodes. Used for input aggregation during data distributed training;

[0078] 4) Reduce: Exchange data between multiple nodes and aggregate the data, usually aggregating data from multiple nodes to one node;

[0079] 5) ReduceScatter: Aggregates data from multiple nodes and distributes it to each node. Contrast with AllGather.

[0080] The following advantages can be achieved by using NCCL:

[0081] 1) Hardware acceleration: NCCL takes full advantage of NVIDIA’s hardware features, such as NVLink (for high-bandwidth communication between GPUs) and NVIDIA RDMA, providing extremely high-bandwidth and low-latency communication;

[0082] 2) Multi-GPU support: NCCL is designed specifically for GPUs and supports single-node and multi-GPU configurations. It also supports distributed training across nodes, which can effectively improve training efficiency.

[0083] 3) Efficient algorithms: NCCL implements a variety of optimized communication algorithms that are optimized for different network topologies and hardware characteristics to minimize communication bottlenecks;

[0084] 4) Ease of use and compatibility: NCCL provides an easy-to-integrate API interface that can be seamlessly integrated with popular deep learning frameworks. Its API design is concise and clear, allowing users to focus on application layer logic without having to worry about the underlying communication implementation;

[0085] 5) Cross-platform support: NCCL supports a variety of Linux systems and GPU hardware in the CUDA environment, and is suitable for various multi-GPU configurations.

[0086] In addition, NCCL is used as the default communication library by multiple deep learning frameworks (such as TensorFlow, PyTorch, MXNet, etc.) to accelerate communication during multi-GPU or distributed training. Specifically, these frameworks call the NCCL library to handle tasks such as gradient synchronization and model parameter broadcasting. In this way, the framework can achieve efficient distributed training, especially in scenarios where multiple GPUs are used. For example, the torch.distributed module in PyTorch provides integration with the NCCL library, and the communication backend can be specified as NCCL through configuration to perform efficient communication operations in a multi-GPU environment.

[0087] That is, this step makes full use of the existing NCCL technology to provide appropriate intra-cluster transmission parameters for communication within the computing power cluster. However, the existing NCCL technology cannot identify whether the communication is intra-cluster communication or cross-cluster communication, and only provides a set of transmission parameter determination solutions that are more suitable for intra-cluster communication. Therefore, this step directly uses NCCL technology to provide intra-cluster transmission parameters to achieve intra-cluster communication when it is clear that the current communication belongs to intra-cluster communication.

[0088] This embodiment provides a method for determining whether it is intra-cluster communication or cross-cluster communication based on the actual delay between the two ends of communication through steps 301-305, and provides corresponding methods for determining matching transmission parameters for two different types of communication, thereby performing different types of communication through different transmission parameters, thereby ultimately improving the overall communication efficiency and data throughput.

[0089] Of course, other means of determining whether it is intra-cluster communication or inter-cluster communication may be combined with the above embodiments to form new embodiments, which are not listed here one by one.

[0090] To deepen your understanding of how to increase the number of RDMA connections to ultimately determine the target number of connections, see Figure 4 , Figure 4 A flowchart of a method for increasing the number of RDMA connections and determining a target number of connections provided by an embodiment of the present disclosure, wherein the process 400 includes the following steps:

[0091] Step 401: increasing the number of RDMA connections based on the initial number of connections, and detecting and obtaining a new actual bandwidth value after the connection number increase process is completed;

[0092] This step is intended to increase the number of RDMA connections based on the initial number of connections by the above-mentioned execution subject, and detect the new actual bandwidth value after the connection number increase process is completed. Since the initial number of connections is the initial value of the number of connections, the increase process described in this step can be understood as the first increase process of the number of connections. Assuming that the initial number of connections is 2, and assuming that the increase process is performed according to a fixed increase step size, and the increase step size is specifically 3, then the new number of connections obtained after the first increase process is: 5.

[0093] After the first increase in the number of connections is completed, it is necessary to re-detect the actual bandwidth value of the current communication link (hereinafter referred to as the new actual bandwidth value, which is used to distinguish the original actual bandwidth value detected before the current round of connection increase processing).

[0094] Step 402: determine whether the new actual bandwidth value is greater than the original actual bandwidth value detected before the connection number increase process is performed. If so, execute step 403; otherwise, execute step 404;

[0095] Based on step 401, this step aims to determine by the execution subject whether the new actual bandwidth value is greater than the original actual bandwidth value detected before the connection number increase process is performed, and execute different processing branches according to the two determination results.

[0096] Step 403: Continue to increase the current number of RDMA connections.

[0097] This step is based on the judgment result in step 402 that the new actual bandwidth value is greater than the original actual bandwidth value, indicating that the processing method of increasing the number of connections is effective and there is still room for the number of connections to continue to increase.

[0098] In order to confirm whether the new round of increase in the number of connections is effective, the process will continue to jump back to step 402 to check whether the bandwidth values ​​detected before and after the new round of increase are increased in the same way. If the same increase continues, step 403 will be executed until there is no more increase, and it can be considered that there is no room for the number of connections to continue to increase.

[0099] Step 404: Determine the current number of RDMA connections as the target number of connections.

[0100] This step is based on the judgment result of step 402 that the new actual bandwidth value is not greater than the original actual bandwidth value, that is, the new actual bandwidth value should be equal to the original actual bandwidth value, and it can be considered that the last increase in the number of connections is invalid. Therefore, the number of connections after the last valid increase in the number of connections is determined as the target number of connections.

[0101] The connection number increase process includes:

[0102] A uniform increase strategy with a fixed step size (such as the increase method with an increase step size of 3 mentioned in the above example) or a non-uniform increase strategy with a non-fixed step size. Compared with the uniform increase strategy, the non-uniform increase strategy aims to reduce the number of increases by a larger step size when the number of connections should be increased quickly, and to accurately approach the target number of connections by a smaller step size when the number of connections should be increased slowly.

[0103] An implementation method including but not limited to may be:

[0104] When the connection number increase processing is a non-uniform increase strategy with an unfixed step size, the RDMA connection number is processed by first exponential growth and then linear growth on the basis of the initial connection number until the detected actual bandwidth value no longer continues to increase, thereby obtaining the target connection number; wherein, the switching timing from exponential growth to linear growth is: the growth rate of the new actual bandwidth value detected after the exponential growth processing is lower than the growth rate of the connection number compared with the original actual bandwidth value detected before the corresponding exponential growth processing.

[0105] The reason why exponential growth is used first is that the initial number of connections is usually small and "far" from the upper limit of the number of connections. Therefore, the upper limit of the number of connections is quickly approached by using exponential growth. It should be understood that if the increase in the bandwidth value detected before and after each exponential increase in the number of connections is not less than the increase in the number of connections, it means that the increase in the number of connections this time has not actually reached the upper limit of the bandwidth value. Therefore, the exponential growth method can be continued, and the growth rate of exponential growth can be further increased. If it is found that the increase in the bandwidth value detected before and after a certain exponential increase in the number of connections is less than the increase in the number of connections, it means that the upper limit of the bandwidth value under the connection adjustment method has been initially found, that is, the increase in the number of connections this time has exceeded the increase in the bandwidth value. Therefore, the number of connections after this increase should be inaccurate and does not match the actual bandwidth value currently detected. Therefore, it is necessary to switch to the linear growth method based on the number of connections after the last valid exponential growth to more accurately explore the most accurate target number of connections. That is, the target number of connections should be the number of connections after the last valid linear growth.

[0106] Furthermore, the growth step of the linear growth may not be fixed, and may be large at first and small later. That is, the growth range of the first linear growth may be determined based on the actual bandwidth growth range before and after the last exponential growth process, and the growth range of the non-first linear growth may be determined based on the actual bandwidth growth range before and after the last linear growth process.

[0107] Furthermore, the upper limit of the number of RDMA connections can be determined based on the bandwidth upper limit of the network card hardware, and after the upper limit of the number of connections is determined, the above-mentioned execution entity can also control the number of connections after the last exponential growth to be less than the upper limit of the number of connections, so as to avoid the last exponential growth increasing the number of connections to exceed the upper limit of the number of connections.

[0108] Similar to Figure 4 For solutions to increase the number of connections, see Figure 5 , Figure 5 A flowchart of a method for increasing the buffer size and determining the target size provided by an embodiment of the present disclosure, wherein the process 500 includes the following steps:

[0109] Step 501: the buffer size is increased based on the initial size, and after the size increase process is completed, a new actual bandwidth value is detected;

[0110] This step is intended to increase the size of the buffer based on the initial size by the execution subject, and detect the new actual bandwidth value after the size increase. Since the initial size is the initial value of the buffer size, the increase process described in this step can be understood as the first increase process of the buffer size. Assuming the initial size is 2MB, assuming that the increase process is performed according to a fixed increase step size, and the increase step size is specifically 1MB, then the new buffer size obtained after the first increase process is: 3MB.

[0111] After the first increase in the buffer size is completed, it is necessary to re-detect the actual bandwidth value of the current communication link (hereinafter referred to as the new actual bandwidth value, which is used to distinguish the original actual bandwidth value detected before the current round of connection number increase).

[0112] Step 502: determine whether the new actual bandwidth value is greater than the original actual bandwidth value detected before the size increase process is performed, if so, execute step 503, otherwise execute step 504;

[0113] Based on step 501, this step aims to allow the execution subject to determine whether the new actual bandwidth value is greater than the original actual bandwidth value detected before the size increase process, and to execute different processing branches according to the two determination results.

[0114] Step 503: Continue to increase the current size of the Buffer;

[0115] This step is based on the fact that the result of the determination in step 502 is that the new actual bandwidth value is greater than the original actual bandwidth value, indicating that the current processing method of increasing the buffer size is effective and there is still room for the buffer size to continue to increase.

[0116] In order to confirm whether the new round of buffer size increase processing is effective, the process will continue to jump back to step 502 to check whether the bandwidth value detected before and after the new round of increase processing has increased in the same way. If the same increase continues, step 53 will be executed until there is no more increase, and it can be considered that there is no room for the buffer size to continue to increase.

[0117] Step 504: Determine the current size of the Buffer as the target size.

[0118] This step is based on the judgment result of step 502 that the new actual bandwidth value is not greater than the original actual bandwidth value, that is, the new actual bandwidth value should be equal to the original actual bandwidth value, and it can be considered that the last increase in the buffer size is invalid. Therefore, the buffer size after the last valid size increase is determined as the target size.

[0119] The size increase process includes:

[0120] A uniform increase strategy with a fixed step size (such as the increase method with an increase step size of 1MB mentioned in the above example) or a non-uniform increase strategy with a non-fixed step size. Compared with the uniform increase strategy, the non-uniform increase strategy aims to reduce the number of increases by a larger step size when the buffer size should be increased quickly, and to accurately approach the target size by a smaller step size when the buffer size should be increased slowly.

[0121] An implementation method including but not limited to may be:

[0122] When the size increase processing is a non-uniform increase strategy with an unfixed step size, the buffer size is processed by first exponential growth and then linear growth on the basis of the initial size until the detected actual bandwidth value no longer continues to increase, thereby obtaining the target size; wherein, the switching timing from exponential growth to linear growth is: the growth rate of the new actual bandwidth value detected after the exponential growth processing is lower than the growth rate of the buffer size compared with the original actual bandwidth value detected before the corresponding exponential growth processing.

[0123] The reason why exponential growth is used first is that the initial size is usually small and "far" from the upper limit of the buffer size. Therefore, the exponential growth method is used to quickly approach the upper limit of the size. It should be understood that if the increase in the bandwidth value detected before and after each exponential growth of the buffer size is not less than the increase in the number of connections, it means that the increase in the buffer size this time has not actually reached the upper limit of the bandwidth value. Therefore, the exponential growth method can be continued, and the growth rate of the exponential growth can be further increased. If it is found that the increase in the bandwidth value detected before and after a certain exponential growth of the buffer size is less than the increase in the buffer size, it means that the upper limit of the bandwidth value under the buffer size adjustment method has been initially found, that is, the increase in the buffer size this time has exceeded the increase in the bandwidth value. Therefore, the buffer size after this increase should be inaccurate and does not match the actual bandwidth value currently detected. Therefore, it is necessary to switch to the linear growth method based on the last valid exponential growth of the buffer size to more accurately explore the target size. That is, the target size should be the buffer size after the last valid linear growth.

[0124] Furthermore, the growth step of the linear growth may not be fixed, and may be large at first and small later. That is, the growth range of the first linear growth may be determined based on the actual bandwidth growth range before and after the last exponential growth process, and the growth range of the non-first linear growth may be determined based on the actual bandwidth growth range before and after the last linear growth process.

[0125] Furthermore, the upper limit of the buffer size can be based on the bandwidth limit of the network card hardware and the upper limit of the number of RDMA connections (the upper limit of the number of connections is determined based on the bandwidth limit, see Figure 4 The corresponding description part recorded in the embodiment shown in the figure) is determined, and after the size upper limit is determined, the above-mentioned execution subject can also control the size after the last exponential growth to be smaller than the size upper limit, so as to avoid the last exponential growth increasing the Buffer size to exceed the size upper limit.

[0126] On the basis of any of the above embodiments, considering that cross-cluster communication is more susceptible to uncontrollable network environment or network quality fluctuations, and once the network quality is affected, the actual available bandwidth will be reduced, it is necessary to timely adjust the cross-cluster transmission parameters according to the actual available bandwidth so that the cross-cluster communication effect is optimized when the cross-cluster transmission parameters adapt to the actual network bandwidth.

[0127] For details, see Figure 6 , Figure 6A flowchart of a method for adjusting cross-cluster transmission parameters according to a bandwidth reduction degree provided by an embodiment of the present disclosure, wherein process 600 includes the following steps:

[0128] Step 601: In response to the current available bandwidth detected after the preset communication time according to the cross-cluster transmission parameter is less than the original stable available bandwidth, determine the degree of reduction of the bandwidth value of the current available bandwidth compared with the original stable available bandwidth;

[0129] This step is intended to determine the degree of reduction of the bandwidth value of the current available bandwidth compared to the original stable available bandwidth when the current available bandwidth detected by the above-mentioned execution subject after communicating for a preset time according to the cross-cluster transmission parameters is less than the original stable available bandwidth.

[0130] Step 602: lowering the target number of connections and the target size based on the degree of bandwidth reduction, and obtaining the lowered number of connections and the lowered size;

[0131] Based on step 601, this step aims to have the above-mentioned execution subject adjust the target number of connections and the target size downward based on the degree of bandwidth reduction, and obtain the number of connections and the size after the adjustment. When adjusting the target number of connections and the target size downward based on the degree of bandwidth reduction, the downward adjustment range of the target number of connections and the target size can be slightly higher than the degree of bandwidth reduction, and then continue to probe upward based on the initial number of connections and the initial size after the adjustment, until the number of connections and the size after the adjustment corresponding to the current available bandwidth at the actual moment are obtained in the upward exploration process.

[0132] Step 603: determining adjusted cross-cluster transmission parameters based on the RDMA connection number reset to the reduced connection number and the Buffer size reset to the reduced size;

[0133] Step 604: Communicate with communication receiving terminals in different computing power clusters according to the adjusted cross-cluster transmission parameters.

[0134] This embodiment provides a solution for readjusting the cross-cluster transmission parameters to obtain new cross-cluster transmission parameters that match the actual situation when network fluctuations occur during cross-cluster communication through steps 601 to 604, so as to communicate with communication receiving ends in different computing power clusters according to the adjusted cross-cluster transmission parameters, so as to optimize the cross-cluster communication effect when the cross-cluster transmission parameters are adapted to the actual network bandwidth.

[0135] To deepen understanding, the present disclosure also combines specific application scenarios, and gives detailed explanations of the problems existing in the application scenarios, the ideas for solving the problems, and the specific implementation solutions:

[0136] With the gradual expansion of training computing volume in the era of large models, more and more chips and larger computing power clusters are needed. Although building a single computing power cluster of 100,000 cards can achieve extreme computing efficiency, it has extremely high requirements for infrastructure and may not be able to be placed in one computer room. At the same time, for scenarios where 10,000-card training is required, if multiple 1,000-card computing power clusters can be utilized, it will not only reduce the construction cost of the 10,000-card computing power cluster, but also efficiently utilize the fragmented 1,000-card computing power cluster and activate the computing power supply. Therefore, multi-computer room (multi-computer room here refers to a simplified expression in which each computing power cluster is equivalent to each computer room) hybrid training in different locations is a possible option for large-scale 10,000-card training in the future.

[0137] Resources across data centers need to communicate through the DCI (Data Center Interconnect Network), such as Figure 7-1 The core challenge of remote hybrid training lies in the high latency and low bandwidth of the DCI network. The latency of the DCI network is usually between 1-2ms, and the bandwidth is usually only about 1T due to cost constraints. Therefore, it is necessary to carefully optimize the interconnection and communication technology and adjust the appropriate parallel strategy, otherwise the communication time will become the biggest bottleneck. Figure 7-1 The DCN (Data Center Network) shown in the figure is a common component interconnected by the DCI network.

[0138] It is a common strategy to split the model into different computer rooms according to the pipeline, because pipeline segmentation, i.e., pp (point point) transmission, usually only involves point-to-point data transmission, and does not involve complex AllreduceRing (an implementation of Allreduce operation in distributed computing) or Tree algorithm, so pp segmentation into different computer rooms is preferred. For cross-computer room pp transmission, it is necessary to adjust the sending parameters of the communication library, such as the transmission buffer and the number of transmission connections, to adapt to the high latency of the DCI network. For different latency and bandwidth scenarios, the most suitable transmission parameters are usually different and need to be carefully designed and adjusted. pp transmission occurs in the same computer room and across computer rooms. Since the latency and bandwidth capacity of the same computer room and across computer rooms are different, different transmission parameters need to be configured for these two scenarios. However, the collective communication library NCCL used for GPU training, or other collective communication libraries such as gloo, do not support the function of automatically adjusting transmission parameters according to link bandwidth and latency. They can only be statically configured when starting the training task, which makes the communication library unable to adapt to the scenario of cross-data center training.

[0139] Therefore, this embodiment needs to implement the following key functions to overcome the defects of the prior art:

[0140] 1) Ability to automatically detect the communication delay across computer rooms and the available bandwidth of the link;

[0141] 2) It can automatically configure the appropriate transmission buffer size and number of connections based on communication delay and bandwidth, and make dynamic adjustments in real time based on the link status.

[0142] That is, this embodiment intends to provide a strategy for adaptively adjusting pp transmission parameters by link bandwidth delay. The collective communication library can automatically configure the number of transmission connections and transmission parameters such as buffer according to the bandwidth and delay of the link. It is especially suitable for scenarios such as remote mixed training where there are pp transmissions in the same machine room and across machine rooms, and the delay and bandwidth are inconsistent, so as to overcome the technical defects that the current related solutions only support static configuration, can only adapt to a single scenario, and are not flexible enough. The specific implementation means will be described in detail below:

[0143] 1. Detection of inter-computer room communication delay and bandwidth

[0144] The collective communication library NCCL uses RDMA in the process of cross-machine transmission. The start time is recorded when RDMA sends data, and the end time is recorded after receiving the returned confirmation. In this way, the round-trip delay of data at the communication library transmission level can be counted. At the same time, the communication bandwidth can be calculated based on the transmitted msgsize (actual data size). In this way, a relatively accurate transmission delay and transmission bandwidth can be obtained, which is a reserve for the subsequent automatic adjustment of transmission parameters. At the same time, if this delay is greater than a certain threshold (for example, 100us), it is considered to be a cross-computer room transmission scenario. The following strategy will only take effect in the cross-computer room transmission scenario.

[0145] 2. Automatically configure appropriate transmission parameters based on communication delay and bandwidth

[0146] Assume the default value of the maximum send buffer is known and delay , then the maximum bandwidth that can be utilized The calculation formula is:

[0147] (Formula 1);

[0148] The default buffer size of the RDNA transmission of the collective communication library NCCL is 4MB. When the RTT delay is 2ms, according to Formula 1, only about 16Gbps of bandwidth can be used at most. If you want to use a full 200G network, at least 13 connections are required; for a communication library such as gloo that is mainly used for CPU communication, it does not even have the ability to have multiple connections and can only use a single connection. Therefore, if the delay and bandwidth across computer rooms are known, setting appropriate transmission parameters is the focus of this embodiment. This embodiment will use the following strategy to set appropriate transmission parameters:

[0149] According to the measured transmission delay, it is determined whether it is a cross-computer room scenario. In this embodiment, the delay threshold is set to 100us based on the empirical value, because the delay of transmission in the same computer room is usually tens of us. If the delay exceeds 100us, it is considered to be cross-computer room transmission. When it is determined to be a non-cross-computer room scenario (that is, communication within the computer room or communication within the cluster), the original strategy of the communication library is maintained without any change; if it is determined to be a cross-computer room scenario, the subsequent strategy takes effect:

[0150] The overall strategy uses a heuristic algorithm to gradually increase the buffer size of the number of connections based on the default parameters. The bandwidth value is calculated each time when the new parameters are used. If the bandwidth value increases, it means that new available bandwidth has been detected. It continues to increase until the bandwidth value stops increasing, indicating that a suitable bandwidth value has been found. This set of connections and buffer sizes that can achieve the appropriate bandwidth value will be used as subsequent transmission parameters. Specifically:

[0151] 1) Adjust the number of transmission connections and buffer by exponential growth first and then linear growth, and adjust the number of connections first to find the best parameters

[0152] The reason why we choose to increase the number of connections first and then the Buffer is that the increase of the Buffer usually results in more video memory usage, and increasing the number of connections itself implies the increase of the Buffer, because there is one Buffer for each connection; therefore, if the bandwidth utilization can be increased by increasing the number of connections, try to delay the adjustment of the Buffer to ensure that the use of video memory is saved as much as possible.

[0153] In other words, if you do not consider the above problems and do not need to worry about the remaining video memory, you can first increase the buffer size and then increase the number of RDMA connections.

[0154] The increase in the number of connections and buffer in the detection phase is described as follows: the first transmission uses the default 4MB transmission buffer and 1 connection for transmission. After the bandwidth calculation is completed, the number of transmission connections is adjusted to 2 and the transmission continues. The number of connections increases exponentially, and the new bandwidth value The calculation method is:

[0155] (Formula 2);

[0156] If the number of connections In the case of doubling, the new bandwidth value If there is no doubling growth, the number of connections will be increased by one connection each time until the bandwidth value stops growing. The subsequent adjustment will increase the transmission buffer, and the method of exponential growth followed by linear growth will be adopted. Here, the number of connections will be increased first, and when it cannot continue to grow, the buffer will be increased to avoid consuming too much buffer and wasting video memory. Usually, this method can be used to find a set of parameters. If the bandwidth stops growing after setting, it is considered that a set of suitable parameters has been found.

[0157] 2) Dynamic calculation of the upper limit of the transmission buffer and the number of connections

[0158] The transmission buffer and the number of connections cannot grow indefinitely. The upper limit can be determined dynamically using the following strategies: Usually, the network card bandwidth of the GPU server is fixed. You can choose to set it in advance or use the TOPO detection results of the collective communication library NCCL as input. Then the upper limit of the number of connections The calculation method can be obtained by the following formula, where the upper limit bandwidth of the network card hardware is :

[0159] (Formula 3);

[0160] Assume that the network card bandwidth available to each GPU is 400Gbps, the detected RTT delay is 2ms, and the send buffer is set to the default 4MB. According to Formula 3, approximately 25 connections are required, so the maximum number of connections is set to 25. The entire process can be dynamically detected. For different computer room delays, the RTT may be different, so the calculated maximum number of connections may also be different. The upper limit of the send buffer is calculated after the optimal number of connections has been detected. The maximum adjustable value of the send buffer can be calculated as follows:

[0161] (Formula 4);

[0162] Assuming that the bandwidth does not continue to increase after the number of connections increases to 16, then the formula is equal to 16, so we can calculate it based on the bandwidth limit of 400Gbps. Equal to 400Gbps, the highest sending buffer, is equal to 2ms, then according to formula 4, It can only reach 6.25MB, so the upper limit is rounded down to 6MB. The maximum increase is 2M.

[0163] There are two options for pp transmission interconnection in different computer rooms: When the backend RDMA network card used by the GPU server has a DCI network that can be directly interconnected, the transmission bandwidth of the network card that each GPU can use is usually relatively large, so the theoretical upper limit of the parameter is calculated as described above; when the backend RDMA network cards of different computer rooms cannot be directly interconnected, pp transmission across computer rooms needs to use the main network card of the GPU server for transmission. At this time, all GPUs of the entire machine can only share the bandwidth of the main network card for transmission. Therefore, the theoretical upper limit of the calculated parameter will not be very high, and the detection process will not last for a long time.

[0164] 3) Parameter adjustment after bandwidth stabilization

[0165] When the appropriate bandwidth value has been detected and the appropriate parameters are used for stable transmission, the link itself may become less available due to congestion. This can be detected from latency and bandwidth detection. When this happens, in order to avoid continuous performance degradation due to the use of expired parameters, the Buffer and the number of connections need to be adjusted accordingly. For the Buffer, one strategy that can be adopted is to halve the processing, but the minimum is not less than the default value of the Buffer. The calculation method is as follows: Formula 5:

[0166] (Formula 5);

[0167] For the adjustment of the number of connections, the new connection value is dynamically calculated based on the adjusted Buffer value and the latest detected reduced bandwidth value. Assume that the new bandwidth value is , the current number of connections is , then the adjusted number of connections The calculation method is shown in the following formula:

[0168] (Formula 6);

[0169] Assuming the number of connections is 16, the bandwidth is reduced from 160Gbps to 120Gbps, and assuming the buffer remains at 4MB, according to the above formula, the adjusted number of connections should be reduced to 7.

[0170] After adjustment, you need to find a suitable parameter again. At this time, the bandwidth will be re-detected. Since it is in the bandwidth recovery stage, both the connection and the buffer will directly enter the linear increase stage instead of the exponential increase stage.

[0171] To facilitate understanding, each step will be described in detail below using a specific example. Figure 7-2 View and understand:

[0172] ①: Assume that the NCCL communication library is initialized, each connection buffer is 4MB, the number of RDMA connections used is 1, and the Rtt is 2ms. Since the latency is greater than 100us, the bandwidth and latency detection phase is entered;

[0173] ②: Calculation of the upper limit of the number of connections: According to formula 3, the upper limit of the number of connections is 25;

[0174] ③: Connection number index detection: Increase the number of connections to 2. According to formula 2, the new bandwidth value is about 32 Gbps. Assuming that the detected bandwidth reaches the expected value, the number of connections is increased to 4. According to formula 2, the new bandwidth value is about 64 Gbps. Assuming that the detected bandwidth value reaches the expected value, the number of connections is increased to 8. According to formula 2, the new bandwidth value is about 128 Gbps. Assuming that the detected bandwidth value does not reach the expected value, but it has also increased to 96 Gbps, it enters the linear increase stage;

[0175] ④: Linear detection of the number of connections: At this time, the number of connections increases from 8 to 9. Assuming that the bandwidth value has not increased, the number of connections is 8 and directly enters the buffer growth phase;

[0176] ⑤: Calculation of buffer upper limit: According to formula 4, since the number of connections is 8, the calculated buffer upper limit is 12.8MB, which is rounded down to 12MB;

[0177] ⑥: Exponential increase in the number of buffers: Increase the buffer from 4MB to 8MB. Assume that the detected bandwidth has increased to 100Gbps, which is not the expected doubling. Therefore, the number of buffers is increased linearly.

[0178] ⑦: Buffer quantity increases linearly: Buffer increases from 8MB to 9MB by 1MB. Assuming that the detected bandwidth remains at 100Bps without significant increase, the buffer is kept at 8MB and no further detection is performed. At this point, a set of optimal parameters has been obtained, with 8 connections and 8MB of send buffer.

[0179] ⑧: Bandwidth drops, and the re-detection phase begins: The detected bandwidth drops from 100 Gbps to 70 Gbps, a 30% drop in bandwidth. Therefore, according to Formula 5, the buffer is halved to 4 MB, and the number of connections drops from 8 to 4 (rounded down) according to Formula 6, and then the bandwidth re-detection phase begins.

[0180] ⑨: At this time, the number of connections is 4 and the buffer is 4MB. Using this set of values ​​to restart, the number of connections increases linearly from 4 to 5. The subsequent process is the same and will not be repeated here.

[0181] Further references Figure 8 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a cross-computing power cluster communication device. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0182] like Figure 8 As shown, the cross-computing power cluster communication device 800 of this embodiment may include: a connection number increase processing unit 801, a buffer increase processing unit 802, a cross-cluster transmission parameter determination unit 803, and a first cross-cluster communication unit 804. Among them, the connection number increase processing unit 801 is configured to increase the number of remote direct data access RDMA connections on the basis of the initial number of connections in response to the communication receiving end belonging to different computing power clusters respectively until the detected actual bandwidth value no longer continues to increase, thereby obtaining the target number of connections; the buffer increase processing unit 802 is configured to increase the size of the buffer buffer on the basis of the initial size when the RDMA connection number remains at the target connection number until the detected actual bandwidth value no longer continues to increase, thereby obtaining the target size; the cross-cluster transmission parameter determination unit 803 is configured to determine the cross-cluster transmission parameters based on the RDMA connection number set as the target number of connections and the buffer size set as the target size; the first cross-cluster communication unit 804 is configured to communicate with the communication receiving end belonging to different computing power clusters according to the cross-cluster transmission parameters.

[0183] In this embodiment, in the cross-computing power cluster communication device 800, the specific processing of the connection number increase processing unit 801, the buffer increase processing unit 802, the cross-cluster transmission parameter determination unit 803, and the first cross-cluster communication unit 804 and the technical effects thereof can be referred to respectively. Figure 2 The relevant descriptions of steps 201 - 204 in the corresponding embodiment are not repeated here.

[0184] In some optional implementations of this embodiment, the cross-computing power cluster communication device 800 may further include:

[0185] The different computing power clusters belonging determination unit is configured to determine that the communication receiving end belongs to different computing power clusters respectively in response to the actual delay between the communication receiving end and the communication receiving end exceeding the preset delay.

[0186] In some optional implementations of this embodiment, the cross-computing power cluster communication device 800 may further include: an actual delay calculation unit, which may include:

[0187] A starting time determination subunit, configured to determine the time of sending data to the communication receiving end as the starting time;

[0188] The end time determination subunit is configured to determine the time of receiving the confirmation notification returned by the communication receiving end after receiving the data as the end time;

[0189] The actual delay calculation subunit is configured to calculate the actual delay based on the start time and the end time.

[0190] In some optional implementations of this embodiment, the actual delay calculation subunit may be further configured as follows:

[0191] Calculate the delay values ​​corresponding to the start and end time pairs consisting of the start time and the matching end time of the preset number of logarithms, and obtain the delay set;

[0192] The average of the delay values ​​in the delay set is calculated to obtain the actual delay.

[0193] In some optional implementations of this embodiment, the cross-computing power cluster communication device 800 may further include: a bandwidth value detection unit, and the bandwidth value detection unit may be further configured to:

[0194] Obtain and calculate the actual data transmission size at the time corresponding to the actual delay;

[0195] The detected bandwidth value is determined based on the actual data transmission size and the actual delay.

[0196] In some optional implementations of this embodiment, the connection number increase processing unit 801 may be further configured to:

[0197] The number of RDMA connections is increased based on the initial number of connections, and a new actual bandwidth value is detected after the number of connections is increased.

[0198] In response to the new actual bandwidth value being greater than the original actual bandwidth value detected before the connection number increase process is performed, the connection number increase process is continued for the current connection number of the RDMA connection number until the subsequently obtained new actual bandwidth value is equal to the original actual bandwidth value detected before the corresponding connection number increase process is performed;

[0199] In response to the new actual bandwidth value being equal to the original actual bandwidth value detected before the corresponding connection number increase process is performed, the current connection number of the RDMA connection number is determined as the target connection number.

[0200] In some optional implementations of this embodiment, the connection number increase process may include:

[0201] A uniform increase strategy with a fixed step size or a non-uniform increase strategy with a non-fixed step size.

[0202] In some optional implementations of this embodiment, the connection number increase process is a non-uniform increase strategy with an unfixed step size, and the connection number increase processing unit 801 can be further configured to:

[0203] The number of RDMA connections is processed by exponential growth first and then linear growth on the basis of the initial number of connections until the detected actual bandwidth value no longer increases, thereby obtaining the target number of connections. The timing of switching from exponential growth to linear growth is as follows: the growth rate of the new actual bandwidth value detected after the exponential growth processing is lower than the growth rate of the number of connections compared with the original actual bandwidth value detected before the corresponding exponential growth processing.

[0204] In some optional implementations of this embodiment, the cross-computing power cluster communication device 800 may further include:

[0205] A connection number upper limit determination unit is configured to determine a connection number upper limit of the number of RDMA connections based on a bandwidth upper limit of the network card hardware;

[0206] The first value control unit is configured to control the number of connections after the last exponential growth to be less than the upper limit of the number of connections.

[0207] In some optional implementations of this embodiment, the growth amplitude of the first linear growth is determined based on the actual bandwidth growth amplitude before and after the last exponential growth processing; the growth amplitude of the non-first linear growth is determined based on the actual bandwidth growth amplitude before and after the last linear growth processing.

[0208] In some optional implementations of this embodiment, the Buffer increase processing unit 802 may be further configured to:

[0209] The buffer size is increased based on the initial size, and a new actual bandwidth value is detected after the size increase.

[0210] In response to the new actual bandwidth value being greater than the original actual bandwidth value detected before the size increase process is performed, the size increase process is continued for the current size of the buffer until the subsequently obtained new actual bandwidth value is equal to the original actual bandwidth value detected before the corresponding size increase process is performed;

[0211] In response to the new actual bandwidth value being equal to the original actual bandwidth value detected before the corresponding size increase process is performed, the current size of the buffer size is determined as the target size.

[0212] In some optional implementations of this embodiment, the size increase process may include:

[0213] A uniform increase strategy with a fixed step size or a non-uniform increase strategy with a non-fixed step size.

[0214] In some optional implementations of this embodiment, the size increase process is a non-uniform increase strategy with an unfixed step size, and the Buffer increase processing unit 802 may be further configured to:

[0215] The buffer size is processed by first exponential growth and then linear growth on the basis of the initial size until the detected actual bandwidth value stops increasing and the target size is obtained. The switching timing from exponential growth to linear growth is: the growth rate of the new actual bandwidth value detected after the exponential growth processing is lower than the growth rate of the buffer size compared with the original actual bandwidth value detected before the corresponding exponential growth processing.

[0216] In some optional implementations of this embodiment, the cross-computing power cluster communication device 800 may further include:

[0217] The buffer size upper limit determination unit is configured to determine the buffer size upper limit based on the bandwidth upper limit of the network card hardware and the connection number upper limit of the RDMA connection number; wherein the connection number upper limit is determined based on the bandwidth upper limit;

[0218] The second value control unit is configured to control the size after the last exponential growth to be smaller than the upper limit.

[0219] In some optional implementations of this embodiment, the growth amplitude of the first linear growth is determined based on the actual bandwidth growth amplitude before and after the last exponential growth processing; the growth amplitude of the non-first linear growth is determined based on the actual bandwidth growth amplitude before and after the last linear growth processing.

[0220] In some optional implementations of this embodiment, the cross-computing power cluster communication device 800 may further include:

[0221] A same computing power cluster belonging determination unit is configured to determine that both the communication receiving end and the communication receiving end belong to the same computing power cluster in response to the actual delay between the communication receiving end and the communication receiving end not exceeding the preset delay;

[0222] The intra-cluster communication unit is configured to communicate with a communication receiving end belonging to the same computing power cluster according to the intra-cluster transmission parameters determined by the collective communication library.

[0223] In some optional implementations of this embodiment, different computing power clusters include: computing power clusters for training generative large language models composed of graphics processors in different computer rooms located at different locations; correspondingly, the same computing power cluster includes: computing power clusters for training generative large language models composed of graphics processors in the same computer room located at the same location.

[0224] In some optional implementations of this embodiment, the cross-computing power cluster communication device 800 may further include:

[0225] The bandwidth value reduction degree determination unit is configured to determine the bandwidth value reduction degree of the current available bandwidth compared with the original stable available bandwidth in response to the current available bandwidth detected after the preset communication time according to the cross-cluster transmission parameter is less than the original stable available bandwidth; wherein the original stable available bandwidth is the bandwidth value matching the cross-cluster transmission parameter;

[0226] A downward adjustment unit is configured to adjust the target number of connections and the target size downward based on the degree of bandwidth value reduction to obtain the number of connections and the size after the reduction;

[0227] The cross-cluster transmission parameter adjustment unit is configured to determine the adjusted cross-cluster transmission parameter based on the RDMA connection number reset to the connection number after the reduction and the buffer size reset to the size after the reduction;

[0228] The second cross-cluster communication unit is configured to communicate with the communication receiving end in different computing power clusters according to the adjusted cross-cluster transmission parameters.

[0229] This embodiment exists as an apparatus embodiment corresponding to the above-mentioned method embodiment. The cross-computing power cluster communication apparatus provided by this embodiment provides a method for re-determining cross-cluster transmission parameters that match the cross-cluster communication environment for the situation where the communication initiator and the communication receiver belong to different computing power clusters respectively. That is, for the situation that cross-cluster communication has a significantly increased latency compared to intra-cluster communication, a method for successively increasing the number of remote direct data access RDMA connections and the size of the buffer BUFFER is provided for cross-cluster communication to determine the target number of connections and target size that match the actual bandwidth value, and then communicate with the communication receivers belonging to different computing power clusters according to the cross-cluster transmission parameters determined based on the target number of connections and the target size, so that cross-cluster communication and intra-cluster communication respectively use transmission parameters that match the actual communication environment, thereby improving the overall communication efficiency and data transmission volume, thereby improving the training efficiency of the generative large language model and shortening the time required for training.

[0230] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the cross-computing power cluster communication method described in any of the above embodiments can be implemented when the at least one processor executes.

[0231] According to an embodiment of the present disclosure, the present disclosure further provides a readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to implement the cross-computing power cluster communication method described in any of the above embodiments when executed.

[0232] According to an embodiment of the present disclosure, the present disclosure also provides a computer program product, which, when executed by a processor, can implement the cross-computing power cluster communication method described in any of the above embodiments.

[0233] Fig. 9 A schematic block diagram of an example electronic device 900 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0234] like Fig. 9 As shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0235] A number of components in the device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0236] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as a cross-computing cluster communication method. For example, in some embodiments, the cross-computing cluster communication method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the cross-computing cluster communication method described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to perform the cross-computing cluster communication method in any other appropriate manner (e.g., by means of firmware).

[0237] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0238] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0239] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0240] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0241] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0242] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client-server relationship is generated by computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and virtual private server (VPS) services.

[0243] According to the technical solution of the embodiments of the present disclosure, a method for re-determining cross-cluster transmission parameters that match the cross-cluster communication environment is provided for the situation where the communication initiator and the communication receiver belong to different computing power clusters respectively. That is, since cross-cluster communication has a significantly increased latency compared to intra-cluster communication, a method for successively increasing the number of remote direct data access (RDMA) connections and the size of the buffer zone BUFFER is provided for cross-cluster communication to determine the target number of connections and the target size that match the actual bandwidth value, and then communicating with the communication receivers belonging to different computing power clusters according to the cross-cluster transmission parameters determined based on the target number of connections and the target size, so that cross-cluster communication and intra-cluster communication respectively use transmission parameters that match the actual communication environment, thereby improving the overall communication efficiency and data transmission volume, thereby improving the training efficiency of the generative large language model and shortening the time required for training.

[0244] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0245] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A cross-computing power cluster communication method, applied to a communication initiator, comprising: In response to the communication receiving end belonging to different computing power clusters, the number of remote direct data access RDMA connections is increased based on the initial number of connections until the detected actual bandwidth value no longer increases, thereby obtaining a target number of connections; When the number of RDMA connections remains at the target number of connections, the size of the buffer is increased based on the initial size until the detected actual bandwidth value stops increasing, thereby obtaining the target size; Determining cross-cluster transmission parameters based on the number of RDMA connections set as the target number of connections and the buffer size set as the target size; Communicate with communication receiving terminals belonging to different computing power clusters according to the cross-cluster transmission parameters.

2. The method according to claim 1, further comprising: In response to an actual delay between the communication receiving end and the communication receiving end exceeding a preset delay, it is determined that the communication receiving end and the communication receiving end belong to different computing power clusters.

3. The method according to claim 2, wherein: The actual delay is calculated by the following steps: Determine the time when data is sent to the communication receiving end as the starting time; Determine the time when the confirmation notification returned by the communication receiving end after receiving the data is received as the end time; The actual delay is obtained by calculation based on the start time and the end time.

4. The method according to claim 3, wherein: The calculating the actual delay based on the start time and the end time includes: Calculate the delay values ​​corresponding to the start and end time pairs consisting of the start time and the matching end time of the preset number of logarithms, and obtain the delay set; An average of the delay values ​​in the delay set is calculated to obtain the actual delay.

5. The method according to claim 2, wherein: The broadband value is detected by the following steps: Acquire and calculate the actual data transmission size at the time corresponding to the actual delay; A detected bandwidth value is determined based on the actual data transmission size and the actual delay.

6. The method according to claim 1, wherein: The process of increasing the number of remote direct data access (RDMA) connections based on the initial number of connections until the detected actual bandwidth value stops increasing to obtain the target number of connections includes: Performing the connection number increase processing on the RDMA connection number based on the initial connection number, and detecting and obtaining a new actual bandwidth value after completing the connection number increase processing; In response to the new actual bandwidth value being greater than the original actual bandwidth value detected before the connection number increase process is performed, continuing to perform the connection number increase process on the current connection number of the RDMA connection number until the subsequently obtained new actual bandwidth value is equal to the original actual bandwidth value detected before the corresponding connection number increase process is performed; In response to the new actual bandwidth value being equal to the original actual bandwidth value detected before the corresponding connection number increase process is performed, the current connection number of the RDMA connection number is determined as the target connection number.

7. The method according to claim 6, wherein: The connection number increase process includes: A uniform increase strategy with a fixed step size or a non-uniform increase strategy with a non-fixed step size.

8. The method according to claim 7, wherein: The connection number increase process is a non-uniform increase strategy with an unfixed step size, and the RDMA connection number is increased based on the initial connection number until the detected actual bandwidth value no longer increases, thereby obtaining the target connection number, including: The RDMA connection number is processed by first exponential growth and then linear growth on the basis of the initial connection number until the detected actual bandwidth value no longer continues to increase, thereby obtaining the target connection number; wherein, the switching timing from the exponential growth to the linear growth is: the growth rate of the new actual bandwidth value detected after the exponential growth processing is lower than the growth rate of the connection number compared with the original actual bandwidth value detected before the corresponding exponential growth processing.

9. The method according to claim 8, further comprising: Determine the upper limit of the number of RDMA connections based on the upper limit of the bandwidth of the network card hardware; The number of connections after the last exponential growth is controlled to be less than the upper limit of the number of connections.

10. The method according to claim 8, wherein: The growth rate of the first linear growth is determined based on the actual bandwidth growth rate before and after the last exponential growth process; the growth rate of the non-first linear growth is determined based on the actual bandwidth growth rate before and after the last linear growth process.

11. The method according to claim 1, wherein: The process of increasing the size of the buffer buffer based on the initial size until the detected actual bandwidth value stops increasing to obtain the target size includes: Performing the size increase processing on the buffer size based on the initial size, and detecting and obtaining a new actual bandwidth value after the size increase processing is completed; In response to the new actual bandwidth value being greater than the original actual bandwidth value detected before the size increase process is performed, continuing to perform the size increase process on the current size of the buffer until the subsequently obtained new actual bandwidth value is equal to the original actual bandwidth value detected before the corresponding size increase process is performed; In response to the new actual bandwidth value being equal to the original actual bandwidth value detected before the corresponding size increase process is performed, the current size of the buffer size is determined as the target size.

12. The method according to claim 11, wherein: The size increase process includes: A uniform increase strategy with a fixed step size or a non-uniform increase strategy with a non-fixed step size.

13. The method according to claim 12, wherein: The size increase process is a non-uniform increase strategy with an unfixed step size, and the size of the buffer buffer is increased based on the initial size until the detected actual bandwidth value no longer increases, thereby obtaining a target size, including: The buffer size is processed by first exponential growth and then linear growth on the basis of the initial size until the detected actual bandwidth value no longer increases, thereby obtaining the target size; wherein, the switching timing from the exponential growth to the linear growth is: the growth rate of the new actual bandwidth value detected after the exponential growth processing is lower than the growth rate of the buffer size compared with the original actual bandwidth value detected before the corresponding exponential growth processing.

14. The method according to claim 13, further comprising: Based on the bandwidth upper limit of the network card hardware and the connection number upper limit of the RDMA connection number, determining the size upper limit of the buffer size; wherein the connection number upper limit is determined based on the bandwidth upper limit; The size of the control after the last exponential growth is less than the upper size limit.

15. The method according to claim 13, wherein: The growth rate of the first linear growth is determined based on the actual bandwidth growth rate before and after the last exponential growth process; the growth rate of the non-first linear growth is determined based on the actual bandwidth growth rate before and after the last linear growth process.

16. The method according to claim 2, further comprising: In response to the actual delay between the communication receiving end and the communication receiving end not exceeding the preset delay, determining that the communication receiving end and the communication receiving end belong to the same computing power cluster; Communicate with the communication receiving end belonging to the same computing power cluster according to the intra-cluster transmission parameters determined by the collective communication library.

17. The method according to claim 1, wherein: Different computing power clusters include: computing power clusters for training generative large language models, which are composed of graphics processors in different computer rooms located at different locations; correspondingly, the same computing power cluster includes: computing power clusters for training generative large language models, which are composed of graphics processors in the same computer room located at the same location.

18. The method according to any one of claims 1 to 17, further comprising: In response to the current available bandwidth detected after the preset communication time according to the cross-cluster transmission parameter being performed being less than the original stable available bandwidth, determining the degree of reduction of the bandwidth value of the current available bandwidth compared to the original stable available bandwidth; wherein the original stable available bandwidth is a bandwidth value matching the cross-cluster transmission parameter; Based on the degree of reduction of the bandwidth value, the target number of connections and the target size are adjusted downward to obtain the number of connections and the size after the adjustment; Determining adjusted cross-cluster transmission parameters based on the RDMA connection number reset to the reduced connection number and the Buffer size reset to the reduced size; Communicate with communication receiving terminals in different computing power clusters according to the adjusted cross-cluster transmission parameters.

19. A cross-computing power cluster communication device, applied to a communication initiator, comprising: The connection number increasing processing unit is configured to, in response to the communication receiving end belonging to different computing power clusters, increase the number of remote direct data access (RDMA) connections based on the initial number of connections until the detected actual bandwidth value no longer increases, thereby obtaining a target number of connections; A buffer increase processing unit is configured to increase the size of the buffer based on the initial size when the number of RDMA connections remains at the target number of connections until the detected actual bandwidth value stops increasing and the target size is obtained; a cross-cluster transmission parameter determination unit configured to determine the cross-cluster transmission parameter based on the number of RDMA connections set as the target number of connections and the buffer size set as the target size; The first cross-cluster communication unit is configured to communicate with communication receiving ends belonging to different computing power clusters according to the cross-cluster transmission parameters.

20. The apparatus according to claim 19, further comprising: The different computing power cluster belonging determination unit is configured to determine that the communication receiving end belongs to different computing power clusters respectively in response to the actual delay between the communication receiving end and the communication receiving end exceeding the preset delay.

21. The apparatus according to claim 20, further comprising: An actual delay calculation unit, the actual delay calculation unit comprising: A starting time determination subunit, configured to determine the time when data is sent to the communication receiving end as the starting time; The end time determination subunit is configured to determine the time of receiving the confirmation notification returned by the communication receiving end after receiving the data as the end time; The actual delay calculation subunit is configured to calculate the actual delay based on the start time and the end time.

22. The device according to claim 21, wherein The actual delay calculation subunit is further configured to: Calculate the delay values ​​corresponding to the start and end time pairs consisting of the start time and the matching end time of the preset number of logarithms, and obtain the delay set; An average of the delay values ​​in the delay set is calculated to obtain the actual delay.

23. The apparatus according to claim 20, further comprising: A bandwidth value detection unit, wherein the bandwidth value detection unit is further configured to: Acquire and calculate the actual data transmission size at the time corresponding to the actual delay; A detected bandwidth value is determined based on the actual data transmission size and the actual delay.

24. The device according to claim 19, wherein The connection number increase processing unit is further configured to: Performing the connection number increase processing on the RDMA connection number based on the initial connection number, and detecting and obtaining a new actual bandwidth value after completing the connection number increase processing; In response to the new actual bandwidth value being greater than the original actual bandwidth value detected before the connection number increase process is performed, continuing to perform the connection number increase process on the current connection number of the RDMA connection number until the subsequently obtained new actual bandwidth value is equal to the original actual bandwidth value detected before the corresponding connection number increase process is performed; In response to the new actual bandwidth value being equal to the original actual bandwidth value detected before the corresponding connection number increase process is performed, the current connection number of the RDMA connection number is determined as the target connection number.

25. The device according to claim 24, wherein: The connection number increase process includes: A uniform increase strategy with a fixed step size or a non-uniform increase strategy with a non-fixed step size.

26. The device according to claim 25, wherein The connection number increase process is a non-uniform increase strategy with an unfixed step size, and the connection number increase processing unit is further configured to: The RDMA connection number is processed by first exponential growth and then linear growth on the basis of the initial connection number until the detected actual bandwidth value no longer continues to increase, thereby obtaining the target connection number; wherein, the switching timing from the exponential growth to the linear growth is: the growth rate of the new actual bandwidth value detected after the exponential growth processing is lower than the growth rate of the connection number compared with the original actual bandwidth value detected before the corresponding exponential growth processing.

27. The apparatus according to claim 26, further comprising: A connection number upper limit determining unit, configured to determine a connection number upper limit of the RDMA connection number based on a bandwidth upper limit of the network card hardware; The first value control unit is configured to control the number of connections after the last exponential growth to be less than the upper limit of the number of connections.

28. The device according to claim 26, wherein The growth rate of the first linear growth is determined based on the actual bandwidth growth rate before and after the last exponential growth process; the growth rate of the non-first linear growth is determined based on the actual bandwidth growth rate before and after the last linear growth process.

29. The device according to claim 19, wherein The Buffer increase processing unit is further configured to: Performing the size increase processing on the buffer size based on the initial size, and detecting and obtaining a new actual bandwidth value after the size increase processing is completed; In response to the new actual bandwidth value being greater than the original actual bandwidth value detected before the size increase process is performed, continuing to perform the size increase process on the current size of the buffer until the subsequently obtained new actual bandwidth value is equal to the original actual bandwidth value detected before the corresponding size increase process is performed; In response to the new actual bandwidth value being equal to the original actual bandwidth value detected before the corresponding size increase process is performed, the current size of the buffer size is determined as the target size.

30. The device according to claim 29, wherein: The size increase process includes: A uniform increase strategy with a fixed step size or a non-uniform increase strategy with a non-fixed step size.

31. The device according to claim 29, wherein: The size increase process is a non-uniform increase strategy with an unfixed step size, and the buffer increase processing unit is further configured to: The buffer size is processed by first exponential growth and then linear growth on the basis of the initial size until the detected actual bandwidth value no longer increases, thereby obtaining the target size; wherein, the switching timing from the exponential growth to the linear growth is: the growth rate of the new actual bandwidth value detected after the exponential growth processing is lower than the growth rate of the buffer size compared with the original actual bandwidth value detected before the corresponding exponential growth processing.

32. The apparatus of claim 31 , further comprising: A buffer size upper limit determining unit is configured to determine the upper limit of the buffer size based on the bandwidth upper limit of the network card hardware and the connection number upper limit of the RDMA connection number; wherein the connection number upper limit is determined based on the bandwidth upper limit; The second value control unit is configured to control the size after the last exponential growth to be smaller than the upper limit.

33. The device according to claim 31, wherein The growth rate of the first linear growth is determined based on the actual bandwidth growth rate before and after the last exponential growth process; the growth rate of the non-first linear growth is determined based on the actual bandwidth growth rate before and after the last linear growth process.

34. The apparatus of claim 20, further comprising: A same computing power cluster belonging determination unit is configured to determine that both the communication receiving end and the communication receiving end belong to the same computing power cluster in response to the actual delay between the communication receiving end and the communication receiving end not exceeding the preset delay; The intra-cluster communication unit is configured to communicate with a communication receiving end belonging to the same computing power cluster according to the intra-cluster transmission parameters determined by the collective communication library.

35. The apparatus of claim 19, wherein: Different computing power clusters include: computing power clusters for training generative large language models, which are composed of graphics processors in different computer rooms located at different locations; correspondingly, the same computing power cluster includes: computing power clusters for training generative large language models, which are composed of graphics processors in the same computer room located at the same location.

36. The apparatus according to any one of claims 19 to 35, further comprising: a bandwidth value reduction degree determining unit, configured to determine a bandwidth value reduction degree of the current available bandwidth compared to the original stable available bandwidth in response to the current available bandwidth detected after the preset communication time according to the cross-cluster transmission parameter being less than the original stable available bandwidth; wherein the original stable available bandwidth is a bandwidth value matching the cross-cluster transmission parameter; A downward adjustment unit is configured to adjust the target number of connections and the target size downward based on the degree of reduction of the bandwidth value to obtain the number of connections and the size after the reduction; The cross-cluster transmission parameter adjustment unit is configured to determine the adjusted cross-cluster transmission parameter based on the RDMA connection number reset to the connection number after the reduction and the buffer size reset to the size after the reduction; The second cross-cluster communication unit is configured to communicate with the communication receiving end in different computing power clusters according to the adjusted cross-cluster transmission parameters.

37. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the cross-computing power cluster communication method described in any one of claims 1-18.

38. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute the cross-computing power cluster communication method described in any one of claims 1-18.

39. A computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the cross-computing power cluster communication method according to any one of claims 1-18.

Images