Communication method and device for GPU cluster, electronic equipment and medium

By dynamically constructing hybrid logical topologies and selecting communication protocols within GPU clusters, the problem of low communication efficiency in existing technologies is solved, achieving an efficient, flexible, and reliable communication strategy that adapts to dynamic changes in hardware and networks, thus shortening model training time.

CN121239682APending Publication Date: 2025-12-30BAIDU (CHINA) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202511397733.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-09-26
Publication Date
2025-12-30

AI Technical Summary

Technical Problem

Existing technologies in GPU cluster communication use fixed communication topologies and single communication protocols, which cannot adapt to the heterogeneity of node hardware performance and dynamic changes in network conditions, resulting in low communication efficiency and becoming a performance bottleneck in distributed computing.

Method used

By acquiring network performance data and hardware specification data of nodes in the GPU cluster, a hybrid logical topology is dynamically constructed. Combining tree, ring, and bus topologies, the optimal communication protocol is selected to form an adaptive communication strategy.

Benefits of technology

It significantly improves the performance, flexibility and robustness of communication, adapts to complex heterogeneous environments, reduces communication bottlenecks, and shortens model training time.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN121239682A_ABST
    Figure CN121239682A_ABST
Patent Text Reader

Abstract

The invention provides a communication method and device for a GPU cluster, electronic equipment and a medium, and relates to the technical field of artificial intelligence, in particular to the technical field of distributed computing, GPU cluster communication optimization and the like. According to the implementation scheme, the method comprises the steps of obtaining network performance data among a plurality of GPU nodes in a GPU cluster and hardware specification data of each GPU node in the plurality of GPU nodes; generating a hybrid logic communication topology for the plurality of GPU nodes based on the network performance data and the hardware specification data, the hybrid logic communication topology being a combination of at least two of a tree topology, a ring topology and a bus topology; determining a communication protocol adopted by a communication link in the hybrid logic communication topology based on the network performance data; and communicating among the plurality of GPU nodes according to the generated hybrid logic communication topology and communication protocol.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to the fields of distributed computing and GPU cluster communication optimization, specifically to a communication method, apparatus, electronic device, computer-readable storage medium, and computer program product for GPU clusters. Background Technology

[0002] Artificial intelligence (AI) is the study of enabling computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It encompasses both hardware and software technologies. AI hardware technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, and big data processing. AI software technologies primarily include computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graph technologies.

[0003] With the rapid development of artificial intelligence technology, the scale and complexity of deep learning models (such as large language models) are increasing daily. To shorten the training cycle, distributed training has become a mainstream technical solution, utilizing clusters of multiple GPU nodes to work collaboratively and accelerate the model convergence process. In distributed training, data communication between GPU nodes, especially ensemble communication operations used to synchronize model parameters or gradients, is a key factor determining overall training efficiency. Therefore, in training clusters with increasingly large node sizes, designing efficient, flexible, and reliable communication strategies is an important direction for current technological development.

[0004] The methods described in this section are not necessarily methods that had been previously conceived or adopted. Unless otherwise specified, no method described in this section should be assumed to be prior art simply because it is included in this section. Similarly, unless otherwise specified, the issues mentioned in this section should not be considered to be accepted in any prior art. Summary of the Invention

[0005] This disclosure provides a communication method, apparatus, electronic device, computer-readable storage medium, and computer program product for GPU clusters.

[0006] According to one aspect of this disclosure, a communication method for a GPU cluster is provided, comprising: acquiring network performance data among multiple GPU nodes in the GPU cluster and hardware specification data of each GPU node; generating a hybrid logical communication topology for the multiple GPU nodes based on the network performance data and the hardware specification data, wherein the hybrid logical communication topology is a combination of at least two of tree topology, ring topology and bus topology; determining the communication protocol used by the communication links in the hybrid logical communication topology based on the network performance data; and performing communication among the multiple GPU nodes according to the generated hybrid logical communication topology and the communication protocol.

[0007] According to another aspect of this disclosure, a communication apparatus for a GPU cluster is provided, comprising: an acquisition module configured to acquire network performance data among a plurality of GPU nodes in the GPU cluster and hardware specification data of each of the plurality of GPU nodes; a generation module configured to generate a hybrid logical communication topology for the plurality of GPU nodes based on the network performance data and the hardware specification data, wherein the hybrid logical communication topology is a combination of at least two of a tree topology, a ring topology, and a bus topology; a determination module configured to determine, based on the network performance data, a communication protocol used by the communication links in the hybrid logical communication topology; and a communication module configured to communicate among the plurality of GPU nodes according to the generated hybrid logical communication topology and the communication protocol.

[0008] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the methods described above.

[0009] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform the above-described method.

[0010] According to another aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the above-described method.

[0011] According to one or more embodiments of this disclosure, a communication method for GPU clusters is provided. Through this data-driven dynamic optimization, the most efficient communication strategy can be tailored for the GPU cluster, significantly improving communication performance, flexibility, and robustness in complex heterogeneous environments.

[0012] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0013] The accompanying drawings exemplify embodiments and form part of the specification, serving together with the textual description to explain exemplary implementations of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of the claims. Throughout the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0014] Figure 1 This is a schematic diagram illustrating example systems in which the various methods described herein can be implemented according to exemplary embodiments. Figure 2 A flowchart illustrating a communication method for a GPU cluster according to an embodiment of the present disclosure is shown; Figure 3 A schematic diagram of a hybrid logic communication topology according to an embodiment of the present disclosure is shown; Figure 4 A structural block diagram of a communication apparatus for a GPU cluster according to embodiments of the present disclosure is shown; and Figure 5 A structural block diagram of an exemplary electronic device that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0015] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0016] In this disclosure, unless otherwise stated, the use of terms such as "first," "second," etc., to describe various elements is not intended to limit the positional, temporal, or importance relationships of these elements; such terms are merely used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of that element, while in other cases, based on the context, they may refer to different instances.

[0017] The terminology used in the description of the various examples described in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context explicitly indicates otherwise, an element may be one or more unless the number of elements is specifically limited. Furthermore, the term "and / or" as used in this disclosure covers any one of the listed items and all possible combinations thereof.

[0018] In related technologies, GPU cluster communication typically employs a fixed communication topology (such as a pure ring or tree topology) and a single communication protocol. This static and singular strategy cannot adapt to the complex real-world environment of heterogeneous node hardware performance and dynamically changing network conditions, resulting in low communication efficiency and becoming a performance bottleneck in distributed computing.

[0019] To address the aforementioned issues, this disclosure provides a communication method for GPU clusters that can dynamically construct the optimal communication architecture and intelligently select communication protocols based on real-time hardware and network conditions. This enables the creation of the most efficient communication strategy tailored to the GPU cluster, significantly improving communication performance, flexibility, and robustness in complex heterogeneous environments.

[0020] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0021] Figure 1 A schematic diagram of an exemplary system 100 in which the various methods and apparatus described herein can be implemented according to embodiments of this disclosure is shown. Reference Figure 1 The system 100 includes one or more client devices 101, 102, 103, 104, 105 and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120. The client devices 101, 102, 103, 104, 105 and 106 can be configured to execute one or more applications.

[0022] In embodiments of this disclosure, server 120 may run one or more services or software applications that enable the execution of communication methods for GPU clusters.

[0023] In some embodiments, server 120 may also provide other services or software applications that may include non-virtual and virtual environments. In some embodiments, these services may be provided as web-based services or cloud services, such as to users of client devices 101, 102, 103, 104, 105 and / or 106 under a Software as a Service (SaaS) model.

[0024] exist Figure 1In the configuration shown, server 120 may include one or more components that implement the functions performed by server 120. These components may include software components, hardware components, or combinations thereof that can be executed by one or more processors. Users operating client devices 101, 102, 103, 104, 105, and / or 106 can sequentially interact with server 120 using one or more client applications to utilize the services provided by these components. It should be understood that various different system configurations are possible and may differ from system 100. Therefore, Figure 1 This is an example of a system used to implement the methods described herein, and is not intended to be limiting.

[0025] Users can use client devices 101, 102, 103, 104, 105, and / or 106 to execute communication methods for the GPU cluster. The client devices can provide an interface that allows users to interact with the client devices. The client devices can also output information to users through this interface. Although... Figure 1 Only six client devices are described, but those skilled in the art will understand that this disclosure can support any number of client devices.

[0026] Client devices 101, 102, 103, 104, 105, and / or 106 may include various types of computer devices, such as portable handheld devices, general-purpose computers (such as personal computers and laptops), workstation computers, wearable devices, smart screen devices, self-service terminal devices, service robots, gaming systems, thin clients, various messaging devices, sensors, or other sensing devices. These computer devices can run various types and versions of software applications and operating systems, such as Microsoft Windows, Apple iOS, UNIX-like operating systems, Linux or Linux-like operating systems (such as Google Chrome OS); or include various mobile operating systems, such as Microsoft Windows Mobile OS, iOS, Windows Phone, and Android. Portable handheld devices may include cellular phones, smartphones, tablets, personal digital assistants (PDAs), etc. Wearable devices may include head-mounted displays (such as smart glasses) and other devices. Gaming systems may include various handheld gaming devices, internet-enabled gaming devices, etc. Client devices are capable of executing various applications, such as various internet-related applications, communication applications (such as email applications), short message service (SMS) applications, and can use various communication protocols.

[0027] Network 110 can be any type of network well known to those skilled in the art, and can support data communication using any of a variety of available protocols (including but not limited to TCP / IP, SNA, IPX, etc.). By way of example only, one or more networks 110 can be a local area network (LAN), an Ethernet-based network, a token ring network, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., Bluetooth, WIFI), and / or any combination of these and / or other networks.

[0028] Server 120 may include one or more general-purpose computers, special-purpose server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and / or combination. Server 120 may include one or more virtual machines running a virtual operating system, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that can be virtualized to maintain virtual storage devices for servers). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.

[0029] The computing unit in server 120 can run one or more operating systems, including any of the aforementioned operating systems and any commercially available server operating system. Server 120 can also run any of a variety of additional server applications and / or middleware applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.

[0030] In some implementations, server 120 may include one or more applications to analyze and merge data feeds and / or event updates received from users of client devices 101, 102, 103, 104, 105, and 106. Server 120 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of client devices 101, 102, 103, 104, 105, and 106.

[0031] In some implementations, server 120 can be a server for a distributed system or a server integrated with blockchain. Server 120 can also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. A cloud server is a host product in the cloud computing service system, designed to address the shortcomings of traditional physical hosts and Virtual Private Server (VPS) services, such as high management difficulty and weak business scalability.

[0032] System 100 may also include one or more databases 130. In some embodiments, these databases may be used to store data and other information. For example, one or more of the databases 130 may be used to store information such as audio files and video files. Databases 130 may reside in various locations. For example, a database used by server 120 may be local to server 120, or it may be located away from server 120 and may communicate with server 120 via a network-based or dedicated connection. Databases 130 may be of different types. In some embodiments, the database used by server 120 may be, for example, a relational database. One or more of these databases may store, update, and retrieve data from and from the databases in response to commands.

[0033] In some embodiments, one or more of the databases 130 may also be used by an application to store application data. The databases used by the application may be of different types, such as key-value stores, object stores, or regular stores supported by a file system.

[0034] Figure 1 The system 100 can be configured and operated in various ways to enable the application of the various methods and apparatus described in this disclosure.

[0035] Figure 2 A flowchart of a communication method for a GPU cluster according to an embodiment of the present disclosure is shown.

[0036] like Figure 2 As shown, the communication method 200 for GPU clusters includes: Step S201: Obtain network performance data among multiple GPU nodes in the GPU cluster and hardware specification data of each GPU node in the multiple GPU nodes; Step S202: Based on the network performance data and the hardware specification data, generate a hybrid logical communication topology for the multiple GPU nodes, wherein the hybrid logical communication topology is a combination of at least two of tree topology, ring topology and bus topology; Step S203: Based on the network performance data, determine the communication protocol used by the communication links in the hybrid logic communication topology; and Step S204: Communicate among the multiple GPU nodes according to the generated hybrid logic communication topology and the communication protocol.

[0037] Compared to existing GPU cluster communication schemes, the communication method 200 for GPU clusters no longer employs a rigid, single communication strategy, but instead constructs a data-driven dynamic optimization framework. Method 200 first performs a comprehensive performance analysis of the hardware capabilities of each GPU node in the cluster and the real-time network conditions between them. Then, based on this comprehensive data, it intelligently generates an optimal hybrid logical topology composed of various structures such as tree and ring topologies, and selects the most efficient communication protocol for the communication links within this topology. Finally, it executes data transmission according to the communication scheme obtained from this dynamic decision-making process. In this way, Method 200 can dynamically achieve an optimal match between the communication strategy and the current hardware and network environment, thereby significantly improving communication performance, flexibility, and overall system robustness in heterogeneous cluster environments.

[0038] In the embodiments disclosed herein, the first step of optimization is to conduct comprehensive data collection to provide a basis for subsequent decision-making. This step collects data in two dimensions: first, network performance data; second, hardware specification data.

[0039] According to some embodiments, step S201 includes: using a network testing tool to obtain at least one of the network bandwidth and network latency between any two nodes in the plurality of GPU nodes as the network performance data. For example, this step can utilize a network testing tool (e.g., nccl-tests) to perform actual measurements on key network metrics between any two nodes in the GPU cluster, obtaining data such as network bandwidth and network latency.

[0040] Therefore, by collecting real, quantitative network performance indicators, rather than static theoretical values, we can ensure that subsequent optimization decisions accurately reflect the current state of the network, thus guaranteeing the effectiveness of the optimization results.

[0041] According to some embodiments, the hardware specification data includes at least one of the following for each GPU node: the number of CUDA cores, computing power, and memory bandwidth. For example, the hardware specification data may include the number of CUDA cores, computing power (e.g., TFLOPS), and memory bandwidth for each GPU node. These parameters are the most core and relevant hardware performance indicators reflecting the GPU's computing and communication capabilities. They provide a solid data foundation for subsequent high-performance-oriented node clustering and communication load allocation, enabling the entire optimization scheme to possess hardware awareness.

[0042] In step S202, an optimized hybrid logical communication topology is dynamically constructed based on the comprehensive data collected in step S201. This topology consists of at least two combinations of tree topology, ring topology, and bus topology, rather than a single fixed structure. This process can be implemented using a hierarchical approach: first, multiple GPU nodes are grouped and clustered; then, the optimal connection method is determined within each group; finally, these independent groups are connected to achieve interconnection between clusters. This achieves a balance between local optimization and global connectivity, making it possible to construct efficient large-scale hybrid topologies.

[0043] Specifically, step S202 includes: clustering the multiple GPU nodes based on the network performance data and the hardware specification data to generate at least one node cluster; determining the intra-cluster communication topology for each of the at least one node cluster based on a preset topology selection rule; and interconnecting the at least one node cluster to generate the hybrid logical communication topology.

[0044] As previously described, firstly, multiple GPU nodes are clustered based on performance data to generate at least one node cluster. According to some embodiments, this clustering process is performance-oriented: the multiple GPU nodes are weighted and scored based on the hardware specification data; at least one central node is determined from the multiple GPU nodes based on the scores; and a k-means algorithm is used to assign each non-central node among the multiple GPU nodes to a corresponding central node among the at least one central node, based on the network performance data and the hardware specification data, to generate the at least one node cluster.

[0045] For example, all GPU nodes are first weighted and scored based on hardware specifications, and the node with the best performance is selected as the "central node." Then, the k-means algorithm is used to comprehensively consider the hardware similarity and network latency between other non-central nodes and these central nodes, and they are assigned to the appropriate clusters. This method ensures that the node with the strongest hardware performance becomes the communication center, and nodes with similar performance and short communication distances are grouped together, laying the foundation for efficient communication.

[0046] Secondly, based on the preset topology selection rules, the intra-cluster communication topology is determined for each node cluster.

[0047] According to some embodiments, determining the intra-cluster communication topology for the at least one node cluster based on a preset topology selection rule includes: for each node cluster in the at least one node cluster, in response to determining that the number of GPU nodes in the node cluster is greater than a first threshold, determining the node cluster as a tree-shaped communication topology; and in response to determining that the number of GPU nodes in the node cluster is not greater than the first threshold and the network latency variance between the GPU nodes in the node cluster is less than a second threshold, determining the node cluster as a ring-shaped communication topology.

[0048] Understandably, the above topology selection rules are based on automated decision-making using quantitative indicators. For example, for each node cluster, if the number of GPU nodes within it is greater than the first threshold, a tree-shaped communication topology with good scalability is determined for it; if the number of nodes is not greater than the first threshold and the network latency variance between nodes is less than the second threshold, which indicates that the network is highly homogeneous, a ring-shaped communication topology with extremely high efficiency is determined for it.

[0049] This refined optimization ensures that each substructure adopts the communication mode best suited to its own characteristics, maximizing local communication efficiency.

[0050] Finally, the individual node clusters are interconnected to form a complete hybrid logical communication topology. According to some embodiments, interconnection between clusters can be achieved by connecting the central nodes of each cluster, for example, forming a higher-level tree structure. This hierarchical construction strategy achieves a balance between local optimization and global connectivity, making it possible to build efficient, large-scale hybrid topologies.

[0051] Figure 3 A schematic diagram of a hybrid logic communication topology according to an embodiment of the present disclosure is shown. Figure 3 This clearly demonstrates how Method 200 organically combines three different basic topologies—tree, bus, and ring—into a unified communication framework. Specifically, nodes 1, 2, and 3 form a top-level tree structure, with node 1 as the root node. Under the branch of node 2, nodes 2, 5, 6, and 4 connect to form a bus topology (or chain topology). Under the branch of node 3, nodes 7, 8, 9, and 10 form an efficient ring topology. In this way, the topology generated by Method 200 can adapt to the characteristics of different local network environments, achieving optimal global communication efficiency.

[0052] In the embodiments of this disclosure, in addition to optimizing the topology, the method also performs link-level fine-grained selection of communication protocols to further improve performance.

[0053] According to some embodiments, the communication link in the hybrid logic communication topology includes multiple sub-links.

[0054] Step S203 includes: based on the network performance data, evaluating whether the path quality of each of the plurality of sub-links meets preset conditions, the preset conditions including the network latency of the sub-link being lower than a third threshold and / or the network bandwidth of the sub-link being higher than a fourth threshold; for each sub-link, in response to determining that the path quality of the sub-link meets the preset conditions, determining that the sub-link uses the TCP protocol for communication; and in response to the path quality of the sub-link not meeting the preset conditions, determining that the sub-link uses the UDP protocol for communication.

[0055] First, the path quality of each sub-link is evaluated to determine if it meets preset conditions. For each sub-link in the hybrid logical communication topology, its path quality is evaluated based on the network performance data (e.g., latency and bandwidth) collected in S201. According to some embodiments, the preset conditions may include the sub-link's network latency being lower than a third threshold and / or the sub-link's network bandwidth being higher than a fourth threshold. The specific threshold values ​​can be set according to actual needs, and this disclosure does not limit them.

[0056] If preset conditions are met, the TCP protocol is selected. This means that for each sub-link, the TCP protocol is chosen in response to the determination that its path quality meets preset conditions (e.g., a high-quality link with low latency and high bandwidth, also known as a "short path").

[0057] If the preset conditions are not met, the UDP protocol will be used. It is understandable that, in response to the determination that the path quality of a sub-link does not meet the preset conditions (e.g., it is a high-latency or low-bandwidth link, also known as a "long path"), the UDP protocol (e.g., the QUIC protocol) will be selected for it.

[0058] In this way, this approach provides fine-grained link-level optimization. It allows TCP to be used on high-quality links where reliability is required, while leveraging the speed advantage of UDP on high-latency, bottleneck-prone links, thereby maximizing the overall communication throughput of the hybrid topology.

[0059] Therefore, the GPU cluster can perform actual communication operations based on the hybrid logical topology generated in step S202 and the communication protocol allocation scheme determined in step S203. The final generated hybrid topology and protocol allocation scheme constitute a complete communication plan, which can be broadcast to all participating nodes via methods such as `torch.distributed.broadcast` at the beginning of all service startup. Each node runs a dedicated service process (e.g., `dyna_client`), which is responsible for parsing the received communication plan and strictly executing subsequent data sending and receiving operations according to the plan.

[0060] Furthermore, this method also possesses dynamic adaptive capabilities. According to some embodiments, the communication method 200 for a GPU cluster further includes: periodically collecting network performance data among the plurality of GPU nodes in the GPU cluster; and updating the hybrid logic communication topology in response to detecting a change in the network performance data.

[0061] For example, network performance data between GPU nodes can be periodically reacquired, and the hybrid logic communication topology can be updated in real time in response to the detection of significant changes in network performance (such as congestion or jitter). This allows the communication strategy to remain optimal, greatly enhancing the stability and high performance of the solution in real, dynamic network environments.

[0062] According to some embodiments, this method may also add a verification step before formal communication. The communication method 200 for a GPU cluster further includes: performing a network connectivity test on the plurality of GPU nodes based on the generated hybrid logic communication topology before communication occurs between the plurality of GPU nodes.

[0063] For example, before communication occurs between multiple GPU nodes, a network connectivity test is performed on the nodes using scripts such as torchrun, based on the generated hybrid logical communication topology. This pre-test ensures the generated topology is feasible, effectively preventing program crashes or communication deadlocks caused by configuration errors, thereby improving the reliability and robustness of the method.

[0064] In summary, this disclosure provides an innovative and efficient adaptive hybrid communication optimization framework for GPU clusters. This framework systematically addresses the problems of existing technologies through a data-driven closed-loop optimization process. This process includes four core stages: 1) Performance profiling: Comprehensively collecting hardware specifications of GPU nodes and real-time network performance data between nodes. 2) Intelligent topology generation: Dynamically constructing an optimal hybrid logical topology composed of tree, ring, and bus structures through hardware-aware intelligent clustering combined with rule-based judgment. 3) Link-level protocol decision: Fine-tuning the selection of the most efficient communication protocol (TCP or UDP / QUIC) for each communication link in the hybrid topology. 4) Communication execution and dynamic adjustment: Executing communication according to the generated scheme and periodically monitoring network changes for dynamic adjustments.

[0065] Compared to existing technologies, this solution offers significant advantages. First, it significantly improves communication efficiency. By tailoring the optimal topology and protocol combination for heterogeneous clusters, it minimizes communication bottlenecks and fully utilizes network bandwidth. Second, it enhances system adaptability. It can automatically adapt to and cope with complex environments of heterogeneous hardware and dynamically changing networks, ensuring the system always operates in a near-optimal communication mode. Furthermore, it enables refined, layered optimization. Optimization extends from the macro-level topology structure to the micro-level protocol selection for each link, achieving more thorough performance optimization.

[0066] This solution can be widely applied in distributed computing fields with extremely high communication performance requirements. In large-scale deep learning model training scenarios, this solution can significantly shorten the time consumed by collective communication operations such as AllReduce, reduce GPU idle time caused by communication bottlenecks, thereby effectively shortening the total training time of the model and accelerating model convergence.

[0067] In addition, in the field of high-performance computing (HPC), such as scientific computing and physical simulation, this solution can handle large-scale data exchange between nodes more efficiently, improving the overall throughput and solution efficiency of complex computing tasks.

[0068] According to another aspect of this disclosure, a communication method apparatus for GPU clusters is also provided.

[0069] like Figure 4 As shown, a communication method apparatus 400 for a GPU cluster includes: an acquisition module 401 configured to acquire network performance data among multiple GPU nodes in the GPU cluster and hardware specification data of each GPU node; a generation module 402 configured to generate a hybrid logical communication topology for the multiple GPU nodes based on the network performance data and the hardware specification data, wherein the hybrid logical communication topology is a combination of at least two of tree topology, ring topology and bus topology; a determination module 403 configured to determine the communication protocol used by the communication links in the hybrid logical communication topology based on the network performance data; and a communication module 404 configured to communicate among the multiple GPU nodes according to the generated hybrid logical communication topology and the communication protocol.

[0070] The operation of modules 401-404 of the communication method apparatus 400 for GPU clusters is similar to the operation of steps S201-S204 described above, and will not be repeated here.

[0071] Therefore, compared to existing GPU cluster communication schemes, the communication method apparatus 400 for GPU clusters no longer employs a rigid, single communication strategy, but instead constructs a data-driven dynamic optimization framework. The communication method apparatus 400 for GPU clusters first performs a comprehensive performance analysis of the hardware capabilities of each GPU node in the cluster and the real-time network conditions between them; then, based on this comprehensive data, it intelligently generates an optimal hybrid logical topology composed of various structures such as tree and ring topologies, and selects the most efficient communication protocol for the communication links in this topology; finally, it executes data transmission according to the communication scheme obtained from this dynamic decision-making process. In this way, the communication method apparatus 400 for GPU clusters can dynamically achieve the best match between the communication strategy and the current hardware and network environment, thereby significantly improving the communication performance, flexibility, and overall system robustness in heterogeneous cluster environments.

[0072] It is understood that the various modules of the communication method apparatus 400 for GPU clusters can be configured to execute the various method steps defined in method 200 to achieve efficient and reliable communication within the GPU cluster, which will not be elaborated further here.

[0073] According to another aspect of this disclosure, an electronic device is also provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a communication method for a GPU cluster.

[0074] According to another aspect of this disclosure, a non-transitory computer-readable storage medium storing computer instructions is also provided, wherein the computer instructions are used to cause the computer to perform a communication method for a GPU cluster.

[0075] According to another aspect of this disclosure, a computer program product is also provided, including a computer program, wherein the computer program, when executed by a processor, implements a communication method for a GPU cluster.

[0076] like Figure 5 As shown, the electronic device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. The RAM 503 may also store various programs and data required for the operation of the electronic device 500. The computing unit 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0077] Multiple components in electronic device 500 are connected to I / O interface 505, including: input unit 506, output unit 507, storage unit 508, and communication unit 509. Input unit 506 can be any type of device capable of inputting information to electronic device 500. Input unit 506 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of electronic device, and may include, but is not limited to, a mouse, keyboard, touchscreen, trackpad, trackball, joystick, microphone, and / or remote control. Output unit 507 can be any type of device capable of presenting information, and may include, but is not limited to, a monitor, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 508 may include, but is not limited to, disk and optical disk. Communication unit 509 allows electronic device 500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers, and / or chipsets, such as Bluetooth. TM Devices, 802.11 devices, WiFi devices, WiMax devices, cellular communication devices and / or the like.

[0078] The computing unit 501 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above, such as communication methods for GPU clusters. For example, in some embodiments, the communication methods for GPU clusters may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by the computing unit 501, one or more steps of the communication methods for GPU clusters described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to perform communication methods for GPU clusters by any other suitable means (e.g., by means of firmware).

[0079] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0080] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0081] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0082] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0083] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0084] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0085] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0086] While embodiments or examples of this disclosure have been described with reference to the accompanying drawings, it should be understood that the methods, systems, and devices described above are merely exemplary embodiments or examples, and the scope of the invention is not limited by these embodiments or examples, but only by the granted claims and their equivalents. Various elements in the embodiments or examples may be omitted or replaced by their equivalents. Furthermore, the steps may be performed in a different order than that described in this disclosure. Further, various elements in the embodiments or examples may be combined in various ways. Importantly, as the technology evolves, many elements described herein can be replaced by equivalents that appear after this disclosure.

Claims

1. A method for communication of a GPU cluster, comprising: obtaining network performance data among a plurality of GPU nodes in the GPU cluster and hardware specification data of each of the plurality of GPU nodes; generating a hybrid logical communication topology for the plurality of GPU nodes based on the network performance data and the hardware specification data, wherein the hybrid logical communication topology is a combination of at least two of a tree topology, a ring topology and a bus topology; determining a communication protocol for a communication link in the hybrid logical communication topology based on the network performance data; and communicating among the plurality of GPU nodes according to the generated hybrid logical communication topology and the communication protocol. The obtaining the network performance data among the plurality of GPU nodes in the GPU cluster comprises:

2. The method of claim 1, wherein, obtaining at least one of network bandwidth and network latency between any two of the plurality of GPU nodes as the network performance data by using a network testing tool. The generating the hybrid logical communication topology for the plurality of GPU nodes based on the network performance data and the hardware specification data comprises:

3. The method of claim 1 or 2, wherein, clustering the plurality of GPU nodes based on the network performance data and the hardware specification data to generate at least one node cluster; determining an intra-cluster communication topology for each of the at least one node cluster based on a preset topology selection rule; and interconnecting the at least one node cluster to generate the hybrid logical communication topology. The clustering the plurality of GPU nodes based on the network performance data and the hardware specification data to generate at least one node cluster comprises:

4. The method of claim 3, wherein, weighting the plurality of GPU nodes based on the hardware specification data to determine at least one center node from the plurality of GPU nodes based on the weighting; and assigning each non-center node of the plurality of GPU nodes to a corresponding center node of the at least one center node based on the network performance data and the hardware specification data by using a k-means algorithm to generate the at least one node cluster. The determining the intra-cluster communication topology for each of the at least one node cluster based on the preset topology selection rule comprises:

5. The method of claim 3 or 4, wherein, for each of the at least one node cluster, in response to determining that a number of GPU nodes in the node cluster is greater than a first threshold, determining the node cluster as a tree communication topology; and in response to determining that the number of GPU nodes in the node cluster is not greater than the first threshold and a variance of network latency among the GPU nodes in the node cluster is less than a second threshold, determining the node cluster as a ring communication topology. The communication link in the hybrid logical communication topology comprises a plurality of sub-links, and wherein the determining the communication protocol for the communication link in the hybrid logical communication topology based on the network performance data comprises:

6. The method of any one of claims 1-5, wherein, evaluating whether a path quality of each of the plurality of sub-links meets a preset condition based on the network performance data, the preset condition comprising that a network latency of a sub-link is lower than a third threshold and / or a network bandwidth of the sub-link is higher than a fourth threshold. ​ For each sub-link, in response to determining that a path quality of the sub-link satisfies the preset condition, determining that the sub-link uses a TCP protocol for communication; and in response to the path quality of the sub-link not satisfying the preset condition, determining that the sub-link uses a UDP protocol for communication.

7. The method of any one of claims 1-6, further comprising: periodically collecting network performance data among the plurality of GPU nodes in the GPU cluster; and in response to detecting a change in the network performance data, updating the hybrid logical communication topology.

8. The method of any one of claims 1-7, further comprising: prior to communicating among the plurality of GPU nodes, performing a network connectivity test on the plurality of GPU nodes based on the generated hybrid logical communication topology.

9. The method of any one of claims 1-8, wherein, The hardware specification data includes at least one of a number of CUDA cores, a computing power, and a memory bandwidth of each GPU node.

10. A communication apparatus for a GPU cluster, comprising: an obtaining module configured to obtain network performance data among a plurality of GPU nodes in the GPU cluster and hardware specification data of each GPU node in the plurality of GPU nodes; a generating module configured to generate a hybrid logical communication topology for the plurality of GPU nodes based on the network performance data and the hardware specification data, wherein the hybrid logical communication topology is a combination of at least two of a tree topology, a ring topology, and a bus topology; a determining module configured to determine a communication protocol used by a communication link in the hybrid logical communication topology based on the network performance data; and a communication module configured to communicate among the plurality of GPU nodes according to the generated hybrid logical communication topology and the communication protocol.

11. An electronic device, comprising: at least one processor; and a memory communicatively connected with the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-9.

12. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to cause the computer to perform the method of any one of claims 1-9.

13. A computer program product comprising a computer program, wherein, The computer program, when executed by a processor, implements the method of any one of claims 1-9. The computer program, when executed by a processor, implements the method of any one of claims 1-9.