Image processing unit (GPU) cluster communication method and electronic device

Abstract

The embodiment of the invention provides an image processing unit (GPU) cluster communication method and an electronic device, and the method comprises the steps: splitting a GPU cluster, and obtaining at least one algorithm isomorphic group and at least one algorithm heterogeneous group; the protocol distribution communication and the full collection communication are performed through the isomorphic GPUs in each algorithm isomorphic group, and the full protocol communication is performed through the heterogeneous GPUs in each algorithm heterogeneous group. Therefore, through the embodiment of the invention, the problems that the communication traffic between heterogeneous GPUs is increased and extra data copy overhead is introduced due to the difference of communication libraries between different GPUs in a heterogeneous GPU cluster environment can be solved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of communications, and more specifically, to a method and electronic device for communicating with a graphics processing unit (GPU) cluster. Background Art

[0002] With the rapid development of information technology, the information society has entered a new stage dominated by deep learning models and large-scale language models. Machine intelligence has become the new mainstream productivity, having a profound impact on society as a whole. As a critical infrastructure for large-scale model training and inference, large-scale distributed GPU (Graphics Processing Unit) clusters and their software stacks provide strong support for the stable and efficient operation of models. With the rapid replacement and upgrade of GPUs, it is difficult to ensure that a data center has only one type of GPU chip. Large-scale distributed GPU clusters are increasingly heterogeneous, that is, using two or more different types of GPUs in a single GPU cluster, which brings new challenges to large-scale model training.

[0003] Regarding communication between different GPUs, the lack of unified interfaces and protocols in GPU communication libraries prevents efficient direct communication between heterogeneous GPUs. Traditional solutions fall into two main categories: heterogeneous communication that does not require CPU (Central Processing Unit) transit (such as communication architectures based on RDMA (Remote Direct Memory Access) technology) and heterogeneous communication based on CPU transit. Although the former can achieve efficient transmission, each introduction of a new GPU requires back-end adaptation, resulting in poor scalability. The latter uses the CPU as a transit station, standardizing the interface and reducing adaptation costs, but the additional CPU-GPU data copy overhead is significant, affecting communication efficiency.

[0004] In summary, no effective solution has been proposed in the related art. Summary of the Invention

[0005] The embodiments of the present application provide an image processing unit (GPU) cluster communication method and electronic device to at least solve the problem of increased communication volume and introduction of additional data copy overhead between heterogeneous GPUs due to differences in communication libraries between different GPUs in a heterogeneous GPU cluster environment.

[0006] According to one embodiment of the present application, a method for communication of an image processing unit (GPU) cluster is provided, comprising: splitting a GPU cluster to obtain at least one algorithm homogeneous group and at least one algorithm heterogeneous group; performing protocol distribution communication and full collection communication through the homogeneous GPUs in each algorithm homogeneous group, and performing full protocol communication through the heterogeneous GPUs in each algorithm heterogeneous group.

[0007] According to another embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored. The computer program is configured to execute the steps of any one of the above method embodiments when run.

[0008] According to another embodiment of the present application, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in the above method embodiment.

[0009] According to another embodiment of the present application, a computer program product is provided, including a computer program, which implements the steps in the above method embodiment when executed by a processor.

[0010] Through the above-mentioned embodiments of the present application, a method for communication of an image processing unit GPU cluster is provided, wherein the GPU cluster is split to obtain at least one algorithm isomorphic group and at least one algorithm heterogeneous group, and protocol distribution communication and full collection communication are performed by the homogeneous GPUs in each algorithm isomorphic group, and full protocol communication is performed by the heterogeneous GPUs in each algorithm heterogeneous group. That is, by splitting the GPU cluster, full utilization of efficient communication of homogeneous GPUs is achieved, and the frequency and amount of heterogeneous GPU communication are reduced, thereby reducing the difficulty of heterogeneous GPU communication and improving the efficiency and performance of large-scale distributed computing. Therefore, the embodiments of the present application can solve the problem of increased communication volume and introduction of additional data copy overhead between heterogeneous GPUs due to differences in communication libraries between different GPUs in a heterogeneous GPU cluster environment, thereby achieving the effect of improving the communication efficiency of the GPU cluster. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 This is a hardware structure block diagram of a computer terminal according to the GPU cluster communication method of an embodiment of the present application;

[0012] Figure 2 This is a network architecture diagram of a method for running a GPU cluster communication according to an embodiment of the present application;

[0013] Figure 3 is a flowchart of a GPU cluster communication method according to an embodiment of the present application;

[0014] Figure 4 This is a schematic diagram of the splitting of a GPU cluster according to an embodiment of the present application;

[0015] Figure 5 is an overall flow chart of the GPU cluster communication method according to an embodiment of the present application;

[0016] Figure 6 4 is a flow chart of a GPU cluster communication method according to an embodiment of the present application. DETAILED DESCRIPTION

[0017] The embodiments of the present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0018] It should be noted that the terms "first", "second", etc. in the description and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0019] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a computer terminal as an example, Figure 1 : is a hardware structure block diagram of a computer terminal according to the GPU cluster communication method of the embodiment of the present application. Figure 1 As shown, the computer terminal may include one or more ( Figure 1 Only one is shown) a processor 102 (the processor 102 may include but is not limited to a microprocessor MCU or a programmable logic device FPGA and other processing devices) and a memory 104 for storing data. The computer terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above-mentioned computer terminal. For example, the computer terminal may also include Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0020] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the GPU cluster communication method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, implementing the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories may be connected to the computer terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0021] The transmission device 106 is used to receive or transmit data via a network. A specific example of the aforementioned network may include a wireless network provided by a communications provider of a computer terminal. In one embodiment, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0022] Currently, the difficulty of mixed training with heterogeneous GPU chips lies in the lack of unified interfaces and protocols in the communication libraries of different types of GPUs, which makes communication between heterogeneous GPUs difficult. When a training task requires the simultaneous use of different types of GPUs, communication between heterogeneous environments across different providers is impossible. Traditional solutions for communication between GPUs from different providers fall into the following two categories:

[0023] (1) Heterogeneous communication without CPU transfer: Through a communication architecture based on RDMA technology, efficient data transmission is achieved between GPUs from different providers. Although this solution can achieve data transmission between GPUs, it requires re-adaptation every time a new GPU chip from a new provider is introduced, resulting in poor scalability.

[0024] (2) Heterogeneous communication based on CPU transfer: This solution copies data from the source GPU to the CPU of the source node, uses inter-CPU communication to transfer the data to the CPU of the target GPU, and finally copies the data to the target GPU. This solution standardizes the relevant interfaces. The CPU plays a transfer and coordination role in the entire communication process. Although it effectively avoids the obstacles caused by differences in communication libraries between GPUs from different providers, it introduces additional CPU-GPU data copy overhead.

[0025] The above-mentioned communication scheme between GPUs from different providers not only increases the amount of data transmitted across GPUs, but also requires additional data copies between the CPU and GPU.

[0026] In view of this, this application embodiment proposes an Allreduce algorithm network architecture diagram for a heterogeneous GPU cluster. Figure 2 is a network architecture diagram of a GPU cluster communication method according to an embodiment of the present application. Figure 2 As shown in the figure, the network architecture diagram includes: training framework, PyTorch, patch package, communication backend module and hardware GPU.

[0027] Training framework: A version based on the deeply optimized Megatron-LM framework, used for building large model topology structures, distributed parallel strategy scheduling, and cross-device communication call triggering.

[0028] PyTorch layer: Based on the upgrade of PyTorch's native ProcessGroup architecture, it is used to implement the device perception matrix analysis engine, build a multi-dimensional communication path decision tree, and integrate a dynamic backend switching controller.

[0029] Patch package: Functional enhancements are achieved through Megatron-LM framework injection technology, and a non-intrusive patch mechanism is integrated into the PyTorch communication stack to deploy device heterogeneous recognition units and build a communication backend dynamic routing module.

[0030] Communication backend module: Contains multiple communication libraries, such as ACCL (A Collective Communications Library) and BCCL (B Collective Communications Library), which are collective communication libraries from different providers; XCCL (X Collective Communications Library), a heterogeneous communication protocol based on CPU cache transit (supporting data transmission between GPUs from different providers).

[0031] Hardware GPU: GPU hybrid deployment cluster.

[0032] Figure 3 is a flowchart of a GPU cluster communication method according to an embodiment of the present application. Figure 3 As shown, the GPU cluster communication process includes the following steps:

[0033] Step S302: Split the GPU cluster to obtain at least one algorithm homogeneous group and at least one algorithm heterogeneous group;

[0034] In this embodiment, since the communication backends corresponding to GPUs of different providers are different, a communication group is split into multiple subgroups, that is, different types of GPUs in the GPU cluster are split into at least one algorithm isomorphic group and at least one algorithm heterogeneous group.

[0035] In some embodiments, before splitting the GPU cluster to obtain at least one algorithm homogeneous group and at least one algorithm heterogeneous group, the GPU cluster communication method further includes: detecting data parameters of the GPU cluster and establishing a data analysis matrix based on the data parameters; and determining communication paths corresponding to the homogeneous GPUs and the heterogeneous GPUs respectively based on the data analysis matrix.

[0036] In this embodiment, before the GPU cluster is split to obtain at least one algorithm isomorphic group and at least one algorithm heterogeneous group, the upper-layer Megatron training framework needs to initiate a distributed communication request. When Megatron-LM executes the Allreduce heterogeneous communication algorithm, the underlying communication operation is triggered through the torch.distributed interface of the PyTorch layer.

[0037] Furthermore, the patch enhancement module of the PyTorch communication stack captures distributed communication requests and starts the communication routing scheduling mechanism:

[0038] A. The system detects the GPU data parameters of both communicating parties (such as GPU type, model, performance indicators, etc.), collects and organizes the data parameters, and constructs a data analysis matrix. This data analysis matrix can intuitively display the characteristics and status of each GPU in the GPU cluster, as well as the compatibility or differences between GPUs.

[0039] B. Based on the data analysis matrix constructed above, determine the communication paths for the homogeneous algorithm group (homogeneous GPUs) and the heterogeneous algorithm group (heterogeneous GPUs) according to the data analysis results. For homogeneous GPUs, direct routing is performed to standard collective communication libraries such as ACCL / BCCL. For heterogeneous GPUs, since GPU communication libraries from different providers cannot communicate directly, data exchange between GPUs is achieved by scheduling the XCCL communication protocol and using CPU cache relay.

[0040] In some embodiments, the GPU cluster is split to obtain at least one algorithm isomorphic group and at least one algorithm heterogeneous group, including: splitting the GPU cluster into at least one algorithm isomorphic group according to a preset splitting rule; and combining the homogeneous GPUs in each algorithm isomorphic group to obtain at least one algorithm heterogeneous group.

[0041] In the embodiment, the preset splitting rule can include at least one of the following splitting manners: splitting based on the greatest common divisor, splitting based on the number of GPUs in the server. If the preset splitting rule is splitting based on the number of GPUs in the server, different numbers and types of GPUs are configured on each server, and at least one algorithm homogeneous group and at least one algorithm heterogeneous group can be split according to the different numbers and types of GPUs; if the preset splitting rule is splitting based on the greatest common divisor, that is, splitting according to the greatest common divisor of the number of different GPU types in the GPU cluster, Figure 4 is a splitting schematic diagram of the GPU cluster according to the embodiment of the application, as shown in Figure 4 the same background is the GPU of the same provider, for example, splitting according to the greatest common divisor, obtaining four algorithm homogeneous groups and two algorithm heterogeneous groups, wherein the algorithm heterogeneous group is defined as follows:

[0042] algorithm homogeneous group: named algo_homo_pg, algorithm homogeneous group split based on the greatest common divisor, used for allgather communication and allreduce communication;

[0043] algorithm heterogeneous group: named algo_hetero_pg, algorithm heterogeneous group split based on the greatest common divisor, used for allgather communication and allreduce communication.

[0044] For example, the algorithm homogeneous group is the GPU with the same model or belonging to the same provider in the GPU cluster. Within the algorithm homogeneous group, the GPUs use the same communication backend for efficient communication. Since the communication library is closely matched with the hardware, the communication efficiency in the algorithm heterogeneous group is high, and the additional overhead caused by heterogeneous communication is avoided.

[0045] The algorithm heterogeneous group is a subset of the cluster containing different provider GPUs. Within the algorithm heterogeneous group, communication needs to cross different hardware platforms, which is more complex than communication in the homogeneous group. Since different GPU communication libraries are incompatible, in order to optimize heterogeneous communication, the algorithm homogeneous group communication is used, and XCCL is used in the algorithm heterogeneous group for communication to reduce CPU-GPU data copy overhead. XCCL, as a component in the heterogeneous GPU cluster, can realize data transmission between different provider GPUs, even if the GPUs use different communication backends (such as one GPU using ACCL and the other GPU using BCCL). XCCL can realize communication between heterogeneous GPUs through intelligent routing and CPU cache relay technology.

[0046] Figure 5 is a whole flowchart of the GPU cluster communication method according to the embodiment of the application, as shown in Figure 5As shown in Figure 2, the GPU cluster communication method can run on the above network architecture. When implementing the Allreduce primitive, the communication group (GPU cluster) is split into an algorithm homogeneous group and an algorithm heterogeneous group by the greatest common divisor. Figure 5 There are three types of GPU chips in total. Among them, the GPUs of the same provider are of the same background. There are 4 GPU chips of the first type, 2 GPU chips of the second type, and 2 GPU chips of the third type. The greatest common divisor of 4, 2, and 2 is 2, which is divided into four algorithm isomorphism groups, each of which contains two GPU chips.

[0047] Split according to the greatest common divisor, use corresponding supporting communication backends between homogeneous GPUs, and use heterogeneous communication between heterogeneous GPUs. This can not only achieve heterogeneous communication, but also maximize the reuse of the communication backends of homogeneous GPUs for communication, thereby improving overall communication efficiency.

[0048] In this embodiment, the homogeneous GPUs in each algorithm homogeneous group are combined to obtain at least one algorithm heterogeneous group.

[0049] For example, if the preset splitting rule is to split based on the greatest common divisor rule, after the GPU cluster is split into at least one algorithm isomorphic group, the i-th homogeneous GPU in each algorithm isomorphic group is combined to obtain at least one algorithm heterogeneous group, where i is an integer greater than 0, such as Figure 5 As shown, for example, if i=1, then the first homogeneous GPU in each algorithm isomorphic group is combined, and the first homogeneous GPU in the first algorithm isomorphic group (i.e., GPU0), the first homogeneous GPU in the second algorithm isomorphic group (i.e., GPU2), the first homogeneous GPU in the third algorithm isomorphic group (i.e., GPU4), and the first homogeneous GPU in the fourth algorithm isomorphic group (i.e., GPU6) are combined to obtain {GPU0, GPU2, GPU4, GPU6}; for example, if i=2, then the second homogeneous GPU in each algorithm isomorphic group is combined. The second homogeneous GPU in the first algorithm isomorphic group (i.e., GPU1), the second homogeneous GPU in the second algorithm isomorphic group (i.e., GPU3), the second homogeneous GPU in the third algorithm isomorphic group (i.e., GPU5), and the second homogeneous GPU in the fourth algorithm isomorphic group (i.e., GPU7) are combined to obtain {GPU1, GPU3, GPU5, GPU7}, that is, the two algorithm heterogeneous groups are: {GPU0, GPU2, GPU4, GPU6} and {GPU1, GPU3, GPU5, GPU7}.

[0050] In some embodiments, after combining the homogeneous GPUs in each algorithm isomorphic group to obtain at least one algorithm heterogeneous group, the GPU cluster communication method further includes: pairing adjacent heterogeneous GPUs in each algorithm heterogeneous group to obtain multiple algorithm heterogeneous subgroups; and performing full protocol communication based on the heterogeneous GPUs in each algorithm heterogeneous subgroup.

[0051] In this embodiment, for each algorithm heterogeneous group, the Ring algorithm can be used to pair adjacent heterogeneous GPUs in each algorithm heterogeneous group. The algorithm heterogeneous subgroups composed of the first algorithm heterogeneous group {GPU0, GPU2, GPU4, GPU6}, that is, the GPU pairs for heterogeneous communication are {GPU2, GPU4}, {GPU4, GPU6}, {GPU6, GPU0} respectively; the algorithm heterogeneous subgroups composed of the second algorithm heterogeneous group {GPU1, GPU3, GPU5, GPU7}, that is, the GPU pairs for heterogeneous communication are {GPU3, GPU5}, {GPU5, GPU7}, {GPU7, GPU1} respectively, and all-reduce communication is performed according to the heterogeneous GPUs in each algorithm heterogeneous subgroup.

[0052] Step S304: Execute the protocol distribution communication and the full gather communication through the homogeneous GPUs in each algorithm homogeneous group, and execute the full protocol communication through the heterogeneous GPUs in each algorithm heterogeneous group.

[0053] In this embodiment, the algorithm isomorphic group and the algorithm heterogeneous group are split in a certain way, for example, according to the greatest common divisor rule, to obtain at least one algorithm isomorphic group and at least one algorithm heterogeneous group, at least one algorithm isomorphic group performs reduce-scatter communication, at least one algorithm heterogeneous group performs all-reduce communication, and then at least one algorithm isomorphic group performs all-gather communication.

[0054] For example, Figure 5 As shown in Figure 1, completing the Allreduce of 8 chips in the communication group includes the following three steps:

[0055] Step 1: Four algorithm isomorphism groups execute ReduceScatter;

[0056] Step 2: Two algorithm heterogeneous groups execute Allreduce;

[0057] Step 3: The four algorithm isomorphism groups execute Allgather again.

[0058] Through the three steps above, the size of the algorithm heterogeneous group is optimized to the minimum, thereby reducing the scope of communication between GPU heterogeneous chips. Most data transmission is carried out through homogeneous GPUs (i.e., the same communication backend) to improve communication efficiency.

[0059] Furthermore, the specific communication process of completing the Allreduce of 8 chips in the communication group is as follows (for example, the data volume of a single GPU chip is There are three types of GPU chips. The same background shows GPUs from the same provider. There are four GPU chips of the first type, two GPU chips of the second type, and two GPU chips of the third type.

[0060] (1) In the initial stage, a total of 8 GPU chips of different types are prepared. According to the greatest common divisor principle (the greatest common divisor of 4, 2, and 2 is 2), they are divided into four algorithm isomorphism groups. Each algorithm isomorphism group contains two GPU chips. The four algorithm isomorphism groups execute homo-ReduceScatter, that is, all GPU chips with the same number in the same algorithm isomorphism group complete ReduceScatter communication through the same communication backend. Figure 5 As shown in Figure 1, the four algorithm isomorphic groups are: {GPU0, GPU1}, {GPU2, GPU3}, {GPU4, GPU5}, and {GPU6, GPU7}. The two GPUs in each algorithm isomorphic group perform ReduceScatter communication, and the amount of heterogeneous communication data is zero.

[0061] (2) Two algorithm heterogeneous groups execute hetero-Allreduce, that is, all GPU chips with different numbers in the same algorithm heterogeneous group complete Allreduce communication through the heterogeneous communication backend. Figure 5 As shown, the two algorithm heterogeneous groups are: {GPU0, GPU2, GPU4, GPU6}, {GPU1, GPU3, GPU5, GPU7}. For each algorithm heterogeneous group, Allreduce communication can obtain the reduction results of all 8 chips for the data block reduced in step (1). Through the Ring algorithm, the first group of GPU pairs that need heterogeneous communication are {GPU2, GPU4}, {GPU4, GPU6}, {GPU6, GPU0}; the second group of GPU pairs that need heterogeneous communication are {GPU3, GPU5}, {GPU5, GPU7}, {GPU7, GPU1}. At this time, the heterogeneous communication volume of GPU0, GPU1, GPU2, and GPU3 is The heterogeneous communication data volume of GPU4, GPU5, GPU6, and GPU7 is

[0062] (3) The four algorithm isomorphism groups perform homo-Allgather communication again, that is, all GPU chips with the same number in the same algorithm isomorphism group complete the Allgather communication through the same communication backend. Figure 5 As shown in Figure 1, the four algorithm isomorphic groups are {GPU0, GPU1}, {GPU2, GPU3}, {GPU4, GPU5}, and {GPU6, GPU7}. The two GPUs in each algorithm isomorphic group perform allgather communication, and the amount of heterogeneous communication data is zero.

[0063] Therefore, the total communication volume and heterogeneous communication volume of a single card that needs to communicate across providers are:

[0064] GPU0, GPU1, GPU2, GPU3: Total communication volume of a single card: Heterogeneous traffic:

[0065]

[0066] GPU4, GPU5, GPU6, GPU7: Total communication volume per card: Heterogeneous traffic:

[0067] Through the above embodiments, in the context of a heterogeneous GPU cluster, heterogeneous communication traffic is significantly reduced. This is achieved by splitting the algorithm homogeneous groups and the algorithm heterogeneous groups in a certain manner, such as by the greatest common divisor. While fully utilizing the efficient communication of homogeneous GPUs, the number of GPUs involved in heterogeneous GPU communication is reduced, and the communication traffic between heterogeneous cards is only one-half of the greatest common divisor of the original communication traffic, thereby reducing the difficulty and efficiency of heterogeneous communication.

[0068] In some embodiments, protocol distribution communication and full-collection communication are executed by the homogeneous GPUs in each algorithm isomorphic group, including: executing protocol distribution communication and full-collection communication by the homogeneous GPUs in each algorithm isomorphic group according to the flow control mechanism, network conditions and the communication path corresponding to the homogeneous GPUs.

[0069] In this embodiment, after the communication routing scheduling mechanism is started as described above, communication is performed based on the flow control mechanism, network conditions, and the communication paths corresponding to the homogeneous GPUs. The communication backend determined by the scheduling executes the AllReduce algorithm. Specifically, the homogeneous GPUs in each algorithm isomorphism group execute ReduceScatter communication and AllGather communication as follows:

[0070] Step 1: Data preparation and partitioning: The communication backend of the homogeneous GPU first prepares the data to be sent. This data is usually extracted from the local calculation results of each computing node. After proper formatting and processing, it will be divided into multiple data blocks as needed.

[0071] Step 2: Communication protocol selection and execution: The communication backend of the homogeneous GPU performs data communication according to the selected communication path (such as ACCL / BCCL, etc.).

[0072] Exemplarily, the communication backend of the homogeneous GPU performs ReduceScatter communication and Allgather communication through the homogeneous communication library.

[0073] Step 3: Data transmission and flow control: During data communication, the communication backend of the homogeneous GPU uses a flow control mechanism to prevent network congestion. Each computing node uses rate limiting to ensure that data can flow smoothly to the target node.

[0074] Step 4: Network Bandwidth and Latency Optimization: The communication backend of the homogeneous GPU can also optimize scheduling based on current network conditions (network bandwidth and network latency, etc.). Furthermore, the communication backend may use methods such as compression and packet consolidation to improve network bandwidth utilization, reduce communication latency, and further ensure smooth data transmission.

[0075] Step 5: Synchronization and consistency: After communication between homogeneous GPUs is completed, synchronization operations are required to ensure data consistency between all computing nodes. For example, mechanisms such as barriers are used to ensure that all computing nodes have completed data transmission before performing subsequent calculations.

[0076] In some embodiments, full protocol communication is performed by the heterogeneous GPUs in each algorithm heterogeneous group, including: performing full protocol communication on the heterogeneous GPUs in each algorithm heterogeneous group according to the flow control mechanism, network conditions and the communication path corresponding to the heterogeneous GPUs.

[0077] In this embodiment, after the communication routing scheduling mechanism is started as described above, communication is performed based on the flow control mechanism, network conditions, and the communication paths corresponding to the heterogeneous GPUs. The communication backend determined by the scheduling executes the Allreduce algorithm. Specifically, the Allreduce communication process for the heterogeneous GPUs in each algorithm heterogeneous group is as follows:

[0078] Step 6: Data preparation and partitioning: The communication backend of the heterogeneous GPU first prepares the data to be sent. This data is usually extracted from the local calculation results of each computing node. After proper formatting and processing, it will be divided into multiple data blocks as needed.

[0079] Step 7: Communication protocol selection and execution: The communication backend of the heterogeneous GPU performs data communication according to the selected communication path (e.g., XCCL).

[0080] Exemplarily, the communication backend of the heterogeneous GPU performs Allreduce communication through a heterogeneous communication library.

[0081] Step 8. Data transmission and flow control: During data communication, the communication backend of the heterogeneous GPU uses a flow control mechanism to prevent network congestion. Each computing node uses rate limiting to ensure that data can flow smoothly to the target node.

[0082] Step 9: Network Bandwidth and Latency Optimization: The heterogeneous GPU communication backend can also optimize scheduling based on current network conditions (network bandwidth and network latency, etc.). Furthermore, the communication backend may use methods such as compression and packet consolidation to improve network bandwidth utilization, reduce communication latency, and further ensure smooth data transmission.

[0083] Step 10: Synchronization and consistency: After communication between heterogeneous GPUs is completed, synchronization operations are required to ensure data consistency between all computing nodes. For example, mechanisms such as barriers are used to ensure that all computing nodes have completed data transmission before performing subsequent calculations.

[0084] In some embodiments, after executing protocol distribution communication and full collection communication through the homogeneous GPUs in each algorithm isomorphic group, and executing full protocol communication through the heterogeneous GPUs in each algorithm heterogeneous group, the GPU cluster communication method further includes: obtaining communication results, and determining whether the communication results meet preset conditions, wherein the preset conditions are that no data transmission anomalies occur during the data communication process; if the communication results do not meet the preset conditions, starting a data anomaly recovery mechanism and retransmitting the data; if the communication results meet the preset conditions, uploading the communication results to the system to continue executing the computing task.

[0085] In this embodiment, the homogeneous GPUs in the GPU cluster execute ReduceScatter communication and Allgather communication. After the heterogeneous GPUs execute Allreduce communication, they obtain the communication results for feedback to determine whether the communication results meet the expected results, including detection of data integrity, correctness, etc., that is, to determine whether the communication results meet the preset conditions. The preset conditions are that no data transmission anomalies occur during the data communication process. For example, data anomalies may include: data packet loss, data corruption, etc.

[0086] In this embodiment, after the homogeneous GPUs and heterogeneous GPUs in the GPU cluster perform a communication operation, the process of feeding back the communication results is as follows:

[0087] Step 11: Data Verification and Validation: The upper-layer system first verifies whether the data returned from the communication backend meets the expected results, including data integrity and correctness. If an anomaly occurs during the communication process (such as data packet loss or data corruption), the anomaly recovery mechanism is immediately activated to retransmit the data or resend the lost data packet.

[0088] Step 12: Result Delivery and Update: Once data verification is complete, the communication results are delivered to the upper-level system (e.g., Megatron) and used in subsequent computational tasks. The upper-level system updates the global state based on the received communication results and may trigger the next stage of computational tasks.

[0089] Step 13: Task Scheduling and Execution: The upper-level system uses the communication results as input to continue executing subsequent tasks. For example, in Megatron, the communication results may be used to continue the training steps of the deep learning model, such as gradient updates and parameter optimization.

[0090] Step 14: Synchronize and Continue Execution: In a distributed computing environment, task execution is typically synchronized. All computing nodes must proceed to the next computing phase after data communication is complete. Synchronization ensures consistent progress on each computing node, preventing computational discrepancies or delays between nodes.

[0091] Figure 6 FIG. 1 is a flow chart of a GPU cluster communication method according to an embodiment of the present application, such as Figure 6 As shown in the figure, when the configuration is TP (Tensor Parallelism) = 8, PP (Pipeline Parallelism) = 1, and DP (Data Parallelism) = 8, 8 DP communication groups are generated after distributed initialization. During training, collective communication operations (such as allreduce / allgather) are completed within the DP communication group. For example, each DP communication group contains 1 card with 6 frames and 2 cards with 2 frames, using the communication backends ACCL and BCCL respectively.

[0092] Specifically, in the initial stage: prepare 8 GPU chips in total, for example, {0, 8, 16, 24, 32, 40, 48, 56}, where the same background refers to GPUs from the same provider;

[0093] Step 1: Based on the greatest common divisor criterion, the algorithm is split into four isomorphic groups. Each group contains two GPU chips. The four groups execute ReduceScatter. The four groups are: {0, 8}, {16, 24}, {32, 40}, and {48, 56}. The two GPUs in each group perform ReduceScatter communication. At this time, the amount of heterogeneous communication data is 0.

[0094] Step 2: Two algorithm heterogeneous groups execute Allreduce, that is, all GPU chips with different numbers in the same algorithm heterogeneous group complete Allreduce communication through the heterogeneous communication backend. Figure 6 As shown in Figure 2, the two algorithm heterogeneous groups are: {0, 16, 32, 48} and {8, 24, 40, 56}.

[0095] Step 3: The four algorithm-isomorphic groups perform another Allgather communication. That is, all GPU chips with the same ID within the same algorithm-isomorphic group complete the Allgather communication through the same communication backend. The four algorithm-isomorphic groups are: {0, 8}, {16, 24}, {32, 40}, and {48, 56}. The two GPUs in each algorithm-isomorphic group perform Allgather communication, and the amount of heterogeneous communication data at this time is zero.

[0096] Through the above-mentioned embodiments of the present application, a method for communication of an image processing unit GPU cluster is provided, wherein the GPU cluster is split to obtain at least one algorithm isomorphic group and at least one algorithm heterogeneous group, and protocol distribution communication and full collection communication are performed by the homogeneous GPUs in each algorithm isomorphic group, and full protocol communication is performed by the heterogeneous GPUs in each algorithm heterogeneous group. That is, by splitting the GPU cluster, full utilization of efficient communication of homogeneous GPUs is achieved, and the frequency and amount of heterogeneous GPU communication are reduced, thereby reducing the difficulty of heterogeneous GPU communication and improving the efficiency and performance of large-scale distributed computing. Therefore, the embodiments of the present application can solve the problem of increased communication volume and introduction of additional data copy overhead between heterogeneous GPUs due to differences in communication libraries between different GPUs in a heterogeneous GPU cluster environment, thereby achieving the effect of improving the communication efficiency of the GPU cluster.

[0097] Scenario Example

[0098] 1. Gradient Aggregation in Distributed Training

[0099] In training large-scale deep learning models, distributed training is usually performed using multiple GPUs from different providers. Each device calculates the corresponding gradient. The Allreduce algorithm for heterogeneous GPU clusters in this embodiment of the application aggregates the gradients to update the global model parameters. The specific steps are as follows:

[0100] Step 15: Each GPU performs forward propagation and back propagation on the local data to calculate the corresponding local gradient;

[0101] Step 16: Use the Allreduce algorithm to aggregate the gradients of all GPUs. Each GPU sends the calculated gradients to other GPUs and receives gradients from other GPUs.

[0102] Step 17: After the Allreduce operation is completed, each GPU obtains the sum of the gradients of all nodes to calculate the global average gradient;

[0103] Step 18: Each GPU uses the global average gradient to update the local model parameters for the next round of training.

[0104] 2. Parameter Synchronization in Model Parallelism

[0105] In model parallel training, different layers of the model may be distributed on different GPUs. To maintain model consistency, synchronize parameters between GPUs from different providers. The specific steps are as follows:

[0106] Step 19: Each node calculates the forward and backward propagation of the local layer to obtain the corresponding gradient;

[0107] Step 20: Aggregate the gradients using the Allreduce algorithm for heterogeneous GPU clusters in this embodiment of the application to ensure that each node receives globally consistent parameter updates.

[0108] Step 21: After the Allreduce operation is completed, each node obtains the weight update information of all layers;

[0109] Step 22: Each node adjusts the weight of its local model based on the aggregated update information and proceeds to the next round of training.

[0110] 3. Loss Aggregation in Multi-Task Learning

[0111] In multi-task learning, the loss functions of multiple tasks may be distributed on different computing nodes. The losses of each task need to be aggregated for overall optimization. The specific steps are as follows:

[0112] Step 23: Each node calculates the loss value of the task it is responsible for;

[0113] Step 24: Aggregate the loss values ​​of each node using the Allreduce algorithm for the heterogeneous GPU cluster of this embodiment to calculate the global loss;

[0114] Step 25: Broadcast the global loss to all nodes for timely gradient updates and model adjustments.

[0115] Step 26: Based on the global loss, each node updates its parameters and proceeds to the next round of training.

[0116] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0117] It should be noted that the above modules can be implemented through software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0118] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. The computer program is configured to execute the steps of any of the above method embodiments when run.

[0119] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0120] An embodiment of the present application further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0121] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0122] According to yet another embodiment of the present disclosure, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the steps of the method described in each embodiment of the present disclosure are implemented.

[0123] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail here.

[0124] Obviously, those skilled in the art should understand that the modules or steps of the present application described above can be implemented using a general-purpose computing device, they can be concentrated on a single computing device, or distributed across a network composed of multiple computing devices, they can be implemented using program code executable by the computing device, and thus, they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be performed in a different order than herein, or they can be fabricated into separate integrated circuit modules, or multiple modules or steps can be fabricated into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.

[0125] The above description is merely a preferred embodiment of the present application and is not intended to limit the present application. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present application shall be included within the scope of protection of the present application.

Claims

1. A GPU cluster communication method, characterized in that: include: Splitting the GPU cluster to obtain at least one algorithm isomorphic group and at least one algorithm heterogeneous group; Reduce-distribute communication and full-gather communication are performed by the homogeneous GPUs in each algorithm homogeneous group, and full-reduce communication is performed by the heterogeneous GPUs in each algorithm heterogeneous group.

2. The method according to claim 1, characterized in that Splitting the GPU cluster to obtain at least one algorithm homogeneous group and at least one algorithm heterogeneous group includes: Splitting the GPU cluster into at least one algorithm isomorphic group according to a preset splitting rule; The homogeneous GPUs in each algorithm homogeneous group are combined to obtain at least one algorithm heterogeneous group.

3. The method according to claim 1, characterized in that in, The homogeneous GPUs communicate through a homogeneous communication library, and the heterogeneous GPUs communicate through a heterogeneous communication library.

4. The method according to claim 2, characterized in that After combining the homogeneous GPUs in each algorithm homogeneous group to obtain at least one algorithm heterogeneous group, the method further includes: Pair adjacent heterogeneous GPUs in each algorithm heterogeneous group to obtain multiple algorithm heterogeneous subgroups; Fully reduce communication is performed on the heterogeneous GPUs in each algorithm heterogeneous subgroup.

5. The method according to claim 1, wherein Before splitting the GPU cluster to obtain at least one algorithm homogeneous group and at least one algorithm heterogeneous group, the method further includes: Detecting data parameters of the GPU cluster and establishing a data analysis matrix according to the data parameters; According to the data analysis matrix, communication paths corresponding to the homogeneous GPU and the heterogeneous GPU are determined respectively.

6. The method according to claim 1, characterized in that The method of executing reduce-distribute communication and all-gather communication by the homogeneous GPUs in each algorithm homogeneous group includes: According to the flow control mechanism, network conditions and the communication paths corresponding to the homogeneous GPUs, the homogeneous GPUs in each algorithm homogeneous group will execute the protocol distribution communication and full collection communication.

7. The method according to claim 1, characterized in that The performing of full protocol communication by the heterogeneous GPUs in each algorithm heterogeneous group includes: According to the flow control mechanism, network conditions and the communication paths corresponding to the heterogeneous GPUs, the heterogeneous GPUs in each algorithm heterogeneous group will perform full protocol communication.

8. The method according to claim 1, characterized in that After performing the protocol distribution communication and the full gather communication by the homogeneous GPUs in each algorithm homogeneous group, and performing the full protocol communication by the heterogeneous GPUs in each algorithm heterogeneous group, the method further includes: Obtaining a communication result, and determining whether the communication result meets a preset condition, wherein the preset condition is that no data transmission anomaly occurs during the data communication process; If the communication result does not meet the preset conditions, start the data anomaly recovery mechanism and retransmit the data; When the communication result meets the preset conditions, the communication result is uploaded to the system to continue executing the computing task.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 8 are implemented.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images