Aggregate communication acceleration method and device of GPU, equipment, medium and program product

Abstract

The embodiment of the invention discloses a set communication acceleration method and device for GPUs, equipment, a medium and a program product. The method comprises the following steps: receiving input data sent by a source end GPU in a plurality of GPUs; performing expert routing calculation according to the input data, and determining a target expert for processing the input data in the hybrid expert model; the input data is sent to a GPU corresponding to the target expert, and the GPU corresponding to the target expert is used for processing the input data through the target expert to obtain an output result; obtaining an output result corresponding to the activated expert in the hybrid expert model; and carrying out specification on the output result corresponding to the activated expert in the hybrid expert model.

Description

Technical Field

[0001] This application belongs to the technical field of artificial intelligence, and relates to, but is not limited to, a method, apparatus, device, medium and program product for accelerating the collective communication of a graphics processing unit (GPU). Background Technology

[0002] Large-scale models are currently evolving from hundreds of billions to trillions of models, from single-modal to multi-modal, and from dense to sparse. Traditional 8-GPU servers suffer from interconnect limitations in training efficiency. As the scale of large models increases, the underlying GPU hardware is required to evolve towards large-scale interconnects, for example, the number of computing nodes in the scale-up domain is evolving towards hundreds of GPUs. In related technologies, ensemble communication operations and some computational tasks for Mixed Experts (MoE) models need to be executed through GPUs. Ensemble communication operations consume GPU resources, thereby reducing the efficiency of model training and inference using GPUs. Summary of the Invention

[0003] This application provides a method, apparatus, device, medium, and program product for accelerating GPU-based collective communication.

[0004] This application proposes a method for accelerating GPU collective communication, applied in a first device, wherein the first device forms a communication connection with multiple GPUs, and the method includes:

[0005] Receive input data sent by the source GPU among the multiple GPUs;

[0006] Based on the input data, perform expert routing calculations to determine the target expert in the MoE for processing the input data;

[0007] The input data is sent to the GPU corresponding to the target expert, and the GPU corresponding to the target expert is used to process the input data by the target expert to obtain the output result;

[0008] Obtain the output results corresponding to the activated experts in the hybrid expert model;

[0009] The output results corresponding to the activated experts in the hybrid expert model are reduced.

[0010] This application also proposes a GPU ensemble communication acceleration device, located in a first device, which forms a communication connection with multiple GPUs. The device includes:

[0011] A receiving module is used to receive input data sent by the source GPU among the multiple GPUs;

[0012] The first processing module is used to perform expert routing calculation based on the input data to determine the target expert in the hybrid expert model for processing the input data.

[0013] The second processing module is used to send the input data to the GPU corresponding to the target expert, and the GPU corresponding to the target expert is used to process the input data through the target expert to obtain the output result; obtain the output result corresponding to the activated expert in the hybrid expert model; and reduce the output result corresponding to the activated expert in the hybrid expert model.

[0014] This application also proposes an electronic device in which multiple GPUs form a communication connection, the electronic device comprising:

[0015] A communication device used to communicate with the GPU;

[0016] Memory, used to store data to be calculated and computer programs;

[0017] A processor for running the computer program to execute any of the aforementioned GPU-based aggregate communication acceleration methods.

[0018] This application also provides a computer storage medium storing a computer program that, when executed by a processor, implements any of the above-described GPU collective communication acceleration methods.

[0019] This application also provides a computer program product, including a computer program that, when executed by a processor, implements any of the above-described GPU aggregate communication acceleration methods.

[0020] As can be seen, in the GPU's collective communication process, the embodiments of this application can execute the expert routing calculation process through the first device. That is, the first device replaces the GPU in executing the expert routing calculation process, which can increase the efficiency of the GPU in executing other calculation operations in the collective communication operation, thereby improving the efficiency of model training and inference through the GPU to a certain extent. Attached Figure Description

[0021] Figure 1 A flowchart of the All Reduce operation provided for related technologies;

[0022] Figure 2 Another flowchart for the All Reduce operation provided for related technologies;

[0023] Figure 3This diagram illustrates the proportion of All-Reduce operation time for different throughput sizes in offline computing scenarios related to the relevant technologies.

[0024] Figure 4 This diagram illustrates the proportion of All-Reduce operation time for different numbers of GPUs in offline computing scenarios related to the relevant technologies.

[0025] Figure 5 A schematic diagram of a network architecture employing on-network computing technology, provided for related technologies;

[0026] Figure 6 A flowchart illustrating a GPU-based method for accelerating aggregated communication in relation to related technologies;

[0027] Figure 7 This is a schematic diagram of the hardware structure of the first device provided in an embodiment of this application;

[0028] Figure 8 This is a flowchart of a GPU collection communication acceleration method according to an embodiment of this application;

[0029] Figure 9 This is a flowchart illustrating the expert routing calculation using the first device in an embodiment of this application.

[0030] Figure 10 This is a schematic diagram of the structure of the data packet sent via the GPU in an embodiment of this application;

[0031] Figure 11 An architecture diagram for implementing the specification operation provided in the embodiments of this application;

[0032] Figure 12 The interactive flowchart for implementing the specification operation is provided in the embodiments of this application;

[0033] Figure 13 A flowchart illustrating the aggregated communication based on GPU card interconnection provided in this application embodiment;

[0034] Figure 14 Another architecture diagram for implementing the specification operation provided in the embodiments of this application;

[0035] Figure 15 A schematic diagram of the core process of sparse sensing reduction provided in the embodiments of this application;

[0036] Figure 16 A flowchart illustrating the execution of protocol operations based on a first device, provided as an embodiment of this application;

[0037] Figure 17 Another flowchart illustrating the execution of protocol operations based on a first device, provided in an embodiment of this application;

[0038] Figure 18 This is a schematic diagram of the structure of the GPU-based collective communication acceleration device according to an embodiment of this application;

[0039] Figure 19 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0040] Currently, computing nodes in the Scale Up domain are evolving towards a scale of hundreds of GPUs. However, the distributed training strategies in related technologies are insufficient to meet the communication resource requirements of gradient aggregation during model training within Scale Up servers. The resources within the GPU computing cores are still occupied by collective communication tasks, which cannot effectively reduce the overhead of the GPU in performing All Reduce operations.

[0041] All Reduce operations are many-to-many communication primitives, requiring multiple data senders and receivers to execute the operation. For example, the same reduce operation can be performed on all nodes in a cluster, and the results of the reduce operation on all nodes can be sent to all nodes. All Reduce operations can be widely applied to parallel strategies such as data parallelism and model parallelism. For instance, when applying All Reduce operations to data parallelism strategies, they can be based on various communication topologies, such as ring AllReduce or tree AllReduce.

[0042] Reference Figure 1 GPU 0, GPU 1, GPU 2, and GPU 3 represent different GPUs, and A, B, C, and D represent the data in GPU 0, GPU 1, GPU 2, and GPU 3, respectively. All Reduce operations can be implemented through Reduce and Broadcast operations. First, a Reduce operation is performed on the master node GPU 0 to obtain the reduced sum (e.g., 26). Then, the reduced sum is broadcast to other nodes, ensuring that the value on each node remains consistent throughout the cluster. (See reference...) Figure 2A0, A1, A2, and A3 represent data split from A; B0, B1, B2, and B3 represent data split from B; C0, C1, C2, and C3 represent data split from C; and D0, D1, D2, and D3 represent data split from D. All Reduce operations can also be implemented through Reduce Scatter and All Gather operations. In related technologies, a software-implemented All Reduce scheme can be Ring AllReduce, which first executes Reduce Scatter operations and then All Gather operations. Each node in the cluster (e.g., GPU) only exchanges data with its neighboring nodes, without centralized transmission to a central node. Ring AllReduce can achieve even distribution of bandwidth and computation among each GPU, resulting in load balancing and strong scalability.

[0043] In related technologies, the software-implemented All Reduce scheme weakens the optimization capability of the overlap between computation and communication. This scheme has at least the following two drawbacks: 1) It requires GPU computing resources for communication, which will affect gradient calculation to some extent; 2) The transmission of gradients will affect the state of the memory system, further affecting gradient calculation.

[0044] By conducting experiments on the transformer model on the V100 DGX-2 system (containing 16 GPUs and 12 switches), the proportion of time spent on gradient all-reduce operations in the training time can be statistically determined. Figure 3 This diagram illustrates the proportion of All-Reduce operation time for different throughput sizes in offline computing scenarios related to the relevant technologies; see reference. Figure 3 The horizontal axis represents the token size per GPU, i.e., the sub-batch size; the vertical axis represents the proportion of training time allocated to the All-Reduce operation. Volta1:1, Volta2:1, Volta4:1, Voltaeth10gb, and Voltaeth100gb represent different throughput sizes.

[0045] Figure 4 This diagram illustrates the time proportions of All-Reduce operations for different numbers of GPUs in offline computing scenarios related to the relevant technologies; see reference. Figure 4The horizontal axis represents the number of GPUs, and the vertical axis represents the proportion of training time allocated to the All-Reduce operation. 345M, 2500M, 8300M, 1200M, and 4200M represent different numbers of model parameters. Figure 4 As can be seen, in the V100 DGX-2 system, the time spent on All-Reduce operations can account for up to 30% of the training time. In future GPU clusters, this proportion will increase significantly. In horizontally scaled (Sacale-out) systems, the proportion of training time spent on All-Reduce operations will be extremely high.

[0046] From an intuitive perspective, for parallel computing models, the gradients on each GPU are reduced proportionally (because the weights become smaller), and the proportion of time spent on all-reduce operations in the training time should decrease. However, in real-world scenarios, the proportion of time spent on all-reduce operations in the training time is actually larger (up to 30%). This is because model parallelism reduces GPU computation time (reducing the number of floating-point operations per second (FLOPS), making training more sensitive to bandwidth when model parallelism is used.

[0047] All Reduce is a common communication operation in parallel and distributed computing used to aggregate data across multiple computing nodes. It is frequently used in distributed training processes in deep learning to integrate gradient information from various nodes for parameter updates.

[0048] In related technologies, optimizing distributed computing through a design approach that tightly couples the various communication stages of networks and distributed applications is an important development direction. Novel aggregated communication acceleration can break through existing computing models, reconstruct application processing logic, and bring about a qualitative leap in system computing efficiency. Among related technologies, aggregated communication acceleration technology is an in-network computing technology. Currently, the most known in-network computing technology is the Infiniband network, proposed for a new generation of distributed parallel computing architectures. This in-network computing technology is a communication acceleration technology developed using a collaborative design concept. It utilizes network devices such as network interface cards (NICs) and switches to perform online computation on data during data transmission, aiming to reduce communication latency and improve overall computing efficiency, becoming a computing unit as important as GPUs and Central Processing Units (CPUs). For example, SHArP technology has begun to be introduced in some switches, integrating a computing engine unit that can support various computations and operations, significantly reducing aggregated communication latency, reducing network congestion, and improving the scalability of cluster systems. (Reference) Figure 5 A network architecture employing on-network computing technology can include multiple servers and switches. Server 1, Server 2, Server 3, and Server 4 represent different servers, which can communicate with each other through switches. Computing engine units can be integrated into the switches.

[0049] The on-network computing methods based on message semantics provided by related technologies can only support data parallelism (DP) strategies in distributed training models. They are mainly used in data center external networks and cannot meet the requirements of low latency and memory consistency for on-premises scale-up GPU card interconnection, nor can they support expert parallelism (EP) strategies.

[0050] In related technologies, when using dynamic tree allocation, Quasi-Fat Tree (QFT) guidance logic, and reservation mode, the resource limitations of the GPU may be ignored, leading to unreasonable resource allocation and unbalanced load, which in turn affects the stability and performance of the switch system.

[0051] In related technologies, the expert routing calculation process in MoE is typically completed by individual GPUs. (See reference...) Figure 6 Device1 to Device n There can be n different GPUs. Device1 can process the first input data and obtain the first output data. n The nth input data can be processed to obtain the nth output data, where n is an integer greater than 1. Device1 to Device nEach GPU in the system can process the input data through a multi-head attention layer, a gate network, and an encoder layer, from Device1 to Device... n The individual GPUs within the system can interact with each other to perform an All-to-All Dispatch operation on the output of their respective coding layers. Here, All-to-All is a common communication mechanism in the collection communication library used for data exchange between multiple devices. The All-to-All operation allows each participating device to distribute its local data to other devices while simultaneously receiving data from other devices. Device1 to Device n Each GPU in the array can be selected from Expert1 to Expert based on the result of the All to All Dispatch operation. n Select the expert who processes the data. Figure 6 In the middle, Expert1 to Expert n These represent different experts, each representing a model for processing the data. After processing the data using these experts, Device1 to Device... n The individual GPUs within the system can interact with each other to perform an all-to-all combine operation on the data processing results from various experts. Within each GPU, the result of the all-to-all combine operation can be processed through a decoding layer, and the corresponding GPU's output data can be obtained from the output of the decoding layer.

[0052] The on-network computing solutions provided by related technologies suffer from poor dynamic load adaptation and cannot adjust the aggregation path according to the real-time data volume. Specifically, traditional All Reduce on-network computing technologies and SHArP technology require full synchronization of all parameters; however, during MoE training, only activated expert gradients need to be synchronized, which places higher demands on the flexibility of its aggregate communication. This manifests as difficulty in adapting dynamic topologies, irregular communication patterns, and unbalanced loads, making traditional All Reduce solutions unsuitable for practical needs. To address this issue, related technologies have not proposed solutions adapted to the actual needs of large sparse models like MoE.

[0053] In view of the technical problems existing in related technologies, this application proposes technical solutions based on its embodiments.

[0054] The embodiments of this application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the embodiments provided herein are merely illustrative of the embodiments of this application and are not intended to limit the embodiments of this application. Furthermore, the embodiments provided below are some embodiments for implementing this application, and not all embodiments for implementing this application. Unless otherwise specified, the technical solutions described in the embodiments of this application can be implemented in any combination.

[0055] It should be noted that, in the embodiments of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a method or apparatus that includes a list of elements includes not only the elements expressly described, but also other elements not expressly listed, or elements inherent to implementing the method or apparatus. Without further limitations, an element defined by the phrase "comprising a..." does not exclude the presence of other related elements (e.g., steps in the method or units in the apparatus, such as portions of circuitry, processors, programs, or software, etc.) in the method or apparatus that includes that element.

[0056] The GPU ensemble communication acceleration method provided in this application includes a series of steps. However, the GPU ensemble communication acceleration method provided in this application is not limited to the steps described. Similarly, the GPU ensemble communication acceleration device provided in this application includes a series of modules. However, the device provided in this application is not limited to the modules explicitly described, but may also include modules that need to be set up for obtaining relevant information or processing based on information.

[0057] This application provides a method for accelerating GPU ensemble communication. This method is applied to a first device, which can establish communication connections with multiple GPUs. The first device can be a switch or other device with communication and computing capabilities. This application can offload GPU computing resources to the first device to improve GPU computing power, reduce the overhead of ensemble communication, and improve the efficiency of large model training and inference. Exemplarily, the first device is mainly used in a 100-GPU supernode interconnect system, including but not limited to various general architectures such as single-machine eight-GPU switching topologies, tree topologies, and star topologies. It should be noted that the architecture used by the first device is not bound to the specific topologies described above.

[0058] Reference Figure 7 The first device may include a forwarding unit 701, a computing unit 702 and a storage unit 703. The first device supports multiple functions such as switching forwarding and protocol calculation (e.g., addition, subtraction, multiplication, division, MAX, MIN and other protocol calculations).

[0059] The forwarding unit 701 is used to implement the switching and forwarding functions of a general-purpose switch or switching chip; the storage unit 703 is used to store the data to be calculated, the calculation results, and the computer program; the calculation unit 702 is used to implement functions such as data calculation and running computer programs. For example, the calculation unit 702 may include at least one of the following: a first arithmetic unit, a second arithmetic unit, and a third arithmetic unit; the first arithmetic unit is used to implement tensor calculation, the second arithmetic unit is used to implement at least one of logical operation, address management, and data sorting, and the third arithmetic unit is used to implement floating-point calculation.

[0060] For example, the first computational unit can be a Tensor Core. Tensor Cores are used to implement tensor-level data aggregation and matrix acceleration. Using Tensor Cores can significantly improve the computational throughput of ALL Reduce operations and route calculations in MoE. During route weight calculation, Tensor Cores can accelerate matrix multiplication and quickly generate expert probability distributions, enabling the first device to also have tensor computation capabilities.

[0061] The second processing unit can be an Arithmetic Logic Unit (ALU). The ALU is responsible for low-latency logic operations, address management, and data sorting algorithms, thereby ensuring transmission efficiency and real-time routing. The ALU can handle the mapping and conversion of GPU memory addresses, ensuring the ordered storage of reduction results (such as calculating the offset of consecutive memory addresses). In the Top-K expert selection process, the ALU can implement parallel algorithms such as radix sort to quickly select the K experts with the highest probability; at the same time, it handles the routing table indexing logic and dynamically adjusts the expert allocation.

[0062] The third arithmetic unit can be a floating-point unit (FPU). The FPU focuses on high-precision floating-point calculations, ensuring reduction accuracy and hardware acceleration for complex functions such as Softmax. In calculations of functions like Softmax and Sigmoid, the FPU performs normalization operations on expert probability distributions. For example, the FPU can perform exponentiation, summation, and division operations. Here, exponentiation can be the calculation of e... W(x) W(x) represents the product of input data x and routing weight H, and the division operation can be used to calculate Softmax(W(x)). The FPU supports low-latency floating-point operations, ensuring real-time routing decisions. In the addition, dot product, MAX reduction, and MIN reduction operations of ALLReduce, the FPU can perform high-precision floating-point accumulation, avoiding precision loss. The FPU works in conjunction with Tensor Cores to handle parameter updates in MOE (such as floating-point weight aggregation).

[0063] Figure 8 This is a flowchart of a GPU-based aggregate communication acceleration method according to an embodiment of this application, as follows: Figure 8 As shown, the process includes:

[0064] Step 801: Receive input data sent by the source GPU among multiple GPUs.

[0065] Here, the source GPU can be any one of multiple GPUs. The source GPU can offload the expert routing calculation function to the first device. In this case, the source GPU can send an expert routing calculation request carrying input data to the first device.

[0066] In this embodiment of the application, the input data sent by the source GPU can be one input data or multiple input data. For each input data sent by the source GPU, the subsequent steps of the set communication acceleration method can be executed, that is, steps 802 to 805 can be executed.

[0067] Step 802: Perform expert routing calculation based on the input data to determine the target expert in the hybrid expert model used to process the input data.

[0068] In this embodiment, an Expert Route Unit (ERU) can be configured in the first device to perform expert route calculations. Exemplarily, the ERU may include a combination of at least two of a tensor core, an ALU, and an FPU. (Refer to...) Figure 9 During the inference or training phase of MoE, the data processing of the source GPU can include the data processing flow of the attention layer and layer normalization processing. After layer normalization processing, the input data of the MoE layer is obtained. The MoE layer can perform expert routing calculation on the input data. In related technologies, the expert routing calculation function of the MoE layer is usually implemented by the GPU through software algorithms. In the embodiments of this application, the expert routing calculation function of the MoE layer can be implemented by ERU. ERU can determine the target expert in MoE used to process the input data, that is, determine which expert in the model loaded by the GPU is used for calculation.

[0069] For example, ERU can employ expert routing algorithms, such as a dedicated Softmax kernel or a Sigmoid kernel, or a hardware Softmax algorithm implemented through segmented exponent calculation, summation, and normalization. In ERU, the product W(x) of the input data x and the routing weight H can be calculated. Based on W(x), the probability distribution G(x) of each expert is obtained. This probability distribution determines which experts are needed for the computational task, which refers to the task of processing the input data. The probability distribution G(x) can be calculated using formula (1) or (2).

[0070] G(x)=softmax(W(x)) (1)

[0071] G(x)=sigmoid(W(x)) (2)

[0072] In ERU, a heap-based parallel sorting algorithm (such as radix sort) can directly replace the Top-K algorithm (which selects the K largest or smallest elements from a set of data) to sort and select the experts with the highest probability. The final normalization operation in the expert routing algorithm can be implemented using an addition tree and division unit in hardware to ensure that the sum of the weights is 1.

[0073] When multiple experts process data, if they share some parameters, these parameters can be stored in a shared memory unit. Hardware-supported dynamic memory management can then handle variable batch sizes, avoiding frequent memory copies. The cache space in the GPU used for reduction calculations (storing relevant data) is arranged based on the memory nodes of the GPU edge nodes and the task groups issued by the application layer. This ensures the ordered arrangement of data within the cache space.

[0074] Step 803: Send the input data to the GPU corresponding to the target expert. The GPU corresponding to the target expert is used to process the input data by the target expert and obtain the output result.

[0075] In this embodiment, experts can be pre-assigned to each GPU. Different experts are assigned to different GPUs; a GPU can be assigned one expert or multiple experts. After assigning experts to each GPU, the address of each expert on the GPU can be determined. The GPU can send the expert assignment results and expert addresses to the first device, thus allowing the first device to know the expert assignment results sent by each GPU and the address of each expert. After determining the target expert, the first device can send input data to the target expert according to the target expert's address. For example, in the GPU, the target expert can use the GPU kernel to perform calculations on the input data to obtain the output result.

[0076] Step 804: Obtain the output results corresponding to the activated experts in the hybrid expert model.

[0077] Understandably, a hybrid expert model can include multiple experts. When the source GPU sends the input data, the experts processing the data in the hybrid expert model can be some of the experts. The experts processing the data are the activated experts, while the experts not processing the data are the inactive experts. In real-world scenarios, the parameters of inactive experts do not need to be synchronized.

[0078] Step 805: Reduce the output of the activated experts in the hybrid expert model.

[0079] For example, the first device can perform reduction operations such as addition, dot product, MAX, MIN, etc. on the output results corresponding to the activated expert.

[0080] In practical applications, steps 801 to 805 can be implemented based on a processor, which can be at least one of the following: Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), CPU, controller, microcontroller, and microprocessor.

[0081] As can be seen, in the GPU's collective communication process, the embodiments of this application can execute the expert routing calculation process through the first device. That is, the first device replaces the GPU in executing the expert routing calculation process, which can increase the efficiency of the GPU in executing other calculation operations in the collective communication operation, thereby improving the efficiency of model training and inference through the GPU to a certain extent.

[0082] Furthermore, it is only necessary to use the first device to reduce the output results corresponding to the activated experts, without synchronizing the data of the inactive experts in each GPU to each GPU. Since there is no need to synchronize the parameters of the inactive experts in real-world scenarios, this embodiment can avoid invalid data transmission to a certain extent, thereby improving the bandwidth utilization and communication efficiency of the GPU.

[0083] In this embodiment, after obtaining the output results corresponding to the activated experts in the hybrid expert model, the first device can perform a first reduction operation on the output results corresponding to the activated experts in the hybrid expert model to obtain a first reduction result; then, the first reduction result can be sent to the source GPU. It can be seen that when the source GPU sends input data to the first device, the source GPU can obtain the corresponding reduction result. Therefore, when each of the multiple GPUs acts as the source GPU, each GPU can obtain the corresponding reduction result, thus satisfying the requirement for each GPU to obtain the reduction result in the GPU aggregation communication process.

[0084] In this application embodiment, there are multiple ways for the first device to obtain the output results corresponding to the activated experts in the hybrid expert model. In some embodiments, the first device can receive the output results corresponding to each expert in the hybrid expert model. The output result corresponding to the first expert in the hybrid expert model includes an indicator field, which indicates whether the first expert is an activated expert. The first expert can be any expert in the hybrid expert model. After receiving the output results corresponding to each expert, the first device can obtain the output result corresponding to the activated expert from the output results corresponding to each expert based on the indicator field in the output results corresponding to each expert.

[0085] For example, the indicator field can be a sparse metadata field (Sparse_Sens); in practical applications, a sparse metadata field can be added to the header of the data packet, using the identity document (ID) of the GPU corresponding to the activated expert as the identifier in the data packet. Figure 10 The data packets sent by the GPU to the first device may include a member group number, source address, destination address, data type, operation type, model type, Sparse_Sens, and payload. Here, the member group number indicates the member group to which the data to be reduced in the first device belongs; the member group number can be an ID or other identifier. The first device can perform reduction operations on the data corresponding to the same member group number in each received data packet. The operation type indicates the type of reduction operation performed by the first device. For example, operation types may include Reduce, Broadcast, Scatter, Reduce Scatter, All Gather, Gather, and All to All. The model type indicates the model type used to process the data in the GPU, and the payload carries the data to be reduced.

[0086] The output of each expert can be sent via data packets. When the sparse metadata field in the data packet is 1, it indicates that the corresponding output is from an activated expert. When the sparse metadata field in the data packet is not 1 (e.g., 0), it indicates that the corresponding output is from an inactive expert. By adding a sparse metadata field to the header of the data packet, the first device can route and reduce only the data of these activated experts. The first device can quickly parse the corresponding header and fill the expert routing table (sparse-aware routing table) with the data whose sparse metadata field is set to 1. Based on the expert routing table, only reduction operations (such as SUM, MAX, MIN, etc.) on the sparse data can be performed.

[0087] For example, the message format of the data packet can be a custom Collective Communication Accelerate (CCA) message or other types of messages. The CCA message can be the data message required by the ALL Reduce computation process. This application embodiment does not limit the message format of the data packet. In practical applications, after the first device performs the first reduction operation on the output result corresponding to the activated expert, it can send the first reduction result to the source GPU through a message. For example, the first reduction result can be sent to the source GPU according to the address and the field representing the ID of the source GPU in the message.

[0088] Reference Figure 11 GPU 1 and GPU 2 represent different GPUs, Token 1 and Token 2 are tokens corresponding to different GPUs, and Expert k, Expert m, Expert n and Expert j represent different experts in the hybrid expert model. The GPU can send a message carrying the output results corresponding to the experts to the first device. After receiving the messages sent by each GPU, the first device can determine the member group to perform the reduction operation based on the dynamic expert parallel member group ID (Group EPID). For example, the Group EPID corresponding to data Data1 and data Data3 is 1, and the Group EPID corresponding to data Data x and data Data m is 2. At this time, the reduction operation can be performed on Data1 and Data2, and the reduction operation can be performed on Data x and Data m. The result of the reduction operation can be returned to the source GPU. For example, the source GPU can be GPU0. Figure 11 In this context, EPID is the expert calculation table pre-allocated to the first device. The expert calculation table is a dedicated space for subsequent execution of the first protocol operation.

[0089] Reference Figure 12In this process, the source GPU is GPU k, which serves as the token source. GPUs m and n can be GPUs containing activated experts. When an All-to-All Dispatch operation is initiated via GPU k, GPUs m, n, and others can send a reduction data read request to the first device. This request can carry a data packet including sparse metadata fields. The first device can store the reduction data read request, identify the data packet corresponding to the activated expert based on the sparse metadata fields, allocate an address to the activated expert, and reply to the GPU corresponding to the activated expert (i.e., send reply information to the GPU corresponding to the activated expert). This reply information can carry the address allocated to the activated expert. GPU m or n can read the corresponding data based on the address in the reply information and send a data packet carrying the read data and the address of the token source to the first device, thereby enabling the first device to load data from the GPU. The first device can store data from the GPUs corresponding to the activated experts. After waiting for data collection in each member group to complete, it performs reduction calculations on the data in each member group and sends the result of the reduction calculation as a reply to GPU k. GPU k can store the received data.

[0090] In some other embodiments, without performing expert routing calculations through the first device, the first device may also receive the output results corresponding to each expert in the hybrid expert model, where the output result corresponding to the first expert in the hybrid expert model includes an indicator field. For example, when performing expert routing calculations through a GPU, the first device may also determine whether the first double-click is an activated expert based on the output result corresponding to the first expert; that is, the method by which the first device identifies activated experts is decoupled from the expert routing calculation process.

[0091] As can be seen, the first device can accurately obtain the output result corresponding to the activated expert based on the indicator field in the output result of each expert. In this case, it is only necessary to use the first device to reduce the output result corresponding to the activated expert, without synchronizing the data of the inactive experts in each GPU to each GPU. Since there is no need to synchronize the parameters of the inactive experts in the actual scenario, the embodiments of this application can avoid invalid data transmission to a certain extent, thereby improving the bandwidth utilization and communication efficiency of the GPU.

[0092] In some embodiments, regarding the implementation of the first device acquiring the output results corresponding to the activated experts in the hybrid expert model, the process of the first device sending input data to the GPU corresponding to the target expert may include: sending input data with an added data payload to the GPU corresponding to the target expert, the data payload being used to enable the GPU to determine the activated expert. Correspondingly, the first device may receive data packets sent by the GPU corresponding to the activated expert, the data packets including at least the output results corresponding to the activated expert.

[0093] In practical applications, when the first device performs expert routing calculations, it can send input data with added data payload to the GPU corresponding to the target expert, who is an activated expert. The GPU corresponding to the activated expert can proactively send a data packet without an indicator field to the first device, which can carry the output result corresponding to the activated expert; the GPU will not send data from inactive experts to the first device. The first device can re-aggregate the output results (e.g., Result 1, ..., Result n) corresponding to multiple activated experts into the corresponding token data, then perform sparse data reduction operations (such as SUM, MAX, etc.), and then distribute the results of the reduction operations to the source GPU.

[0094] As can be seen, the GPU corresponding to the activated expert can actively send the output result of the activated expert to the first device. In this way, the first device can reliably obtain the output result of the activated expert. In this case, it is only necessary to use the first device to reduce the output result of the activated expert, without synchronizing the data of the inactive experts in each GPU to each GPU. Since there is no need to synchronize the parameters of the inactive experts in the actual scenario, the embodiments of this application can avoid invalid data transmission to a certain extent, thereby improving the bandwidth utilization and communication efficiency of the GPU.

[0095] In some embodiments, the method by which the first device determines the target expert can be implemented by performing expert routing calculations based on the input data described above, determining the expert allocation result, and then determining the target expert in the hybrid expert model for processing the input data based on the expert allocation result. Accordingly, the data load described above includes the expert allocation result. For example, the expert allocation result may include the index k of the target expert and its routing weight.

[0096] It can be seen that when the data load includes the expert assignment results, by sending the input data of the data load to the GPU corresponding to the target expert, the GPU can reliably determine the activated expert.

[0097] For example, the data packet sent by the GPU corresponding to the activated expert may also include a data payload. Thus, the first device can determine the expert allocation result based on the data payload, and then, based on the expert allocation result, perform a first reduction operation on the output result corresponding to the same activated expert to obtain a first reduction result.

[0098] In this embodiment, each activated expert can process multiple input data to obtain multiple output results. In this way, the first device can perform a first reduction operation on the output results corresponding to the same activated expert to obtain the reduction result corresponding to the same activated expert, which helps to process the reduction result corresponding to the same activated expert in the future.

[0099] The GPU collection communication process of this application embodiment will be described exemplarily below with reference to the accompanying drawings.

[0100] Reference Figure 13 The aggregated communication process based on GPU card interconnection can include:

[0101] Step A1: Routing layer segmentation.

[0102] In this step, before the parallel computing task begins, the routing layer is partitioned, and the expert routing computation function is implemented by replacing the routing layer in the GPU with a first device. That is, the routing layer in the MoE layer is offloaded to the first device. In practical applications, different experts can be assigned to different GPUs, and one GPU can be assigned multiple experts. For example, each GPU can be divided into task groups and expert category numbers, and send the number of expert categories in the model and the number of experts required by the GPU to the first device. In this way, the first device can reasonably allocate experts to the GPU.

[0103] Step A2: Initialize the first device.

[0104] In this step, address mapping can be performed between each GPU and the first device, ensuring that each GPU maps to the same address space as the first device. When a parallel computing task is initiated, task groups are created, and the source GPU can send input data and the destination address (the address of the first device) to the first device. Before the GPU sends input data to the first device, the first device can pre-configure an expert routing table, which is an additional space allocated specifically for subsequent expert routing calculations. For example, the source GPU can send input data to the first device by sending a request message.

[0105] Step A3: Expert route calculation.

[0106] In this step, the first device can perform expert routing calculations on the input data from the GPU to obtain expert assignment results. Based on the expert assignment results, a target expert can be selected to process the input data. It should be noted that the default shared expert in the hybrid expert model is not selected. The default shared expert can be a shared expert that is traversed by different partitioning tasks across different GPUs; that is, the target expert is not the default shared expert.

[0107] Step A4: Expert assignment.

[0108] In this step, the first device can send the expert assignment result to the destination address of the target expert. For example, the first device can group tokens destined for the same expert based on the index and routing weight in the expert assignment result. The first device can send the input data, the expert assignment result, and the address number of the target expert to the destination address of the selected target expert.

[0109] Step A5: GPU computation.

[0110] In this step, the GPU can read the data from the target address in the target expert and obtain the corresponding output result through the corresponding expert's calculation. For example, if the target expert is Expert3 and the input data that the target expert needs to process is Data1, the input data Data1 will be processed by the corresponding expert Expert3 to obtain the corresponding output tensor result Result1.

[0111] Reference Figure 14 GPU 0 is the source GPU. GPU 0 can send input data, task group number, and the numbers of the experts it deploys to the first device. In the first device, the expert allocation result can be calculated using ERU. For example, the expert allocation result includes the following three target experts: Expert 0, Expert 3, and Expert m. The first device can send input data to Expert 3, and Expert 3 can calculate the corresponding output tensor result Result 1.

[0112] A6: Parameter specification synchronization.

[0113] In this step, the GPU can send the expert's output to the first device, where it performs reduction aggregation and updates all parameters using expert sparse-aware aggregation technology. Since inactive expert parameters in MoE do not need to be synchronized, related technologies synchronize all expert parameters across GPUs during the All Reduce operation, requiring a full transmission of all expert parameters, which wastes bandwidth to some extent. However, in this embodiment, the reduction of the output corresponding to the activated expert can be achieved based on the sparse-aware reduction core process, avoiding the synchronous transmission of inactive expert parameters between different GPUs.

[0114] For example, refer to Figure 15 The core process of sparse perception reduction includes:

[0115] Step B1: Identify the activated experts.

[0116] In this step, the first device can identify the output of the activated expert during the training or inference phase of the model.

[0117] Step B2: Allocate space to the activated experts.

[0118] In this step, the first device can allocate corresponding storage space to each activated expert and store the output results in the corresponding storage space.

[0119] Step B3: Reduce the output of the activated experts.

[0120] Step B4: Forward the reduction result to the source GPU.

[0121] In summary, this application proposes a ensemble communication acceleration scheme based on GPU-to-GPU interconnection, which can achieve expert routing in sparse model scenarios (such as MoE). In this application, a first device replaces the GPU to perform expert routing calculations, expert selection, and expert allocation in the model, thereby accelerating ensemble communication during the training process of sparse model layers. This application can directly identify sparse data (e.g., data corresponding to activated experts) for sparse models such as MoE, achieving accelerated communication of sparse data, filtering data corresponding to inactive experts, avoiding invalid data transmission, and improving bandwidth utilization and communication efficiency. This application can guarantee low latency and memory consistency requirements based on memory semantics, and solve the dynamic communication problems brought about by various models through the sparse-aware reduction core process.

[0122] In this embodiment, the first device can not only reduce the output results corresponding to the activated experts in the hybrid expert model, but also perform other reduction operations. For example, before receiving input data from the source GPU among multiple GPUs, the first device can also receive a read data request from a first GPU among the multiple GPUs, where the first GPU is any one of the multiple GPUs. After receiving the read data request from the first GPU, the first device can, according to the identifier of the reduction group carried in the read data request, send the read data request to each GPU in the reduction group, which includes GPUs that need to read data to be reduced; then, it receives the data to be reduced sent by each GPU in the reduction group; performs a second reduction operation on the data to be reduced sent by each GPU in the reduction group to obtain a second reduction result; and sends the second reduction result to the first GPU.

[0123] Here, the first GPU can be a GPU participating in the reduction calculation. The first GPU, as the initiator, can send a message carrying a read data request to the first device. For example, the message carrying the read data request can include the following information: the identifier of the reduction group, the source address (i.e., the address of the first GPU), the destination address (the address of the first device), the data type (e.g., various precision types such as INT32 / Float32 / Float16 / BFloat16), the operation type (including but not limited to various operations such as addition, multiplication, MAX, MIN, AND, OR, XOR), and the model type. The model type can be a normal model, MoE, multimodal model, etc.

[0124] After the first device requests and completes its internal processing, it can reply to the read data request in a multicast manner to all GPU members participating in the ensemble communication acceleration (i.e., GPU members in the protocol group). For example, the first device can identify other GPUs in the same protocol group based on the identifier of the protocol group in each read data request. The identifier of the protocol group can be the sequence number of the protocol group or other identifiers.

[0125] When a GPU in the reduction group receives a read data request, it can load the data to be reduced from its internal memory according to the operation type and reply to the first device with the information carrying the data to be reduced.

[0126] For example, before the first device receives input data sent by the source GPU among multiple GPUs, address mapping is performed between each GPU and the first device, so that each GPU maps to the same address space of the first device. Task grouping can be performed when a task related to the second reduction operation (e.g., an ALL Reduce task) is initiated. Embodiments of this application can also pre-set multicast groups or broadcast groups (MCIDs) and set members of the multicast groups or broadcast groups (e.g., GPU 1, GPU 2, ..., GPUn, etc.), where the multicast group or broadcast group represents the space allocated in the first device for subsequent reduction operations.

[0127] After receiving a read data request, the first device can modify the destination address in the read data request to the address mapped from the first GPU to the first device. The address mapped from the first GPU to the first device can be the address of the relevant computing unit in the first device's ERU. The first device can store information such as the data type and operation type in the read data request for subsequent data aggregation or forwarding. After receiving a message carrying a read data request, each GPU in the protocol group can parse the message to determine the source address of the data to be read and the destination address of the peer (e.g., the address of the relevant computing unit in the first device's ERU). The GPU can read data from the specified source address (e.g., DATA2, DATA7, DATAn) and prepare to send it to the peer (i.e., the first device) to complete the aggregation.

[0128] The first device can parse the destination address in the message carrying the data to be reduced sent by the GPU, and route the data to be reduced to the relevant computing unit according to the destination address. After collecting the data to be reduced from all GPUs in the same reduction group, the first device can perform a second reduction operation to obtain a second reduction result, ensuring that the reduction calculation is completed after all the data to be reduced from all GPUs in the same reduction group has been collected in the same computing unit.

[0129] After receiving the second reduction result, the first device can determine the destination address corresponding to the second reduction result from the source address in the read data request sent by the first GPU. Then, it packages the second reduction result and the corresponding destination address and sends them to the first GPU. Each GPU, acting as the initiator of the read data request, can store its own received reduction results. Each GPU is only responsible for aggregating and retaining its own corresponding data.

[0130] As can be seen, the embodiments of this application can collect the data to be reduced sent by each GPU in the reduction group according to the identifier of the reduction group carried in the read data request, so as to reliably and accurately perform the second reduction operation according to the data to be reduced sent by each GPU in the reduction group.

[0131] To further improve the reliability of the second reduction operation, in some embodiments, the process of the first device sending a read data request to each GPU in the reduction group includes: generating an alignment identifier for the reduction group, and sending a read data request carrying the alignment identifier to each GPU in the reduction group. Correspondingly, the process of receiving the data to be reduced sent by each GPU in the reduction group includes: receiving the data to be reduced carrying the alignment identifier sent by each GPU in the reduction group. The process of performing a second reduction operation on the data to be reduced sent by each GPU in the reduction group to obtain a second reduction result includes: aligning the data to be reduced sent by each GPU according to the alignment identifier to obtain aligned data to be reduced; and performing a second reduction operation on the aligned data to be reduced to obtain a second reduction result.

[0132] For example, after receiving a message carrying a read data request, the first device can assign an alignment identifier to the message, for example, the alignment identifier is TAG 1. After sending the read data request carrying the alignment identifier to each GPU in the reduction group, each GPU can support aligning data with the same alignment identifier sent by multiple GPUs. However, each GPU does not process the alignment identifier, but instead returns the data to be reduced carrying the alignment identifier to the first device. The first device can then reduce the data to be reduced with the same alignment identifier. After receiving the data to be reduced with the alignment identifier, the first device can align the data to be reduced with the same alignment identifier to obtain the aligned data to be reduced.

[0133] In this embodiment of the application, during distributed parallel training, the first device can replace the GPU to perform ALLReduce reduction computation. The All Reduce operation can also be implemented through Reduce Scatter and All Gather operations. The following describes... Figure 16 This describes the process of executing the Reduce Scatter operation based on the first device, referring to... Figure 16GPU 0 is the first GPU. GPU 0 can send a read data request to the first device. The read data request can carry the destination address and the data to be reduced by GPU 0, which is Data 0. The first device sends the read data request to each GPU in the reduction group, for example, GPU 0, GPU 1, and GPU n. The first device can receive the data to be reduced sent by each GPU in the reduction group, and then perform a second reduction operation on the data to be reduced sent by each GPU in the reduction group to obtain the second reduction result. For example, if the data to be reduced sent by GPU 1 is Data 1, and the data to be reduced sent by GPU n is Data n, then in the first device, an addition operation can be performed on Data 0, Data 1, GPU n, and Data n to obtain Data A0, which is the second reduction result. The first device can send Data A0 to GPU 0; similarly, GPU 1, GPU 7, and GPU n can also obtain the corresponding second reduction results Data A1, Data A7, and Data An.

[0134] It can be seen that the first device can perform a second reduction operation on the aligned data to be reduced sent by each GPU after aligning the data to be reduced. This is beneficial for reliably implementing the reduction operation on the data to be reduced.

[0135] In order to achieve synchronization of the second protocol result, in some embodiments, after the first device sends the second protocol result to the first GPU, it can also receive a message carrying the second protocol result sent by the first GPU; and send the second protocol result to each GPU in the protocol group via multicast.

[0136] In this embodiment, after the first device sends the second reduction result to the first GPU, the first GPU can send a broadcast request message to the first device. The broadcast request message can carry a broadcast group, a destination address (i.e., the address of the first device), and data to be reduced. The first device copies the message according to the broadcast group and forwards it to all members of the broadcast group. It should be noted that the first device only forwards the data and does not modify its content. Each GPU in the reduction group can pre-establish a correspondence between each reduction group and the address offset in the GPU address space. Thus, after receiving a message carrying the second reduction result, each GPU in the reduction group can parse the destination address in the message, determine the address offset corresponding to the reduction group, and write the second reduction result into the GPU's buffer and store it in memory based on the destination address and the address offset. The following describes... Figure 17 This describes the process of performing the All Gather operation based on the first device, referring to... Figure 17GPU 0 sends a broadcast request message to the first device. The first device can send the broadcast request message to all members of the broadcast group. For example, all members of the broadcast group include... Figure 17 GPU 1, GPU 7, and GPU n.

[0137] As can be seen, the embodiments of this application can achieve synchronization of the second reduction result between different GPUs through multicast.

[0138] The embodiments of this application can be applied to scenarios such as artificial intelligence, distributed training, intelligent computing interconnection, and basic hardware. Compared with the SHArP technology in related technologies, the embodiments of this application can not only offload traditional ensemble communication operations, but also perform routing and computation offloading for specific experts in sparse MoE models. This solves the problems of irregular high-latency scheduling in dynamic communication, as well as load imbalance and local hardware overload, thereby improving the computational communication efficiency of large MoE models, offloading GPU ensemble communication computation capabilities, and improving GPU tensor computation utilization.

[0139] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0140] This application also proposes a GPU-based collective communication acceleration device, which is located in a first device. Figure 18 This is a schematic diagram of the structure of the GPU's collective communication acceleration device according to an embodiment of this application, as shown below. Figure 18 As shown, the device includes:

[0141] The receiving module 1800 is used to receive input data sent by the source GPU among the plurality of GPUs;

[0142] The first processing module 1801 is used to perform expert routing calculation based on the input data to determine the target expert in the hybrid expert model for processing the input data.

[0143] The second processing module 1802 is used to send the input data to the GPU corresponding to the target expert, the GPU corresponding to the target expert is used to process the input data through the target expert to obtain the output result; obtain the output result corresponding to the activated expert in the hybrid expert model; and reduce the output result corresponding to the activated expert in the hybrid expert model.

[0144] In some embodiments, the second processing module 1802 is specifically used to receive the output results corresponding to each expert in the hybrid expert model. The output result corresponding to the first expert in the hybrid expert model includes an indicator field, which is used to indicate whether the first expert is an activated expert. The first expert is any one of the experts in the hybrid expert model.

[0145] The second processing module 1802 is specifically used to obtain the output result corresponding to the activated expert from the output results corresponding to each expert based on the indicator field in the output results corresponding to each expert.

[0146] In some embodiments, the second processing module 1802 is specifically used to send the input data with added data load to the GPU corresponding to the target expert, wherein the data load is used to enable the GPU to determine the activated expert;

[0147] The second processing module 1802 is specifically used to receive a data packet sent by the GPU corresponding to the activated expert, the data packet including at least the output result corresponding to the activated expert.

[0148] In some embodiments, the first processing module 1801 is specifically configured to perform expert routing calculation based on the input data, determine expert allocation results, and determine the target expert in the hybrid expert model for processing the input data based on the expert allocation results; the data load includes the expert allocation results.

[0149] In some embodiments, the data packet further includes the data payload; the second processing module 1802 is specifically configured to reduce the output results corresponding to the same activated expert according to the expert allocation result.

[0150] In some embodiments, the second processing module 1802 is specifically used to perform a first reduction operation on the output results corresponding to the activated experts in the hybrid expert model to obtain a first reduction result;

[0151] The second processing module 1802 is further configured to send the first reduction result to the source GPU after obtaining the first reduction result.

[0152] In some embodiments, the apparatus further includes a third processing module, the third processing module being configured to receive a read data request sent by a first GPU among the plurality of GPUs; send the read data request to each GPU in the reduction group according to the identifier of the reduction group carried in the read data request, the reduction group including GPUs that need to read data to be reduced; receive the data to be reduced sent by each GPU in the reduction group; perform a second reduction operation on the data to be reduced sent by each GPU in the reduction group to obtain a second reduction result; and send the second reduction result to the first GPU.

[0153] In some embodiments, the third processing module is further configured to, after sending the second reduction result to the first GPU, receive a message carrying the second reduction result sent by the first GPU; and send the second reduction result to each GPU in the reduction group via multicast.

[0154] In some embodiments, the third processing module is specifically used to generate an alignment identifier for the protocol group and send the read data request carrying the alignment identifier to each GPU in the protocol group;

[0155] The third processing module is specifically used to receive the data to be reduced, which carries the alignment identifier, sent by each GPU in the reduction group;

[0156] The third processing module is specifically used to align the data to be reduced sent by each GPU according to the alignment identifier to obtain aligned data to be reduced; and to perform a second reduction operation on the aligned data to be reduced to obtain a second reduction result.

[0157] In practical applications, the receiving module 1800, the first processing module 1801, the second processing module 1802, and the third processing module can be implemented based on a processor and a communication device.

[0158] It should be noted that the description of the above device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects. For technical details not disclosed in the device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0159] It should be noted that, in the embodiments of this application, if the above methods are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a terminal, server, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0160] Correspondingly, this application embodiment further provides a computer program product, the computer program product including computer executable instructions, which are used to implement any of the GPU collective communication acceleration methods provided in this application embodiment.

[0161] Accordingly, this application embodiment further provides a computer storage medium storing computer-executable instructions, which are used to implement any of the GPU collective communication acceleration methods provided in the above embodiments.

[0162] This application also provides an electronic device. Figure 19 This is a schematic diagram of the composition structure of an electronic device provided in an embodiment of this application, such as... Figure 19 As shown, the electronic device 190 may include:

[0163] Communication device 1900, used for communicating with the GPU;

[0164] Memory 1901 is used to store data to be calculated and computer programs;

[0165] Processor 1902 is used to run the computer program to execute any of the above-described GPU collective communication acceleration methods.

[0166] The processor 1902 mentioned above can be at least one of ASIC, DSP, DSPD, PLD, FPGA, CPU, controller, microcontroller, and microprocessor.

[0167] The aforementioned computer-readable storage medium and memory 1901 may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM), etc.; it may also be various terminals that include one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0168] In some embodiments, the functions or modules of the apparatus provided in this application can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0169] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0170] The methods disclosed in the various method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0171] The features disclosed in the various product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0172] The features disclosed in the various method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0173] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0174] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims. All of these forms are within the protection scope of this application.

Claims

1. A method for accelerating ensemble communication in a GPU, characterized in that, Applied to a first device, the first device forming a communication connection with multiple graphics processing units (GPUs), the method includes: Receive input data sent by the source GPU among the multiple GPUs; Based on the input data, perform expert routing calculations to determine the target expert in the hybrid expert model used to process the input data; The input data is sent to the GPU corresponding to the target expert, and the GPU corresponding to the target expert is used to process the input data by the target expert to obtain the output result; Obtain the output results corresponding to the activated experts in the hybrid expert model; The output results corresponding to the activated experts in the hybrid expert model are reduced.

2. The method according to claim 1, characterized in that, The step of obtaining the output results corresponding to the activated experts in the hybrid expert model includes: The system receives the output results corresponding to each expert in the hybrid expert model. The output result corresponding to the first expert in the hybrid expert model includes an indicator field, which is used to indicate whether the first expert is an activated expert. The first expert is any one of the experts in the hybrid expert model. Based on the indicator field in the output results corresponding to each expert, the output result corresponding to the activated expert is obtained from the output results corresponding to each expert.

3. The method according to claim 1, characterized in that, Sending the input data to the GPU corresponding to the target expert includes: The input data with added data payload is sent to the GPU corresponding to the target expert, and the data payload is used to enable the GPU to determine the activated expert; The step of obtaining the output results corresponding to the activated experts in the hybrid expert model includes: Receive data packets sent by the GPU corresponding to the activated expert, wherein the data packets include at least the output results corresponding to the activated expert.

4. The method according to claim 3, characterized in that, The step of performing expert routing calculation based on the input data to determine the target expert in the hybrid expert model for processing the input data includes: performing expert routing calculation based on the input data to determine the expert allocation result, and determining the target expert in the hybrid expert model for processing the input data based on the expert allocation result; the data load includes the expert allocation result.

5. The method according to claim 4, characterized in that, The data packet also includes the data payload; The reduction of the output results corresponding to the activated experts in the hybrid expert model includes: Based on the expert allocation results, the output results corresponding to the same activated expert are reduced.

6. The method according to claim 1, characterized in that, The reduction of the output results corresponding to the activated experts in the hybrid expert model includes: performing a first reduction operation on the output results corresponding to the activated experts in the hybrid expert model to obtain a first reduction result; After obtaining the first reduction result, the method further includes: sending the first reduction result to the source GPU.

7. The method according to any one of claims 1 to 6, characterized in that, Before receiving the input data sent by the source GPU among the plurality of GPUs, the method further includes: Receive a read data request sent by the first GPU among the plurality of GPUs; Based on the identifier of the reduction group carried in the read data request, the read data request is sent to each GPU in the reduction group, the reduction group including the GPUs that need to read the data to be reduced; Receive the data to be reduced sent by each GPU in the reduction group; For the data to be reduced sent by each GPU in the reduction group, perform the second reduction operation to obtain the second reduction result; The second specification result is sent to the first GPU.

8. The method according to claim 7, characterized in that, After sending the second reduction result to the first GPU, the method further includes: Receive the message sent by the first GPU carrying the result of the second protocol; The second reduction result is sent to each GPU in the reduction group via multicast.

9. The method according to claim 7, characterized in that, Sending the read data request to each GPU in the protocol group includes: generating an alignment identifier for the protocol group, and sending the read data request carrying the alignment identifier to each GPU in the protocol group; The step of receiving the data to be reduced sent by each GPU in the reduction group includes: receiving the data to be reduced sent by each GPU in the reduction group, which carries the alignment identifier; The step of performing a second reduction operation on the data to be reduced sent by each GPU in the reduction group to obtain a second reduction result includes: aligning the data to be reduced sent by each GPU according to the alignment identifier to obtain aligned data to be reduced; and performing a second reduction operation on the aligned data to be reduced to obtain a second reduction result.

10. A GPU-based collective communication acceleration device, characterized in that, The device is located in a first device, which is in communication connection with multiple graphics processing units (GPUs). The device includes: A receiving module is used to receive input data sent by the source GPU among the multiple GPUs; The first processing module is used to perform expert routing calculation based on the input data to determine the target expert in the hybrid expert model for processing the input data. The second processing module is used to send the input data to the GPU corresponding to the target expert, and the GPU corresponding to the target expert is used to process the input data through the target expert to obtain the output result; obtain the output result corresponding to the activated expert in the hybrid expert model; and reduce the output result corresponding to the activated expert in the hybrid expert model.

11. An electronic device, characterized in that, The electronic device forms a communication connection with multiple GPUs, and the electronic device includes: A communication device used to communicate with the GPU; Memory, used to store data to be calculated and computer programs; A processor for running the computer program to perform the method according to any one of claims 1 to 9.

12. The electronic device according to claim 11, characterized in that, The processor includes at least one of the following: a first arithmetic unit, a second arithmetic unit, and a third arithmetic unit; the first arithmetic unit is used to implement tensor computation, the second arithmetic unit is used to implement at least one of logical operations, address management, and data sorting, and the third arithmetic unit is used to implement floating-point computation.

13. A computer storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the method described in any one of claims 1 to 10.

14. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1 to 10.

Images