Data processing method and device for hybrid expert model, equipment and medium
By performing one inter-node communication and one intra-node broadcast within the target node, the problem of high latency in All-to-All communication is solved, the communication efficiency of the hybrid expert model is improved, and the requirements of low-latency scenarios are met.
Patent Information
- Application Number
- CN202511351589.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-09-19
- Publication Date
- 2025-12-19
AI Technical Summary
In large-scale expert parallelism, the long communication latency of All-to-All communication becomes the main bottleneck for inference and decoding in hybrid expert models, resulting in low communication efficiency.
By conducting an inter-machine communication within the target node, data packets are sent to the target processing unit, and then broadcast within that node via high-speed NVL, reducing inter-machine communication redundancy and improving communication efficiency.
It effectively reduces communication latency, improves the efficiency of all-switched communication, reduces inter-machine communication redundancy, and meets the requirements of low-latency scenarios.
Smart Images
Figure CN121166601A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of artificial intelligence technology, and more particularly to the field of deep learning and hybrid expert model technology. More specifically, this disclosure provides a data processing method, apparatus, electronic device, storage medium, and computer program product for hybrid expert models. Background Technology
[0002] With the rapid development of MOE (Mixture of Experts) models, expert parallelism has become a common parallel approach. All-to-All communication is a key operation in expert parallelism. In large-scale expert parallelism, all-to-All communication is used to distribute and merge hidden state data across different experts on different machines. However, the implementation of all-to-All often results in long communication latency, which is a major bottleneck in the inference-decoding truncation of large models. Summary of the Invention
[0003] This disclosure provides a data processing method, apparatus, electronic device, storage medium, and computer program product for hybrid expert models.
[0004] According to a first aspect, a data processing method using a hybrid expert model is provided, wherein multiple expert sub-models of the hybrid expert model are distributed across processing units of multiple nodes. The method includes: using the current processing unit of a source node to send a data packet containing source data to a peer processing unit in a target node corresponding to the current processing unit, wherein the data packet further includes synchronization information, including target location information to which the source data is to be received; using the peer processing unit to send the source data to at least one target processing unit in the target node indicated by the target location information; using the expert sub-models of each of the at least one target processing unit to process the source data, obtaining at least one processing result; using the peer processing unit to obtain the processing results of the expert sub-models of each target processing unit, and merging the processing results to obtain target data; and using the peer processing unit to send the target data to the current processing unit.
[0005] According to a second aspect, a data processing apparatus for a hybrid expert model is provided, wherein multiple expert sub-models of the hybrid expert model are distributed in processing units of multiple nodes; the apparatus includes: a first sending module, used to send a data packet containing source data to a peer processing unit corresponding to the current processing unit in a target node using the current processing unit of the source node, wherein the data packet further includes synchronization information, the synchronization information including target location information to which the source data is to be received; a second sending module, used to send the source data to at least one target processing unit in the target node indicated by the target location information using the peer processing unit; a processing module, used to process the source data using the expert sub-models of each of the at least one target processing unit to obtain at least one processing result; a first merging module, used to obtain the processing result of the expert sub-model of each target processing unit using the peer processing unit, and to merge the processing results to obtain target data; and a third sending module, used to send the target data to the current processing unit using the peer processing unit.
[0006] According to a third aspect, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a method provided according to the present disclosure.
[0007] According to a fourth aspect, a non-transitory computer-readable storage medium is provided that stores computer instructions for causing a computer to perform the methods provided in this disclosure.
[0008] According to a fifth aspect, a computer program product is provided, comprising a computer program stored on at least one of a readable storage medium and an electronic device, wherein the computer program, when executed by a processor, implements the method provided in this disclosure.
[0009] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description
[0010] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:
[0011] Figures 1A-1B This is a general framework diagram of ALL to ALL fully switched communication using a hybrid expert model in related technologies;
[0012] Figure 2 This is a flowchart of a data processing method for a hybrid expert model according to an embodiment of the present disclosure;
[0013] Figure 3 This is an overall framework diagram of a data processing method for a hybrid expert model according to an embodiment of the present disclosure;
[0014] Figure 4 This is a schematic diagram of a data packet according to an embodiment of the present disclosure;
[0015] Figure 5 This is a schematic diagram of a data processing method using a hybrid expert model according to an embodiment of the present disclosure;
[0016] Figure 6A This is a schematic diagram of a distribution operation in a data processing method according to an embodiment of the present disclosure;
[0017] Figure 6B This is a schematic diagram of a combination operation in a data processing method according to an embodiment of the present disclosure;
[0018] Figure 7 This is a block diagram of a data processing apparatus for a hybrid expert model according to an embodiment of the present disclosure; and
[0019] Figure 8 This is a block diagram of an electronic device for a data processing method using a hybrid expert model according to an embodiment of the present disclosure. Detailed Implementation
[0020] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.
[0021] Hybrid expert models consist of multiple expert sub-models and gated networks. Expert parallelism strategies include data parallelism and expert parallelism. Data parallelism involves splitting a batch of data across multiple GPUs, while model parallelism involves distributing multiple expert sub-models across processing units on different nodes. For example, processing units can be GPUs (Graphics Processing Units), TPUs (Tensor Processing Units), NPUs (Natural-network Processing Units), etc. Each node can include multiple processing units, and each processing unit can have one or more expert sub-models distributed across it. ALL-to-ALL communication is the communication mode for expert parallelism, requiring expert sub-models distributed across processing units on different nodes to communicate with each other.
[0022] For example, the data on each processing unit can include multiple tokens. The gating network calculates which expert sub-model each token should be sent to for processing based on its characteristics. Taking a GPU as the processing unit, for any given GPU, according to the gating network's routing strategy, a portion of the data on that GPU needs to be sent to the corresponding expert sub-models on other GPUs for processing. The current GPU can send the data that needs to be sent to other GPUs, and simultaneously receive data from other GPUs that needs to be processed by its local expert sub-model. After the current GPU has processed the data that needs to be processed by its local expert sub-model, it also needs to return the processing results from the data received from other GPUs to the GPUs that sent the data.
[0023] Figures 1A-1B This is a general framework diagram of ALL to ALL fully switched communication using a hybrid expert model in related technologies.
[0024] like Figures 1A-1B As shown, the overall framework of ALL-to-ALL fully switched communication includes a first computing node 110 and a second computing node 120. The first computing node 110 may include multiple processing units, such as a first processing unit 111, a second processing unit 112, a third processing unit 113, and a fourth processing unit 114. The second computing node 120 may also include multiple processing units, such as a first processing unit 121, a second processing unit 122, a third processing unit 123, and a fourth processing unit 124. The processing units in the first computing node 110 and the second computing node 120 can be GPUs, TPUs, or NPUs, etc.
[0025] like Figure 1A As shown, in the fully switched communication of the expert parallel strategy, each processing unit, whether communicating with other processing units within the same node or with other processing units outside the node, uses Remote Direct Memory Access (RDMA) communication. "Within the same node" refers to units within the same node, while "outside the node" refers to units belonging to different computing nodes. For example, the first processing unit 111 communicates with the second computing unit 112 of the first computing node 110 via RDMA. The first processing unit 111 also communicates with the first processing unit 121 and the third processing unit 123 of the second computing node 120 via RDMA. RDMA communication transmits data through the InfiniBand (IB) network port. The IB network port has low bandwidth and poor performance, resulting in high ALL-to-ALL communication latency, making it difficult to meet the requirements of low-latency scenarios.
[0026] like Figure 1B As shown, in the fully switched communication of the expert parallel strategy, each processing unit communicates with other processing units within the machine using NVIDIA Link (NVL, a high-speed point-to-point communication technology developed by NVIDIA for connecting GPUs to GPUs or GPUs to CPUs), and with other processing units outside the machine using Remote Direct Access (RDMA). For example, the first processing unit 111 communicates with the second processing unit 112 of the first compute node 110 via NVIDIA Link. The first processing unit 111 communicates with the first processing unit 121 and the third processing unit 123 of the second compute node 120 via RDMA. NVIDIA Link has higher bandwidth compared to... Figure 1A Communication methods Figure 1B The communication method shown reduces communication latency to some extent. However, when a current processing unit needs to communicate with multiple processing units belonging to the same external node, the same data needs to be transmitted separately to these multiple processing units via Remote Direct Access (RDMA). For example, the first processing unit 111 needs to transmit the same data twice via RDMA to the first processing unit 121 and the third processing unit 123 of the second computing node 120. In other words, the same data needs to be transmitted to the same external machine multiple times, resulting in redundancy in inter-machine data transmission.
[0027] In view of this, embodiments of the present disclosure provide a data processing method using a hybrid expert model, which transmits data from multiple processing units that need to be transmitted to the same external node to that external node through a single RDMA communication, and then broadcasts the data within that external node via high-speed NVL, thereby further improving the communication efficiency of the fully switched communication.
[0028] The following is combined Figures 2-5 , Figures 6A-6B The data processing method of the hybrid expert model of the present disclosure will be described.
[0029] Figure 2 This is a flowchart of a data processing method for a hybrid expert model according to an embodiment of the present disclosure.
[0030] like Figure 2 As shown, the data processing method 200 for the hybrid expert model includes operations S210 to S250. The expert parallel strategy distributes multiple expert sub-models of the hybrid expert model across processing units on multiple nodes. Each node can include multiple processing units, and each processing unit can have one or more expert sub-models distributed across it. The expert parallel strategy also splits the data to be processed into different processing units.
[0031] In operation S210, the source node's current processing unit sends a data packet containing source data to the peer processing unit in the target node corresponding to the current processing unit.
[0032] The current processing unit can be the processing unit that initiates the communication operation, and the node where the current processing unit resides is the source node. For example, according to the routing strategy of the gating network, a portion of the data (source data) on the current processing unit needs to be sent to the expert sub-model of the processing unit on other nodes for processing. The current processing unit extracts the source data and packages it into a data packet. The node where this data packet needs to reach is the target node.
[0033] The target node comprises multiple processing units. To ensure that data packets are sent to the correct processing unit, synchronization information can be incorporated into the data packets. This synchronization information can include the target location information to which the source data is to reach. Specifically, the target location information can include which target node the source data is to reach, which target processing unit(s) on the target node it is to reach, and the specific location on the processing unit.
[0034] When sending a data packet containing source data to a target node, the current processing unit can send the data packet to the corresponding processing unit on the target node. This corresponding processing unit can be called the peer processing unit. For example, the correspondence between the current processing unit and the peer processing unit can mean that the number of the current processing unit on the source node is the same as the number of the peer processing unit on the target node. Therefore, inter-machine communication uses processing units with the same number to transmit data, making it easy to determine which processing unit on the target node the source data was sent to without introducing additional synchronization signals.
[0035] The current processing unit on the source node sends the data packet containing the source data to the peer processing unit on the target node corresponding to the current processing unit. The source data can be sent to the target node in one inter-machine communication, without the need for multiple inter-machine transmissions, thus reducing inter-machine communication redundancy.
[0036] In operation S220, the source data is sent to at least one target processing unit in the target node indicated by the target location information using the peer processing unit.
[0037] After the source data arrives at the peer processing unit of the target node, the peer processing unit can broadcast the source data within the second node so that the source data can be sent to the target processing unit on the target node.
[0038] For example, the source data can be broadcast to the target processing units based on which target processing units on the target node the source data needs to reach, without having to broadcast it to all processing units on the target node.
[0039] In operation S230, the source data is processed using the expert sub-models of at least one target processing unit to obtain at least one processing result.
[0040] After the target processing unit obtains the source data that needs to be processed by its own expert sub-model, it can use its own expert sub-model to process the source data and obtain the processing result.
[0041] In operation S240, the processing results of the expert sub-model of each target processing unit are obtained by using the peer processing unit, and the processing results are merged to obtain the target data.
[0042] After obtaining the processing results of the expert sub-model of each target processing unit, the processing results need to be summarized and returned to the current processing unit that initiated the communication operation.
[0043] The processing results of the expert sub-models of each target processing unit on the target node can be collected by the peer processing unit of the target node. Then, the processing results are merged to obtain the target data, and the target data is returned to the current processing unit.
[0044] In operation S250, the target data is sent to the current processing unit using the peer processing unit.
[0045] The processing unit at the other end corresponds to the current processing unit, for example, by having the same number. Therefore, inter-machine communication of the target data uses the same numbered processing unit for transmission, maintaining consistency with inter-machine communication of the source data.
[0046] The target processing unit collects the processing results from the target processing unit and performs a merging operation to obtain the target data. The target processing unit then returns the target data to the current processing unit. This allows the target data to be returned to the source node in a single inter-machine communication, eliminating the need for multiple inter-machine transmissions and reducing inter-machine communication redundancy.
[0047] According to embodiments of this disclosure, the current processing unit of the source node transmits source data that needs to be transmitted to multiple processing units of the same target node to the target node through only one inter-machine communication, eliminating inter-machine communication redundancy. Within the target node, the source data is sent to the target processing unit through internal broadcast. After the target processing unit processes the source data, the processing results are merged and returned to the source node through one inter-machine communication, which can improve the communication efficiency of the fully switched communication.
[0048] Figure 3 This is an overall framework diagram of data processing using a hybrid expert model according to an embodiment of the present disclosure.
[0049] like Figure 3As shown, this embodiment includes a first computing node 310 and a second computing node 320. The first computing node 310 may include multiple processing units, such as a first processing unit 311, a second processing unit 312, a third processing unit 313, and a fourth processing unit 314. The second computing node 320 may also include multiple processing units, such as a first processing unit 321, a second processing unit 322, a third processing unit 323, and a fourth processing unit 324. The processing units in the first computing node 310 and the second computing node 320 may be GPUs, TPUs, or NPUs, etc.
[0050] For example, the first processing unit 311 initiates a communication operation, requiring the source data to be sent to the target processing unit. The target processing unit includes a second processing unit 312 located in the first computing node 310, a first processing unit 321 located in the second computing node 320, and a third processing unit 323. Within the first computing node 310, the first processing unit can directly send the source data to the second processing unit 312 via the high-speed interconnect NVL.
[0051] When sending source data to the target processing unit in the second node 320, the source data can be packaged first, and synchronization information can be added to the data packet. The synchronization information can be used to indicate which target node the source data needs to be sent to, which target processing unit(s) on the target node it needs to be sent to, and the location of the target processing unit it needs to be sent to. For example, the synchronization information may include the target node number (such as the second computing node 320), the target processing unit number (such as the first processing unit 321, the third processing unit 323), and the specific location information within the target processing unit.
[0052] The first processing unit 311 of the first computing node 310 and the first processing unit 321 of the second computing node 320 can be corresponding, for example, both being the first processing unit within their respective nodes. Therefore, the first processing unit 311 can send data packets to the first processing unit 321. For example, the first processing unit 311 can send data packets to the first processing unit 321 via Remote Direct Access (RDMA) through the IB network port.
[0053] Inside the second computing node 320, after the first processing unit 321 receives a data packet containing source data, it can send the data packet to the third processing unit 323 in the second computing node 320 according to the synchronization information in the data packet. For example, the first processing unit 321 can send the source data to the third processing unit 323 through a high-speed interconnect NVL.
[0054] The embodiments of this disclosure transmit source data that needs to be transmitted to multiple processing units on the same external node to one processing unit on that external node via a single Remote Direct Access (RDMA). This eliminates inter-machine communication redundancy. Within that external node, the source data is sent to the target processing unit using higher bandwidth NVL communication, which reduces communication latency and improves the communication efficiency of the fully switched communication.
[0055] Figure 4 This is a schematic diagram of a data packet according to an embodiment of the present disclosure.
[0056] like Figure 4 As shown, the data packet may include source data location 410, source data 420, and synchronization information 430 for multiple target nodes. The synchronization information 430 for multiple target nodes may include synchronization information for the first target node and synchronization information for the nth target node, where n is an integer greater than 1. The synchronization information for each target node includes the number of target processing units 431, the target processing unit number 432, the target processing unit location 433, and a weighting coefficient 434.
[0057] Taking the current GPU in the source node that initiates the communication operation as an example, which wants to send source data 420 to the target GPU of the target node, the data packet of this embodiment will be described.
[0058] Source data location 410 refers to the location of source data 420 in the current GPU. Since source data 420 needs to be sent to GPUs on multiple target nodes, the data packet contains synchronization information for these target nodes. In each target node's synchronization information, the target processing unit number 431 indicates the number of GPUs in the target nodes to which source data 420 needs to reach. The target processing unit number 432 indicates the GPU number to which the source data needs to reach. The target processing unit location 433 indicates the location of the target processing unit to which the source data needs to reach. The weight coefficient 434 represents the weight coefficient of the expert sub-model in the target processing unit, which indicates the weight of the processing result of the expert sub-model in that target processing unit during the merge operation.
[0059] The embodiments of this disclosure manage the synchronization status of data between multiple nodes, within a single node, and within a processing unit through multi-level synchronization information management, which can ensure the accuracy of data transmission and improve communication performance.
[0060] According to embodiments of this disclosure, obtaining the processing results of the expert sub-models of each target processing unit using the peer processing unit, and performing a merging operation on the processing results to obtain target data includes: obtaining the processing results of the expert sub-models of each target processing unit using the peer processing unit, and performing a merging operation on the processing results according to the weight coefficients of the expert sub-models to obtain target data.
[0061] For example, the peer processing unit can multiply the processing result of the expert sub-model of each target processing unit with the weight coefficient of the expert sub-model to obtain the weighted processing result of the target processing unit, and then sum the weighted processing results of multiple target processing units to obtain the target data.
[0062] In the embodiments of this disclosure, by adding synchronization information to the data packets to represent the weight coefficients of the expert sub-models in the target processing unit, the weight coefficients are sent to the target node in an implicit synchronization manner, so that the processing results of each target processing unit can be merged in the target node, and then the merged target data is returned to the source node through a single inter-machine communication, thereby improving communication efficiency.
[0063] Figure 5 This is a schematic diagram of a data processing method for a hybrid expert model according to an embodiment of the present disclosure.
[0064] like Figure 5 As shown, the data processing method in this embodiment includes a distribution operation 510 and a combination operation 520, which are performed by the GPU0 of the source node and the GPU0 of the target node. The GPU0 of the source node can be the GPU that initiates the communication, and the GPU0 of the target node can be the GPU corresponding to the GPU0 of the source node. The GPU0 of the source node is divided into RDMA transmitter / RDMA receiver and NVL transmitter / NVL receiver according to different communication methods. The GPU0 of the target node is also divided into RDMA transmitter / RDMA receiver and NVL transmitter / NVL receiver according to different communication methods. Among them, the RDMA transmitter / RDMA receiver is used for inter-machine communication through remote direct access RDMA, and the NVL transmitter / NVL receiver is used for intra-machine communication through high-speed interconnect NVL.
[0065] In distribution operation 510, the RDMA transmitter of the source node's GPU0 sends the data to be processed to the RDMA receiver of the target node's GPU0. The RDMA receiver of the target node's GPU0 sends the data to be processed to the NVL transmitter of the target node's GPU0, and the NVL transmitter of the target node's GPU0 sends the data to be processed to the NVL receiver of the target node's GPU0, thus completing the distribution operation.
[0066] According to embodiments of this disclosure, the distribution operation can support both single-QP (Queue Pair, the basic communication unit for RDMA operations) and multi-QP modes. In single-QP mode, multiple batches of data to be processed can be combined and then sent via remote RDMA in a single transmission. In multi-QP mode, each batch of data to be processed can be sent via RDMA individually.
[0067] In combination operation 520, the NVL transmitter of GPU0 on the target node sends the processed result data to the NVL receiver of GPU0 on the target node. The NVL receiver of GPU0 on the target node performs a merge operation on the processed result data to obtain the target data. The NVL receiver of GPU0 on the target node sends the target data to the RDMA transmitter of GPU0 on the target node. The RDMA transmitter of GPU0 on the target node sends the target data to the RDMA receiver of GPU0 on the source node. The RDMA receiver of GPU0 on the source node performs a merge operation on the target data from multiple target nodes.
[0068] According to embodiments of this disclosure, remote direct access to RDMA inter-machine communication is performed via RDMA transmitter / RDMA receiver, and high-speed interconnection NVL intra-machine communication is performed via NVL transmitter / NVL receiver. This can improve the stability of both inter-machine and intra-machine communication and enhance communication performance.
[0069] Figure 6A This is a schematic diagram of a distribution operation in a data processing method according to an embodiment of the present disclosure.
[0070] like Figure 6A As shown, the distribution operation in this embodiment includes operations S601 to S606.
[0071] According to embodiments of this disclosure, a data packet is sent from the global memory of the current processing unit to the inter-machine shared memory of the current processing unit using the current processing unit; a data packet is also sent from the inter-machine shared memory of the current processing unit to the inter-machine shared memory of the peer processing unit using the current processing unit.
[0072] For example, the current processing unit is GPU0 of the first compute node, and the global memory of the current processing unit is the global memory of GPU0 of the first compute node. The inter-machine shared memory of the current processing unit is the RDMA memory of GPU0 of the first compute node. The peer processing unit corresponding to the current processing unit is GPU0 of the second compute node, and the inter-machine shared memory of the peer processing unit is the RDMA memory of GPU0 of the second compute node.
[0073] In operation S601, GPU0 of the first compute node initiates a communication operation, determines the source data from the global memory of GPU0 of the first compute node, and packages it into a data packet.
[0074] In operation S602, GPU0 of the first compute node sends data packets from the global memory of GPU0 of the first compute node to the RDMA memory of GPU0 of the first compute node.
[0075] In operation S603, GPU0 of the first compute node sends data packets from the RDMA memory of GPU0 of the first compute node to the RDMA memory of GPU0 of the second compute node.
[0076] According to embodiments of this disclosure, a peer processing unit is used to send data packets from the inter-machine shared memory of the peer processing unit to the intra-machine shared memory of the peer processing unit; and a peer processing unit is used to send data packets from the intra-machine shared memory of the peer processing unit to the intra-machine shared memory of at least one target processing unit.
[0077] For example, the shared memory of the peer processing unit is the NVL memory of GPU0 of the second compute node. The target processing unit is GPU1 of the second compute node, and the shared memory of the target processing unit is the NVL memory of GPU1 of the second compute node.
[0078] In operation S604, GPU0 of the second compute node sends data packets from the RDMA memory of GPU0 of the second compute node to the NVL memory of GPU0 of the second compute node.
[0079] In operation S605, GPU0 of the second compute node sends data packets from the NVL memory of GPU0 of the second compute node to the NVL memory of GPU1 of the second compute node.
[0080] According to embodiments of this disclosure, a target processing unit sends data packets from the target processing unit's internal shared memory to the target processing unit's global memory; the target processing unit parses the data packets in the target processing unit's global memory to obtain source data; and the target processing unit uses an expert sub-model to process the source data to obtain the processing result of the target processing unit's expert sub-model.
[0081] When operating S606, GPU1 of the second compute node sends data packets from the NVL memory of GPU1 of the second compute node to the global memory of GPU1 of the second compute node.
[0082] In the global memory of GPU1 on the second computing node, GPU1 parses the source data from the data packet and processes it using the expert sub-model of GPU1 to obtain the processing result of the expert sub-model of GPU1 on the second computing node.
[0083] Figure 6B This is a schematic diagram of a combination operation in a data processing method according to an embodiment of the present disclosure.
[0084] like Figure 6B As shown, the combined operation in this embodiment includes operations S607 to S610.
[0085] According to embodiments of this disclosure, the target processing unit sends the processing results of the expert sub-model of the target processing unit from the global memory of the target processing unit to the shared memory of the target processing unit; the target processing unit sends the processing results from the shared memory of the target processing unit to the shared memory of the peer processing unit; and the peer processing unit performs a merging operation on at least one processing result from at least one target processing unit in the shared memory of the peer processing unit to obtain target data.
[0086] For example, the target processing unit is GPU1 of the second computing node, and the processing result of the expert sub-model of the target processing unit is stored in the global memory of the target processing unit (the global memory of GPU1 of the second computing node). The peer processing unit corresponding to GPU0 of the first computing node that initiated the communication operation is GPU0 of the second computing node.
[0087] During operation of S607, GPU1 of the second compute node sends the processing results of the expert sub-model of GPU1 from the global memory of GPU1 to the NVL memory of GPU1 of the second compute node.
[0088] During operation of S608, GPU1 of the second computing unit sends the received expert processing results from the NVL memory of GPU1 of the second computing node to the NVL memory of GPU0 of the second computing node.
[0089] In the NVL memory of GPU0 in the second computing node, GPU0 of the second computing node performs a merging operation on the expert processing results from GPU1 of the second computing unit to obtain the target data.
[0090] For example, the processing units involved in the merge operation may also include other processing units in the second computing node, not shown in the figure. When the merge operation is performed on GPU0 of the second computing node, the target data can be obtained by weighted summation based on the weight coefficients of the processing results from each target processing unit.
[0091] According to embodiments of this disclosure, the target data is sent from the intra-machine shared memory of the peer processing unit to the inter-machine shared memory using the peer processing unit; the target data is also sent from the inter-machine shared memory of the peer processing unit to the inter-machine shared memory of the current processing unit using the peer processing unit.
[0092] For example, the current processing unit that initiates the communication operation is GPU0 of the first computing node.
[0093] During operation of S609, GPU0 of the second compute node sends target data from the NVL memory of GPU0 of the second compute node to the RDMA memory of the second compute node.
[0094] During operation S610, the GPU0 of the second compute node sends target data from the RDMA memory of the GPU0 of the second compute node to the RDMA memory of the GPU0 of the first compute node.
[0095] According to embodiments of this disclosure, the target data from multiple target nodes is merged in the inter-machine shared memory of the current processing unit to obtain the target processing result; the target processing result is then sent from the inter-machine shared memory of the current processing unit to the global memory of the current processing unit.
[0096] In the RDMA memory of GPU0 on the first compute node, GPU0 performs a merge operation on the target data from multiple target nodes to obtain the target processing result. This target processing result is the final processing result of the source data.
[0097] During operation S611, GPU0 of the first compute node sends the target processing result from the RDMA memory of GPU0 of the first compute node to the global memory of GPU0 of the first compute node.
[0098] For example, GPU0 of the first compute node sends the target processing result of the source data to the location of the source data in global memory.
[0099] According to embodiments of this disclosure, by dividing the memory of the processing unit into global memory, intra-machine shared memory, and inter-machine shared memory, data transmission within a single machine and between multiple machines can be performed more accurately, thereby improving communication accuracy.
[0100] According to embodiments of this disclosure, this disclosure also provides a data processing apparatus for hybrid expert models.
[0101] Figure 7 This is a block diagram of a data processing apparatus for a hybrid expert model according to an embodiment of the present disclosure.
[0102] like Figure 7 As shown, the data processing device 700 for the hybrid expert model includes a first sending module 710, a second sending module 720, a processing module 730, a first merging module 740, and a third sending module 750.
[0103] The first sending module 710 is used to send a data packet containing source data to the peer processing unit corresponding to the current processing unit in the target node using the current processing unit of the source node. The data packet also includes synchronization information, which includes the target location information to which the source data is to be delivered.
[0104] The second sending module 720 is used to send source data to at least one target processing unit among the target nodes indicated by the target location information using the peer processing unit.
[0105] The processing module 730 is used to process the source data using the expert sub-models of at least one target processing unit to obtain at least one processing result.
[0106] The first merging module 740 is used to obtain the processing results of the expert sub-model of each target processing unit using the peer processing unit, and to perform a merging operation on the processing results to obtain the target data.
[0107] The third sending module 750 is used to send target data to the current processing unit using the peer processing unit.
[0108] According to embodiments of this disclosure, the synchronization information further includes the weight coefficients of the expert sub-models in the target processing unit indicated by the target location information. The first merging module 740 is further configured to obtain the processing results of the expert sub-models of each target processing unit using the peer processing unit, and to perform a merging operation on the processing results according to the weight coefficients of the expert sub-models to obtain the target data.
[0109] According to embodiments of this disclosure, the target location information further indicates multiple target nodes. The data processing apparatus 700 for the hybrid expert model also includes a second merging module.
[0110] The second merging module is used to merge target data from multiple target nodes using the current processing unit to obtain the target processing result.
[0111] The first sending module 710 is used to send data packets from the global memory of the current processing unit to the inter-machine shared memory of the current processing unit using the current processing unit; and to send data packets from the inter-machine shared memory of the current processing unit to the inter-machine shared memory of the peer processing unit using the current processing unit.
[0112] The second sending module 720 is used to send data packets from the inter-machine shared memory of the peer processing unit to the intra-machine shared memory of the peer processing unit using the peer processing unit; and to send data packets from the intra-machine shared memory of the peer processing unit to the intra-machine shared memory of at least one target processing unit using the peer processing unit.
[0113] The processing module 730 is used to send data packets from the shared memory of the target processing unit to the global memory of the target processing unit for each target processing unit; to parse the data packets in the global memory of the target processing unit to obtain the source data; and to process the source data using the expert sub-model of the target processing unit to obtain the processing result of the expert sub-model of the target processing unit.
[0114] The first merging module 740 is used to send the processing results of the expert sub-model of the target processing unit from the global memory of the target processing unit to the internal shared memory of the target processing unit; to send the processing results from the internal shared memory of the target processing unit to the internal shared memory of the peer processing unit; and to perform a merging operation on at least one processing result from at least one target processing unit in the internal shared memory of the peer processing unit to obtain the target data.
[0115] The third sending module 750 is used to send target data from the intra-machine shared memory of the peer processing unit to the inter-machine shared memory using the peer processing unit; and to send target data from the inter-machine shared memory of the peer processing unit to the inter-machine shared memory of the current processing unit using the peer processing unit.
[0116] The second merging module is used to merge target data from multiple target nodes in the inter-machine shared memory of the current processing unit to obtain the target processing result; and to send the target processing result from the inter-machine shared memory of the current processing unit to the global memory of the current processing unit.
[0117] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.
[0118] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.
[0119] like Figure 8As shown, device 800 includes a computing unit 801, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 802 or a computer program loaded from storage unit 808 into random access memory (RAM) 803. RAM 803 may also store various programs and data required for the operation of device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via bus 804. Input / output (I / O) interface 805 is also connected to bus 804.
[0120] Multiple components in device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of monitors, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.
[0121] The computing unit 801 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as the hybrid expert model data processing method. For example, in some embodiments, the hybrid expert model data processing method can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the hybrid expert model data processing method described above can be performed. Alternatively, in other embodiments, the computing unit 801 can be configured to perform the hybrid expert model data processing method by any other suitable means (e.g., by means of firmware).
[0122] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.
[0123] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.
[0124] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
[0125] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).
[0126] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.
[0127] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.
[0128] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.
[0129] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.
Claims
1. A data processing method for a hybrid expert model, wherein multiple expert sub-models of the hybrid expert model are distributed across multiple processing units on multiple nodes; the method includes: The source node's current processing unit sends a data packet containing source data to the peer processing unit in the target node corresponding to the current processing unit. The data packet also includes synchronization information, which includes the target location information to which the source data is to be delivered. The source data is sent by the peer processing unit to at least one target processing unit among the target nodes indicated by the target location information; The source data is processed using the expert sub-models of each of the at least one target processing unit to obtain at least one processing result; The processing results of the expert sub-model of each target processing unit are obtained using the peer processing unit, and the processing results are merged to obtain the target data; and The target data is sent to the current processing unit using the peer processing unit.
2. The method according to claim 1, wherein, The synchronization information also includes the weight coefficients of the expert sub-models in the target processing unit indicated by the target location information; the process of obtaining the processing results of the expert sub-models of each target processing unit using the peer processing unit, and merging the processing results to obtain the target data includes: The processing results of the expert sub-model of each target processing unit are obtained by the peer processing unit, and the processing results are merged according to the weight coefficients of the expert sub-model to obtain the target data.
3. The method according to claim 1, wherein, The target location information also indicates multiple target nodes; the method further includes: The current processing unit performs a merging operation on target data from multiple target nodes to obtain the target processing result.
4. The method according to claim 1, wherein, The step of using the current processing unit of the source node to send a data packet containing source data to the peer processing unit in the target node corresponding to the current processing unit includes: The data packet is sent from the global memory of the current processing unit to the inter-machine shared memory of the current processing unit using the current processing unit; The data packet is sent from the inter-machine shared memory of the current processing unit to the inter-machine shared memory of the peer processing unit using the current processing unit.
5. The method according to claim 4, wherein, The step of sending the source data to at least one of the target nodes indicated by the target location information using the peer processing unit includes: The data packet is sent from the inter-machine shared memory of the peer processing unit to the intra-machine shared memory of the peer processing unit using the peer processing unit. The peer processing unit uses the peer processing unit to send the data packet from the peer processing unit's internal shared memory to the respective internal shared memory of the at least one target processing unit.
6. The method according to claim 5, wherein, The step of processing the source data using the expert sub-models of each of the at least one target processing unit to obtain at least one processing result includes: for each target processing unit, The target processing unit uses the target processing unit to send the data packet from the target processing unit's internal shared memory to the target processing unit's global memory; The target processing unit parses the data packets in its global memory to obtain the source data; The source data is processed using the expert sub-model of the target processing unit to obtain the processing result of the expert sub-model of the target processing unit.
7. The method according to claim 1, wherein, The process of obtaining the processing results of the expert sub-model of each target processing unit using the peer processing unit, and merging the processing results to obtain the target data includes: for each target processing unit, The processing results of the expert sub-model of the target processing unit are sent from the global memory of the target processing unit to the on-machine shared memory of the target processing unit. The target processing unit uses its internal shared memory to send the processing result from the target processing unit's internal shared memory to the peer processing unit's internal shared memory. The target data is obtained by merging at least one processing result from at least one target processing unit in the shared memory of the peer processing unit using the peer processing unit.
8. The method according to claim 7, wherein, The step of sending the target data to the current processing unit using the peer processing unit includes: The target data is sent from the on-machine shared memory of the peer processing unit to the inter-machine shared memory using the peer processing unit. The target data is sent from the inter-machine shared memory of the peer processing unit to the inter-machine shared memory of the current processing unit using the peer processing unit.
9. The method according to claim 8, wherein, The target location information also indicates multiple target nodes; the method further includes: The target data from multiple target nodes are merged in the inter-machine shared memory of the current processing unit to obtain the target processing result. The target processing result is sent from the inter-machine shared memory of the current processing unit to the global memory of the current processing unit using the current processing unit.
10. The method according to claim 1, wherein, The correspondence between the current processing unit and the peer processing unit includes the fact that the number of the current processing unit on the source node is consistent with the number of the peer processing unit on the target node.
11. A data processing apparatus for a hybrid expert model, wherein multiple expert sub-models of the hybrid expert model are distributed across multiple node processing units; the apparatus comprises: The first sending module is used to send a data packet containing source data to the peer processing unit in the target node corresponding to the current processing unit using the current processing unit of the source node. The data packet also includes synchronization information, which includes the target location information to which the source data is to be delivered. The second sending module is used to send the source data to at least one target processing unit among the target nodes indicated by the target location information using the peer processing unit; The processing module is used to process the source data using the expert sub-models of each of the at least one target processing unit to obtain at least one processing result; The first merging module is used to obtain the processing results of the expert sub-models of each target processing unit using the peer processing unit, and to perform a merging operation on the processing results to obtain the target data; and The third sending module is used to send the target data to the current processing unit using the peer processing unit.
12. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 10.
13. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 10.
14. A computer program product comprising a computer program stored on at least one of a readable storage medium and an electronic device, the computer program implementing the method according to any one of claims 1 to 10 when executed by a processor.