Data processing method and system and related equipment
By executing computation and communication tasks in parallel in a distributed computing system with hybrid expert models, the problem of low resource utilization is solved, and more efficient model training and inference are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HUAWEI TECH CO LTD
- Filing Date
- 2024-10-17
- Publication Date
- 2026-04-17
Smart Images
Figure CN121882129A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer technology, and in particular to a data processing method, system and related equipment. Background Technology
[0002] The Mixed Expert Model (MOE) is a deep learning framework that extends the feed-forward network (FFN) layer of a transformer model into multiple FFNs, each representing an expert model. Each FFN processes the input tokens. In the MOE model, each input token is not processed by all expert models. Before being input into an FFN, a token passes through a gating model. The gating model calculates the weights of each expert model for that token, and then selects at least one expert model with the largest weight from among the multiple expert models to process that token. Finally, each token is input into its corresponding expert model for processing.
[0003] As current models grow increasingly large, training and inference are typically performed using multiple computing nodes. This process involves communication between these nodes. For example, if multiple expert models are deployed across multiple computing nodes, each node receives multiple tokens as input. The nodes need to calculate the weights of each expert model for each token using a gating model. After each token is assigned a target expert model, the computing node containing that target model might differ from the node where the token was input. This necessitates communication between the nodes: sending each token to the corresponding target expert model and synchronizing information about which expert models should process each token with other nodes. This process involves multiple computations and communications, currently executed serially, resulting in low utilization of computational and communication resources across the nodes. Summary of the Invention
[0004] This application provides a data processing method, system, and related equipment that enables multiple calculation and communication processes involved in a scenario using a hybrid expert model to be executed in parallel, thereby improving the utilization rate of the system's computing and communication resources and increasing the efficiency of data processing.
[0005] In a first aspect, this application provides a data processing method applied to a distributed computing system. The distributed computing system includes multiple computing nodes, each of which deploys at least one expert model. A total of m expert models are deployed across the multiple computing nodes. These m expert models are used to process input tokens. A first computing node executes a first computing task to determine m weights corresponding to each of the multiple tokens input to the first computing node. The m weights corresponding to the first token include the weights of each expert model among the m experts for the first token. The first computing node is one of the multiple computing nodes, and the first token is any one of the multiple tokens. During the execution of the first computing task, the first computing node initiates a first communication task to synchronize the multiple tokens with other computing nodes in the distributed computing system.
[0006] In scenarios applying the MOE model, each input token is not processed by all expert models. Before a token is input into an expert model, the computing node first calculates the weight of that token for each expert model, then selects at least one expert model with a higher weight from among the multiple expert models to process the token, and finally inputs each token into its corresponding expert model for processing. The aforementioned multiple computing nodes deploy m expert models. For multiple tokens input to the first computing node, the first computing node first executes the first computation task, calculating the weight of each expert model for each token to determine which target expert models will ultimately process each token. Since the target expert model corresponding to a token input to the first computing node may not be deployed on the first computing node, the first computing node needs to synchronize the tokens input to it with other computing nodes in the distributed computing system. During the execution of the first computing task, the first computing node can initiate the execution of the first communication task, synchronizing multiple tokens received with other computing nodes in the distributed computing system. This enables parallel execution of computing and communication, improving the utilization rate of computing and communication resources in the distributed computing system, increasing the efficiency of data processing, and enhancing the efficiency of training and inference.
[0007] In one possible implementation, after the first computing node completes the first computing task, it executes a second computing task, which includes: determining the model information corresponding to the first token based on the m weights corresponding to the first token; wherein the model information includes the identifiers of K expert models determined from the aforementioned m expert models for processing the first token, and the weights of these K expert models for the first token; where K is less than m. It should be understood that the first computing node performs this operation on all tokens among the aforementioned multiple tokens.
[0008] After completing the first computing task, the first computing node begins to execute the second computing task, determining the K expert models to process each token.
[0009] In one possible implementation, the time when the first computing node executes the first communication task overlaps with the time when it executes the second computing task.
[0010] The first computing node initiates the execution of the first communication task while performing the first computing task. If the first communication task has not been completed when the first computing task is completed, the first computing node continues to execute the first communication task while performing the second computing task.
[0011] In one possible implementation, the above method further includes: after the first computing node completes the second computing task, it executes a second communication task, which includes synchronizing the identifiers of K expert models used to process the first token with other computing nodes.
[0012] If the first computing node completes the second computing task while the first communication task is already completed, then the first computing node can start executing the second communication task and synchronize the results obtained from the second computing task with other computing nodes. If the first computing node completes the second computing task while the first communication task is not yet completed, then the first computing node continues to execute the first communication task, and starts executing the second communication task after completing the first communication task. This maintains the parallel execution of computing and communication, making full use of the computing and communication resources of the distributed computing system and improving resource utilization.
[0013] In one possible implementation, the above method further includes: after the first computing node completes the second computing task, it executes the third computing task to normalize the weights of the first token by the K expert models and obtain the normalization result; wherein the time for the first computing node to execute the third computing task overlaps with the time for executing the second communication task.
[0014] After the first computing node finishes executing the second computing task, it begins executing the third computing task. If the first computing node starts executing the second communication task at this time, the second communication task and the third computing task are executed in parallel, which can maintain the parallel execution of computing and communication, make full use of the computing and communication resources of the distributed computing system, and improve resource utilization.
[0015] In one possible implementation, the first computing node begins executing the third computing task. The first computing node may also be executing the first communication task. In this case, the first communication task and the third computing task are executed in parallel. After the first communication task ends, the first computing node starts executing the second communication task.
[0016] In one possible implementation, the method further includes: after completing the third computing task, the first computing node executes a third communication task to synchronize the normalization result with other computing nodes in the distributed computing system. The first computing node initiates the execution of the third communication task after executing the third computing task to synchronize the result of the third computing task with other computing nodes. It should be noted that if the second communication task has not yet been executed or has not been completed when the first computing node completes the third computing task, the first computing node waits for the second communication task to complete before starting to execute the third communication task.
[0017] In one possible implementation, the above method further includes: after the first computing node completes the second communication task, it executes the fourth computing task, and determines at least one token to be processed by the at least one expert model deployed in the first computing node based on at least one expert model deployed in the first computing node and the model information corresponding to each token; wherein the time when the first computing node executes the fourth computing task overlaps with the time when it executes the third communication task.
[0018] In this application, the first computing node needs the results obtained from the second computing task by all computing nodes to execute the fourth computing task. Therefore, the first computing node only starts executing the fourth computing task after completing the second communication task. Thus, while the first computing node is executing the fourth computing task, it is also executing the third communication task, thereby achieving parallel execution of computing and communication, making full use of the computing and communication resources of the distributed computing system, and improving resource utilization.
[0019] In one possible implementation, the first computing node mentioned above can be a graphics processing unit (GPU), a neural network processing unit (NPU), a tensor processing unit (TPU), or other dedicated AI processing chip.
[0020] Secondly, this application provides a distributed computing system, which includes multiple computing nodes, and any one of the multiple computing nodes is used to perform the operation performed by the first computing node as described in the first aspect or any possible implementation of the first aspect.
[0021] Thirdly, this application provides a computing node applied to a distributed computing system. The distributed computing system includes multiple computing nodes, each of which deploys at least one expert model. A total of m expert models are deployed within the multiple computing nodes. These m expert models are used to process input tokens. The first computing node in the distributed computing system includes a processing module and a communication module. The processing module is used to execute a first computing task, determining m weights corresponding to each token among the multiple tokens input to the first computing node. The m weights corresponding to the first token include the weights of each expert model among the m experts for the first token. The first computing node is one of the multiple computing nodes, and the first token is one of the multiple tokens. The communication module is used to execute a first communication task during the execution of the first computing task, synchronizing the multiple tokens with other computing nodes in the distributed computing system.
[0022] In one possible implementation, the processing module is further configured to execute a second computation task after completing the first computation task, and determine the model information corresponding to the first token based on the m weights corresponding to the first token; wherein the model information includes the identifiers of K expert models determined from the m expert models for processing the first token and the weights of these K expert models for the first token; wherein the first token is one of multiple tokens input to the first computation node, and K is less than m.
[0023] In one possible implementation, the time when the first computing node executes the first communication task overlaps with the time when it executes the second computing task.
[0024] In one possible implementation, the communication module is further configured to execute a second communication task after the processing module has completed the second computation task. The second communication task includes synchronizing the identifiers of the K expert models that process the first token in the model information with other computing nodes.
[0025] In one possible implementation, the above processing module is further used to execute a third computation task after the second computation task is completed, and to normalize the weights of the K expert models corresponding to each token to obtain a normalized result; wherein the time for the first computing node to execute the third computation task overlaps with the time for executing the second communication task.
[0026] In one possible implementation, the communication module is also used to perform a third communication task after the processing module has completed the third computation task, to synchronize the normalization results with other computing nodes.
[0027] In one possible implementation, the processing module is further configured to execute a fourth computation task after the communication module has completed the second communication task, and to determine at least one token to be processed by the at least one expert model deployed in the first computing node based on at least one expert model deployed in the first computing node and the model information corresponding to each token; wherein the time when the first computing node executes the fourth computation task overlaps with the time when it executes the third communication task.
[0028] In one possible implementation, the computing node can be a graphics processing unit (GPU), a neural network processing unit (NPU), a tensor processing unit (TPU), or other dedicated AI processing chips.
[0029] Fourthly, this application provides a computing device including a processor, a plurality of computing nodes and a memory, wherein the plurality of computing nodes are configured to execute instructions stored in the memory to implement the method implemented by the first computing node as described in the first aspect or any possible implementation thereof.
[0030] Fifthly, this application provides a computer-readable storage medium including computer program instructions that, when executed by a computing device, cause the computing device to implement the method implemented by the first computing node as described in the first aspect or any possible implementation of the first aspect.
[0031] Sixthly, this application provides a computer program product, which includes a computer program that, when run by a computing device, implements the method implemented by the first computing node as described in the first aspect or any possible implementation of the first aspect.
[0032] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description
[0033] Figure 1 This is a schematic diagram of an expert model for determining the correspondence of word units provided in this application;
[0034] Figure 2 This is a schematic diagram of a data processing procedure provided in this application;
[0035] Figure 3 This is a schematic diagram of a data flow for computing and communication provided in this application;
[0036] Figure 4 This is a schematic diagram of a distributed computing system provided in this application;
[0037] Figure 5 This is a flowchart illustrating a data processing method provided in this application;
[0038] Figure 6 This is a schematic diagram of another data processing procedure provided in this application;
[0039] Figure 7 This is another data flow diagram for computing and communication provided in this application;
[0040] Figure 8 This is a schematic diagram of a computing node provided in this application;
[0041] Figure 9 This is a schematic diagram of a computing device provided in this application. Detailed Implementation
[0042] The distributed computing method provided in this application will be described below with reference to the accompanying drawings.
[0043] With the development of information technology, the processing of large-scale data has become a key focus in various fields. The computational processing of large-scale data usually requires distributed computing systems for distributed computing. For example, in the field of artificial intelligence (AI), efficiency is improved by deploying models to multiple computing nodes in a distributed computing system, with each computing node responsible for a portion of the computation in the model training and inference process.
[0044] The Mixed Expert Model (MOE) is a deep learning framework that extends the feed-forward network (FFN) layer of the transformer model into multiple FFNs, each representing an expert model. Each expert model processes the input tokens. In the Mixed Expert Model, each input token is not processed by all experts. Before being input into an FFN, a token passes through a gating model, which determines which expert models will process the token.
[0045] For example, see Figure 1 , Figure 1 This is a schematic diagram of an expert model for determining the correspondence of lexical units provided in this application. Figure 1 In this dataset, the input data includes 8 tokens, and the hybrid expert model includes 4 experts. The 8 tokens are represented by identifiers 0 to 7, and the 4 experts are represented by identifiers 0 to 3. The process of determining which expert models process each token based on the gating model is as follows.
[0046] (1) First, the computing nodes calculate the weights (logits). The computing nodes calculate the weight of each expert model for each token using a gating model. Then, they calculate the top K expert models. For any token, the computing nodes select the K expert models with the highest weights for that token from among multiple experts. These K expert models are the expert models used to process that token. The computing nodes record the identifiers (top_indices) of the K expert models corresponding to that token and the weights (top_logits) of these K experts for that token. The computing nodes then normalize the weights. For any token, the weights of the K expert models corresponding to that token are normalized to obtain the normalized weight (score). For example... Figure 1 As shown, Figure 1 In this context, Sij represents the weight of the normalized expert model j on token i, where i is an integer greater than or equal to 0 and less than or equal to 7, and j is an integer greater than or equal to 0 and less than or equal to 3. Figure 1 Taking K=2 as an example, for token 0, token 0 is processed by expert models labeled 2 and 3. After normalization, the weight of expert model 2 for token 0 is S02, and the weight of expert model 3 for token 0 is S03.
[0047] (2) After determining which expert models will process each token, the input tokens are reordered. Based on the K expert models corresponding to each token, the tokens that each expert model needs to process are determined. For example... Figure 1 As shown, expert model 0 is used to process tokens 1, 3, 5, and 7; expert model 1 is used to process tokens 2, 4, 5, and 6; home model 2 is used to process tokens 0, 1, 3, and 6; home model 1 is used to process tokens 0, 2, 4, and 7; each expert model needs to process 4 tokens, and each token is processed by two expert models.
[0048] (3) Input a token into the corresponding K expert models. Based on the output of each expert model and the weight of each expert model on the token, perform a weighted operation on the output of each expert model to obtain a new token corresponding to the token. Figure 1 In this context, i_O represents the input token i, and i_N represents the output result after token i is processed by K expert models.
[0049] As current models become increasingly large, training and inference are typically achieved using multiple computing nodes. For example, in the MOE model mentioned above, the multiple expert models included in the MOE model are deployed across multiple computing nodes, with each computing node deploying a portion of these expert models. For instance, if the MOE model has 8 expert models and the distributed computing system includes 4 computing nodes, then 2 expert models can be deployed on each computing node, with each model containing a gating module. For the input token to be processed, the token is divided into 4 parts and input into the 4 computing nodes for processing. The processing procedure on each computing node can be found in [reference needed]. Figure 1 The corresponding processing procedure.
[0050] Since multiple expert models are distributed across multiple computing nodes, after a computing node determines the K expert models corresponding to each token, the computing node containing the K expert models corresponding to a token may not be the same computing node from which the token was input. This involves communication between multiple computing nodes to send each token to the computing node containing the corresponding expert model. The above process involves multiple computations and multiple communications. The following section details the above process when multiple computing nodes are involved. See [link to documentation] Figure 2 , Figure 2 This is a schematic diagram of a data processing procedure provided in this application. For example... Figure 2 As shown, the distributed computing system includes four computing nodes: node 0, node 1, node 2, and node 3. The input data for node 0 is data 0, the input data for node 1 is data 1, the input data for node 2 is data 2, and the input data for node 3 is data 3. Each node's input data includes multiple tokens; for example, each node's input data includes 10 tokens. The following describes the process by which each token is processed by multiple expert models to output a new token. This process includes the following:
[0051] (1) Computation nodes calculate weights (logits). For each computation node, the node calculates the weights of each expert model for each input token using the gating model. For example... Figure 2 In this context, `logits_i` includes the weights of each expert model for each token in `data_i`. For example, for `logits_0`, node 0 calculates the weights of each expert model for each token in `data_0` using a gating model, thus obtaining `logits_0`. If `data_0` contains 10 tokens and the MOE model includes 8 expert models, then `logits_0` contains 80 weights, with 8 weights corresponding to each token, representing the weights of the 8 expert models deployed on different computing nodes for that token.
[0052] (2) The computing node determines the model information corresponding to each token. That is, the computing node calculates the top K expert models corresponding to each token. For any token, the computing node selects the K expert models with the highest weights for that token from multiple experts. These K expert models are the expert models used to process that token. The computing node records the identifiers (top_indices) of the K expert models corresponding to that token and the weights (top_logits) of these K experts for that token; this is the model information corresponding to that token. If topKi includes top_indicessi and top_logitssi, where top_indicessi includes the identifiers of the K expert models corresponding to each token in data i, and top_logitssi includes the weights of the K expert models corresponding to each token in data i for that token, then... Figure 2In the data, topK 0 includes top_indices 0 and top_logits 0. Top_indices 0 includes the identifiers of the K expert models corresponding to each token in data 0, and top_logits 0 includes the weights of the K expert models for each token in data 0.
[0053] If K equals 2, data0 includes 10 tokens, then top_indices 0 includes 20 expert model identifiers, with each token corresponding to 2 expert model identifiers; top_logits 0 includes 20 weights, with each token corresponding to the weights of 2 expert models for that token.
[0054] (3) Normalization. For any token, after determining the K expert models to process the token through the gating model, the weights of the K expert models for the token are normalized to obtain the normalized weights (scores). Figure 2 In this context, `scoresi` includes the weights after normalization of the weights in `top_logits 0`. During the normalization operation, the weights of the K expert models corresponding to the same token are normalized on a token-by-token basis.
[0055] (4) Input data is synchronized among computing nodes, ensuring each node has complete input data (data). Complete data includes data 0, data 1, data2, and data3 mentioned above. Data synchronization between multiple computing nodes is achieved through allgather communication. For example... Figure 2 As shown, after synchronizing input data between computing nodes, each computing node includes data 0, data 1, data2, and data3. In this application, the data synchronization operation between multiple computing nodes is denoted as the allgather_data operation.
[0056] (5) Synchronizing the expert model identifiers for processing each token among computing nodes. That is, computing nodes synchronize the aforementioned top_indices in each computing node through the allgather operation, ensuring that each computing node contains complete expert model identifiers (top_indices), including top_indices 0, top_indices 1, top_indices 2, and top_indices 3. In this application, the operation of synchronizing the expert model identifiers for processing each token among computing nodes is denoted as the allgather_indices operation.
[0057] (6) Synchronizing the normalized weights (scores) among computing nodes. That is, the computing nodes synchronize the aforementioned scores (scores) across each node through the `allgather` operation, ensuring that each computing node contains the complete normalized weights (scores). The complete normalized weights (scores) include scores 0, scores 1, scores 2, and scores 3. In this application, the operation of synchronizing the normalized weights among computing nodes is denoted as the `allgather_scores` operation.
[0058] (7) Each computing node performs a reordering operation to determine the tokens that the expert models deployed within each computing node need to process. After the above three synchronization operations, a computing node contains complete input data. However, the expert models deployed within each computing node do not completely input every token in the data. For example, data 0, data 1, data2, and data3 each contain 10 tokens, and each computing node has 2 expert models deployed. After the synchronization operation, each computing node contains 10 tokens, but an expert model within a computing node may only need to process 5 of them. Therefore, the 2 expert models deployed within each computing node can process a maximum of 10 tokens. A computing node determines the tokens that each expert model within the computing node needs to process based on the identifier of the expert model corresponding to each token and the expert models deployed within the computing node.
[0059] The computation and communication process of the aforementioned MOE model includes multiple computation tasks and multiple communication tasks. The computation tasks include calculating weights (logits), computing the top-K expert model, normalization, and reordering operations at the computing nodes. The communication tasks include the `allgather_data` operation, the `allgather_indices` operation, and the `allgather_scores` operation. These multiple computations and communications are executed serially, as follows: Figure 3 As shown, Figure 3 This is a schematic diagram of the computation and communication data flow provided in this application. Each computing node first sequentially executes the calculation of weights (logits), the calculation of the top K expert models, and normalization processing, followed by the sequential execution of the allgather_data, allgather_indices, and allgather_scores operations, and finally a reordering operation. This serial execution method cannot fully utilize the computational and communication resources of the distributed computing system, resulting in low efficiency in model training and inference.
[0060] To address the aforementioned issues, this application provides a data processing method that, through reasonable scheduling of multiple computational and communication tasks involved in the computation and communication processes of the MOE model, enables parallel execution of computational and communication tasks, thereby improving the utilization rate of computational and communication resources in the distributed computing system and enhancing the efficiency of model training and inference.
[0061] See Figure 4 , Figure 4This is a schematic diagram of a distributed computing system provided in this application. The distributed computing system includes multiple computing nodes connected by interconnect links. Each computing node is used to perform matrix multiplication between two matrices and to communicate with other computing nodes. These computing nodes can be graphics processing units (GPUs), neural network processing units (NPUs), tensor processing units (TPUs), or other dedicated AI processing chips. The multiple computing nodes can be located in the same computing device or in different computing devices. If the multiple computing nodes are located in the same computing device, they can be interconnected via interconnect links. These interconnect links can be buses, such as high-speed serial computer expansion (PCIe) buses, NVLink buses, or other buses; they can also be switching chips, such as NVSwitch chips or other chips capable of connecting multiple computing nodes and providing data communication functions. This application does not specifically limit the topology between the multiple computing nodes. If the aforementioned multiple computing nodes are located in multiple different computing devices, then each computing device includes at least one computing node. These multiple computing devices are connected by interconnection links, and the computing nodes in the multiple computing devices can communicate with each other through interconnection links. This application does not specifically limit the topology between the multiple computing devices.
[0062] The data processing method provided in this application is described below with reference to the accompanying drawings, and is applied to... Figure 4 The distributed computing system shown has multiple expert models and one gating model deployed within each computing node, and a total of m expert models deployed across the multiple computing nodes. It should be understood that the number of expert models deployed within each computing node can be the same or different; this application does not impose a specific limitation in this regard. See [link to relevant documentation]. Figure 5 , Figure 5 This is a flowchart illustrating a data processing method provided in this application, which includes the following steps S501 to S507.
[0063] S501. The first computing node executes the first computing task to determine the m weights corresponding to each of the multiple tokens input to the first computing node.
[0064] In this application, the distributed computing system includes n computing nodes, and each computing node includes multiple tokens. For the first computing node in the distributed computing system, the first computing node executes a first computing task, which calculates the weights of each expert model with respect to each token input to the first computing node through a gating model. That is, the first computing task is to calculate the weights (logits).
[0065] Since there are m expert models deployed in the distributed computing system, for a token, the first computing node calculates the weights of each of the m expert models for that token, thus obtaining the m weights corresponding to that token. Each weight corresponding to that token represents the weight of an expert model for that token.
[0066] See Figure 6 , Figure 6 This is a schematic diagram of another data processing procedure provided in this application. Figure 6 In this example, a distributed computing system consists of four computing nodes: N1 (first node), N2 (second node), N3 (third node), and N4 (fourth node). The input data for the first node is data 0, for the second node it is data 1, for the third node it is data 2, and for the fourth node it is data 3. Each node's input data includes multiple tokens; for example, each node's input data may include 10 tokens.
[0067] It should be understood that while the first computing node is executing the first computational task, other computing nodes in the distributed computing system are also executing the first computational task. For example, the second computing node executes the first computational task, obtaining the weights of each expert model for each token input to the second computing node. Figure 6 As shown, the first computing node performs the first computing task, calculating the weight of each expert model in the m expert models for each token in data 0, and obtaining the weight set logits 0; the first computing node performs the first computing task and obtains logits 1. Figure 5 In this context, logits i includes the weights of each expert model for each token in data i. For logits 0, if data0 includes 10 tokens and m is 8, meaning the MOE model includes 8 expert models, then logits 0 includes 80 weights, with each token corresponding to 8 weights, representing the weights of the 8 expert models deployed on different computing nodes for that token.
[0068] S502. During the execution of the first computing task, the first computing node initiates the execution of the first communication task, which is used to synchronize multiple tokens input to the first computing node with other computing nodes in the distributed computing system.
[0069] In this application, since each computing node in the distributed computing system deploys a subset of m expert models (for example, if m is 8 and the distributed computing system includes 4 computing nodes, then each computing node deploys 2 expert models), a gating model is needed to determine which expert models will process the token. If the expert models for processing a token are deployed on the first and second computing nodes, and the token is input to the third computing node, the gating model in the third computing node determines the expert model that will process the token. Since neither the first nor the second computing nodes contain the token, this application requires each computing node to synchronize the tokens it receives, ensuring that each computing node contains all the tokens.
[0070] Before executing the first computation task, the first computing node has already obtained multiple tokens that need to be processed. Similarly, other computing nodes in the distributed computing system also obtain their own multiple tokens that need to be processed. Therefore, during the execution of the first computation task, each computing node can initiate a first communication task to synchronize its multiple tokens through allgather communication, ensuring that each computing node receives all the tokens. For example, the first computing node initiates the first communication task to send its multiple tokens to other computing nodes. In this application, the first communication task is used to synchronize the input data of each computing node through allgather communication, and this first communication task is referred to as allgather_data.
[0071] For example, such as Figure 6 As shown, the input data for the first computing node is data 0, the input data for the second computing node is data 1, the input data for the third computing node is data 2, and the input data for the fourth computing node is data 3. Each computing node's input data includes multiple tokens. Allgather communication is performed between the computing nodes to synchronize their received input data, ensuring that each computing node contains data 0, data 1, data 2, and data 3.
[0072] S503. After completing the first computing task, the first computing node executes the second computing task to determine the model information corresponding to each token input to the first computing node.
[0073] In this application, the model information corresponding to a token includes the identifiers of K expert models determined from m expert models for processing the token, as well as the weights of these K expert models for the token. After executing the first computation task, the first computing node obtains m weights corresponding to each token, and one of the m weights corresponding to a token represents the weight of an expert model for that token.
[0074] After completing the first computation task and obtaining the m weights corresponding to each token, the first computation node executes the second computation task to calculate the top K expert models corresponding to each token and records the model information corresponding to the top K expert models. Specifically, for a token, based on the weights of each expert model for that token, K expert models with larger weights are selected from the m expert models, and the token will subsequently be input into these K expert models for processing.
[0075] After determining the top K expert models corresponding to a token, the first computing node records the expert model identifiers (top_indices) and the weights (top_logits) of these K expert models for that token, which constitutes the model information corresponding to that token. The first computing node then executes a second computing task to obtain the K expert model identifiers and the weights of these K expert models for each token. In other words, the model information for a token includes the K expert model identifiers and K weights corresponding to that token.
[0076] For example, such as Figure 6As shown, `topKi` includes `top_indicesi` and `top_logitsi`. `top_indicesi` includes the K expert model identifiers corresponding to each token in `data i`, and `top_logitsi` includes the K weights corresponding to each token in `data i`. For example, `topK0` includes `top_indices0` and `top_logits0`. `top_indices0` includes the K expert model identifiers corresponding to each token in `data 0`, and `top_logits0` includes the K weights corresponding to each token in `data 0`. If K equals 2, and `data0` contains 10 tokens, then `top_indices0` contains 20 expert model identifiers, with one token corresponding to two expert model identifiers; `top_logits0` contains 20 weights, with one token corresponding to two expert model weights for that token. It should be understood that there may be duplicate identifiers among the 20 expert model identifiers included in `top_indices0`, for example, multiple tokens may need to be processed by the same expert model.
[0077] In one possible implementation, the time when the first computing node executes the first communication task may overlap with the time when it executes the second computing task. For example, when the first computing node finishes executing the first computing task, the first communication task has not yet been completed, and the first computing node starts executing the second computing task. In this case, the time when the first computing node executes the first communication task may overlap with the time when it executes the second computing task.
[0078] S504. After the first computing node executes the second computing task, it executes the second communication task.
[0079] In this application, after executing the second computing task, the first computing node executes a second communication task. This second communication task is used to synchronize the K expert model identifiers (top_indices) corresponding to each token with other computing nodes, so that each computing node stores information about the K expert models that indicate the processing of each token. For example... Figure 6As shown, each computing node performs a second communication task, synchronizing the top_indices included in topKi across all computing nodes via allgather communication. This ensures that each computing node contains complete expert model identifiers (top_indices), including top_indices 0, top_indices 1, top_indices 2, and top_indices 3. In this application, the operation of synchronizing the expert model identifiers for processing each token between computing nodes is denoted as the allgather_indices operation.
[0080] In this application, each computing node can execute the second communication task after it has completed the second computing task. If a computing node has not completed the first communication task after completing the second computing task, the computing node can wait for all computing nodes to complete the first communication task before executing the second communication task.
[0081] S505. After the first computing node executes the second computing task, it executes the third computing task, which normalizes the K weights corresponding to each token to obtain the normalized weights corresponding to each token.
[0082] In this application, for any token, after determining K expert models to process the token through a gating model, the weights of these K expert models for the token are normalized to obtain the normalized weights (scores) corresponding to the token. For multiple tokens input to the first computing node, the first computing node input normalizes the K weights corresponding to each token to obtain the normalized weights corresponding to the input data within the first computing node.
[0083] like Figure 6 As shown, scores i includes the weights after normalization of the weights in top_logitsi. During the normalization operation, the weights in top_logitsi are normalized in units of tokens, and the weights of the K expert models corresponding to the same token are normalized for that token. For example, normalizing top_logits 0 in the first computing node in units of tokens yields scores 0.
[0084] In one possible implementation, the time when the first computing node executes the second communication task may overlap with the time when it executes the third computing task, or they may not overlap. For example, if the first computing node starts executing the second communication task after completing the first communication task while executing the third computing task, then the execution time of the second communication task overlaps with the execution time of the third computing task. If the first communication task has a large communication volume, and the first computing node completes the third computing task before the first communication task is finished, then the execution time of the second communication task overlaps with the execution time of the third computing task.
[0085] S506. After the first computing node executes the third computing task, it executes the third communication task to synchronize the normalized results with other computing nodes.
[0086] After each computing node completes its third computation task, it begins its third communication task. This involves synchronizing the scores *i* obtained from the third computation task across computing nodes via the *allgather* operation. This ensures that each computing node contains the complete normalized weights (scores), with complete expert model identifiers including scores 0, scores 1, scores 2, and scores 3. In this application, the operation of synchronizing the normalized weights between computing nodes is denoted as the *allgather_scores* operation.
[0087] S507. After completing the third computing task, the first computing node executes the fourth computing task to determine the tokens that need to be processed by each expert model deployed within the first computing node.
[0088] After each computing node completes the third computation task, it performs a reordering operation. This reordering operation includes determining the tokens that the expert models deployed within each computing node need to process. After the three synchronization operations described above, a computing node contains the complete input data. However, the expert models deployed within each computing node do not process every token in the complete input data. For example, data 0, data 1, data2, and data3 each contain 10 tokens, and each computing node has 2 expert models deployed. After the Allgather_data synchronization operation, each computing node contains 40 tokens, but an expert model within a computing node may only need to process 5 of them. Therefore, the 2 expert models deployed within each computing node can process a maximum of 10 tokens. The first computing node determines the tokens that each expert model within the computing node needs to process based on the K expert model identifiers corresponding to each token and the expert models deployed within the computing node.
[0089] In one possible implementation, the time it takes for the first computing node to execute the third communication task may overlap with the time it takes to execute the fourth computing task.
[0090] For example, see Figure 7 , Figure 7 This is another data flow diagram for computing and communication provided in this application. Figure 5 As described in the relevant examples, in this application, the execution order of the multiple computing tasks and multiple communication tasks can be as follows: Figure 7 As shown, during the execution of the first computation task, the computing node begins executing the first communication task. After completing the first computation task, the computing node executes the second computation task while continuing to execute the first communication task. After the second computing node completes its second computation task, it executes the third computation task. At this point, since the first communication task is still not complete, the computing node continues to execute the first communication task until it is completed, after which it begins executing the second communication task. After the computing node completes its second communication task, since the third communication task is not yet fully completed, the computing node does not execute the third communication task immediately until it has completed the third computation task, after which it begins executing the third communication task.
[0091] It should be noted that the above Figure 5 The corresponding embodiment takes the first computing node in a distributed computing system as an example, and introduces the process by which the first computing node implements the data processing method provided in this application. The process by which other computing nodes in the distributed computing system execute the data processing method provided in this application is the same as that of the first computing node, except that the data being processed is different, and will not be described in detail here.
[0092] For the sake of simplicity, the above method embodiments are described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to this application. Therefore, other reasonable combinations of steps that those skilled in the art can conceive of based on the above description also fall within the scope of protection of this application.
[0093] For example, the above Figure 5 In a corresponding embodiment, the first computing node may start executing the first communication task only after starting the second computing task. For example, if the first communication task is completed when the first computing node finishes the second computing task, the first computing node can start executing the second communication task and synchronize the results obtained from the second computing task with other computing nodes. If the first communication task is not yet completed when the first computing node finishes the second computing task, the first computing node continues executing the first communication task, and starts executing the second communication task after completing the first communication task. This also maintains the parallel execution of computing and communication, fully utilizing the computing and communication resources of the distributed computing system and improving resource utilization. For example, if the first computing node starts executing the third computing task while still executing the first communication task, the first communication task and the third computing task are executed in parallel, and the first computing node starts executing the second communication task after the first communication task ends.
[0094] The above text combined Figures 1 to 7 This application details the data processing method it seeks to protect. Next, it will further combine... Figures 8 to 9 The computing nodes and computing devices that this application seeks to protect are described separately.
[0095] See Figure 8 , Figure 8 This is a schematic diagram of a computing node provided in this application. The computing node includes a processing module 810 and a communication module 820. The processing module 810 is used to execute a first computing task, determining m weights corresponding to each of a plurality of tokens input to the first computing node. The m weights corresponding to the first token include the weights of each expert model among the m experts for the first token. The first computing node is one of a plurality of computing nodes, and the first token is one of a plurality of tokens.
[0096] The communication module 820 is used to perform a first communication task during the execution of the first computing task, and to synchronize the multiple tokens with other computing nodes in the distributed computing system.
[0097] The aforementioned processing module 810 is further configured to execute a second computation task after completing the first computation task, and determine the model information corresponding to the first token based on the m weights corresponding to the first token; wherein the aforementioned model information includes the identifiers of the K expert models determined from the m expert models for processing the first token and the weights of these K expert models for the first token; wherein the first token is one of the multiple tokens input to the first computation node, and K is less than m.
[0098] In one possible implementation, the time when the first computing node executes the first communication task overlaps with the time when it executes the second computing task.
[0099] In one possible implementation, the communication module 820 is further configured to execute a second communication task after the processing module 810 has completed the second computation task. The second communication task includes synchronizing the identifiers of the K expert models that process the first token in the model information with other computing nodes.
[0100] In one possible implementation, the processing module 810 is further configured to execute a third computation task after the second computation task is completed, and to normalize the weights of the K expert models corresponding to each token to obtain a normalized result; wherein the time for the first computing node to execute the third computation task overlaps with the time for executing the second communication task.
[0101] In one possible implementation, the communication module 820 is also used to perform a third communication task after the processing module 810 has completed the third computing task, to synchronize the normalization results with other computing nodes.
[0102] In one possible implementation, the processing module 810 is further configured to execute a fourth computing task after the communication module 820 has completed the second communication task, and to determine at least one token to be processed by the at least one expert model deployed in the first computing node based on at least one expert model deployed in the first computing node and the model information corresponding to each token; wherein the time when the first computing node executes the fourth computing task overlaps with the time when it executes the third communication task.
[0103] The first, second, third, and fourth computational tasks can be referred to the above. Figure 5 The descriptions of the corresponding embodiments, the first communication task, the second communication task, and the third communication task can be referred to above. Figure 5 The operations performed by the processing module 810 can be referred to in the corresponding embodiments. Figure 5The computation-related operations performed by the first computing node in the corresponding embodiment, and the operations performed by the communication module 820, can be referred to as follows. Figure 5 The communication-related operations performed by the first computing node in the corresponding embodiment will not be described in detail here.
[0104] See Figure 9 , Figure 9 This is a schematic diagram of a computing device provided in this application, such as... Figure 9 As shown, the computing device 900 includes a bus 901, a processor 902, multiple computing nodes 903, a communication interface 904, and a memory 905. The processor 902, the multiple computing nodes 903, the communication interface 904, and the memory 905 communicate with each other via the bus 901. It should be understood that this application does not limit the number of processors 902, computing nodes 903, and memory 905 in the computing device 900. The computing device can be a server, such as a server in a cloud data center, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer or a laptop computer.
[0105] The 901 bus can be a Peripheral Component Interconnect Express (PCIe) bus, an Extended Industry Standard Architecture (EISA) bus, a Unified Bus (Ubus or UB), a Compute Express Link (CXL) bus, a Cache Coherent Interconnect for Accelerators (CCIX) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 9 The bus 901 is represented by only one line, but this does not mean that there is only one bus or one type of bus. Bus 901 may include a path for transmitting information between various components of computing device 900. Among them, a unified bus is, for example, the Lingqu bus.
[0106] The processor 902 may be a central processing unit (CPU), or may include a CPU and other hardware chips. The aforementioned hardware chips may be of various types, such as any one of the following: a programmable logic device (PLD), a complex programmable logic device (CPLD), a field programmable gate array (FPGA), or a digital signal processor (DSP). The computing device 900 may include one or more of the aforementioned types of hardware chips, or may include multiple types of the aforementioned hardware chips. This application embodiment does not make specific limitations.
[0107] The computing node 903 can be a graphics processing unit (GPU), a neural network processing unit (NPU), a tensor processing unit (TPU), or other dedicated AI processing chips, etc., and this application does not impose specific restrictions.
[0108] Memory 905 can be non-volatile memory, such as read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Memory 105 can also be volatile memory, which can be random access memory (RAM) used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM). In addition, the memory 905 can also be implemented using storage class memory (SCM), phase change memory (PCM), or other types of storage media.
[0109] It should be noted that the same type of storage medium can be configured in the same computing device to realize the function of memory 905, or two or more types of storage media can be configured to realize the function of memory 905. This application does not limit this.
[0110] The memory 905 stores executable program code, which the processor 902 executes to issue tasks to the computing node 903, enabling the computing node to perform tasks. Figures 5 to 7 The data processing methods described in the corresponding method embodiments.
[0111] The communication interface 904 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 900 and other devices or communication networks.
[0112] This application also provides a computer program product containing instructions, which may be a software or program product containing instructions capable of running on a computing device or stored on any usable medium. When the computer program product runs on the computing device, it causes the computing device to perform... Figures 5 to 7 The data processing methods described in the corresponding method embodiments.
[0113] This application also provides a computer-readable storage medium, which can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium, or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to implement... Figures 5 to 7 The data processing methods described in the corresponding method embodiments.
[0114] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this application.
Claims
1. A data processing method, characterized by, This method is applied to a distributed computing system, which includes multiple computing nodes. Each of these nodes deploys at least one expert model, and a total of m expert models are deployed across the nodes. These m expert models are used to process input tokens. The first computing node executes the first computing task to determine m weights corresponding to each of the multiple tokens input to the first computing node; wherein, the m weights corresponding to the first token include the weights of each expert model among the m experts for the first token, the first computing node is one of the multiple computing nodes, and the first token is one of the multiple tokens; During the execution of the first computing task by the first computing node, a first communication task is initiated, which is used to synchronize the multiple tokens with other computing nodes in the distributed computing system.
2. The method of claim 1, wherein, The method further includes: After the first computing node completes the first computing task, it executes the second computing task to determine the model information corresponding to the first token based on the m weights corresponding to the first token. The model information includes the identifiers of K expert models selected from the m expert models for processing the first token and the weights of the K expert models for the first token. K is less than m.
3. The method of claim 2, wherein, The time when the first computing node executes the first communication task overlaps with the time when it executes the second computing task.
4. The method according to claim 2 or 3, characterized in that, The method further includes: After the first computing node completes the second computing task, it executes the second communication task, which includes synchronizing the identifiers of the K expert models that process the first token with the other computing nodes.
5. The method of claim 4, wherein, The method further includes: After the first computing node completes the second computing task, it executes the third computing task to normalize the weights of the first token in the K expert models and obtain the normalization result. The time when the first computing node executes the third computing task overlaps with the time when it executes the second communication task.
6. The method of claim 5, wherein, The method further includes: After completing the third computing task, the first computing node performs the third communication task to synchronize the normalization result with the other computing nodes.
7. The method of claim 6, wherein, The method further includes: After the first computing node completes the second communication task, it executes the fourth computing task to determine at least one token to be processed by the at least one expert model deployed in the first computing node and the model information corresponding to each token, based on at least one expert model deployed in the first computing node. The time when the first computing node executes the fourth computing task overlaps with the time when it executes the third communication task.
8. The method according to any one of claims 1 to 7, characterized in that, The computing nodes include graphics processing units (GPUs), neural network processing units (NPUs), tensor processing units (TPUs), or other dedicated AI processing chips.
9. A distributed computing system, characterized by The distributed computing system includes multiple computing nodes, and any one of the multiple computing nodes is used to perform the operation performed by the first computing node as described in any one of claims 1-8.
10. A computing node, characterized in that, This invention is applied to a distributed computing system, which includes multiple computing nodes. Each computing node deploys at least one expert model, and a total of m expert models are deployed across the multiple computing nodes. These m expert models are used to process input tokens. The first computing node in the distributed computing system includes a processing module and a communication module. The processing module is configured to perform a first computation task, determining m weights corresponding to each token among the multiple tokens input to the first computation node; wherein, the m weights corresponding to the first token include the weights of each expert model among the m experts for the first token, the first computation node is one of the multiple computation nodes, and the first token is one of the multiple tokens; The communication module is used to perform a first communication task during the execution of the first computing task, and to synchronize the multiple tokens with other computing nodes in the distributed computing system.
11. The computing node of claim 10, wherein, The processing module is also used for: After the first computation task is completed, the second computation task is executed to determine the model information corresponding to the first token based on the m weights corresponding to the first token. The model information includes the identifiers of K expert models determined from the m expert models for processing the first token and the weights of the K expert models on the first token. The first token is one of the plurality of tokens input to the first computation node, and K is less than m.
12. The computing node of claim 11, wherein, The time when the first computing node executes the first communication task overlaps with the time when it executes the second computing task.
13. The computing node of claim 11 or 12, wherein, The communication module is also used for: After the processing module completes the second computation task, it executes the second communication task, which includes synchronizing the identifiers of the K expert models that processed the first token with the other computing nodes.
14. The computing node of claim 13, wherein, The processing module is also used for: After the second calculation task is completed, the third calculation task is executed to normalize the weights of the K expert models corresponding to each token and obtain the normalization result. The time when the first computing node executes the third computing task overlaps with the time when it executes the second communication task.
15. The computing node according to claim 14, characterized in that, The communication module is also used for: After the processing module completes the third computation task, it executes the third communication task to synchronize the normalization result with the other computing nodes.
16. The computing node according to claim 15, characterized in that, The processing module is also used for: After the communication module completes the second communication task, the fourth computing task is executed. Based on the at least one expert model deployed in the first computing node and the model information corresponding to each token, the at least one token to be processed by the at least one expert model deployed in the first computing node is determined. The time when the first computing node executes the fourth computing task overlaps with the time when it executes the third communication task.
17. The computing node according to any one of claims 10-16, characterized in that, The computing nodes include graphics processing units (GPUs), neural network processing units (NPUs), tensor processing units (TPUs), or other dedicated AI processing chips.
18. A computer-readable storage medium, characterized in that, It includes computer program instructions that, when executed by a computing device, cause the computing device to perform the method as described in any one of claims 1 to 9.