Inference method, device and equipment of large model and storage medium

Abstract

Embodiments of the present application provide a large model inference method and device, equipment and a storage medium, including in the network layer, calculating the corresponding hidden state result according to a plurality of to-be-processed word units and attention mechanism; determining the routing decision corresponding to the plurality of to-be-processed word units according to the hidden state result and the gating network; creating a plurality of parallel processing task queues; sending the hidden state result to the first task queue, and sending the routing decision and the corresponding hidden state result to the second task queue; determining the first calculation result through the hidden state result in the first task queue and the shared expert network; sending the corresponding hidden state result to the GPU where the corresponding routing expert network is deployed through the routing decision in the second task queue to calculate the second calculation result; and determining the output result of the network layer according to the first calculation result and the second calculation result. The mutual waiting problem of communication and calculation, and the mutual waiting problem of calculation and communication in the execution model are eliminated.

Description

Technical Field

[0001] This application relates to the field of computers, and more particularly to a method, apparatus, device, and storage medium for reasoning large models. Background Technology

[0002] Current large language models mostly adopt a Mixture of Experts (MoE) architecture. During the communication phase of MoE inference, each GPU (Graphics Processing Unit) needs to establish connections with dozens or even hundreds of remote GPUs simultaneously. The massive concurrent transmission of small-granular data leads to low network link utilization and frequent congestion. As a result, remote direct memory access (RDMA) links across nodes become a hard bottleneck for service throughput. The larger the computing cluster and the more dispersed the experts are, the higher the proportion of communication latency in end-to-end inference latency. Summary of the Invention

[0003] This application provides, in this application, a method, apparatus, device, and storage medium for reasoning large models.

[0004] This application provides an inference method for a large model, the large model including multiple network layers, each network layer including multiple expert networks, the expert networks including a shared expert network and a routing expert network. The method includes: acquiring multiple words to be processed; calculating corresponding hidden state results in the network layers based on the multiple words to be processed and an attention mechanism, wherein during the calculation, the hidden state results are calculated on a unit basis, with each word to be processed as a batch of data; determining the routing decision corresponding to the word to be processed based on the hidden state results and a gating network, for determining the routing expert network; and creating multiple parallel processing task queues for each word to be processed as a batch of data. The hidden state results corresponding to the batch of data are sent to a first task queue, and the routing decisions and hidden state results corresponding to the unprocessed lexical units are sent to a second task queue. A first calculation result corresponding to the batch of data is determined using the hidden state results in the first task queue and the shared expert network. The hidden state results corresponding to the unprocessed lexical units are sent to the GPUs deployed by the corresponding routing expert networks using the routing decisions in the second task queue to calculate a second calculation result corresponding to the batch of data. The output result of the network layer corresponding to the batch of data is determined based on the first calculation result and the second calculation result.

[0005] This application provides an inference device for a large model, the large model including multiple network layers, each network layer including multiple expert networks, the expert networks including a shared expert network and a routing expert. The device includes: a calculation module, used to acquire multiple words to be processed, and in the network layers, calculate the corresponding hidden state results based on the multiple words to be processed and an attention mechanism, calculating the corresponding hidden state results on a unit of the words to be processed as a batch of data; a first determination module, used to determine the routing decision corresponding to the words to be processed based on the hidden state results and a gating network, and used to determine the routing expert network; a creation module, used to create multiple parallel processing task queues for the words to be processed as a batch of data; and a first sending module, used to... The system comprises: a first determination module, which sends the hidden state result corresponding to the batch of data to a first task queue; a second sending module, which sends the routing decision corresponding to the batch of data and the hidden state result corresponding to the unprocessed lexical to a second task queue; a third determination module, which determines the first calculation result corresponding to the batch of data based on the hidden state result in the first task queue and the shared expert network; a fourth sending module, which sends the hidden state result corresponding to the unprocessed lexical to the GPU deployed in the corresponding routing expert network based on the routing decision in the second task queue, so as to calculate the second calculation result corresponding to the batch of data; and a fifth determination module, which determines the output result of the network layer corresponding to the batch of data based on the first calculation result and the second calculation result.

[0006] This application provides an embodiment of a device, including: a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program for: acquiring multiple words to be processed; calculating corresponding hidden state results in a network layer of a large model based on the multiple words to be processed and an attention mechanism; during the calculation process, calculating the corresponding hidden state results for words to be processed as units in a batch of data; the network layer includes multiple expert networks, including a shared expert network and a routing expert network; determining the routing decision corresponding to the words to be processed based on the hidden state results and a gating network, for determining the routing expert network; and creating a routing decision for each word to be processed as a batch of data. Multiple parallel processing task queues are established; the hidden state results corresponding to the batch of data are sent to the first task queue, and the routing decisions and hidden state results corresponding to the unprocessed words corresponding to the batch of data are sent to the second task queue; the first calculation result corresponding to the batch of data is determined by the hidden state results in the first task queue and the shared expert network; the hidden state results corresponding to the unprocessed words are sent to the GPUs deployed by the corresponding routing expert networks by the routing decisions in the second task queue to calculate the second calculation result corresponding to the batch of data; the output result of the network layer corresponding to the batch of data is determined based on the first calculation result and the second calculation result.

[0007] This application provides a computer-readable storage medium storing a computer program, characterized in that, when the computer program is executed by one or more processors, it causes the one or more processors to implement the steps in the above-described method.

[0008] In this disclosed technical solution, multiple words to be processed are acquired. In the network layer, the corresponding hidden state results are calculated based on these multiple words and an attention mechanism. During the calculation, the hidden state results are calculated on a unit basis, with each word in a batch of data as a unit. Based on the hidden state results and a gating network, the routing decision corresponding to each word is determined, which is used to determine the routing expert network. Multiple parallel processing task queues are created for each word in a batch of data. The hidden state results corresponding to a batch of data are sent to the first task queue, and the routing decision and the hidden state results corresponding to each word in a batch of data are sent to the second task queue. The first calculation result corresponding to the batch of data is determined using the hidden state results in the first task queue and the shared expert network. The hidden state results corresponding to each word in a batch of data are sent to the GPU deployed by the corresponding routing expert network using the routing decision in the second task queue to calculate the second calculation result corresponding to the batch of data. Based on the first and second calculation results, the output result of the network layer corresponding to the batch of data is determined.

[0009] The process involves sending the hidden state results corresponding to a batch of data to a first task queue, and sending the routing decisions and hidden state results corresponding to the unprocessed words to a second task queue. The first computational result for the batch of data is determined using the hidden state results in the first task queue and the shared expert network. Then, using the routing decisions in the second task queue, the hidden state results corresponding to the unprocessed words are sent to the GPU deployed with the corresponding routing expert network to compute the second computational result for the batch of data. Both queues are parallel processing queues. This allows for continuous execution of the shared expert network's computational operations while communicating with the GPU deployed with the routing expert network during the processing of a batch of data. This hides communication latency within the computation time, fundamentally eliminating the mutual waiting problem between communication and computation in the serial execution model. Attached Figure Description

[0010] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0011] Figure 1 This is a flowchart illustrating an exemplary reasoning method for a large model in this application;

[0012] Figure 2 This is a schematic diagram illustrating the temporal overlap of computational and communication operations in a large-scale model inference example of an exemplary embodiment of this application.

[0013] Figure 3 This is a schematic diagram illustrating the projection calculation and aggregation overlap execution of an exemplary embodiment of this application;

[0014] Figure 4 A schematic diagram of the structure of a large-scale inference device as an exemplary embodiment of this application;

[0015] Figure 5 This is a schematic diagram of the structure of a device that is an exemplary embodiment of this application. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0017] The Mixture of Experts (MoE) architecture has become the mainstream design paradigm for large-scale language models. It replaces the feed-forward network (FFN) in the Transformer layer with multiple parallel expert sub-networks. While maintaining manageable inference computation, the MoE architecture expands the number of model parameters to the tens or even trillions, significantly improving model capacity and performance ceilings.

[0018] In distributed MoE inference, routing experts are typically deployed across multiple GPU nodes. Full-to-All communication is an essential step in MoE inference and a core performance bottleneck. Each GPU needs to establish concurrent connections with a large number of remote GPUs in the cluster simultaneously, with each data transmission having a small granularity but a large number of connections. This many-to-many, small-granularity, and high-concurrency communication characteristic leads to low network bandwidth utilization, especially in cross-node scenarios, where the establishment of RDMA links and data transmission overhead become a hard bottleneck for system throughput. As the cluster size expands from 8 GPUs per machine to dozens or even hundreds of GPUs per machine, the proportion of communication time in end-to-end inference latency also increases, reaching over 50% in some scenarios. Effectively hiding this communication latency and preventing GPU computing power from idling during communication is a key issue in improving MoE inference efficiency.

[0019] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0020] Figure 1 This is a flowchart illustrating a large-scale model inference method as an exemplary embodiment of this application. The method 100 provided in this embodiment can be deployed on a server, such as a server, service cluster, or cloud platform. The large-scale model includes multiple network layers, each network layer including multiple expert networks, and the expert networks including a shared expert network and a routing expert network. The method 100 includes the following steps:

[0021] Step 101: Obtain multiple words to be processed. In the network layer, calculate the corresponding hidden state results based on the multiple words to be processed and the attention mechanism. During the calculation process, the corresponding hidden state results are calculated using the words to be processed as a batch of data as units.

[0022] Step 102: Based on the hidden state results and the gating network, determine the routing decision corresponding to the token to be processed, which is used to determine the routing expert network.

[0023] Step 103: Create multiple parallel processing task queues for the unprocessed words in a batch of data.

[0024] Step 104: Send the hidden state results corresponding to a batch of data to the first task queue, and send the routing decisions and hidden state results corresponding to the unprocessed words to the second task queue.

[0025] Step 105: Determine the first calculation result corresponding to a batch of data by using the hidden state results in the first task queue and the shared expert network.

[0026] Step 106: Through the routing decision in the second task queue, send the hidden state result corresponding to the word to be processed to the GPU deployed by the corresponding routing expert network to calculate the second calculation result corresponding to a batch of data.

[0027] Step 107: Based on the first calculation result and the second calculation result, determine the output result of the network layer corresponding to a batch of data.

[0028] It should be noted that the large model includes hybrid expert models. The large model consists of multiple neural network layers, which include shared experts (i.e., shared expert networks or models) and routing experts (i.e., routing expert networks or models).

[0029] This method 100 can be applied to a distributed cluster, which includes multiple physical nodes, each physical node comprising a physical device or physical server. Each physical device or physical server can have multiple GPUs. Multiple GPUs distributed across different physical devices or physical servers are used to deploy a shared expert network and a routing expert network. The shared expert network can be deployed on each GPU, meaning each GPU has a shared expert network. The routing expert network is distributed and deployed across multiple GPUs.

[0030] The above steps are explained in detail below:

[0031] Step 101: Obtain multiple words to be processed. In the network layer, calculate the corresponding hidden state results based on the multiple words to be processed and the attention mechanism. During the calculation process, the corresponding hidden state results are calculated using the words to be processed as a batch of data as units.

[0032] In this context, a token refers to the basic discrete unit used by a large language model when processing text. The large model does not directly perform calculations on the original input string, but first maps the input text into a token sequence through a token segmenter, and then converts it into a vector representation before feeding it into the neural network.

[0033] The token for a word can be obtained by receiving an inference request from the client and then retrieving the token to be processed based on that request. Alternatively, it can be obtained from other servers; details will not be elaborated here.

[0034] like Figure 2 As shown, after obtaining multiple tokens, each token needs to be calculated for each network layer. Upon receiving the request and obtaining the tokens, the distributed cluster uses a scheduling engine to distribute the tokens to the corresponding GPUs via a queue, i.e., to the physical nodes where the corresponding GPUs reside. During the calculation of any network layer, the GPU deployed on the physical node that obtained the tokens can first perform Q, K, V projection calculations on the hidden state of the tokens using the weights held on the local GPU, obtaining the corresponding Q, K, V values. Then, the calculation is performed according to the attention mechanism to obtain the result. During the calculation, the tokens to be processed are treated as a batch of data. The attention mechanism is used to calculate and capture the contextual dependencies between tokens, and the self-attention weighted representation is used as the output. All obtained tokens undergo the same attention calculation. The output dimension is then projected back to the model's hidden dimension through output projection to obtain the hidden state result. Thus, the calculation of the hidden state results of multiple tokens in a batch of data at this network layer is completed.

[0035] It should be noted that for other network layers, the output of the previous network layer is directly used as the input of the next network layer. The Q, K, V projection calculations and intention mechanism calculations are performed based on the output results. However, as mentioned above, these are still processed as a batch of data, so they will not be elaborated on further.

[0036] Step 102: Based on the hidden state results and the gating network, determine the routing decision corresponding to the token to be processed, which is used to determine the routing expert network.

[0037] In this context, the Gating Network or Router refers to the lightweight neural network module in the MoE model responsible for dynamic routing decisions.

[0038] The gating network receives the hidden states of the current layer, outputs the selection probability distribution of each token to all routing experts through linear transformation and Softmax normalization, and selects the Top-K experts with the highest scores as the routing targets for that token. It then determines the routing strategy.

[0039] Routed Experts refer to expert subnetworks dynamically selected by a gating network in the MoE model. Each Token is assigned to at least a number of routing experts for processing based on the Top-K score output by the gating network, and different Tokens may be routed to different expert sets.

[0040] The computational cost of the aforementioned gated network is far less than that of the expert FFN, but its output determines the target topology and data flow direction for subsequent all-to-all communication. In the MoE architecture, the FFN is further extended into multiple parallel expert FFNs, including a shared expert FFN and several routing expert FFNs.

[0041] All-to-All is a type of aggregated communication operation used for full data exchange between multiple parallel computing units. In All-to-All communication, each participating physical node sends a set of data to all other physical nodes and simultaneously receives a set of data from all other physical nodes. Ultimately, each physical node obtains an aggregated result of partial data from all physical nodes.

[0042] As mentioned above, Figure 2 As shown, the hidden state results are input into the gating network to determine the routing decisions corresponding to multiple unprocessed terms in a batch of data, which will not be elaborated further.

[0043] Step 103: Create multiple parallel processing task queues for the unprocessed words in a batch of data.

[0044] It should be noted that step 103 can be executed after step 102, before step 101, or before step 102; no special restrictions are imposed here.

[0045] The task queue can include CUDA (Compute Unified Device Architecture) streams, which are task queues in the GPU programming model. In the GPU programming model, task queues are abstractions used to manage and schedule asynchronous parallel operations.

[0046] The distributed cluster deploys the execution process of single-network-layer MoE inference across at least two independent Streams. Operations within the same Stream are executed strictly in sequence, while operations between different Streams can be executed concurrently. Streams can be created using the CPUs deployed on the current physical nodes in the distributed cluster. Furthermore, operations between different Streams can be scheduled for concurrent execution by GPU hardware. Taking step 103, executed after step 102, as an example, as described above, two independent Streams can be created after the routing decision is determined. For other GPUs, the corresponding two independent Streams can also be created by the CPUs deployed on their respective physical nodes, as described above.

[0047] This implementation submits computation and communication operations that do not have data dependencies to different streams, allowing the GPU's computing units and communication links to be operational simultaneously.

[0048] Step 104: Send the hidden state results corresponding to a batch of data to the first task queue, and send the routing decisions and hidden state results corresponding to the unprocessed words to the second task queue.

[0049] As mentioned above, the GPU will calculate the hidden state results corresponding to a batch of data and send or submit them to one of the two independent Streams' task queues, which can be used as the first task queue. The routing decisions corresponding to the aforementioned batch of data, along with the hidden state results corresponding to the tokens to be processed with those routing decisions, will be sent or submitted to the other task queue, which can be used as the second task queue.

[0050] For example, the GPU submits the task corresponding to a batch of data for shared expert computation to StreamA, and submits the All-to-All Dispatch communication corresponding to a batch of data to StreamB. The two streams can be started synchronously and executed in parallel after the gated network GATE has been computed.

[0051] like Figure 2 As shown, on the timeline, the data transfer process of Dispatch communication is overlaid on the matrix multiplication and activation calculation processes of the shared expert network. That is, the computation and distribution communication of the shared expert network can be executed in parallel. Ideally, if the computation time of the shared expert network is greater than the time spent on Dispatch communication, the Dispatch communication latency can be completely hidden, without contributing additional to the end-to-end inference latency.

[0052] Step 105: Determine the first calculation result corresponding to a batch of data by using the hidden state results in the first task queue and the shared expert network.

[0053] Among them, Shared Expert refers to a special type of expert in the MoE model, who does not select routes and all tokens are forced to be calculated by them.

[0054] Shared experts are used to capture common knowledge patterns shared by all tokens, effectively complementing the sparse activations of routing experts. The computation of shared experts depends only on the hidden states of the current layer input, and is completely independent of which routing experts each token is routed to.

[0055] The shared expert operates on all tokens in the current layer, including the complete FFN computation, which includes up projection, gate projection, activation function, and down projection. Its input depends only on the Hidden States output by the current layer's Attention and is independent of the token routing results.

[0056] The Feed-Forward Network (FFN) is a core submodule of the Transformer layer, located after the Multi-Head Self-Attention (MoE) module. An FFN can consist of two linear projection layers and a non-linear activation function. The first linear projection layer raises the input from the hidden dimension to an intermediate dimension (2 to 8 times the hidden dimension), called up-projection. The second linear projection layer lowers the input back to the hidden dimension from the intermediate dimension, called down-projection. Each token, after Attention, is computed by a shared expert FFN and top-K routing expert FFNs. The outputs are then weighted and aggregated to serve as the final output of the MoE module for that layer.

[0057] As described above, the GPU can sequentially extract the hidden state results of the corresponding tokens from the first task queue. Based on these results, it performs the FFN calculation of the shared expert network described above to obtain the calculation results corresponding to the final batch of data. Finally, the calculation results of the shared expert network for each token currently handled by the GPU are calculated, thus completing the calculation of the shared expert network for the current network layer.

[0058] It should be understood that the tokens handled by other GPUs are also calculated in the same way as described above, so we will not go into details.

[0059] Specifically, the first calculation result corresponding to a batch of data is determined by using the hidden state results in the first task queue and the shared expert network, including: extracting multiple hidden state results corresponding to multiple unprocessed lexical units in the first task queue using the GPU; and performing calculations based on the shared expert network deployed on the GPU and the extracted hidden state results to obtain the first calculation result corresponding to a batch of data.

[0060] As mentioned above, when the GPU responsible for the calculation of the shared expert network extracts the hidden state results from the queue, the GPU performs the FFN calculation based on its own deployed shared expert network and the hidden results to obtain the calculation results corresponding to a batch of data.

[0061] Step 106: Through the routing decision in the second task queue, send the hidden state result corresponding to the word to be processed to the GPU deployed by the corresponding routing expert network to calculate the second calculation result corresponding to a batch of data.

[0062] Routed Experts: Each token is dynamically selected from the Top-K through a lightweight gating network for computation, used to capture differentiated and specialized knowledge.

[0063] The MoE architecture expands the number of model parameters to the hundreds of billions or even trillions while keeping the amount of inference computation controllable, significantly improving the model capacity and performance ceiling.

[0064] The feedforward network (FFN) structure of the routing expert is typically consistent with that of the shared expert's feedforward network (FFN), comprising four sub-stages: up-projection, gate-projection, activation function, and down-projection. The sparse activation mechanism of the routing expert allows the computational cost to increase only by a constant order while the parameter size of the MoE increases significantly. The FFN computation of the routing expert can be divided into two stages: the first stage includes up-projection, gate-projection, and the activation function, outputting intermediate activation values; the second stage includes down-projection, mapping the intermediate activation values ​​back to the model's hidden dimensions.

[0065] As described above, the aforementioned GPU sequentially extracts the corresponding routing decisions and the hidden state results corresponding to the tokens from StreamB, and initiates All-to-All Dispatch communication. Based on the routing decisions, it distributes the hidden state results corresponding to the tokens to the GPUs where the corresponding routing expert networks reside, and indicates the routing expert identifier (e.g., ID) for each GPU. Upon receiving the hidden state results corresponding to the tokens, the GPU performs the routing expert network calculation based on the corresponding expert ID.

[0066] There is no data dependency between shared expert computation and All-to-All Dispatch communication. Shared experts directly consume the Hidden States input from the current layer without needing to know which routing expert each token will be routed to; All-to-All Dispatch communication only relies on the Top-K routing decisions output by the gating network and does not need to wait for the computation results of shared experts.

[0067] The above implementation submits the shared expert network and Dispatch communication to the first and second task queues respectively, and executes them in parallel through two independent task queues. Specifically, the computation of the shared expert network depends only on the hidden state results of the current network layer input, and has no data dependency with the Dispatch communication. Based on this, this application achieves overlapping computation by distributing the shared expert computation task and the Dispatch communication task to different CUDA streams, thereby realizing the parallel utilization of computational and communication resources.

[0068] In distributed MoE inference across nodes, all-to-all communication can be based on RDMA links. However, when each GPU needs to establish RDMA connections with a large number of remote GPUs simultaneously and perform fine-grained data transfer, the overhead of connection establishment and maintenance will lead to a decrease in network link utilization, and the communication latency of RDMA links will become a hard bottleneck restricting system throughput.

[0069] In MoE inference, All-to-All communication can include two phases: the Dispatch phase, in which each GPU sends its local token to the remote GPU where the corresponding expert is located according to the routing decision; and the Combine phase, in which each GPU sends the intermediate activation value after completing the expert calculation back to the GPU where the token originally resides.

[0070] Computations in MoE inference, such as matrix multiplication and activation function calculations for FFN, can be performed by the GPU's Compute Engine. Communication in MoE inference, such as all-to-all data transfers, can be handled independently by the GPU's CopyEngine or network interface.

[0071] In this embodiment, the input of the shared expert network is independent of the routing results, meaning there is no data dependency between the shared expert computation and the All-to-All Dispatch communication, allowing them to be executed in parallel.

[0072] When executing a single-layer Transformer MoE module, the MoE inference framework can also use a strict serial execution model to execute the four stages of shared expert computation, All-to-All Dispatch communication, routing expert computation, and All-to-All Combine communication in a fixed order.

[0073] Taking DeepSpeed-MoE as an example, its single-layer MoE inference strictly follows a serial progression. Shared experts possess independent parameters and computational resources, performing forward propagation on all tokens. During their computation, the All-to-All communication links are idle. Conversely, in the subsequent Dispatch communication phase, the shared expert's computation has finished, and GPU computing power is in a waiting state. This computation-while-communication, communication-while-computation pattern leads to significant resource waste, which worsens as the model size increases and the proportion of cross-node communication rises. However, this approach causes a large amount of GPU computing power to idle during communication, resulting in low hardware utilization. After the shared expert's computation is complete, the GPU immediately enters the All-to-All Dispatch communication phase. In this communication phase, the GPU's computing units are completely idle and cannot perform any effective computation. Taking a typical MoE model as an example, the time spent on single-layer dispatch communication can reach 50% to 80% of the time spent on shared expert computation. The wasted computing power during this period is amplified layer by layer as the number of model layers accumulates, resulting in the GPU utilization always being unsaturated and the hardware procurement cost not being fully converted into effective computing power output.

[0074] Furthermore, the increased end-to-end inference latency negatively impacts the online service experience. Because shared expert network computation and Dispatch communication are forced to be serialized, the execution latency of a single-layer MoE module equals the sum of the latency of the four stages. However, in this implementation, the time windows for shared expert network computation and Dispatch communication can completely overlap. As mentioned earlier, the former shared expert network only depends on the current layer's input Hidden States, and the latter routing expert network's communication only depends on the Top-K routing results of the gating network; there is no data dependency between them. However, the aforementioned serial framework artificially lengthens the execution time of each layer, accumulating significantly to end-to-end latency across dozens of layers, directly affecting the first token delay (TTFT) and per token generation delay (TPOT) in online service scenarios.

[0075] Furthermore, this approach fails to coordinate the use of network bandwidth and computing resources, limiting throughput due to a bottleneck. In the serial execution model, the cluster's instantaneous bottleneck switches back and forth between computation and communication; that is, network bandwidth is idle during shared expert computation, while GPU computing power is idle during dispatch communication. Both are constantly in a staggered state of activity, preventing the formation of an ideal parallel saturation state for computation and communication. Especially in large-scale cross-node deployments, when the proportion of communication time increases further due to RDMA link latency, the absolute time of GPU idling also increases, exacerbating the bottleneck effect and resulting in the cluster's overall inference throughput falling far below the theoretical hardware peak.

[0076] Systems like FasterMoE and Tutel have optimized expert placement strategies and load balancing, for example, by reducing cross-node communication hops through topology-aware expert mapping or improving load balancing through fine-grained expert group scheduling. However, these efforts focus on making communication itself faster, i.e., compressing the absolute latency of a single communication within existing communication patterns. They do not change the fundamental paradigm of serial execution of shared expert computation and dispatch communication, and the hidden problem of communication latency remains unresolved.

[0077] Specifically, the hidden state results corresponding to the tokens to be processed are sent to the GPUs deployed with the corresponding routing expert networks based on the routing decisions in the second task queue. This includes: extracting the routing decisions and corresponding hidden state results from the second task queue; and sending the corresponding hidden state results to the GPUs deployed with the corresponding routing expert networks based on the routing decisions. Since this has already been explained, it will not be repeated here.

[0078] In addition, after the transmission is completed, a second calculation result can be obtained by calculating based on the routing expert network deployed on the GPU and the hidden state results corresponding to the received tokens to be processed.

[0079] This implementation achieves parallel execution of shared expert network computation and communication by using two parallel task queues, thereby eliminating GPU idle time during shared expert computation and improving hardware utilization.

[0080] Furthermore, this implementation proposes to execute shared expert computation and the aforementioned All-to-All Dispatch communication in parallel on two independent streams. While Dispatch communication is underway, the GPU's computing units continuously perform matrix multiplication and other computational operations by the shared expert, thus hiding communication latency within the computation time. This fundamentally eliminates the mutual waiting problem between communication and computation, and between computation and communication, inherent in the serial execution model. Simultaneously, it shortens the effective execution latency of a single-layer MoE module and reduces end-to-end inference latency. By transforming the original serial relationship between shared expert computation and Dispatch communication into a parallel relationship, the effective execution latency of a single-layer MoE module is reduced from the sum of computation and communication to the larger of the two. In the ideal case, the latency of Dispatch communication can be completely masked by the latency of shared expert computation, achieving a significant reduction in inference latency and significantly improving response speed in online service scenarios. In addition, it enables the coordinated saturation utilization of computing and communication resources, improving the overall cluster throughput. This implementation, through a multi-stream parallel scheduling strategy, ensures that computing and communication resources are actively working at any given time, avoiding peak-shifting phenomena where a single resource is idle. In large-scale deployments, this synergistic advantage can be linearly accumulated to multi-machine, multi-GPU scenarios, bringing cluster throughput close to the theoretical upper limit of the hardware and fully leveraging the performance potential of the MoE architecture in large-scale inference.

[0081] The specific execution process of the second calculation result mentioned above may include: calculating up projection, gate projection, and activation function based on the routing expert network deployed on the GPU and the hidden state result corresponding to the received token to be processed, to obtain intermediate activation values; returning the intermediate activation values ​​to the GPU where the shared expert network corresponding to the token to be processed is located through the GPU with the routing expert network deployed; and calculating down projection through the GPU where the shared expert network is located and the intermediate activation values ​​to obtain the final second calculation result.

[0082] As mentioned earlier, after the GPU with the corresponding routing expert network receives the hidden state result corresponding to the token and the corresponding expert ID, such as... Figure 2 As shown, Up projection, Gate projection, and activation function calculations are performed to obtain intermediate activation values. After determining the intermediate activation values, the GPU returns the intermediate activation values, along with the token's ID or the hidden state result corresponding to the token, to the source GPU for final down projection calculation, resulting in a second calculation result for a batch of data.

[0083] In this embodiment, since the calculation of the second-stage down projection in the above-mentioned routing expert network occurs on the GPU where the Token originally resides, there can be an overlapping window in the backhaul process of the result of the backhaul Combine communication.

[0084] Since the decomposition and down projection of the Routing Expert Network (FFN) can overlap with the pipeline of All-to-All Combine communication, the FFN computation of the Routing Expert Network is decomposed into up projection, gate projection, activation function stage, and down projection stage. Taking advantage of the block-by-block arrival characteristic of Combine communication data, the down projection computation and Combine communication data reception are interleaved in a pipelined manner on independent streams, thus hiding the latency of the latter half of Combine communication beneath the down projection computation. This achieves pipeline overlap between down projection and Combine communication after the FFN decomposition of the Routing Expert Network.

[0085] Specifically, the method 100 further includes: calculating up-projection, gate-projection, and activation functions based on the routing expert network deployed on the GPU and the received hidden state results corresponding to the tokens to be processed, to obtain intermediate activation values; returning the intermediate activation values ​​to the GPU of the shared expert network corresponding to the tokens to be processed in a batch of data through the GPU with the routing expert network deployed; submitting the receipt of intermediate activation values ​​to the third task queue and submitting the calculation of down-projection of intermediate activation values ​​to the fourth task queue through the GPU of the shared expert network; extracting intermediate activation values ​​from the third task queue according to a preset batch and sending them to the fourth task queue; and calculating down-projection through the GPU of the shared expert network and the intermediate activation values ​​in the fourth task queue to obtain the second calculation result corresponding to the final batch of data.

[0086] The above implementation method can also be a specific implementation method that calculates the second calculation result based on the routing expert network deployed on the GPU and the hidden state result corresponding to the received token to be processed.

[0087] The third and fourth task queues can be processed in parallel. Task queues can include CUDA (Compute Unified Device Architecture) streams, which are task queue abstractions used in the GPU programming model to manage and schedule asynchronous parallel operations. Further details will not be provided.

[0088] As mentioned above, multiple parallel processing task queues can be created simultaneously, as described previously. Alternatively, parallel processing task queues can be created in the implementation, for example, after the intermediate activation value is returned to the source GPU before creation.

[0089] As mentioned earlier, after the routing expert's FFN calculation is split into two stages, the first stage calculation on each expert's GPU, including up projection, gate projection, and activation function completion, generates intermediate activation values ​​that are transmitted back to the GPU where each token originally resides via All-to-AllCombine communication. On the original token GPU, the second stage calculation of down projection can only be executed after the intermediate activation values ​​transmitted back from the Combine have arrived. Since there is a pipeline parallelism opportunity between Combine communication and down projection, Combine communication arrives sequentially, one token at a time, one data block at a time, rather than all at once. Therefore, as described earlier, the data reception from Combine communication and the calculation of down projection are submitted to different Streams on the original token GPU. This allows the sending GPU or the receiving source GPU to directly submit data to the receiving data Stream. Then, the source GPU used for down calculation sequentially extracts the received data from the Stream and, according to the read instructions, transfers the arrived intermediate activation values ​​to the calculation Stream in batches. The calculation Stream then starts the down projection calculation for that batch, while the receiving Stream continues to receive the next batch of data simultaneously. The source GPU can sequentially extract intermediate activation values ​​from the computation stream for corresponding batches and perform down computation on the source GPU. Through this fine-grained pipeline orchestration, the down projection computation process overlaps temporally with the latter half of the Combine communication, further reducing the latency exposed on the inference critical path. For example... Figure 2 As shown, the Down projection calculation and the intermediate activation value backpropagation can be performed in parallel.

[0090] It should be noted that at least one GPU enjoys a corresponding set of third and fourth task queues, and at least one GPU refers to the source GPU where the shared expert network resides. For example, each source GPU can enjoy a corresponding set of queues, or the source GPUs on a single physical node can enjoy a corresponding set of queues.

[0091] The aforementioned double time overlap further conceals the Combine communication latency, approaching the theoretical upper limit of complete overlap between computation and communication. This embodiment not only achieves the first layer of overlap between the shared expert network and Dispatch, but also further extracts the down projection in the routing expert FFN and performs a second layer of pipeline overlap with Combine communication. Within a single-layer MoE inference, a computation-integrated scheduling architecture with double overlap between computation and communication is implemented. The first layer of overlap includes computation of the shared expert network and Dispatch communication, while the second layer of overlap includes computation of the down projection and data return Combine, compressing the original four or five serial latencies into the larger of each overlap group. The synergistic effect of the two time overlaps allows the latencies of both communication stages—Dispatch and Combine—in a single-layer MoE to be partially or completely covered by computation time, minimizing the proportion of communication latency exposed on the inference critical path.

[0092] The overlapping parallelism of down projection computation and Combine communication differs fundamentally from the parallelism of shared expert computation and Dispatch communication. A data dependency exists between computation and communication; down projection computation must wait for Combine communication to send intermediate activation values ​​back to the local machine before execution. To achieve overlap between computation and communication in this scenario, the unprocessed terms or hidden states within a batch of data need to be divided into n data blocks, allowing Combine communication to reach the local GPU in units of data blocks. Simultaneously, a corresponding control signal is allocated for each data block in the GPU's high-bandwidth memory, totaling n control signals to achieve synchronization between the two streams.

[0093] Specifically, the process involves dividing intermediate activation values, sending communication operators to the communication stream, requesting control signals corresponding to the division results in high-bandwidth memory, and using the communication operators to transmit the divided intermediate activation values. The computation operators corresponding to the down projection calculation are sent to the computation stream, and spinlocks are set on the computation operators to monitor the status of each control signal in the high-bandwidth memory. When the communication stream completes sending the divided intermediate activation values ​​to the GPU where the shared expert network resides, the corresponding control signals are set to the ready state, and the spinlocks on the computation stream are unlocked, and the corresponding down projection calculations are executed through the computation operators.

[0094] The method for determining the number of segments n mentioned above includes: determining the number n based on hardware attributes and the computational intensity of down projection. Furthermore, n should be made as large as possible when performing the segmentation.

[0095] As mentioned above, since communication is required before computation, the Combine communication of the first partitioned data block cannot overlap with any communication before the down projection is calculated. Similarly, the down projection calculation of the last partitioned data block cannot overlap with any computation after all Combine communication is completed. Therefore, in order to minimize the overhead caused by these two non-overlapping parts, n should be made as large as possible to eliminate the additional overhead introduced by the head and tail.

[0096] Furthermore, the computational intensity of down projection calculations is directly proportional to the size of the data block processed in a single operation. If the number of partitions, n, is too large, the size of each data block after partitioning will be too small, resulting in insufficient computational intensity to fully utilize the hardware's computing power. Therefore, the selection of the number of partitions, n, needs to strike a balance between the number of pipeline stages and the computational intensity.

[0097] The number of segments, n, is determined based on hardware characteristics and the computational intensity of the down computation. The host machine divides the down computation into n data blocks and transmits them on communication stream 0 via a communication operator. It also requests n control signals from high-bandwidth memory, each corresponding to a data block. On computation stream 1, communication data blocks are transmitted via a computation operator, and a spinlock is used to monitor the synchronization control signals on the high-bandwidth memory. When the i-th computational data block on communication stream 0 is completed, the i-th control signal is pulled high. Once computation stream 1 detects the i-th control signal going high, it unlocks and performs down projection computation on the completed i-th computational data block.

[0098] Furthermore, by performing up projection, gate projection, activation function, and down projection calculations using the routing expert network deployed on the GPU and the hidden state results corresponding to the received tokens to be processed, a second calculation result is obtained, namely the second calculation result corresponding to a batch of data.

[0099] Since this has already been stated above, it will not be repeated here. It will only be explained that down projection calculations are performed on GPUs with routing expert networks deployed.

[0100] The second calculation result is returned to the GPU containing the shared expert network corresponding to the unprocessed tokens of a batch of data by the GPU with the routing expert network deployed. Step 107 is executed by the GPU containing the shared expert network: based on the first calculation result and the second calculation result, the output result of the network layer corresponding to a batch of data is determined to realize the combine aggregation.

[0101] Specifically, the GPUs deployed with the routing expert network submit the calculation of the down projection of intermediate activation values ​​to the third task queue, and submit the calculation results of the down projection of the GPUs sharing the expert network to the fourth task queue. When submitting to the third task queue, intermediate activation values ​​can be obtained according to a preset batch and sent to the third task queue. Through the GPUs deployed with the routing expert network and the calculation of the down projection in the third task queue, the second calculation result corresponding to the final batch of data is obtained. Then, through the GPUs sharing the expert network and the fourth task queue, the aggregated result of the first and second calculation results is obtained. The third and fourth task queues can process data in parallel.

[0102] Specifically, the computational cost of the down projection corresponding to the intermediate activation values ​​is divided. The computation operator is sent to the computation stream to perform the down projection calculation after the division. The control signal corresponding to the division result is requested in the high-bandwidth memory. The communication operator corresponding to the calculation result of the down projection after the division is sent to the communication stream, and a spinlock is set on the communication operator to monitor the status of each control signal in the high-bandwidth memory. When the computation stream completes the calculation of the down projection after the division, the corresponding control signal is set to the ready state, and the spinlock on the communication stream is unlocked and the calculation result of the down projection after the division is executed through the communication operator. As mentioned above, the result is transmitted to the GPU where the shared expert network corresponding to the unprocessed tokens of a batch of data is located, and the first and second calculation results are aggregated.

[0103] As mentioned above, there are two constraints for the segmentation n, including:

[0104] Constraint 1: Since computation is required before communication, the down projection computation of the first data block cannot overlap with any communication before the Combine communication arrives, and the Combine communication of the last data block cannot overlap with any computation after all down projection computations are completed. Therefore, in order to minimize the overhead caused by these two non-overlapping parts, n should be made as large as possible when splitting n to eliminate the additional overhead introduced by the head and tail.

[0105] The number of segments, n, is negatively correlated with the overhead at the beginning and end of the pipeline, and positively correlated with the computational intensity of the data blocks.

[0106] Constraint 2: The computational intensity of down projection calculation is directly proportional to the size of the data block processed in a single operation. If the number of partitions n is too large, the size of each data block after partitioning will be too small, resulting in insufficient computational intensity to fully utilize the hardware's computing power. Therefore, the selection of the number of partitions n needs to strike a balance between the number of pipeline stages and the computational intensity.

[0107] like Figure 5As shown, firstly, the number of segments n is determined based on hardware characteristics and the computational intensity of the down-process. The host divides the down-process computation into n data blocks and emits them on the first computation stream (stream0), such as the 0th computation data block, the 1st computation data block, etc. Additionally, n control signals are allocated on high-bandwidth memory, each control signal corresponding to a data block, and each data block corresponding to a computation operator. Communication data blocks are emitted on the second communication stream (stream1), corresponding to communication operators, and a spinlock is applied to monitor the synchronization control signals on the high-bandwidth memory. When the i-th computation data block on the first computation stream (stream0) is completed, the i-th control signal is pulled high. Upon detecting the i-th control signal going high, the second communication stream (stream1) unlocks the signal and communicates with the completed i-th computation data block, transmitting the data back to the source GPU for aggregation.

[0108] Step 107: Based on the first calculation result and the second calculation result, determine the output result of the network layer corresponding to a batch of data.

[0109] As mentioned above, the source GPU aggregates the output of the shared expert with the output of the routing expert after down projection, such as by weighted summation, to obtain the final output of the current MoE layer, which is then passed to the next layer Transformer as the input of the next layer.

[0110] It should be noted that this implementation method does not depend on a specific model structure or hardware topology; it is applicable as long as the model includes both shared expert networks and routing expert networks in its MoE architecture. In other words, this application provides a universal application for various dual-track MoE models that combine shared expert networks and routing expert networks.

[0111] When the inference hardware supports multi-stream parallel execution and the aforementioned MoE model includes two types of FFN modules—shared expert and routing expert—it can be directly applied. This allows for seamless deployment on GPU platforms from different vendors and various mainstream shared expert MoE models, demonstrating broad engineering practical value and industrialization prospects.

[0112] In the arrangement of multiple network layers, the first network layer is the network layer preceding the second network layer. After determining the output result of the first network layer, the method 100 further includes: for the second network layer, based on the output result of the first network layer, performing the calculation of the corresponding hidden state result based on multiple unprocessed words and the attention mechanism. During the calculation process, the step of calculating the corresponding hidden state result is performed using unprocessed words in a batch of data as units, until the output result of the second network layer is determined, until the final network layer outputs the result.

[0113] As described above, the output of a batch of data from the previous layer is used as the input of the next layer, and this process is repeated to generate the corresponding hidden state result. This hidden state result still corresponds to the original token in the batch of data. The output result of this layer is calculated until the final network layer is completed, which will not be elaborated further.

[0114] A specific example is as follows: The Hidden States output by the current layer's Attention are simultaneously fed into the shared expert branch and the gating network. The gating network calculates the Top-K routing expert allocation scheme for each Token. After the gating network completes its calculation, the cluster simultaneously starts two Streams. Stream A performs the complete FFN calculation for the shared expert, such as up projection, gate projection, activation function, and down projection. Stream B initiates All-to-All Dispatch communication, distributing the hidden state results corresponding to the Tokens in a batch of data to the GPUs where the corresponding routing experts reside. The two streams execute in parallel, with the shared expert calculation and Dispatch communication overlapping on the timeline. After each expert GPU receives the distributed hidden state results corresponding to the Tokens, it performs the first stage of the routing expert's FFN calculation, including up projection, gate projection, and activation function, producing intermediate activation values. All-to-All Combine communication is then initiated, sending the intermediate activation values ​​back to the original GPUs of each Token. Simultaneously, on the original GPUs of the Tokens, the intermediate activation values ​​received by Combine are submitted in batches to the down projection calculation Stream, achieving pipeline overlap between down projection and Combine communication. The output of the shared expert is aggregated with the output of the routing expert after down projection, such as by weighted summation, to obtain the final output of the current MoE layer, which is then passed to the next layer Transformer.

[0115] The aforementioned method of serially executing shared expert computation and dispatch communication results in an effective latency of a single-layer MoE module equal to the sum of the latencies of each stage. This invention, through multi-stream parallel scheduling, transforms the serial execution of shared expert computation and dispatch communication into parallel execution, reducing the effective latency of a single layer from the sum of computation and communication latency to the maximum value of the time consumed between computation and communication.

[0116] Taking a typical MoE model as an example, if the Dispatch communication time is approximately 50% to 80% of the time spent on shared expert computation, under ideal overlap conditions, this 50% to 80% communication latency can be completely hidden within the computation time. This benefit accumulates layer by layer in each MoE layer of the model, significantly reducing end-to-end inference latency. Eliminating GPU computing power idle during communication significantly improves hardware utilization. In the serial scheme, the GPU computing unit is completely idle during Dispatch communication, causing GPU utilization to drop to zero within the communication window. By deploying shared expert computation on independent Streams and executing it in parallel with Dispatch communication, the GPU computing unit continuously performs effective computational tasks while communication is in progress. Whether in a single-machine 8-GPU scenario or a multi-machine cross-node scenario, the GPU computing unit remains active during communication, and hardware procurement costs are more fully converted into effective computing power output.

[0117] The aforementioned time overlap granularity is at the complete operator level, with the entire shared expert computation and the entire Dispatch communication running in parallel. The computation of the shared expert FFN can also be further divided into smaller tile blocks along the Token dimension or the Hidden States dimension. For example, using a single matrix multiplication block as the scheduling unit, the corresponding communication resources are released immediately after each tile computation is completed, forming a tile-level interleaved pipeline.

[0118] Specifically, the method 100 further includes: dividing each unprocessed term in a batch of data; calculating the hidden state result after division based on the division result and the attention mechanism; determining the first calculation result after division based on the hidden state result after division and the shared expert network; sending the hidden state result after division to the GPU where the corresponding routing expert network is located according to the routing decision to calculate the second calculation result after division; determining the output result of the network layer corresponding to the batch of data based on the first calculation result after division and the second calculation result after division; during the calculation of the first calculation result after division, completing multiple calculation parts of the shared expert network based on the hidden state result after division; releasing the corresponding computing resources after each calculation part is completed to calculate the first calculation result of the next or subsequent hidden state results after division; and sending the hidden state results after division to the corresponding GPU in parallel; releasing the corresponding communication resources after sending.

[0119] Alternatively, the hidden state result corresponding to a batch of data can be directly divided into segments. Based on the segmented hidden state results and the shared expert network, the first calculation result after the segmentation is determined. The segmented hidden state results are then sent to the GPU where the corresponding routing expert network is located according to the routing decision to calculate the second calculation result after the segmentation. Based on the first and second calculation results after the segmentation, the output result of the network layer corresponding to the batch of data is determined. During the calculation of the first calculation result after the segmentation, multiple calculation parts of the shared expert network are completed based on the segmented hidden state results. After each calculation part is completed, the corresponding computing resources are released to calculate the first calculation result of the next or subsequent segmented hidden state results. The segmented hidden state results can be sent to the corresponding GPU in parallel. After the sending is completed, the corresponding communication resources are released.

[0120] For example, as mentioned above, further dividing the data along the Token dimension or Hidden States dimension into smaller Tile blocks yields the Hidden States of the divided Tile blocks. The up projection, gate projection, activation function, and down projection of the shared expert are directly calculated based on the Tile division, eliminating the need for the aforementioned queue. Once the calculation of each Tile is completed (e.g., the up projection, gate projection, activation function, or down projection), the GPU register file and shared memory it occupies are released. At this point, the corresponding sub-operation of Dispatch communication can be initiated on the same GPU, directly dispatch communication of the Hidden States of the divided Tile blocks to complete the routing expert calculation, without the aforementioned queue. Finally, the first and second calculation results of the corresponding tokens are fused according to the partitioning rules, and a final aggregation is performed. Compared to the aforementioned operator-level overlap, the Tile-level implementation can achieve interleaving of computation and communication with finer time granularity, achieving a higher overlap rate in scenarios where single computation time is long and communication time is relatively dispersed.

[0121] Furthermore, shared expert computation and all-to-all dispatch communication logic can be cohesively integrated into a single mega-kernel function, Mega-Kernel. This Mega-Kernel runs persistently on the GPU as a persistent thread block, internally managing both computational and communication initiation tasks through cooperative thread groups.

[0122] Specifically, the method 100 further includes: dividing a preset thread group into a first thread bundle and a second thread bundle. The first thread bundle is used for the computation of the shared expert network, and the second thread bundle is used to initiate a request to send the terminology to be processed to the GPU where the corresponding routing expert network is located. The first thread bundle and the second thread bundle can be executed simultaneously. Using the first thread bundle, an idle buffer is selected from multiple buffers to load the hidden state results corresponding to a batch of data, and the computation of the shared expert network is performed based on the hidden state results to obtain a first computation result, and then the buffer is released. Using the second thread bundle, an idle buffer is selected from multiple buffers to load the routing decisions and terms to be processed corresponding to a batch of data, a communication descriptor is constructed based on the routing decisions, and a request is initiated to send the terminology to be processed to the corresponding GPU, and then the buffer is released.

[0123] As described above, within the Mega-Kernel, one set of threads performs matrix multiplication and activation function calculations for the shared expert FFN, while another set of threads simultaneously constructs the Dispatch communication descriptor and initiates data transfer requests. These two types of operations are alternately scheduled at the Warp level within the streaming multiprocessor (SM), sharing multiple buffers to achieve instruction-level computation and communication overlap. During the shared expert FFN calculation, the hidden state results are loaded into the available buffers of multiple buffers, and the calculations are performed before releasing the buffer. Simultaneously, constructing the Dispatch communication descriptor and initiating data transfer requests utilizes other available buffers to load routing decisions and the hidden state results corresponding to the terms to be processed. This allows communication requests to be constructed using the aforementioned communication method, the calculation of the second result to be completed, and the corresponding buffer to be released. By scheduling the available buffers of different thread sets, the aforementioned communication and computation can be executed in parallel, achieving alternating scheduling.

[0124] This implementation is equivalent to the multi-stream scheme in terms of external performance, both eliminating serialization between shared experts and Dispatch. This implementation achieves this through operator fusion rather than multi-queue scheduling.

[0125] It should be noted that the aforementioned Tile-level segmentation, Mega-Kernel fusion, and multi-Stream scheduling utilize the independence between shared expert computation and Dispatch communication to achieve parallel execution of both on the GPU. Other similar variations are all within the scope of this application.

[0126] like Figure 3 As shown, this application also proposes a large-scale model inference device applicable to service devices. The large-scale model includes multiple network layers, each network layer including multiple expert networks, and the expert networks including a shared expert network and a routing expert. The device 300 includes:

[0127] The calculation module 301 is used to acquire multiple words to be processed. In the network layer, it calculates the corresponding hidden state results based on the multiple words to be processed and the attention mechanism. During the calculation process, the corresponding hidden state results are calculated using the words to be processed as a batch of data as units.

[0128] The first determining module 302 is used to determine the routing decision corresponding to the word to be processed based on the hidden state result and the gating network, and is used to determine the routing expert network.

[0129] Create module 303 to create multiple parallel processing task queues for the unprocessed words in a batch of data.

[0130] The first sending module 304 is used to send the hidden state result corresponding to the batch of data to the first task queue, and to send the routing decision corresponding to the batch of data and the hidden state result corresponding to the unprocessed word to the second task queue.

[0131] The second determining module 305 is used to determine the first calculation result corresponding to the batch of data by using the hidden state results in the first task queue and the shared expert network.

[0132] The second sending module 306 is used to send the hidden state result corresponding to the word to be processed to the GPU deployed by the corresponding routing expert network through the routing decision in the second task queue, so as to calculate the second calculation result corresponding to the batch of data.

[0133] The third determining module 307 is used to determine the output result of the network layer corresponding to the batch of data based on the first calculation result and the second calculation result.

[0134] In the layered arrangement of the multiple network layers, the first network layer is the network layer preceding the second network layer. After determining the output result of the first network layer, for the second network layer, based on the output result of the first network layer, the calculation module 301 executes the step of calculating the corresponding hidden state result based on multiple unprocessed words and the attention mechanism. During the calculation process, the step of calculating the corresponding hidden state result is performed using unprocessed words in a batch of data as units until the output result of the second network layer corresponding to the batch of data is determined, until the final network layer outputs the result corresponding to the batch of data.

[0135] The device 300 is used in a distributed cluster, which includes multiple GPUs distributed across different devices for deploying a shared expert network and a routing expert network; the shared expert network is deployed on each GPU, and the routing expert network is distributed across multiple GPUs.

[0136] Specifically, the second determining module 305 includes: a first extraction unit, used to extract multiple hidden state results corresponding to multiple unprocessed lexical units in the first task queue through the GPU; and a calculation unit, used to perform calculations based on the shared expert network deployed on the GPU and the extracted hidden state results to obtain a first calculation result corresponding to a batch of data.

[0137] Specifically, the second sending module 306 includes: a second extraction unit, used to extract routing decisions and corresponding hidden state results from the second task queue; and a sending unit, used to send the corresponding hidden state results to the GPU deployed with the corresponding routing expert network according to the routing decisions.

[0138] In addition, the calculation module 301 is also used to calculate up projection, gate projection and activation function based on the routing expert network deployed on the GPU and the hidden state result corresponding to the received token to be processed, to obtain intermediate activation values; the second sending module 306 is also used to return the intermediate activation values ​​to the GPU where the shared expert network corresponding to the token to be processed of the batch of data is located through the GPU where the routing expert network is deployed; the calculation module 301 is also used to calculate down projection through the GPU where the shared expert network is located and the intermediate activation values, to obtain the second calculation result corresponding to the final batch of data.

[0139] Furthermore, the calculation module 301 is also used to calculate up projection, gate projection, and activation function based on the routing expert network deployed on the GPU and the hidden state results corresponding to the received tokens to be processed, to obtain intermediate activation values; the second sending module 306 is also used to return the intermediate activation values ​​to the GPU of the shared expert network corresponding to the tokens to be processed corresponding to the batch of data through the GPU with the routing expert network deployed; the device 300 also includes a submission module, used to submit the receipt of the intermediate activation values ​​to the third task queue and submit the calculation of the down projection of the intermediate activation values ​​to the fourth task queue through the GPU of the shared expert network; the submission module is used to extract the intermediate activation values ​​in the third task queue according to a preset batch and send them to the fourth task queue; the calculation module 301 is also used to calculate the down projection through the GPU of the shared expert network and through the intermediate activation values ​​in the fourth task queue to obtain the second calculation result corresponding to the final batch of data, and the third task queue and the fourth task queue can be processed in parallel.

[0140] In this context, at least one GPU enjoys a corresponding set of the third task queue and the fourth task queue, and the at least one GPU refers to the GPU where the shared expert network is located.

[0141] Furthermore, the second determining module 305 is also used to divide each of the unprocessed tokens in the batch of data, calculate the hidden state result after division based on the division result and the attention mechanism, and determine the first calculation result after division based on the hidden state result after division and the shared expert network; the second sending module 306 is also used to send the hidden state result after division to the GPU where the corresponding routing expert network is located according to the routing decision, so as to calculate the second calculation result after division; the third determining module 307 is also used to determine the output result of the network layer corresponding to the batch of data based on the first calculation result after division and the second calculation result after division.

[0142] During the calculation of the first calculation result after partitioning, multiple calculation parts of the shared expert network are completed based on the hidden state results after partitioning. After each calculation part is completed, the corresponding computing resources are released to calculate the first calculation result of the hidden state results of the next or subsequent partitions. The hidden state results after partitioning can be sent to the corresponding GPU in parallel. After the sending is completed, the corresponding communication resources are released.

[0143] or,

[0144] The second determining module 305 is further configured to directly divide each of the hidden state results corresponding to a batch of data, and determine the first calculation result after division based on the divided hidden state results and the shared expert network; the second sending module 306 is further configured to send the divided hidden state results to the GPU where the corresponding routing expert network is located according to the routing decision, so as to calculate the second calculation result after division; the third determining module 307 is further configured to determine the output result of the network layer corresponding to the batch of data based on the first calculation result after division and the second calculation result after division.

[0145] During the calculation of the first calculation result after partitioning, multiple calculation parts of the shared expert network are completed based on the hidden state results after partitioning. After each calculation part is completed, the corresponding computing resources are released to calculate the first calculation result of the hidden state results of the next or subsequent partitions. The hidden state results after partitioning can be sent to the corresponding GPU in parallel. After the sending is completed, the corresponding communication resources are released.

[0146] In addition, the device 300 further includes: a partitioning module, used to partition a preset thread group into a first thread bundle and a second thread bundle, wherein the first thread bundle is used for the computation of a shared expert network, and the second thread bundle is used to initiate a request to send the term to be processed to the GPU where the corresponding routing expert network is located, and the first thread bundle and the second thread bundle can be executed simultaneously; a selection module, used to select an idle buffer from multiple buffers through the first thread bundle to load the hidden state result corresponding to the batch of data, and perform the computation of the shared expert network according to the hidden state result to obtain a first computation result, and release the buffer; and a selection module, used to select an idle buffer from multiple buffers through the second thread bundle to load the routing decision corresponding to the batch of data and the hidden state result corresponding to the term to be processed, construct a communication descriptor according to the routing decision and initiate a request to send the hidden state result corresponding to the term to be processed to the corresponding GPU, and release the buffer.

[0147] This will not be elaborated upon further here; for any content that has not been covered in detail, please refer to the content mentioned above.

[0148] like Figure 4 As shown, this application also proposes a device 400, including: a memory 401 and a processor 402; the memory 401 is used to store a computer program; the processor 402 is used to execute the computer program for:

[0149] Multiple unprocessed words are obtained. In the network layer of the large model, the corresponding hidden state results are calculated based on the multiple unprocessed words and the attention mechanism. During the calculation process, the corresponding hidden state results are calculated using the unprocessed words in a batch of data as units. The network layer includes multiple expert networks, including a shared expert network and a routing expert network.

[0150] Based on the hidden state results and the gating network, the routing decision corresponding to the word to be processed is determined, which is used to determine the routing expert network;

[0151] For each word element to be processed in a batch of data, create multiple task queues for parallel processing;

[0152] Send the hidden state results corresponding to a batch of data to the first task queue, and send the routing decisions corresponding to a batch of data and the hidden state results corresponding to the unprocessed words to the second task queue.

[0153] The first calculation result corresponding to a batch of data is determined by using the hidden state results in the first task queue and the shared expert network.

[0154] Through the routing decision in the second task queue, the hidden state result corresponding to the word to be processed is sent to the GPU deployed by the corresponding routing expert network to calculate the second calculation result corresponding to a batch of data;

[0155] Based on the first calculation result and the second calculation result, the output result of the network layer corresponding to a batch of data is determined.

[0156] This will not be elaborated upon further here; for any content that has not been covered in detail, please refer to the content mentioned above.

[0157] This application also proposes a computer-readable storage medium storing a computer program, characterized in that, when the computer program is executed by one or more processors, it causes the one or more processors to implement the steps in method 100.

[0158] This will not be elaborated upon further here; for any content that has not been covered in detail, please refer to the content mentioned above.

[0159] Furthermore, some processes described in the above embodiments and accompanying drawings include multiple operations appearing in a specific order. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or they may be executed in parallel. The operation numbers, such as 101, 102, etc., are merely used to distinguish different operations and do not represent any execution order. Additionally, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions such as "first" and "second" in this document are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types.

[0160] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0161] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of a necessary general-purpose hardware platform, or by a combination of hardware and software. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a computer product. The present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0162] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable multimedia data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable multimedia data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0163] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable multimedia data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0164] These computer program instructions may also be loaded onto a computer or other programmable multimedia data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0165] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0166] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0167] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0168] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A reasoning method for large models, characterized in that, The large model includes multiple network layers, each network layer including multiple expert networks, the expert networks including a shared expert network and a routing expert network, and the method includes: Multiple words to be processed are obtained. In the network layer, the corresponding hidden state results are calculated based on the multiple words to be processed and the attention mechanism. During the calculation process, the corresponding hidden state results are calculated using the words to be processed as a batch of data as a unit. Based on the hidden state results and the gating network, the routing decision corresponding to the word to be processed is determined, which is used to determine the routing expert network; For each word element to be processed in a batch of data, create multiple task queues for parallel processing; Send the hidden state result corresponding to the batch of data to the first task queue, and send the routing decision corresponding to the batch of data and the hidden state result corresponding to the word to be processed to the second task queue; The first calculation result corresponding to the batch of data is determined by the hidden state results in the first task queue and the shared expert network. Through the routing decision in the second task queue, the hidden state result corresponding to the word to be processed is sent to the GPU deployed by the corresponding routing expert network to calculate the second calculation result corresponding to the batch of data; Based on the first calculation result and the second calculation result, the output result of the network layer corresponding to the batch of data is determined.

2. The method according to claim 1, characterized in that, In the layer arrangement of the plurality of network layers, the first network layer is the network layer preceding the second network layer. After determining the output result of the first network layer, the method further includes: For the second network layer, based on the output of the first network layer, the step of calculating the corresponding hidden state result based on multiple unprocessed words and the attention mechanism is performed. During the calculation process, the unprocessed words in a batch of data are used as units to calculate the corresponding hidden state result until the output result of the second network layer corresponding to the batch of data is determined, until the network layer finally outputs the result corresponding to the batch of data.

3. The method according to claim 1, characterized in that, The method is applied to a distributed cluster, which includes multiple GPUs distributed across different devices for deploying a shared expert network and a routing expert network; the shared expert network is deployed on each GPU, and the routing expert network is distributed across multiple GPUs.

4. The method according to claim 1 or 3, characterized in that, The step of determining the first calculation result corresponding to the batch of data through the hidden state results in the first task queue and the shared expert network includes: The GPU is used to extract multiple hidden state results corresponding to multiple unprocessed lexical units in the first task queue. The first calculation result corresponding to the batch of data is obtained by calculating based on the shared expert network deployed on the GPU and the extracted hidden state results.

5. The method according to claim 1 or 3, characterized in that, The step of sending the hidden state result corresponding to the word to be processed to the GPU deployed by the corresponding routing expert network through the routing decision in the second task queue includes: Extract the routing decisions and corresponding hidden state results from the second task queue; Based on the routing decision, the corresponding hidden state result is sent to the GPU on which the corresponding routing expert network is deployed.

6. The method according to claim 1 or 5, characterized in that, The method further includes: Based on the routing expert network deployed on the GPU and the hidden state results corresponding to the received tokens to be processed, up projection, gate projection and activation function are calculated to obtain intermediate activation values; The intermediate activation value is returned to the GPU containing the shared expert network corresponding to the unprocessed lexical unit of the batch of data by the GPU with the routing expert network deployed thereon; By using the GPU where the shared expert network is located and the intermediate activation values, down projection calculations are performed to obtain the second calculation result corresponding to the final batch of data.

7. The method according to claim 1 or 5, characterized in that, The method further includes: Based on the routing expert network deployed on the GPU and the hidden state results corresponding to the received tokens to be processed, up projection, gate projection and activation function are calculated to obtain intermediate activation values; The intermediate activation value is returned to the GPU containing the shared expert network corresponding to the unprocessed lexical unit of the batch of data by the GPU with the routing expert network deployed thereon; The intermediate activation values ​​are received and submitted to the third task queue via the GPU where the shared expert network is located, and the calculation of the down projection of the intermediate activation values ​​is submitted to the fourth task queue. The intermediate activation values ​​in the third task queue are extracted according to the preset batch and sent to the fourth task queue; The second calculation result corresponding to the final batch of data is obtained by using the GPU where the shared expert network is located and by performing down projection calculations using the intermediate activation values ​​in the fourth task queue. The third task queue and the fourth task queue can be processed in parallel.

8. The method according to claim 7, characterized in that, At least one GPU enjoys a corresponding set of the third task queue and the fourth task queue, and the at least one GPU refers to the GPU where the shared expert network is located.

9. The method according to claim 1, characterized in that, The method further includes: Each of the terms to be processed in a batch of data is divided, and the hidden state result after division is calculated based on the division result and the attention mechanism. The first calculation result after division is determined based on the hidden state result after division and the shared expert network. The partitioned hidden state results are sent to the GPU where the corresponding routing expert network is located according to the routing decision, so as to calculate the second calculation result after partitioning; Based on the first calculation result after partitioning and the second calculation result after partitioning, the output result of the network layer corresponding to the batch of data is determined; During the calculation of the first calculation result after partitioning, multiple calculation parts of the shared expert network are completed based on the hidden state result after partitioning. After each calculation part is completed, the corresponding computing resources are released to calculate the first calculation result of the next or subsequent hidden state result after partitioning. The hidden state result after partitioning can be sent to the corresponding GPU in parallel. After the sending is completed, the corresponding communication resources are released. or, The hidden state result corresponding to each batch of data is directly divided, and the first calculation result after division is determined based on the hidden state result after division and the shared expert network. The partitioned hidden state results are sent to the GPU where the corresponding routing expert network is located according to the routing decision, so as to calculate the second calculation result after partitioning; Based on the first calculation result after partitioning and the second calculation result after partitioning, the output result of the network layer corresponding to the batch of data is determined; During the calculation of the first calculation result after partitioning, multiple calculation parts of the shared expert network are completed based on the hidden state results after partitioning. After each calculation part is completed, the corresponding computing resources are released to calculate the first calculation result of the hidden state results of the next or subsequent partitions. The hidden state results after partitioning can be sent to the corresponding GPU in parallel. After the sending is completed, the corresponding communication resources are released.

10. The method according to claim 1, characterized in that, The method further includes: The preset thread group is divided into a first thread bundle and a second thread bundle. The first thread bundle is used to share the computation of the expert network, and the second thread bundle is used to initiate a request to send the token to be processed to the GPU where the corresponding routing expert network is located. The first thread bundle and the second thread bundle can be executed simultaneously. Using the first thread bundle, a free buffer is selected from multiple buffers to load the hidden state result corresponding to the batch of data, and the calculation of the shared expert network is performed according to the hidden state result to obtain the first calculation result, and then the buffer is released; The second thread bundle selects an idle buffer from multiple buffers to load the routing decision corresponding to the batch of data and the hidden state result corresponding to the term to be processed. Based on the routing decision, a communication descriptor is constructed and a request is initiated to send the hidden state result corresponding to the term to be processed to the corresponding GPU, and the buffer is released.

11. A large-scale model reasoning device, characterized in that, The large model includes multiple network layers, each network layer including multiple expert networks, the expert networks including a shared expert network and a routing expert, and the device includes: The calculation module is used to acquire multiple words to be processed. In the network layer, the corresponding hidden state results are calculated based on the multiple words to be processed and the attention mechanism. During the calculation process, the corresponding hidden state results are calculated using the words to be processed as a batch of data as units. The first determining module is used to determine the routing decision corresponding to the word to be processed based on the hidden state result and the gating network, and is used to determine the routing expert network; Create a module to create multiple parallel processing task queues for the unprocessed words in a batch of data; The first sending module is used to send the hidden state result corresponding to the batch of data to the first task queue, and to send the routing decision corresponding to the batch of data and the hidden state result corresponding to the word to be processed to the second task queue. The second determining module is used to determine the first calculation result corresponding to the batch of data by using the hidden state results in the first task queue and the shared expert network. The second sending module is used to send the hidden state result corresponding to the word to be processed to the GPU deployed by the corresponding routing expert network through the routing decision in the second task queue, so as to calculate the second calculation result corresponding to the batch of data; The third determining module is used to determine the output result of the network layer corresponding to the batch of data based on the first calculation result and the second calculation result.

12. A device, characterized in that, include: Memory, processor; The memory is used to store computer programs; The processor is configured to execute the computer program for: Multiple unprocessed words are obtained. In the network layer of the large model, the corresponding hidden state results are calculated based on the multiple unprocessed words and the attention mechanism. During the calculation process, the corresponding hidden state results are calculated using the unprocessed words in a batch of data as units. The network layer includes multiple expert networks, including a shared expert network and a routing expert network. Based on the hidden state results and the gating network, the routing decision corresponding to the word to be processed is determined, which is used to determine the routing expert network; For each word element to be processed in a batch of data, create multiple task queues for parallel processing; Send the hidden state result corresponding to the batch of data to the first task queue, and send the routing decision corresponding to the batch of data and the hidden state result corresponding to the word to be processed to the second task queue; The first calculation result corresponding to the batch of data is determined by the hidden state results in the first task queue and the shared expert network. Through the routing decision in the second task queue, the hidden state result corresponding to the word to be processed is sent to the GPU deployed by the corresponding routing expert network to calculate the second calculation result corresponding to the batch of data; Based on the first calculation result and the second calculation result, the output result of the network layer corresponding to the batch of data is determined.

13. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by one or more processors, it causes the one or more processors to perform the steps of the method according to claims 1-10.

Images