Model reasoning method, host, computer system, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a model reasoning method, a host, a computer system, electronic equipment and a storage medium. The method is executed by a first host in a computer system, and comprises the following steps: in response to receiving a first lexical element sent by first computing equipment, determining at least one target expert sub-network corresponding to the first lexical element from a plurality of expert sub-networks, the first lexical unit is distributed to at least one target computing device deployed with at least one target expert sub-network, so that the at least one target computing device performs computing on the first lexical unit through the target expert sub-network deployed on the target computing device to generate an intermediate computing result, and the first computing device is managed by the first host; and based on the intermediate calculation result generated by the at least one target calculation device, generating a reasoning result corresponding to the first lexical element. According to the method, the communication video memory overhead of the computing equipment can be reduced, the end-to-end reasoning delay of the model is shortened, and the system reasoning throughput rate is improved.

Description

Technical Field

[0001] Embodiments of this disclosure relate to the field of artificial intelligence, specifically to a model reasoning method, a host, a computer system, an electronic device, and a storage medium. Background Technology

[0002] With the rapid development of artificial intelligence technology, the number of parameters in large-scale models is growing exponentially. To further expand the model parameter capacity while maintaining computational efficiency, a Mixture of Experts (MoE) model architecture can be applied. The inference process of a MoE model involves complex steps such as attention network computation, routing decisions, word distribution, expert sub-network computation, and result combination, placing high demands on the computing power and communication bandwidth of computing devices. Therefore, achieving high-efficiency, low-latency model inference has become one of the current technical challenges that needs to be addressed. Summary of the Invention

[0003] This disclosure provides at least one embodiment of a model inference method, wherein the method is applied to a computer system including multiple computing devices, wherein multiple expert subnetworks are distributedly deployed on the multiple computing devices, and the method is executed by a first host in the computer system, comprising: in response to receiving a first lexical unit sent by the first computing device, determining at least one target expert subnetwork corresponding to the first lexical unit from the multiple expert subnetworks, and distributing the first lexical unit to at least one target computing device on which the at least one target expert subnetwork is deployed, so that the at least one target computing device performs calculations on the first lexical unit using the target expert subnetwork deployed thereon to generate intermediate calculation results, wherein the first computing device is managed by the first host; and generating an inference result corresponding to the first lexical unit based on the intermediate calculation results generated by the at least one target computing device.

[0004] In the model inference method provided in at least one embodiment of this disclosure, determining at least one target expert subnetwork corresponding to the first word from the plurality of expert subnetworks includes: determining the index and weight of at least one target expert subnetwork corresponding to the first word from the plurality of expert subnetworks according to routing rules.

[0005] In the model inference method provided in at least one embodiment of this disclosure, the first host and the first computing device are located on the same computing node. The step of distributing the first lexical to at least one target computing device deployed with the at least one target expert subnetwork includes: in response to the at least one target computing device including a second computing device located on a different computing node from the first host, sending the first lexical to the target computing node where the second computing device is located, so that a second host in the target computing node used to manage the second computing device obtains the first lexical for transmission to the second computing device; in response to the at least one target computing device including a first computing device located on the same computing node as the first host, adding the first lexical to a set of lexical to be processed for transmission to the first computing device.

[0006] In the model inference method provided in at least one embodiment of this disclosure, the step of sending the first word to the target computing node where the second computing device is located includes: in response to the existence of multiple computing devices in the target computing node that require the first word, sending the first word to the target computing node only once.

[0007] In the model inference method provided in at least one embodiment of this disclosure, the step of distributing the first lexical unit to at least one target computing device having the at least one target expert sub-network deployed further includes: sending the index and weight of the at least one target expert sub-network corresponding to the first lexical unit to the target computing node, so that a second host in the target computing node used to manage the second computing device can obtain the index and weight.

[0008] In a model inference method provided in at least one embodiment of this disclosure, the first host manages multiple first computing devices, and the method further includes: in response to obtaining a second lexical unit from a remote computing node, adding the second lexical unit to the set of lexical units to be processed; dividing the set of lexical units to be processed into multiple subsets according to the index of the target expert subnetwork corresponding to each lexical unit in the set of lexical units to be processed, and sending the multiple subsets to the multiple first computing devices managed by the first host respectively.

[0009] In the model inference method provided in at least one embodiment of this disclosure, in the computer system, data on the computing device side is stored using a first data arrangement, and data on the host side is stored using a second data arrangement. Before determining at least one target expert subnetwork corresponding to the first term from the plurality of expert subnetworks, the method further includes: converting the data arrangement of the first term from the first data arrangement to the second data arrangement. Before sending the plurality of subsets to the plurality of first computing devices managed by the first host, the method further includes: converting the data arrangement of each term in the plurality of subsets from the second data arrangement to the first data arrangement.

[0010] In the model reasoning method provided in at least one embodiment of this disclosure, the first word token sent by the first computing device is generated based on the result calculated by the first computing device through an attention network.

[0011] In the model reasoning method provided in at least one embodiment of this disclosure, generating the reasoning result corresponding to the first word based on the intermediate calculation results generated by the at least one target computing device includes: aggregating the intermediate calculation results generated by the at least one target computing device based on the weights of the at least one target expert sub-network to obtain the reasoning result corresponding to the first word.

[0012] In the model inference method provided in at least one embodiment of this disclosure, the step of generating the inference result corresponding to the first lexical based on the intermediate calculation results generated by the at least one target computing device is performed during the process of processing a batch of lexicals including the first lexical. The process includes: obtaining multiple intermediate calculation results generated in the computing node where the first host is located, wherein each intermediate calculation result carries lexical identifier information; performing a first aggregation process on the multiple intermediate calculation results based on the lexical identifier information of the multiple intermediate calculation results to obtain at least one local aggregation result; sending the at least one local aggregation result to multiple computing nodes in the computer system according to the lexical-computing node mapping relationship; and in response to obtaining a remote aggregation result from a remote computing node, obtaining the inference result of the lexical corresponding to the computing device managed by the first host based on the obtained remote aggregation result and the local aggregation result, wherein the inference result of the lexical corresponding to the computing device managed by the first host includes the inference result corresponding to the first lexical.

[0013] In the model inference method provided in at least one embodiment of this disclosure, all lexical units in the lexical batch are assigned global codes. The step of performing a first aggregation process on the multiple intermediate calculation results based on the lexical identification information of the multiple intermediate calculation results to obtain at least one local aggregation result includes: for each obtained intermediate calculation result, weighted summing of the intermediate calculation result to the target position corresponding to the global code in the global batch tensor based on the global code of the lexical unit corresponding to the intermediate calculation result and the weight corresponding to the intermediate calculation result; wherein the step of sending the at least one local aggregation result to multiple nodes in the computer system according to the lexical-computing node mapping relationship. The computing node includes: based on the mapping relationship between the global encoding and multiple computing nodes in the computer system, sending multiple sub-batch tensors in the global batch tensor to the multiple computing nodes respectively for acquisition by the host in the multiple computing nodes; wherein, in response to acquiring a remote aggregation result from a remote computing node, obtaining an inference result of the tokens corresponding to the computing device managed by the first host based on the acquired remote aggregation result and local aggregation result includes: in response to acquiring a sub-batch tensor from a remote computing node, performing a second aggregation process on the received sub-batch tensor and the local sub-batch tensor to obtain the inference result of the tokens corresponding to the computing device managed by the first host.

[0014] In the model reasoning method provided in at least one embodiment of this disclosure, multiple intermediate calculation results generated in the computing node where the first host is located are obtained, and the multiple intermediate calculation results are subjected to a first aggregation process based on the lexical identification information of the multiple intermediate calculation results, including: starting multiple worker threads to obtain the intermediate calculation results to be processed in a concurrent manner, and performing the first aggregation process on the intermediate calculation results to be processed based on the lexical identification information of the intermediate calculation results to be processed.

[0015] At least one embodiment of this disclosure provides a first host, wherein the first host is located in a computer system, the computer system including multiple computing devices, and multiple expert subnetworks are distributedly deployed on the multiple computing devices. The first host includes: a routing module configured to, in response to receiving a first term sent by the first computing device, determine at least one target expert subnetwork corresponding to the first term from the multiple expert subnetworks; a distribution module configured to distribute the first term to at least one target computing device on which the at least one target expert subnetwork is deployed, so that the at least one target computing device performs calculations on the first term using the target expert subnetwork deployed thereon to generate intermediate calculation results, wherein the first computing device is managed by the first host; and a combination module configured to generate an inference result corresponding to the first term based on the intermediate calculation results generated by the at least one target computing device.

[0016] At least one embodiment of this disclosure provides a computer system, wherein the computer system includes multiple computing devices, and multiple expert subnetworks are distributedly deployed on the multiple computing devices. A first host in the computer system is configured to: in response to receiving a first term sent by a first computing device, determine at least one target expert subnetwork corresponding to the first term from the multiple expert subnetworks, and distribute the first term to at least one target computing device on which the at least one target expert subnetwork is deployed, so that the at least one target computing device performs calculations on the first term using the target expert subnetwork deployed thereon to generate intermediate calculation results, wherein the first computing device is managed by the first host; and generate an inference result corresponding to the first term based on the intermediate calculation results generated by the at least one target computing device.

[0017] At least one embodiment of this disclosure provides an electronic device, including: at least one processor; at least one memory including one or more computer program modules; wherein the one or more computer program modules are stored in the at least one memory and configured to be executed by the at least one processor, and the one or more computer program modules are used to implement the method provided by at least one embodiment of this disclosure.

[0018] At least one embodiment of this disclosure provides a non-transitory computer-readable storage medium having computer-readable instructions stored thereon, wherein the computer-readable instructions, when executed by at least one processor, perform the method provided in at least one embodiment of this disclosure.

[0019] This disclosure provides at least one embodiment of a model inference method, host, computer system, electronic device, and storage medium, proposing a heterogeneous computing architecture based on host-assisted communication. By offloading routing computation tasks and communication tasks, including lexical distribution and result combination, to the host during model inference, computation and communication are decoupled. The computing device only needs to focus on the computation of the attention network and expert sub-network. This scheme is effectively applicable to computing devices with limited computing power or performance, reducing the communication and memory overhead of the computing device. Furthermore, host-assisted communication enables more parallel communication and computation strategies, thereby shortening the end-to-end inference latency of the model and improving the system inference throughput. Attached Figure Description

[0020] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings of the embodiments will be briefly described below. Obviously, the drawings described below only relate to some embodiments of this disclosure and are not intended to limit this disclosure.

[0021] Figure 1AA schematic block diagram of a monolithic system provided for at least one embodiment of this disclosure;

[0022] Figure 1B A schematic block diagram of a distributed system provided for at least one embodiment of this disclosure;

[0023] Figure 2 A schematic flowchart illustrating a model reasoning method provided for at least one embodiment of this disclosure;

[0024] Figure 3 A schematic block diagram illustrating a model reasoning method provided in at least one embodiment of this disclosure;

[0025] Figure 4 A schematic diagram illustrating a model reasoning method provided in at least one embodiment of this disclosure;

[0026] Figure 5 A schematic block diagram of a first host provided for at least one embodiment of this disclosure;

[0027] Figure 6 A schematic block diagram of a computer system provided for at least one embodiment of this disclosure;

[0028] Figure 7 A schematic block diagram of an electronic device provided for at least one embodiment of this disclosure;

[0029] Figure 8 A schematic block diagram of another electronic device provided for at least one embodiment of this disclosure;

[0030] Figure 9 This is a schematic block diagram of a non-transitory computer-readable storage medium provided for at least one embodiment of the present disclosure. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the described embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0032] Unless otherwise defined, the technical or scientific terms used in this disclosure shall have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms “first,” “second,” and similar terms used in this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as “comprising” or “including” mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as “connected” or “linked” are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as “upper,” “lower,” “left,” and “right” are used only to indicate relative positional relationships, and these relative positional relationships may change accordingly when the absolute position of the described objects changes.

[0033] The present disclosure will now be described through several specific embodiments. To keep the following description of the embodiments of the present disclosure clear and concise, detailed descriptions of known functions and known components may be omitted. When any component of an embodiment of the present disclosure appears in more than one drawing, the component is represented by the same or similar reference numerals in each drawing.

[0034] Mainstream large-scale models are typically implemented based on a Transformer architecture. In this architecture, the main body of the model consists of multiple stacked decoder modules. Each decoder module can include a self-attention module and a feed-forward network (FFN) module. The self-attention module captures the global dependencies and contextual information between tokens in the input sequence, while the feed-forward network module performs independent non-linear transformations and mappings on the features of each token. During model inference, the input data is processed layer by layer by all decoder modules, and the output of the last decoder layer serves as the model's final inference result.

[0035] With the rapid development of artificial intelligence technology, the number of parameters in large-scale models is growing exponentially. To further expand the model parameter capacity while maintaining computational efficiency, the Mixture of Experts (MoE) model architecture can be applied. MoE is a sparse model architecture that replaces the feedforward network module of the traditional dense Transformer architecture with multiple parallel expert sub-networks. Each of these expert sub-networks learns to process different types of data features or sub-tasks. A learnable routing mechanism, also known as a gating network, is introduced into the MoE model. For each term output by the self-attention module, the routing mechanism dynamically computes and selects a few of the most relevant expert sub-networks to participate in subsequent computations, rather than activating all expert sub-network parameters. For example, a MoE model may contain up to 256 expert sub-networks, but for each term, the routing mechanism only distributes it to at least a few expert sub-networks (e.g., 8) for processing. This sparse activation mechanism allows the model to have a huge total number of parameters while maintaining low inference computation, thus significantly improving the model's expressive power.

[0036] The inference process of MoE involves two communication and computation steps: Dispatch and Combine. Dispatch refers to the process of dynamically distributing tokens to one or more selected expert subnetworks based on a routing table generated by the routing mechanism. Combine refers to the process of weighting and fusing the outputs of each expert subnetwork according to routing weights after the selected one or more expert subnetworks have completed their computations on the token, thus restoring it to a single token feature vector. The result of the combiner process can be passed as input to the decoder module following the current decoder module.

[0037] The aforementioned inference process encompasses attention network computation, routing decisions, token distribution, expert subnetwork computation, and result combination, all executed uniformly by computing devices (e.g., GPUs). However, this architecture also introduces new challenges. Because different tokens are routed to different expert subnetworks, and these subnetworks are typically distributed across different computing devices, the data flow during inference is highly dynamic and irregular, leading to complex cross-device communication requirements. This places higher demands on the system architecture for large-scale cluster inference.

[0038] Specifically, large-scale MoE models, due to their massive number of parameters, consume a significant amount of GPU memory during inference for their model weights. To balance the GPU memory requirements of model weights, key-value cache (KVCache), and activation values ​​within limited hardware resources, cluster inference can be employed when the model size is too large. This not only balances GPU memory requirements but also improves the system's concurrent processing capacity and single-card throughput through distributed parallelism. Furthermore, since large-scale MoE models exhibit significant differences in the number of tokens processed and computational characteristics between the Prefill Phase (P phase) and the Decode Phase (D phase) (P phase is computationally intensive, D phase is memory-intensive), a cluster architecture with separate Prefill-Decode (PD) phases is typically adopted, deploying these two phases on dedicated compute node clusters to optimize resource utilization.

[0039] In the Decode cluster, tokens per second (TPS) generated per user is a key metric for measuring real-world user experience. Lower end-to-end inference latency translates to higher TPS per user. However, due to the frequent cross-node data exchanges in the MoE architecture, and the fact that all communication loads—including routing decisions, token distribution, and result aggregation—are directly borne by the computing devices, communication overhead has become a performance bottleneck. For example, communication time accounts for a very high percentage of end-to-end inference time in the Decode cluster, typically exceeding 50%, and even reaching 80% in complex routing or high-load scenarios. This significant communication latency severely limits the improvement of single-user TPS, hindering efficient inference for current large-scale MoE models.

[0040] This disclosure provides at least one embodiment of a model inference method, a host computer system, an electronic device, and a storage medium.

[0041] At least one embodiment of this disclosure provides a model inference method applied to a computer system including multiple computing devices, wherein multiple expert subnetworks are distributedly deployed on the multiple computing devices. The method is executed by a first host in the computer system and includes: in response to receiving a first word character sent by the first computing device, determining at least one target expert subnetwork corresponding to the first word character from the multiple expert subnetworks, and distributing the first word character to at least one target computing device on which at least one target expert subnetwork is deployed, so that the at least one target computing device performs calculations on the first word character using the target expert subnetwork deployed thereon to generate intermediate calculation results, wherein the first computing device is managed by the first host; and generating an inference result corresponding to the first word character based on the intermediate calculation results generated by the at least one target computing device.

[0042] The model inference method provided in at least one embodiment of this disclosure proposes a heterogeneous computing architecture based on host-assisted communication. By offloading the routing computation tasks and communication tasks, including lexical distribution and result combination, during the model inference process to the host, computation and communication are decoupled. The computing device only needs to focus on the computation of the attention network and the expert sub-network. This scheme is effectively applicable to computing devices with limited computing power or performance, reducing the communication and memory overhead of the computing device. Furthermore, host-assisted communication enables more parallel communication and computation strategies, thereby shortening the end-to-end inference latency of the model and improving the system inference throughput.

[0043] Figure 1A This is a schematic block diagram of a monolithic system provided for at least one embodiment of the present disclosure.

[0044] For example, Figure 1A The monolithic system shown may include one or more hosts, each of which can be directly connected to one or more computing devices via an internal bus or dedicated interface. This tightly coupled architecture enables the hosts to directly schedule computing device resources with extremely low communication latency, allowing them to work together to complete computing tasks.

[0045] Figure 1B This is a schematic block diagram of a distributed system provided for at least one embodiment of the present disclosure.

[0046] In distributed computing scenarios, computing nodes are the basic building blocks of distributed systems, enabling efficient parallel processing of large-scale tasks through collaboration and communication between nodes.

[0047] For example, Figure 1B The distributed system shown includes multiple computing nodes, each computing node may include one or more hosts, and each host may connect to one or more computing devices. Hosts and computing devices can collaborate to complete computing tasks. This disclosure does not limit the number of computing nodes in the distributed system, or the number of hosts and computing devices included in each computing node. Within a computing node, data transmission between hosts and computing devices can be achieved in various ways. Computing devices (also referred to as "slave devices") can either be directly connected to the host via a hardware interface (e.g., via a PCI Express, PCIe bus) to achieve low-latency data transmission, or they can collaborate with the host via a network in a software-defined manner (such as virtualization technology).

[0048] It should be noted that, Figure 1B As an example only, in the distributed system described above, each computing node may consist of only one host, and each host may be physically connected to only one computing device. Furthermore, the number of hosts in each computing node may be unequal, and the number of computing devices physically connected to each host may also be unequal.

[0049] Communication between computing nodes is implemented based on network communication mechanisms. These mechanisms can be based on standard protocols (such as Transmission Control Protocol / Internet Protocol (TCP / IP), User Datagram Protocol (UDP), etc.), Remote Procedure Call (RPC), message queues, or communication libraries for high-performance computing (such as Message Passing Interface (MPI), Remote Direct Memory Access (RDMA), etc.). The specific implementation method can be flexibly selected according to the application scenario. Aggregate communication is a communication mode for data exchange between multiple processes in distributed computing, including broadcast, scatter, and gather. It is used to efficiently achieve data synchronization and collaborative computing between computing nodes, and is usually supported by the communication libraries described above on top of the underlying network communication mechanisms.

[0050] For example, the host computer mentioned above may include a central processing unit (CPU), and the computing device may include a graphics processing unit (GPU), a general-purpose graphics processing unit (GPGPU), a tensor processing unit (TPU), a deep learning processing unit (DPU), an accelerated processing unit (APU), a neural network processing unit (NPU), an application-specific integrated circuit (ASIC), or a field-programmable gate array (FPGA), etc. (also known as an artificial intelligence processor).

[0051] This disclosure provides at least one embodiment of a model inference method, applied to a computer system including multiple computing devices, such as... Figure 1A The single-unit system shown or Figure 1BThe example shown is a distributed system. Of course, the application scenarios of this method are not limited to this; it can also be extended to other types of computer systems.

[0052] The models described above can be, for example, large-scale AI models (also known as AI large-scale models, or simply "large models") built from AI networks. Large-scale models can be content generation models based on prompt words, such as Large Language Models (LLMs), visual large-scale models, or multimodal large-scale models. They can also be models based on the Transformer architecture or models built using attention mechanisms. For instance, a model can be a hybrid expert model, comprising multiple stacked decoder modules. Each decoder module includes a cascaded self-attention module and a hybrid expert (MoE) module. The self-attention module is configured to capture global dependencies and contextual information between lexical units in the input sequence based on an attention mechanism. The hybrid expert module includes a routing network (also known as a gating network) and multiple expert subnetworks. These expert subnetworks can be implemented, for example, based on feedforward networks. During data processing, the routing network is configured to calculate the assigned weights corresponding to each expert subnetwork for the input tokens (from the output of the self-attention module), and to select a target number of expert subnetworks from multiple expert subnetworks (e.g., by implementing a Top-K selection strategy) for activation; subsequently, only the activated target expert subnetworks perform calculations on the input tokens. The outputs of each target expert subnetwork are weighted and fused according to the assigned weights to obtain the output of the MoE module. This output may optionally undergo residual calculation and normalization before being passed to the next decoder module. It should be noted that the above description of the structure of the hybrid expert model is only an exemplary embodiment of this disclosure and does not constitute a limitation on the scope of protection of this disclosure. The hybrid expert model may also include other types of functional layers (such as convolutional layers, multilayer perceptron layers, etc.), and the internal structure of the decoder module, the specific topology of the expert subnetworks, the routing network selection algorithm, etc., can all be adjusted and modified according to the actual application scenario. This disclosure does not limit the specific type, structure, or implementation method of the model.

[0053] For example, the aforementioned computer system employs an Expert Parallelism (EP) strategy, dividing the expert set containing all expert sub-networks into segments, distributing these expert sub-networks across multiple computing devices within the cluster (i.e., the computer system). In expert parallelism mode, a single computing device is only responsible for the computation of a portion of the expert sub-networks within the expert set, while the complete model inference capability is achieved through dynamic scheduling and efficient communication coordination via a routing mechanism. For instance, depending on the expert parallelism strategy, a single computing device can be responsible for the computation of one or more expert sub-networks; that is, one or more expert sub-networks can be deployed on a single computing device. This disclosure does not impose any limitations on this aspect.

[0054] The model reasoning method provided in at least one embodiment of this disclosure is executed by a first host in a computer system. Here, "first host" is used only to refer to the subject currently being described. It can be any host in the computer system, without specifically referring to a particular host, nor indicating any particular order, hierarchical relationship, or quantity limitation.

[0055] Figure 2 This is a schematic flowchart illustrating a model reasoning method provided for at least one embodiment of the present disclosure.

[0056] For example, such as Figure 2 As shown, the model reasoning method provided in at least one embodiment of this disclosure includes steps S101 to S102.

[0057] Step S101: In response to receiving the first word character sent by the first computing device, at least one target expert subnetwork corresponding to the first word character is determined from multiple expert subnetworks, and the first word character is distributed to at least one target computing device that has deployed at least one target expert subnetwork, so that at least one target computing device can use the target expert subnetwork deployed thereon to perform calculations on the first word character to generate intermediate calculation results, wherein the first computing device is managed by the first host.

[0058] Step S102: Based on the intermediate calculation results generated by at least one target computing device, generate the reasoning result corresponding to the first word element.

[0059] For example, the first computing device can be one of the aforementioned computing devices directly managed by the first host, such as a computing device directly connected to the first host's PCIe slot. Here, "first" does not imply any specific order, hierarchy, or quantity limitation. The first host is responsible for the PCIe communication of the first computing device. The first computing device can send data to the first host via a Device-to-Host (D2H) communication link; the first host can send data to the first computing device via a Host-to-Device (H2D) communication link. To further improve transmission efficiency, the aforementioned communication links can employ Direct Memory Access (DMA) technology, allowing data transfer operations to be executed independently by the hardware controller without host intervention; or, a zero-copy mechanism can be used to avoid redundant data copying between the host-side user space and kernel space, directly realizing address mapping and data interaction between device memory and host memory, thereby reducing data transfer latency and saving processor resources.

[0060] In at least one embodiment of this disclosure, before the host performs route calculation, the lexical units sent by the computing device to the host are derived from the computational output of the attention network. For example, in step S101, the first lexical unit sent by the first computing device is generated based on the result calculated by the first computing device through the attention network. It should be noted that "first lexical unit" here refers to the lexical unit sent by the first computing device, and does not indicate any specific order, hierarchical relationship or quantity limitation.

[0061] In some examples, the computer system is configured in a one-to-one host-device management topology. In this configuration, each host in the computer system directly manages only one computing device, and each computing device has a complete attention network deployed on it. Specifically, the first host may directly manage only a single first computing device, establishing an exclusive communication channel between them. In this case, the first computing device independently performs the complete attention computation, and the resulting computation is the complete feature vector output by the attention network. This complete feature vector may optionally undergo residual computation and normalization processing, and can be sent directly to the first host as the first term via the D2H communication link without undergoing a cross-device all-reduce operation.

[0062] In other examples, the computer system is configured as a one-to-many host-device management topology. In this configuration, each host in the computer system directly manages multiple computing devices. Specifically, a first host can directly manage multiple first computing devices and interact with each first computing device via independent H2D and D2H communication links. When the computer system employs a Tensor Parallelism (TP) strategy, the multiple first computing devices managed by the first host can collectively form a TP group. Tensor parallelism refers to breaking down a layer (or operator) of the model and distributing some of the operator's weights across different computing devices, thereby reducing the memory usage of a single computing device (e.g., a GPU). Each first computing device only holds a portion of the weights of the attention network and the corresponding activation value portions generated during computation. After the attention network computation is complete, multiple first computing devices need to perform a full TP reduction communication operation to reconstruct the complete semantic features. This operation aggregates the attention calculation results distributed across various first computing devices in segments (e.g., through global summation or concatenation) to logically reconstruct a complete feature vector. This complete feature vector may optionally undergo residual calculation and normalization. To improve communication efficiency and fully utilize multi-link bandwidth, a parallel transmission strategy can be adopted. Each first computing device sends a partial segment of the complete feature vector as a first term to the first host in parallel via independent D2H links. That is, each first computing device sends a different first term.

[0063] For example, in step S101, after receiving the token sent by the computing device, the host performs subsequent route calculations. For example, a routing network used to calculate the routing table is deployed in the host. During the route calculation process, the host can select one or more target expert subnetworks for the input token and determine the indices of these target expert subnetworks. For example, the first host can determine at least one target expert subnetwork corresponding to the received first token from all the multiple expert subnetworks.

[0064] For example, one example of "determining at least one target expert subnetwork corresponding to the first term from multiple expert subnetworks" in step S101 may include: determining the index and weight of at least one target expert subnetwork corresponding to the first term from multiple expert subnetworks according to routing rules.

[0065] For example, the index of the target expert sub-network is used to uniquely identify the selected specific expert among multiple target expert sub-networks, so that the host can correctly distribute the first term to the corresponding expert in subsequent distribution operations; the weight of the target expert sub-network is used for subsequent fusion operations, and the host can use the above weights to weight the output vectors of each target expert sub-network to obtain the inference result corresponding to the term. Further, the host can generate a routing table based on the indexes and weights of the target expert sub-networks corresponding to the received multiple terms, for use in subsequent distribution operations. For example, each entry in the routing table can record term identification information, the index of one or more target expert sub-networks corresponding to the term, and the weight of one or more target expert sub-networks corresponding to the term. It should be noted that the routing table can also record more information, and this embodiment of the disclosure does not limit this. For example, term identification information can be used to uniquely identify terms in the input sequence, such as the position index of the term in the sequence or a globally unique identifier (ID), so that the inference result corresponding to the term can be restored to the corresponding position in the original sequence. It should be noted that the above is only one example of a routing table. Different implementation methods can be used depending on actual needs, such as building an index routing table and a weighted routing table separately. This disclosure does not limit this approach.

[0066] For example, routing rules can be Top K routes, dynamic adaptive routes, hierarchical routes, etc. These rules define the architecture of the routing network, and this disclosure does not limit them.

[0067] Top K routing refers to calculating the gating scores of all expert subnetworks for a single term, selecting the K expert subnetworks with the highest scores as target expert subnetworks, and obtaining their corresponding indices; subsequently, determining the weights based on the gating scores of these K experts. For example, the gating scores can be used directly as weights, or the results of further data processing of the gating scores can be used as weights. The value of K can be set according to actual needs, and this embodiment does not limit this. For example, when K=1, the host determines one target expert subnetwork from multiple expert subnetworks; when K>1, the host determines multiple target expert subnetworks from multiple expert subnetworks.

[0068] Dynamic adaptive routing refers to dynamically determining the number of expert subnetworks to be activated based on the feature complexity of the first lexical unit (i.e., the value of K is not fixed). For simple lexical units, routing is done to a smaller number of expert subnetworks to save computational resources, while for complex lexical units, routing is done to a larger number of expert subnetworks to improve expressive power.

[0069] Hierarchical routing refers to dividing multiple expert subnetworks into multiple levels or groups and using a cascading approach for route selection. First, the first-layer router filters out the target group from multiple groups. Then, the second-layer router further filters out the specific target expert subnetworks within the target group and determines their corresponding weights. Alternatively, a subset of candidate expert subnetworks can be selected first, and then a secondary selection can be performed on the set of candidate expert subnetworks to finally determine the target expert subnetwork used to process the first term.

[0070] This disclosure does not limit the method of calculating the gating score. For example, a linear projection network combined with a normalized exponential (Softmax) function can be used, that is, the feature vector of the first word is mapped to the original score of each expert subnetwork through a learnable linear layer, and then transformed into a probability distribution by the Softmax function; or a similarity-based calculation method can be used, that is, the cosine similarity or Euclidean distance between the feature vector of the first word and the prototype vectors corresponding to each expert subnetwork is calculated, and the negative value of the similarity or distance is used as the gating score; or a multilayer perceptron (MLP) can be used as the gating network to enhance the expressive power of the routing mechanism through nonlinear transformation.

[0071] For example, the index precision can be set to INT16, while the weight precision can be selected as BF16 or FP32. The specific configuration can be determined according to the communication optimization requirements.

[0072] For example, in step S101, after completing the route calculation, the host continues to execute the subsequent dispatch operation, and the communication process is driven by the host. Based on the expert parallel strategy adopted by the computer system, each host can maintain a mapping table between the index of the expert subnetwork and each computing device in the computer system. In the dispatch operation, the host can distribute the corresponding token to the corresponding computing device based on the index of the target expert subnetwork recorded in the routing table and the aforementioned mapping table. For example, it can look up the corresponding computing device in the mapping table based on the index of the target expert subnetwork. For instance, the first host can distribute the first token received from the first computing device to at least one target computing device that has deployed at least one target expert subnetwork.

[0073] In the model inference method provided in at least one embodiment of this disclosure, the routing computation task and the communication tasks, including lexical distribution and result combination, in the model inference process are all offloaded to the host for execution, thus decoupling computation and communication. The computing device only needs to focus on the computation of the attention network and the expert sub-network. This scheme is effectively applicable to computing devices with limited computing power or performance (e.g., computing devices that cannot support direct asynchronous zero-copy communication between GPU memory and network card), reducing the communication and GPU memory overhead of the computing device. Furthermore, host-assisted communication enables more parallel communication and computation strategies, thereby shortening the end-to-end inference latency of the model and improving the system inference throughput.

[0074] For a monolithic system, if the first host identifies multiple target expert sub-networks for the first term and confirms that these multiple target expert sub-networks are located in different target computing devices, the first host can send the first term to these target computing devices respectively; if multiple target expert sub-networks are located in the same target computing device, the first host can send the first term to the target computing device only once to reduce the amount of data sent and avoid redundant sending.

[0075] In a distributed system, when the first host and the target computing device are located in the same computing node, the first word can be directly transmitted to the target computing device (e.g., via an H2D communication link). When the first host and the target computing device are located in different computing nodes, the first word needs to be sent to the corresponding remote computing node (here, "remote" is relative to the first host), and then the host in the remote computing node that manages the target computing device will send the first word to the target computing device.

[0076] For example, in a distributed system, the first host and the first computing device are located on the same computing node. An example of "distributing the first word to at least one target computing device that has deployed at least one target expert subnetwork" in step S101 may include steps S1011 to S1012.

[0077] Step S1011: In response to at least one target computing device including a second computing device located on a different computing node from the first host, the first term is sent to the target computing node where the second computing device is located, so that the second host in the target computing node, which manages the second computing device, can obtain the first term for transmission to the second computing device.

[0078] Step S1012: In response to the fact that at least one target computing device includes the first computing device, the first lexical is added to the set of lexical to be processed for transmission to the first computing device.

[0079] In step S1011, "second computing device" refers to a computing device located on a different computing node than the first host, that is, a device located on a remote computing node. When the first host and the target computing device are located on different computing nodes, the first host needs to send the first token to the corresponding remote computing node, and then the host in the remote computing node that manages the target computing device (i.e., the second host) sends the first token to the target computing device (e.g., via an H2D communication link).

[0080] In step S1012, the first host and the first computing device are located on the same computing node, and the first computing device is managed by the first host. When the target computing device is the first computing device, it indicates that the computing device used to generate the first word and the computing device used to perform expert subnetwork calculation on the first word are the same computing device, that is, the first computing device undertakes both the task of generating the word and performing expert subnetwork calculation on the word. At this time, the first host can add the first word to the set of words to be processed, so that it can be subsequently transmitted to the first computing device through the H2D communication link.

[0081] It should be noted that the terms "first host", "second host", "first computing device" and "second computing device" mentioned above are only used to distinguish different hosts or computing devices, and do not specifically refer to any particular host or computing device, nor do they indicate any specific order, hierarchical relationship or quantity limit.

[0082] After the distribution operation is completed, each target computing device obtains the first word element and can use its deployed target expert sub-network to perform calculations on the first word element to generate intermediate calculation results. This calculation operation is, for example, the forward propagation calculation of the target expert sub-network. It can be understood that each target expert sub-network generates one intermediate calculation result; if there are multiple target experts on a target computing device, then that target computing device will have multiple intermediate calculation results.

[0083] For example, one example of "sending the first word to the target computing node where the second computing device is located" in step S1011 may include: in response to the existence of multiple computing devices in the target computing node that require the first word, sending the first word only once to the target computing node.

[0084] For example, "the computing device that requires the first term" refers to the computing device indicated by the index of the target expert sub-network corresponding to the first term. If the computing devices indicated by the indices of multiple target expert sub-networks are located on the same computing node, the first host only needs to send the first term to that computing node once, without needing to transmit it repeatedly for each target expert sub-network. That is, data interaction is performed on a per-computation-node basis in the distribution operation, which means that the same term data only needs to be transmitted to the target computing node once, thereby effectively avoiding redundant data transmission and improving communication efficiency.

[0085] For example, one example of "distributing the first word to at least one target computing device having at least one target expert subnetwork" in step S101 may also include step S1013.

[0086] Step S1013: Send the index and weight of at least one target expert sub-network corresponding to the first word element to the target computing node, so that the second host in the target computing node used to manage the second computing device can obtain the index and weight.

[0087] For example, when the first host and the target computing device are located in different computing nodes, the first host needs to send the index and weight of at least one target expert sub-network corresponding to the first word to the corresponding remote computing node so that the host in the remote computing node that manages the target computing device can obtain the above index and weight.

[0088] In the model inference method provided in at least one embodiment of this disclosure, after each host in the computer system completes routing calculation, the first host may receive tokens sent by a remote computing node. The tokens sent by the remote computing node are generated based on the results calculated by the computing device in the remote computing node through an attention network. Similarly, the first host may also receive the index and weight of the target expert subnetwork corresponding to the tokens sent by the remote computing node.

[0089] The model inference method provided in at least one embodiment of this disclosure may further include steps S103 to S104. In this example, a first host manages multiple first computing devices, and the first host and the multiple first computing devices are located in the same computing node.

[0090] Step S103: In response to receiving the second word from the remote computing node, the first host adds the second word to the set of words to be processed.

[0091] Step S104: Based on the index of the target expert sub-network corresponding to each word in the word set to be processed, the first host divides the word set to be processed into multiple subsets and sends the multiple subsets to multiple first computing devices managed by the first host.

[0092] For example, in step S103, "remote computing node" refers to a computing node in the computer system other than the computing node where the first host is located. It should be noted that "second term" here refers to a term sent by the remote computing node, used to distinguish it from the "first term" sent by the first computing device, and does not indicate any specific order, hierarchical relationship, or quantity limit. The set of terms to be processed contains terms for the expert subnetwork computation performed by the multiple first computing devices managed by the first host.

[0093] For example, in step S104, the first host can divide the set of words to be processed into multiple subsets specific to the first computing device according to the index of the target expert sub-network corresponding to each word in the set of words to be processed, and transmit them in parallel to their respective matched first computing devices through the H2D communication link.

[0094] For example, suppose a host manages two computing devices, denoted as G0 and G1 for ease of description. The set of terms to be processed contains six terms {T0, T1, T2, T3, T4, T5}. Expert subnetworks {E0, E1} are deployed on G0, and expert subnetworks {E2, E3} are deployed on G1. In one example, the correspondence between the terms and the indexes of the target expert subnetworks is as follows: {T0-E0, T1-E1, T2-E3, T3-E2, T4-E0, T5-E3}. The host can divide the set of terms to be processed into two subsets based on these indexes: S0 = {T0, T1, T4} and S1 = {T2, T3, T5}. Subset S0 is sent to G0, and subset S1 is sent to G1. By merging multiple fine-grained communications targeting the same computing device into a single coarse-grained communication, not only is communication overhead reduced, but the link bandwidth is also fully utilized through parallel transmission, significantly improving inference throughput.

[0095] For example, the first host can also calculate a word-target expert subnetwork index mapping table for each of the multiple first computing devices it manages, based on the index of the target expert subnetwork corresponding to each word in the set of words to be processed, and send it to the corresponding first computing device to instruct the expert subnetwork deployed in the first computing device to process which words.

[0096] For example, in the example above, the host computes two lexical-target expert subnetwork index mapping tables for the two computing devices, namely M0={T0-E0,T1-E1,T4-E0} and M1={T2-E3,T3-E2,T5-E3}. M0 indicates that lexicals T0 and T4 need to be processed by expert E0 deployed on G0, and lexical T1 needs to be processed by expert E1 deployed on G0; M1 indicates that lexicals T2 and T5 need to be processed by expert E3 deployed on G1, and lexical T3 needs to be processed by expert E2 deployed on G1, thereby guiding the operation of the computing devices.

[0097] Tensor layout (also referred to as data arrangement in this embodiment) refers to the arrangement rules of tensor elements in storage space (such as memory or video memory). By indicating the storage order and dimensional arrangement of data, it can affect the data access efficiency and the throughput of hardware computing units, thereby affecting the hardware parallel computing efficiency.

[0098] Common tensor layout types include, but are not limited to, the following:

[0099] Row-major layout: Consecutive rows of data are stored sequentially in contiguous storage locations (e.g., memory). For example, in a two-dimensional array, all data within a row is stored contiguously, while data between different rows may have significant address jumps.

[0100] Column-major layout: Consecutive columns of data are stored sequentially in contiguous memory locations. For example, in a two-dimensional array, all data within a single column is stored contiguously, while data between different columns may have significant memory address jumps.

[0101] Stride Layout: By specifying the stride for each dimension, the jump distance of tensor elements in the storage space is defined. In other words, the stride for each dimension defines how many bytes to jump to the next element, which can flexibly support custom layouts of any dimension.

[0102] Dimensional order layout: This determines the layout of a tensor in storage space by specifying the order in which its dimensions are arranged. For example, a four-dimensional tensor includes batch (B) dimension, channel (C) dimension, height (H) dimension, and width (W) dimension. Two common dimensional order layouts are as follows:

[0103] NCHW layout: Batch, Channel, Height, Width are arranged in dimensional order, which is suitable for mathematical operations on the CPU and facilitates channel-level operations.

[0104] NHWC layout: arranged in dimensional order of Batch, Height, Width, and Channel, suitable for GPU memory access patterns.

[0105] The tensor layout types described above are merely illustrative examples. The tensor layouts (i.e., data arrangement methods) involved in this disclosure are not limited to the aforementioned examples. Various tensor layouts can be flexibly defined and extended according to different application scenarios and hardware characteristics. This disclosure does not impose any restrictions on this.

[0106] In the model inference method provided in at least one embodiment of this disclosure, a heterogeneous data storage strategy can be adopted to address the differences in hardware characteristics between the computing device side (e.g., accelerators such as GPUs and NPUs) and the host side (e.g., CPU): data on the computing device side is stored using a first data arrangement method, and data on the host side is stored using a second data arrangement method. The first and second data arrangement methods are different. For example, the first data arrangement method is a column-first layout, and the second data arrangement method is a row-first layout. The above are merely examples, and the embodiments of this disclosure do not limit the specific data arrangement method; any applicable layout form can be adopted based on the actual application scenario.

[0107] Due to the differences in data arrangement on both sides, when data needs to be exchanged between the host and computing devices, layout transformation or data reordering operations are required. These operations convert the data into a format that matches the optimal access mode of the target hardware, thereby significantly improving data access efficiency, hardware computing unit throughput, and parallel computing efficiency, and maximizing memory bandwidth utilization.

[0108] Correspondingly, before determining at least one target expert subnetwork corresponding to the first word element from multiple expert subnetworks, the model inference method provided in at least one embodiment of this disclosure further includes: the first host converting the data arrangement of the first word element from a first data arrangement to a second data arrangement.

[0109] For example, the data arrangement methods in this disclosure not only cover the layout defining the storage order and dimensional arrangement of data, but also further include the fragmented aggregation form of data in a multi-device parallel environment. For example, the first data arrangement method is a column-first fragmentation form, and the second data arrangement method is a row-first aggregation form. For example, in a scenario where a host (e.g., CPU) manages four computing devices (e.g., GPU), each computing device holds a 4×7168 (representing four tokens, each token being 7168 in length) column-first matrix as the first data arrangement method, while the second data arrangement method on the host side is to merge and rearrange these four independent column-first matrices into a complete 16×7168 row-first matrix, or rearrange them into 16 consecutive 1×7168 row-first vectors. It should be noted that the above matrix size and splicing method are only examples, and this disclosure does not limit them.

[0110] For example, the first term after the data arrangement is transformed can be used as input for both subsequent routing calculations and subsequent distribution operations. This arrangement transformation operation not only improves the efficiency of the host in transmitting term data but also enhances the efficiency of the host in performing subsequent routing calculations and distribution operations. Furthermore, since the arrangement transformation operation is performed by the host, rather than by the computing device, the computing device can focus on executing computational tasks, significantly improving both bandwidth utilization and the parallel computing efficiency and resource utilization of the computer system. Moreover, for operations such as table lookup and data rearrangement, when processing small datasets, the host (e.g., CPU) exhibits a significant performance advantage, with its execution efficiency far exceeding that of computing device-based (e.g., GPU) implementations, effectively reducing end-to-end latency.

[0111] Correspondingly, before sending the multiple subsets to the multiple first computing devices managed by the first host, the model inference method provided in at least one embodiment of this disclosure further includes: the first host converting the data arrangement of each word in the multiple subsets from the second data arrangement to the first data arrangement.

[0112] For example, in a scenario where a host manages four computing devices, the host holds an m×7168 row-major matrix as the second data arrangement method, while each computing device holds an m×7168 row-major matrix. i A column-priority matrix of 7168 (where i = 0, 1, 2, 3, and m0 + m1 + m2 + m3 = m) is used as the first data arrangement method.

[0113] For example, before the computing device performs expert subnetwork computation, the host can convert the data arrangement of the tokens into a data arrangement suitable for the computing device in advance, without the computing device having to perform a rearrangement operation. This allows the computing device to focus on the execution of the computation task, which improves bandwidth utilization and significantly enhances the parallel computing efficiency and resource utilization of the computer system.

[0114] After each target computing device completes the expert subnetwork computation, a combination operation needs to be performed to aggregate the output results of each target expert for each word, restoring them to a single word feature vector. For example, the aggregation can be a reduction operation or other forms of linear or nonlinear aggregation operations, and this disclosure does not limit this. For example, the reduction operation can include summation reduction, weighted reduction, mean reduction, etc. For example, summation reduction is a basic reduction operation, which means summing each output result element-wise, that is, accumulating the elements at the same position in each output result; mean reduction means summing each output result element-wise and then taking the arithmetic mean; weighted reduction means weighting each output result according to the weights corresponding to its source expert subnetwork, and then summing element-wise.

[0115] In the model inference method provided in at least one embodiment of this disclosure, the combination operation is performed by the host, and the communication process involved in the operation is also driven by the host.

[0116] For example, in step S102, the inference result can be defined as an intermediate inference result or a final inference result depending on the network layer where the current calculation is performed. For example, if the current processing flow has not yet reached the last decoder layer of the model, the generated inference result is an intermediate inference result, which will be used as input data to be transferred to the next level decoder layer to continue performing deep feature extraction; if the current process is in the calculation stage of the last decoder layer, the generated inference result is a final inference result, and the model's prediction result can be directly generated based on this inference result.

[0117] An example of step S102 may include: based on the weights of at least one target expert sub-network, the first host aggregates the intermediate computation results generated by at least one target computing device to obtain the inference result corresponding to the first word element.

[0118] It is understandable that for a single word, each target expert subnetwork generates an intermediate computation result. If there are multiple target expert subnetworks on a target computing device, then the target computing device corresponds to multiple intermediate computation results.

[0119] For example, after receiving all intermediate computation results for the first term, the first host can perform a weighted reduction operation on these intermediate computation results based on the weights of at least one target expert sub-network to obtain the inference result corresponding to the first term. For example, the first host can sum the intermediate computation results element-wise after weighting them according to their corresponding weights. The aforementioned intermediate computation results can come from computing devices in remote computing nodes or from computing devices in local computing nodes. It is understood that the terms "remote" and "local" are relative to the first host, which is the object of the current description. It should be noted that in the above example, both inter-node communication and intra-node communication are driven by the host, not by the computing devices. Thus, the communication overhead is borne by the host, while the computing devices can focus on performing computational tasks, thereby improving the system's throughput and computational efficiency. In addition, thanks to the high performance of the CPU's single-core clock speed, CPU-driven RDMA communication is significantly better than the GPU direct-connect RDMA scheme based on the PCIe bus in terms of latency control and throughput efficiency, thereby further reducing communication latency.

[0120] Another example of step S102 may include: the first host performing a summation and reduction operation on the intermediate computation results generated by at least one target computing device to obtain the inference result corresponding to the first word. For example, the first host sums each intermediate computation result element by element.

[0121] Another example of step S102 is performed during the processing of a batch of lexical units including the first lexical unit. This process is performed by the first host and involves intra-node combination operations and inter-node combination operations, and may include the following steps S1021 to S1024.

[0122] Step S1021: Obtain multiple intermediate calculation results generated in the computing node where the first host is located, wherein each intermediate calculation result carries word identifier information.

[0123] Step S1022: Based on the lexical identifier information of multiple intermediate calculation results, perform a first aggregation process on each of the multiple intermediate calculation results to obtain at least one local aggregation result.

[0124] Step S1023: Send at least one local aggregation result to multiple computing nodes in the computer system according to the word-computing node mapping relationship.

[0125] Step S1024: In response to obtaining the remote aggregation result from the remote computing node, obtain the inference result of the word corresponding to the computing device managed by the first host based on the obtained remote aggregation result and the local aggregation result.

[0126] In at least one embodiment of this disclosure, the combination operation of the calculation results of the expert subnetwork corresponding to the first token is performed during the processing of token batches containing the first token. For example, a token batch is the set of all tokens that need to be processed in parallel during a single forward propagation of inference. For example, in the decoding stage, each user request generates only one token in a single forward propagation. If N user requests are processed simultaneously, then the token batch size Token_Batch_Size=N, which represents the number of user requests that the system can process in parallel during a single inference.

[0127] For example, steps S1021 to S1022 represent intra-node combination operations.

[0128] In step S1021, the first host collects multiple intermediate computation results generated in its local computing node, each carrying token identifier information. In other words, the first host collects intermediate computation results generated by all computing devices participating in the expert subnetwork computation within its local computing node (the computing node where the first host resides). These intermediate computation results may correspond to different tokens; the token identifier information carried within them allows for differentiation and association of these intermediate computation results, thereby improving the accuracy of subsequent combination operations. One implementation of the token identifier information is global encoding, which involves uniformly numbering all tokens in a token batch and assigning each token a globally unique encoded identifier.

[0129] In step S1022, the local aggregation result refers to the aggregation result of the intermediate calculation results of the token in the local computing node, and each local aggregation result corresponds to one token. For example, the first aggregation process can be a reduction operation or other forms of linear or nonlinear aggregation operation, and this embodiment of the disclosure does not limit it.

[0130] An example of step S1022 may include: for each intermediate calculation result obtained, based on the global encoding of the word corresponding to the intermediate calculation result and the weight corresponding to the intermediate calculation result, the intermediate calculation result is weighted and accumulated to the target position corresponding to the global encoding in the global batch tensor.

[0131] For example, the computing node where the first host resides can maintain a global batch tensor. Its first dimension has the same length as the word batch size, and its second dimension has the same length as the feature dimension of the word (e.g., the hidden layer size), used to collect multiple local aggregation results. Each element in the global batch tensor is initialized to 0. For each intermediate computation result obtained, the first host first parses the global encoding it carries, locating the unique target position indicated by that global encoding within the global batch tensor. Simultaneously, it multiplies the intermediate computation result by its corresponding weight and accumulates the weighted result at the target position.

[0132] In one example, the computing node where the first host is located contains only one host (i.e., the first host itself). In this case, in step S1021, the first host can collect all the intermediate computing results generated in its computing node; in step S1022, the first host can perform a first aggregation process on all the intermediate computing results based on the word identifier information of all the intermediate computing results to obtain at least one local aggregation result.

[0133] When the primary host is a multi-core host, the above steps can be implemented using a single-process, multi-threaded approach. For example, the primary host can launch multiple worker threads to concurrently acquire intermediate computation results and perform a first aggregation process based on the lexical identifiers of these results. Alternatively, each core of the primary host can launch a worker thread, employing a pipelined approach to concurrently acquire tasks from the task queue and execute the acquisition of intermediate computation results and the first aggregation process according to the tasks. It is understood that the above is merely an example; the specific number of cores used can be set according to actual needs and does not necessarily require the use of all cores.

[0134] In another example, the computing node where the first host is located contains multiple hosts (that is, in addition to the first host, there are other hosts in the local computing node). In this case, in step S1021, multiple hosts will jointly collect all the intermediate computing results generated in their respective computing nodes; in step S1022, multiple hosts will jointly perform a first aggregation process on all the intermediate computing results based on the lexical identifier information of all the intermediate computing results, and obtain at least one local aggregation result.

[0135] The above steps can be implemented using a single-process, multi-threaded approach. For example, the first host and other hosts in the compute node can start multiple worker threads to concurrently acquire intermediate computation results and perform a first aggregation process based on the lexical identifier information of the intermediate computation results. For instance, each core of each host can start a worker thread, using a pipelined approach to concurrently acquire tasks from the task queue and execute the acquisition of intermediate computation results and the first aggregation process according to the tasks. It is understood that the specific number of cores used in each host can be set according to actual needs; the above is merely an example and does not necessarily require the use of all cores.

[0136] The model inference method provided in at least one embodiment of this disclosure effectively utilizes the multi-core parallel capabilities of the host, meets the requirements of high-performance computing for low latency and high concurrency, and effectively improves computing efficiency and throughput.

[0137] For example, steps S1023 to S1024 represent the combination operation between nodes.

[0138] For example, in step S1023, the first host can send at least one local aggregation result to multiple computing nodes in the computer system according to the lexical-computing node mapping relationship. The lexical-computing node mapping relationship defines the correspondence between each lexical in a lexical batch and multiple computing nodes in the computer system, that is, it describes which computing node performs the attention network computation for each lexical in the lexical batch. For example, assuming that computing node Node0 performs the attention computation for lexicals T0~T31 and computing node Node1 performs the attention computation for lexicals T32~T63, then the lexical-computing node mapping relationship can be expressed as: Node0 corresponds to {T0…T31}, and Node1 corresponds to {T32…T63}.

[0139] An example of step S1023 may include: based on the mapping relationship between global encoding and multiple computing nodes in a computer system, the first host sends multiple sub-batch tensors in the global batch tensor to multiple computing nodes respectively, so that the hosts in the multiple computing nodes can obtain them.

[0140] For example, a global batch tensor can be divided into multiple sub-batch tensors according to the above mapping relationship and sent to the corresponding computing nodes. For instance, it can be divided at the computing node granularity, with each computing node corresponding to one sub-batch tensor. It should be noted that during the transmission of sub-batch tensors, both the sender and receiver are hosts, not computing devices. Therefore, the communication overhead is borne by the host, allowing the computing devices to focus on executing computational tasks, thereby improving system throughput and computational efficiency.

[0141] For example, in step S1024, in response to obtaining the remote aggregation result from the remote computing node, the first host obtains the inference result of the tokens corresponding to the computing devices managed by the first host based on the obtained remote aggregation result and the local aggregation result. For example, the remote aggregation result represents the aggregation result obtained by performing the aggregation operation internally in the remote computing node and sent to the computing node where the first host is located.

[0142] An example of step S1024 may include: in response to obtaining a sub-batch tensor from a remote computing node, performing a second aggregation process on the received sub-batch tensor and the local sub-batch tensor to obtain the inference result of the lexical corresponding to the computing device managed by the first host.

[0143] For example, a sub-batch tensor from a remote computing node is an example of the remote aggregation result mentioned above. The host can perform a second aggregation process based on the received sub-batch tensor and the previously computed local sub-batch tensor to obtain the inference result of the lexical corresponding to the computing device managed by the first host.

[0144] In some examples, the first aggregation process is a weighted reduction operation, and the second aggregation process is a summation reduction operation. This can be understood as first performing a weighted accumulation operation within a node to obtain the local reduction result, and then performing a full reduction across nodes, effectively reducing the amount of communication data across nodes and lowering communication overhead. In other examples, both the first and second aggregation processes are summation reduction operations. This can be understood as first performing an accumulation operation within a node to obtain the local reduction result, and then performing a full reduction across nodes. In this example, no weights are used. It should be noted that the above combination of the first and second aggregation processes is only one example, and the embodiments disclosed herein are not limited thereto.

[0145] It is understandable that, since steps S1021 to S1024 above are performed on a batch of words containing the first word, the reasoning result of the word corresponding to the computing device managed by the first host obtained in step S1024 includes the reasoning result corresponding to the first word.

[0146] Figure 3 This is a schematic block diagram illustrating a model reasoning method provided in at least one embodiment of the present disclosure.

[0147] Figure 4 This is a schematic diagram of a model reasoning method provided in at least one embodiment of the present disclosure. Figure 4 for Figure 3 A specific example of routing and distribution operations.

[0148] For example, combining Figure 3 and Figure 4 This distributed system employs a hybrid parallel strategy combining tensor parallelism, data parallelism, and expert parallelism to adapt to inference in large-scale hybrid expert models. Specifically, a combination of 4-way tensor parallelism (TP4) and 8-way data parallelism (DP8) is used in the attention network computation stage, while 32-way expert parallelism (EP32) is used in the expert subnetwork computation stage. The distributed system comprises four computing nodes (Node 0 to Node 3), each configured with two CPUs and eight GPUs. In terms of communication topology, each CPU manages the PCIe communication links for the four GPUs. Figure 4For simplicity, only a single CPU and its corresponding four GPUs are shown in the diagram. The entire distributed system is divided into eight DP groups (DP_0~DP_7), with two DP groups deployed within each compute node, and each DP group comprising four GPUs. Simultaneously, the four GPUs within each DP group form a TP group. This distributed system deploys a total of 256 expert subnetworks EP0~EP255, with eight expert subnetworks deployed on each GPU. Furthermore, to address the load imbalance issue in the hybrid expert model, Figure 3 The document also shows 32 redundant experts (REP 0 ~ REP 31). These redundant experts are dynamically replicated copies of high-load experts to address specific expert overload issues. These copies are distributed across different GPUs to offload the load from the original experts. It should be noted that, for clarity of the system architecture, [further details are needed]. Figure 3 Only two compute nodes are shown as an example.

[0149] The batch size is set to 128 (e.g., the batch includes terms T0 to T127, each with a unique global encoding). Due to the use of 8-way data parallelism, each DP group is responsible for the attention network computation of 16 terms. Therefore, a complete batch can be divided into 8 consecutive batch segments (BS_0 to BS_7). The specific allocation is as follows: DP_0 processes BS_0={T0…T15}, DP_1 processes BS_1={T16…T31}, and so on, DP_6 processes BS_6={T96…T111}, and DP_7 processes BS_7={T112…T127}. Figure 3 and Figure 4 The MLA shown represents Multi-head LatentAttention, as a concrete example of an attention network implementation.

[0150] Taking Node 0 as an example, it includes two hosts, CPU0_0 and CPU0_1. CPU0_0 manages computing devices GPU0_1 to GPU0_3, and CPU0_1 manages computing devices GPU0_4 to GPU0_7.

[0151] In this example, the hidden size of the model is 7168. CPU0_0 is selected as a representative example of the first host mentioned earlier. Each of the computing devices GPU0_1 to GPU0_3 managed by CPU0_0 only holds a partial weight slice of the attention network. GPU0_1 to GPU0_3 all use batch fragments BS_0={T0…T15} as input and execute the attention network computation in parallel. After the attention network computation is completed, in order to reconstruct the complete semantic features, GPU0_1 to GPU0_3 need to perform a TP full reduction communication operation to aggregate the attention computation result slices distributed on GPU0_1 to GPU0_3, thereby logically reconstructing a complete attention feature vector with a dimension of 16×7168.

[0152] To improve communication efficiency and fully utilize multi-link bandwidth, this embodiment employs a parallel transmission strategy. GPU0_1 to GPU0_3 each send a local slice (4×7168 dimensions, column-first arrangement) of their complete attention feature vector to CPU0_0 in parallel via independent D2H links. Each local slice includes four complete tokens and can be processed independently by CPU0_0.

[0153] The host CPU0_0 receives four local fragments from the four GPUs and performs the first rearrangement operation, converting the four 4×7168 column-matrixed matrices into a single 16×7168 row-matrixed matrix (equivalent to 16 complete tokens). This transformed matrix can be used as input for both subsequent routing calculations and distribution operations. This transformation operation not only improves the efficiency of the CPU in transmitting token data but also enhances the efficiency of the CPU in performing subsequent routing calculations and distribution operations.

[0154] Next, CPU0_0 performs routing calculations based on the 16×7168 row-matrix (taking the Top 8 as an example). For each of the 16 terms, the top 8 target expert subnetworks with the highest scores are selected. The routing calculation process can generate two sets of data for each term: the routing expert index and the corresponding routing weight. Tables 1 and 2 show the routing expert indexes and their routing weights for some terms. Specifically, Table 1 shows the indices of the top 8 experts (TOP1~TOP8 experts) in descending order of routing score for different terms; Table 2 corresponds to the expert indices in Table 1, and the values ​​represent the weights assigned to the corresponding experts. It should be noted that the data shown in Tables 1 and 2 below are only illustrative examples and do not constitute a limitation on the embodiments of this disclosure. Taking Table 2 as an example, the weight values ​​listed therein can be normalized values ​​or unnormalized raw values.

[0155] Table 1. Expert Index of Lexical Routing

[0156]

[0157] Table 2. Distribution of lexical routing weights

[0158]

[0159] Next, CPU0_0, based on the obtained lexical routing expert index table and lexical routing weight distribution table, distributes these 16 lexical terms to the target GPUs where the corresponding target experts reside. In this example, GPU0_0 is responsible for calculating experts EP0~EP7, GPU0_1 is responsible for calculating experts EP8~EP15, and so on.

[0160] Referring to Table 1, taking term T0 as an example, its eight corresponding target experts are EP1, EP3, EP64, EP68, EP32, EP31, EP67, and EP192. EP1, EP3, EP31, and EP32 are all located at Node0, EP64, EP67, and EP68 are all located at Node1, and EP192 is located at Node3. Therefore, CPU0_0 needs to send term T0 to Node1 and Node3, and also needs to add term T0 to its own set of terms to be processed. The distribution operation is performed on a per-component basis; therefore, even though there are three target experts corresponding to term T0 in Node1, term T0 only needs to be sent to Node1 once, effectively reducing redundant data communication caused by repeated transmissions. The distribution operation is driven by the CPU; the GPU does not participate. For example, Figure 4 A row-matrix matrix of dimension n×7168 represents the n tokens sent by Node0 to a remote node (e.g., Node1). Similarly, CPU0_0 can also send the calculated token routing expert index table and token routing weight distribution table to Node1 and Node3.

[0161] Each CPU in this distributed system follows the same operational logic as CPU0_0; the specific process can be found in the previous description of CPU0_0, and will not be repeated here. Therefore, CPU0_0 will also receive tokens sent by other nodes and add the received tokens to the set of tokens to be processed. Figure 4 In the example, the set of words to be processed contains m words. CPU0_0 performs the second rearrangement operation, transforming a matrix of dimension m×7168, arranged in row-major order, into m... iA p×7168 matrix (where i = 0, 1, 2, 3, and m0 + m1 + m2 + m3 = m), column-majorly arranged. The m0×7168 matrix consists of m0 words that need to be distributed to GPU0_0 for expert subnetwork computation, the m1×7168 matrix consists of m1 words that need to be distributed to GPU0_1 for expert subnetwork computation, and so on. CPU0_0 can also compute four word-target expert subnetwork index mapping tables for each of the four GPUs to indicate which words the expert subnetworks deployed on each GPU process. Similarly, in a remote node (e.g., Node1), if the set of words to be processed contains p words, the CPU in the remote node performs a second rearrangement operation, transforming a p×7168 row-major matrix into a p... i A matrix of size ×7168 (where i=0,1,2,3, and p0+p1+p2+p3=p), arranged in column order.

[0162] After the distribution operation is completed, each GPU in the computer system performs calculations on the corresponding received tokens using the target expert subnetwork deployed on it, based on the token-target expert subnetwork index mapping table, and generates intermediate calculation results for each token.

[0163] Intermediate computation results generated by each GPU can be transmitted to the corresponding CPU via the D2H communication link. Taking the compute node Node0, where CPU0_0 resides, as an example, the intra-node combination operation is executed first. Each core of the host CPU0_0 and CPU0_1 in compute node Node0 can start a worker thread, which concurrently retrieves tasks from the task queue in a pipelined manner. Based on the tasks, all intermediate computation results generated by multiple GPUs in Node0 are read sequentially. For each retrieved intermediate computation result, based on the global encoding of the corresponding token and the weight of the intermediate computation result, the intermediate computation result is weighted and accumulated to the target position in the global batch tensor corresponding to the global encoding.

[0164] Next, CPU0_0 performs inter-node combination operations. Based on the mapping relationship between the global encoding and multiple computing nodes in the distributed system, it sends multiple sub-batch tensors from the global batch tensor to multiple computing nodes for the CPUs on those nodes to access. For example, ... Figure 3 As shown, CPU0_0 will access the 7th sub-batch tensor in the global batch tensor maintained by Node0. Figure 3 The tensors in the middle are denoted as 96~111) and the 8th sub-batch tensor ( Figure 3The tensors (denoted as 112~127) are sent to compute node Node3. When a sub-batch tensor is received from another compute node, CPU0_0 performs element-wise summation between the received sub-batch tensor and the local sub-batch tensor to obtain the inference results of the terms corresponding to the compute devices GPU0_0~GPU0_3 managed by CPU0_0, i.e. Figure 3 The result group is 0.

[0165] The inference result of a word can be defined as an intermediate inference result or a final inference result depending on the network layer where the current computation takes place. For example, if the current processing flow has not yet reached the last decoder layer of the model, the generated inference result is an intermediate inference result, which will be used as input data to be passed to the next level decoder layer to continue performing deep feature extraction; if the current process is in the computation stage of the last decoder layer, the generated inference result is a final inference result, and the model's prediction result can be directly generated based on this inference result.

[0166] It should also be noted that the execution order of the various steps of the model inference method in the various embodiments of this disclosure is not limited. Although the execution process of each step has been described in a specific order above, this does not constitute a limitation on the embodiments of this disclosure. The various steps in the model inference method can be executed sequentially or in parallel, which can be determined according to actual needs.

[0167] For example, compared to the above description, the model reasoning method provided in at least one embodiment of this disclosure may include more or fewer steps, and the embodiments of this disclosure do not limit this.

[0168] Figure 5 This is a schematic block diagram of a first host provided for at least one embodiment of the present disclosure.

[0169] For example, such as Figure 5 As shown, the first host 500 provided in at least one embodiment of this disclosure includes a routing module 501, a distribution module 502, and a combination module 503. The first host is located in a computer system, which includes multiple computing devices, and multiple expert subnetworks are distributed and deployed on the multiple computing devices.

[0170] For example, routing module 501 is configured to determine at least one target expert subnetwork corresponding to the first term from a plurality of expert subnetworks in response to receiving a first term sent by the first computing device.

[0171] For example, the distribution module 502 is configured to distribute the first word to at least one target computing device that has deployed at least one target expert subnetwork, so that the at least one target computing device can use the target expert subnetwork deployed thereon to perform calculations on the first word to generate intermediate calculation results, wherein the first computing device is managed by a first host.

[0172] For example, the combination module 503 is configured to generate an inference result corresponding to the first word based on intermediate calculation results generated by at least one target computing device.

[0173] In some examples, the routing module is further configured to determine the index and weight of at least one target expert subnetwork corresponding to the first term from multiple expert subnetworks based on routing rules.

[0174] In some examples, the first host and the first computing device are located on the same computing node, and the distribution module is further configured to, in response to at least one target computing device including a second computing device located on a different computing node than the first host, send the first word to the target computing node where the second computing device is located, so that the second host in the target computing node, which manages the second computing device, obtains the first word for transmission to the second computing device; and in response to at least one target computing device including a first computing device located on the same computing node as the first host, add the first word to a set of words to be processed for transmission to the first computing device.

[0175] In some examples, the distribution module is further configured to send the first word only once to the target computing node in response to the presence of multiple computing devices in the target computing node that require the first word.

[0176] In some examples, the distribution module is further configured to send the index and weights of at least one target expert subnetwork corresponding to the first term to the target computing node, so that the second host in the target computing node, which manages the second computing device, can obtain the index and weights.

[0177] In some examples, the first host also includes a partitioning module configured to, in response to receiving a second term from a remote computing node, add the second term to a set of terms to be processed; divide the set of terms to be processed into multiple subsets according to the index of the target expert subnetwork corresponding to each term in the set of terms to be processed; and send the multiple subsets to multiple first computing devices managed by the first host.

[0178] In some examples, in a computer system, data on the computing device side is stored using a first data arrangement, and data on the host side is stored using a second data arrangement. The first host also includes a first re-arrangement module and a second re-arrangement module. The first re-arrangement module is configured to convert the data arrangement of the first term from the first data arrangement to the second data arrangement before determining at least one target expert subnetwork corresponding to the first term from multiple expert subnetworks. The second re-arrangement module is configured to convert the data arrangement of each term in the multiple subsets from the second data arrangement to the first data arrangement before sending the multiple subsets to the multiple first computing devices managed by the first host.

[0179] In some examples, the first word sent by the first computing device is generated based on the result calculated by the first computing device through an attention network.

[0180] In some examples, the combination module is further configured to aggregate intermediate computation results generated by at least one target computing device based on the weights of at least one target expert subnetwork to obtain the inference result corresponding to the first word unit.

[0181] In some examples, the combination module is further configured to obtain multiple intermediate computation results generated in the computing node where the first host is located, wherein each intermediate computation result carries lexical identifier information; based on the lexical identifier information of the multiple intermediate computation results, perform a first aggregation process on the multiple intermediate computation results respectively to obtain at least one local aggregation result; send the at least one local aggregation result to multiple computing nodes in the computer system according to the lexical-computing node mapping relationship; in response to obtaining a remote aggregation result from a remote computing node, obtain an inference result of the lexical corresponding to the computing device managed by the first host based on the obtained remote aggregation result and the local aggregation result, wherein the inference result of the lexical corresponding to the computing device managed by the first host includes the inference result corresponding to the first lexical.

[0182] In some examples, all tokens in a token batch are assigned a global encoding. The combination module is further configured to, for each intermediate calculation result obtained, weight and accumulate the intermediate calculation results to the target position in the global batch tensor corresponding to the global encoding based on the global encoding of the token corresponding to the intermediate calculation result and the weight corresponding to the intermediate calculation result; based on the mapping relationship between the global encoding and multiple computing nodes in the computer system, send multiple sub-batch tensors in the global batch tensor to multiple computing nodes for the host in multiple computing nodes to obtain; in response to obtaining the sub-batch tensor from the remote computing node, perform a second aggregation process on the received sub-batch tensor and the local sub-batch tensor to obtain the inference result of the tokens corresponding to the computing device managed by the first host.

[0183] In some examples, the combination module is further configured to launch multiple worker threads to concurrently obtain the intermediate computation results to be processed, and to perform a first aggregation process on the intermediate computation results based on the lexical identification information of the intermediate computation results to be processed.

[0184] It should be noted that the various modules and units mentioned above can be implemented by software, hardware, firmware or any combination thereof. For example, the routing module, distribution module and combination module can be implemented as routing circuit, distribution circuit and combination circuit respectively. The embodiments of this disclosure do not limit their specific implementation methods.

[0185] It should be understood that the first host provided in at least one embodiment of this disclosure can be used to implement the aforementioned model inference method, and can also achieve similar technical effects as the model inference method, which will not be elaborated here.

[0186] It should be noted that in the embodiments of this disclosure, the first host may include more or fewer modules or units, and the connection relationship between the various modules or units is not limited and can be determined according to actual needs. The specific configuration of each module or unit is not limited and can be constructed from analog devices, digital chips, or other suitable methods according to circuit principles.

[0187] Figure 6 This is a schematic block diagram of a computer system provided for at least one embodiment of the present disclosure.

[0188] For example, such as Figure 6 As shown, the computer system 600 includes multiple computing devices 601, and multiple expert subnetworks are distributedly deployed on the multiple computing devices 601. The first host 602 in the computer system is configured to: in response to receiving a first word character sent by the first computing device, determine at least one target expert subnetwork corresponding to the first word character from the multiple expert subnetworks, and distribute the first word character to at least one target computing device that has deployed at least one target expert subnetwork, so that the at least one target computing device can use the target expert subnetwork deployed on it to perform calculations on the first word character to generate intermediate calculation results, wherein the first computing device is managed by the first host; and generate an inference result corresponding to the first word character based on the intermediate calculation result generated by the at least one target computing device.

[0189] The first host can be any host in the computer system. Figure 6 The first host 602 in the middle can, but is not limited to, using Figure 5 The structure of the first host 500 is shown. Further descriptions of the first host can be found in the above method embodiments or... Figure 5 The descriptions in the embodiments are not repeated here.

[0190] Figure 7 This is a schematic block diagram of an electronic device provided for at least one embodiment of the present disclosure.

[0191] For example, such as Figure 7 As shown, the electronic device 700 includes at least one processor 701 and at least one memory 702. The at least one memory 702 includes one or more computer program modules. These computer program modules are stored in the memory 702 and configured to be executed by the at least one processor 701. The one or more computer program modules include instructions for performing the model inference method described above. When executed by the at least one processor 701, they can perform one or more steps of the model inference method provided in at least one embodiment of this disclosure. The memory 702 and the processor 701 can be interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0192] For example, processor 701 can be a central processing unit (CPU), digital signal processor (DSP), graphics processing unit (GPU), general-purpose graphics processing unit (GPGPU), artificial intelligence (AI) accelerator, or other form of processing unit with data processing and / or program execution capabilities, such as a field-programmable gate array (FPGA); for example, the central processing unit (CPU) can be an x86, ARM, or RISC-V architecture. Processor 701 can be a general-purpose processor or a special-purpose processor, capable of controlling other components in electronic device 700 to perform desired functions.

[0193] For example, memory 702 may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB memory, flash memory, etc.

[0194] Figure 8 This is a schematic block diagram of another electronic device provided for at least one embodiment of the present disclosure.

[0195] The electronic devices in at least one embodiment of this disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (PADs), portable multimedia players (PMPs), in-vehicle terminals (e.g., in-vehicle navigation terminals), wearable electronic devices, and fixed terminals such as digital TVs and desktop computers. Figure 8 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0196] The electronic device includes at least one processor and a memory. The processor may be referred to as processing device 801 as described below, and the memory may include at least one of read-only memory (ROM), random access memory (RAM), and storage device 808 as described below. The memory is used to store programs for performing the methods described in the various method embodiments above; the processor is configured to execute the programs stored in the memory. The processor may include a central processing unit (CPU) or other forms of processing unit having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0197] like Figure 8 As shown, the electronic device 800 may include a processing unit 801 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) or a program loaded from a storage device 808 into a random access memory (RAM). Various programs and data required for the operation of the electronic device 800 are also stored in RAM 803. The processing unit 801, ROM 802, and RAM 803 are interconnected via a bus 804. Input / output (I / O) interfaces are also connected to the bus 804.

[0198] Typically, the following devices can be connected to I / O interface 805: input devices 806 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 807 including, for example, displays, speakers, vibrators, etc.; storage devices 808 including, for example, magnetic tapes, hard disks, etc.; and communication devices 809. Communication device 809 allows electronic device 800 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 8 An electronic device 800 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0199] In particular, according to at least one embodiment of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, at least one embodiment of this disclosure includes a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device 809, or installed from a storage device 808, or installed from a ROM 802. When the computer program is executed by a processing device 801, it performs the functions defined in the methods of at least one embodiment of this disclosure.

[0200] It should be noted that the computer-readable medium described above in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In at least one embodiment of this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In at least one embodiment of this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, radio frequency (RF), etc., or any suitable combination thereof.

[0201] The aforementioned computer-readable medium may be included in the aforementioned electronic device 800; or it may exist independently and not assembled into the electronic device 800.

[0202] Figure 9 This is a schematic block diagram of a non-transitory computer-readable storage medium provided for at least one embodiment of the present disclosure.

[0203] For example, such as Figure 9 As shown, a non-transitory computer-readable storage medium 900 stores computer-readable instructions 901, which, when executed by at least one processor, perform one or more steps of the model inference method described above.

[0204] For example, the storage medium may include a memory card for a smartphone, a storage component for a tablet computer, a hard drive for a personal computer, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), flash memory, or any combination of the above storage media, or other suitable storage media. For example, the readable storage medium may also be... Figure 7 The memory 702 in the memory is described in the foregoing content and will not be repeated here.

[0205] Although the present disclosure has been described in detail above with general descriptions and specific embodiments, modifications or improvements can be made to the embodiments of the present disclosure, which will be obvious to those skilled in the art. Therefore, all such modifications or improvements made without departing from the spirit of the present disclosure are within the scope of protection claimed by the present disclosure.

[0206] The following points should be noted regarding this disclosure:

[0207] (1) The accompanying drawings of the embodiments of this disclosure only involve the structures involved in the embodiments of this disclosure. Other structures can be referred to the general design.

[0208] (2) For clarity, the thickness of layers or regions in the drawings used to describe embodiments of the present disclosure is enlarged or reduced, i.e., these drawings are not drawn to actual scale.

[0209] (3) Where there is no conflict, the embodiments of this disclosure and the features in the embodiments can be combined with each other to obtain new embodiments.

[0210] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. The scope of protection of this disclosure should be determined by the scope of protection of the claims.

[0211] Although the present disclosure has been described in detail above with general descriptions and specific embodiments, modifications or improvements can be made to the embodiments of the present disclosure, which will be obvious to those skilled in the art. Therefore, all such modifications or improvements made without departing from the spirit of the present disclosure are within the scope of protection claimed by the present disclosure.

[0212] The following points should be noted regarding this disclosure:

[0213] (1) The accompanying drawings of the embodiments of this disclosure only involve the structures involved in the embodiments of this disclosure. Other structures can be referred to the general design.

[0214] (2) For clarity, the thickness of layers or regions in the drawings used to describe embodiments of the present disclosure is enlarged or reduced, i.e., these drawings are not drawn to actual scale.

[0215] (3) Where there is no conflict, the embodiments of this disclosure and the features in the embodiments can be combined with each other to obtain new embodiments.

[0216] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. The scope of protection of this disclosure should be determined by the scope of protection of the claims.

Claims

1. A model reasoning method, characterized in that, The method is applied to a computer system comprising multiple computing devices, wherein multiple expert subnetworks are distributedly deployed on the multiple computing devices, and the method is executed by a first host in the computer system, comprising: In response to receiving a first word segment sent by a first computing device, at least one target expert subnetwork corresponding to the first word segment is determined from the plurality of expert subnetworks, and the first word segment is distributed to at least one target computing device on which the at least one target expert subnetwork is deployed, so that the at least one target computing device uses the target expert subnetwork deployed thereon to perform calculations on the first word segment to generate intermediate calculation results, wherein the first computing device is managed by the first host; Based on the intermediate calculation results generated by the at least one target computing device, the reasoning result corresponding to the first word element is generated.

2. The model reasoning method according to claim 1, characterized in that, Determining at least one target expert subnetwork corresponding to the first word element from the plurality of expert subnetworks includes: The index and weight of at least one target expert subnetwork corresponding to the first term are determined from the plurality of expert subnetworks according to the routing rules.

3. The model reasoning method according to claim 1, characterized in that, The first host and the first computing device are located on the same computing node. Distributing the first term to at least one target computing device deployed with the at least one target expert sub-network includes: In response to the fact that the at least one target computing device includes a second computing device located on a different computing node from the first host, the first word is sent to the target computing node where the second computing device is located, so that the second host in the target computing node used to manage the second computing device can obtain the first word for transmission to the second computing device; In response to the at least one target computing device including a first computing device located on the same computing node as the first host, the first lexical is added to the set of lexicals to be processed for transmission to the first computing device.

4. The model reasoning method according to claim 3, characterized in that, Sending the first word to the target computing node where the second computing device is located includes: In response to the presence of multiple computing devices requiring the first word in the target computing node, the first word is sent to the target computing node only once.

5. The model reasoning method according to claim 3, characterized in that, The step of distributing the first word token to at least one target computing device deployed with the at least one target expert sub-network further includes: The index and weight of the at least one target expert subnetwork corresponding to the first word are sent to the target computing node so that the second host in the target computing node, which manages the second computing device, can obtain the index and weight.

6. The model reasoning method according to claim 3, characterized in that, The first host manages multiple first computing devices, and the method further includes: In response to receiving a second word from a remote computing node, the second word is added to the set of words to be processed; Based on the index of the target expert subnetwork corresponding to each word in the word set to be processed, the word set to be processed is divided into multiple subsets, and the multiple subsets are respectively sent to multiple first computing devices managed by the first host.

7. The model reasoning method according to claim 6, characterized in that, In the computer system, data on the computing device side is stored using a first data arrangement, and data on the host side is stored using a second data arrangement. Before determining at least one target expert subnetwork corresponding to the first term from the plurality of expert subnetworks, the method further includes: The data arrangement of the first word element is changed from the first data arrangement method to the second data arrangement method. The method further includes, before sending the plurality of subsets to the plurality of first computing devices managed by the first host, the subsets being sent to each of the subsets respectively. The data arrangement of each word element in the plurality of subsets is changed from the second data arrangement to the first data arrangement.

8. The model reasoning method according to claim 1, characterized in that, The first word token sent by the first computing device is generated based on the result calculated by the first computing device through an attention network.

9. The model reasoning method according to claim 1, characterized in that, The generation of the inference result corresponding to the first word unit based on the intermediate calculation result generated by the at least one target computing device includes: Based on the weights of the at least one target expert sub-network, the intermediate calculation results generated by the at least one target computing device are aggregated to obtain the inference result corresponding to the first word element.

10. The model reasoning method according to claim 1, characterized in that, The generation of the inference result corresponding to the first lexical unit based on the intermediate calculation result generated by the at least one target computing device is performed during the processing of a batch of lexical units including the first lexical unit, and the process includes: Obtain multiple intermediate computation results generated in the computing node where the first host is located, wherein each intermediate computation result carries word identifier information; Based on the lexical identifier information of the multiple intermediate calculation results, the multiple intermediate calculation results are subjected to a first aggregation process to obtain at least one local aggregation result; The at least one local aggregation result is sent to multiple computing nodes in the computer system according to the word-computing node mapping relationship; In response to obtaining the remote aggregation result from the remote computing node, the inference result of the tokens corresponding to the computing device managed by the first host is obtained based on the obtained remote aggregation result and the local aggregation result. The reasoning result of the word corresponding to the computing device managed by the first host includes the reasoning result corresponding to the first word.

11. The model reasoning method according to claim 10, characterized in that, All lexical units in the lexical batch are assigned a global code. Specifically, the step of performing a first aggregation process on the multiple intermediate calculation results based on the lexical identifier information of the multiple intermediate calculation results to obtain at least one local aggregation result includes: For each intermediate calculation result, based on the global encoding of the corresponding word and the weight of the intermediate calculation result, the intermediate calculation result is weighted and accumulated in the global batch tensor to the target position corresponding to the global encoding. The step of sending the at least one local aggregation result to multiple computing nodes in the computer system according to the lexical-computing node mapping relationship includes: Based on the mapping relationship between the global encoding and multiple computing nodes in the computer system, multiple sub-batch tensors in the global batch tensor are sent to the multiple computing nodes respectively, so that the hosts in the multiple computing nodes can obtain them. Specifically, in response to obtaining the remote aggregation result from the remote computing node, the inference result of the lexical units corresponding to the computing device managed by the first host is obtained based on the obtained remote aggregation result and the local aggregation result, including: In response to receiving a sub-batch tensor from a remote computing node, the received sub-batch tensor is combined with the local sub-batch tensor for a second aggregation process to obtain the inference result of the lexical corresponding to the computing device managed by the first host.

12. The model reasoning method according to claim 10, characterized in that, Obtain multiple intermediate computation results generated in the computing node where the first host is located, and perform a first aggregation process on each of the multiple intermediate computation results based on the lexical identifier information of the multiple intermediate computation results, including: Multiple worker threads are started to obtain intermediate calculation results to be processed concurrently, and the intermediate calculation results to be processed are subjected to a first aggregation process based on the word identifier information of the intermediate calculation results to be processed.

13. A first host computer, characterized in that, The first host is located in a computer system, which includes multiple computing devices, and multiple expert subnetworks are distributedly deployed on the multiple computing devices. The first host includes: The routing module is configured to, in response to receiving a first term sent by a first computing device, determine at least one target expert subnetwork corresponding to the first term from the plurality of expert subnetworks; The distribution module is configured to distribute the first word to at least one target computing device on which the at least one target expert subnetwork is deployed, so that the at least one target computing device can perform calculations on the first word using the target expert subnetwork deployed thereon to generate intermediate calculation results, wherein the first computing device is managed by the first host; The combination module is configured to generate the inference result corresponding to the first word unit based on the intermediate calculation result generated by the at least one target computing device.

14. A computer system, characterized in that, The computer system includes multiple computing devices, and multiple expert subnetworks are distributed and deployed on the multiple computing devices. The first host in the computer system is configured as follows: In response to receiving a first word segment sent by a first computing device, at least one target expert subnetwork corresponding to the first word segment is determined from the plurality of expert subnetworks, and the first word segment is distributed to at least one target computing device on which the at least one target expert subnetwork is deployed, so that the at least one target computing device uses the target expert subnetwork deployed thereon to perform calculations on the first word segment to generate intermediate calculation results, wherein the first computing device is managed by the first host; Based on the intermediate calculation results generated by the at least one target computing device, the reasoning result corresponding to the first word element is generated.

15. An electronic device, characterized in that, The electronic device includes: At least one processor; At least one memory, including one or more computer program modules; The one or more computer program modules are stored in the at least one memory and configured to be executed by the at least one processor, and the one or more computer program modules are used to implement the method according to any one of claims 1-12.

16. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer-readable instructions, wherein the computer-readable instructions, when executed by at least one processor, perform the method according to any one of claims 1-12.

Images