Communication system, data processing method and apparatus thereof

Abstract

A communication system. The communication system comprises a first computing unit and a second computing unit, a first expert module being deployed in the second computing unit; the first computing unit is configured to acquire a first feature, the first feature being a feature representation of a data sequence input to the first computing unit; the first computing unit is configured to identify from the first feature first sub-features corresponding to the first expert module; the first computing unit is configured to send a plurality of sub-features to the second computing unit during a communication process with the second computing unit, the plurality of sub-features sent to the second computing unit all being the first sub-features corresponding to the first expert module. In the present application, each feature slice contains only features that need to be executed by one expert module, then only communication of features that need to be executed by a specific expert module is performed during one communication process, and then feature execution is performed by means of the expert module, such that bandwidth occupation during communication can be increased, thus reducing waste of resources.

Description

Communication system, data processing method and apparatus thereof

[0001] The present application claims priority from the Chinese patent application No. 202410868853.0 filed on June 29, 2024, and entitled "Communication system, data processing method and apparatus thereof", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the field of artificial intelligence, and in particular to a communication system, a data processing method and an apparatus thereof. BACKGROUND

[0003] Artificial intelligence (AI) is the use of digital computers or digital computer-controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a branch of computer science that aims to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is the design principle and implementation method of various intelligent machines, enabling machines to have perception, reasoning and decision-making functions.

[0004] With the increase of computing power of computing devices, the increase of bandwidth of communication networks, and the accumulation of data of big data networks, it is being researched to deploy large-scale neural network models based on neural networks on distributed networks for computation and processing, to realize large-scale parallel data processing, distributed storage, elastic topology, high redundancy and non-linear operation of cloud computing and other applications.

[0005] Distributed model training (for example, MoE (Mixure-of-Experts, mixed expert model)) is one of the technical paths to realize super large-scale model training. The idea of this model is to train multiple neural networks (distributed in multiple computing nodes), and each computing node trains a different part of the data set. The input data of different devices passes through the respective routing network (for example, Gating network), and selects different expert nodes, so there will be communication between different computing units to realize the distribution of data. However, the communication across computing units often has the problem of low bandwidth utilization, resulting in a decline in overall computing performance. SUMMARY

[0006] In a first aspect, the present application provides a communication system, the communication system comprising a first computing unit and a second computing unit, the first computing unit being in communication with the second computing unit, the second computing unit being deployed with a first expert module; the first computing unit is configured to obtain a first feature, the first feature being a feature representation of a data sequence input to the first computing unit; the first computing unit is configured to identify a first sub-feature corresponding to the first expert module from the first feature; the first computing unit is configured to send a plurality of sub-features to the second computing unit in a communication process with the second computing unit, wherein the plurality of sub-features sent to the second computing unit are all the first sub-feature corresponding to the first expert module.

[0007] wherein the sub-feature corresponding to the expert module can be understood as a sub-feature that needs to be processed by the expert module, for example, the first sub-feature corresponding to the first expert module can be understood as: the first sub-feature is a feature that needs to be processed by the first expert module.

[0008] wherein the plurality of sub-features are all the sub-features sent by the first computing unit to the second computing unit in a communication process with the second computing unit, which are all corresponding to the same expert module (i.e. the first expert module) and do not correspond to other expert modules.

[0009] In the prior art, when the features are divided, the division is based on the address arrangement of the features in the storage space, for example, when the division number is 2, the features located in the first half of the address arrangement in the storage space can be taken as a feature slice, and the features located in the second half of the address arrangement in the storage space can be taken as a feature slice. When the features in the first half are processed, the expert modules needed to process each token in the features in the first half can be determined, and then the features of the tokens that need to be processed by the expert modules deployed on other nodes or computing units are sent to the other nodes or computing units through a communication process (such as AlltoAll between nodes or Allgather between computing units), which can cause the following problems:

[0010] Since the feature slicing is based on the address arrangement of the features in the storage space, each feature slice often contains the features of the tokens that need to be processed by each expert module in the entire expert module, for example, in the case shown in FIG. 4, each feature slice contains the features of the tokens that need to be processed by the expert module E1, the expert module E2, the expert module E3 and the expert module E4. In the communication process for a feature slice, the computing unit needs to communicate the features of the tokens processed by all the expert modules, for example, for the computing unit deployed with E11 and E21, the features of the tokens processed by the expert module E3 and the expert module E4 need to be sent to the computing unit on the node 12 in one communication process, and after the node 12 on the computing node receives the features corresponding to the expert module E3 and the expert module E4, it needs to respectively perform the communication and calculation of the features corresponding to the expert module E3 and E4 (for example, first perform Allgather and calculation of the features corresponding to E3, and then perform Allgather and calculation of the features corresponding to E4), and each feature slice needs to repeatedly perform this process, and each communication transmits a small amount of data of the expert module (for example, first perform communication of a small part of the features corresponding to E3, then perform communication of a small part of the features corresponding to E4, then perform communication of a small part of the features corresponding to E3, and then perform communication of a small part of the features corresponding to E4), which will cause low bandwidth occupation during communication, thereby causing waste of resources, and when processing data through the expert module, since the processing of the features on each feature slice needs to be performed in sequence, for a computing unit, when processing a feature slice, each expert module needs to be called in sequence for processing, and when processing another feature slice, each expert module needs to be called again for processing, which will cause large resource calling overhead (the main reason is that when calling the expert module, the parameters need to be loaded from the memory to the cache of the computing unit), for example, in the case shown in FIG. 4, each feature slice contains the features of the tokens that need to be processed by the expert module E1, the expert module E2, the expert module E3 and the expert module E4, and taking the computing unit deployed with E11 and E21 as an example, for feature slice 1, the computing unit needs to first call E11 to execute the data on feature slice 1 that needs to be executed by E1, and then call E21 to execute the data on feature slice 1 that needs to be executed by E2, then, for feature slice 2, the computing unit needs to first call E11 to execute the data on feature slice 2 that needs to be executed by E1, and then call E21 to execute the data on feature slice 2 that needs to be executed by E2, which means that in the processing of two feature slices, the expert module is called four times, and the resource calling overhead is large.

[0011] The idea of the embodiments of the present application is that when the features are sliced, it is not based on the address arrangement of the features on the storage space, but the features needed to be executed by each expert module are identified from the features, which is equivalent to each feature slice containing only the features needed to be executed by one expert module, and then only the features needed to be executed by the specific expert module are communicated in one communication process, and then the features are executed by the expert module. The advantage of this is that each feature slice contains the token features that need to be processed by a specific expert module, and each feature slice contains the token features that need to be sent to other nodes (or computing units) for processing by one expert module.For example, taking the case shown in FIG. 4 as an example, one feature slice contains the features of tokens that need to be processed by the expert module E1 and the expert module E3 (only contains the features of tokens that need to be processed by one expert module of the node 12, that is, the features of tokens that need to be processed by the expert module E3), and another feature slice contains the features of tokens that need to be processed by the expert module E2 and the expert module E4 (only contains the features of tokens that need to be processed by one expert module of the node 12, that is, the features of tokens that need to be processed by the expert module E4). During the communication process for one feature slice, the computing unit only needs to communicate the features of tokens that need to be processed by the specified expert module, for example, for the computing unit deployed with E11 and E21, it needs to send the features of tokens that need to be processed by the expert module E3 to the computing unit on the node 12 in one communication process, and does not need to send the features of tokens that need to be processed by the expert module E4 to the computing unit on the node 12. Since the features that need to be processed by the specific expert module are identified in the features, and the features that need to be processed by the specific expert module are sent in one communication process, only one specific expert module needs to be processed in the subsequent communication and calculation process, and different expert modules do not need to be processed at one time, so that the bandwidth occupation during communication can be increased, and the waste of resources is reduced. When the data is processed by the expert module, for one computing unit, only the specific expert module needs to be called for processing when processing one feature slice, and a new expert module can be called for processing when processing another feature slice, so that the calling overhead of resources is reduced when all expert modules need to be called for processing each feature slice. For example, taking the case shown in FIG. 4 as an example, one feature slice (feature slice 1) contains the features of tokens that need to be processed by the expert module E1 and the expert module E3, and another feature slice (feature slice 2) contains the features of tokens that need to be processed by the expert module E2 and the expert module E4. Taking the computing unit deployed with E11 and E21 as an example, for the feature slice 1, the computing unit only needs to call E11 to execute the data on the feature slice 1 that needs to be executed by E1. Then, for the feature slice 2, the computing unit needs to call E21 to execute the data on the feature slice 2 that needs to be executed by E2, and then call E21 to execute the data on the feature slice 2 that needs to be executed by E2. In the processing of the two feature slices, the expert module is called twice, and the calling overhead of resources is small.

[0012] In one possible implementation, the communication process includes AlltoAll communication or Allgather communication.

[0013] In one scenario, the communication system can be a set of computing units, which can be chip-level, such as an NPU, in which case the above-mentioned "one communication process" can be an Allgather communication. In one scenario, the communication system can be a set of nodes, which internally contain computing units, and the nodes can be servers, and the computing units can be chip-level, such as an NPU, in which case the above-mentioned "one communication process" can be an AlltoAll communication.

[0014] In one possible implementation, the first feature specifically refers to a feature representation of the data sequence input to the first computing unit in one batch.

[0015] In one possible implementation, the first computing unit is deployed with a second expert module; the second computing unit is configured to obtain a second feature, which is a feature representation of a data sequence input to the second computing unit; the second computing unit is configured to identify a second sub-feature corresponding to the second expert module from the second feature; and the second computing unit is configured to send a plurality of sub-features to the first computing unit in the one communication process with the first computing unit, wherein the plurality of sub-features sent to the first computing unit are all second sub-features corresponding to the second expert module.

[0016] In one possible implementation, the first computing unit is further configured to identify a third sub-feature corresponding to the second expert module from the first feature; and the first computing unit is further configured to process the third sub-feature and the second sub-feature through the second expert module. For example, the third sub-feature and the second sub-feature (but not limited to these two features) can be fused first, and then the fusion result is processed through the second expert module.

[0017] In one possible implementation, the first computing unit is deployed on a first node, and the first computing unit is further configured to, after sending the first sub-feature to the second computing unit, obtain a fourth sub-feature that needs to be processed by the second expert module in the one communication process (e.g., Allgather) from other computing units on the first node; and the first computing unit can process the third sub-feature, the fourth sub-feature, and the second sub-feature through the second expert module.

[0018] In one possible implementation, the second computing unit is further deployed with a third expert module, which is different from the first expert module; and the first computing unit can further send a feature that needs to be processed by the third expert module in the first feature to the second computing unit in the one communication process with the second computing unit after sending the first sub-feature to the second computing unit.

[0019] In a possible implementation, the first computing unit is deployed with a fourth expert module; and the process of sending the features in the first features corresponding to the third expert module is performed in parallel by the first computing unit when processing the features in the first features corresponding to the fourth expert module by the fourth expert module.

[0020] Specifically, the first computing unit can send the features requiring processing by the first expert module to the second computing unit in a communication process with the second computing unit, and then, in order to accelerate the data execution process by parallelizing the data, the first computing unit can further communicate with the second computing unit in a next communication process while processing the data by the expert module itself, for example, the features requiring processing by the third expert module can be sent to the second computing unit. Parallelizing (that is, masking each other) the computing and communication processes can accelerate the data processing process.

[0021] In a possible implementation, the first sub-feature is a shard of the features in the first features corresponding to the first expert module; and the first computing unit is further configured to send another shard of the features in the first features corresponding to the first expert module to the second computing unit in another communication process different from the communication process.

[0022] In addition, when each computing unit includes fewer expert modules (for example, only two), only the computing communication flow between experts can result in a low communication masking ratio. In order to improve the computing communication masking ratio, the features corresponding to each expert module can be further divided.

[0023] In a possible implementation, the first computing unit and the second computing unit are a graphics processing unit (GPU), a network processing unit (NPU), or a tensor processing unit (TPU), the first computing unit is deployed on a first node, and the second computing unit is deployed on a second node, the first node and the second node are different servers; or the first computing unit and the second computing unit are a graphics processing unit (GPU), a network processing unit (NPU), or a tensor processing unit (TPU), and the first computing unit and the second computing unit are both deployed on a first node.

[0024] In a possible implementation, the first computing unit is further configured to:

[0025] obtain a fifth sub-feature, the fifth sub-feature being obtained by fusing different sub-features through Allgather communication; and the fifth sub-feature including features of multiple tokens;

[0026] According to the fifth sub-feature, Key data and Query data of the multiple tokens are obtained through linear transformation.

[0027] The attention scores between each two different tokens in the plurality of tokens and the attention scores within each token in the plurality of tokens are calculated while keeping the arrangement of the Key data and the Query data of the plurality of tokens in the memory, wherein the attention score between each two different tokens is obtained based on the Key data of one token and the Query data of the other token in the two different tokens, and the attention score within each token is obtained based on the Key data and the Query data of the same token.

[0028] In a second aspect, the present application provides a data processing method, applied to a first computing unit, the first computing unit being in communication with a second computing unit, and the second computing unit being deployed with a first expert module; the method comprising:

[0029] obtaining a first feature, the first feature being a feature representation of a data sequence input to the first computing unit;

[0030] identifying a first sub-feature corresponding to the first expert module from the first feature;

[0031] in a communication process with the second computing unit, sending the first sub-feature to the second computing unit.

[0032] In a possible implementation, the communication process comprises AlltoAll communication or Allgather communication.

[0033] In a possible implementation, the first feature is specifically a feature representation of a data sequence input to the first computing unit in a batch.

[0034] In a possible implementation, the first computing unit is deployed with a second expert module, and the method further comprises:

[0035] identifying a third sub-feature corresponding to the second expert module from the first feature;

[0036] processing the third sub-feature and the second sub-feature by the second expert module.

[0037] In a possible implementation, the first computing unit is deployed on a first node, and the method further comprises: obtaining a fourth sub-feature corresponding to the second expert module sent by another computing unit on the first node;

[0038] processing the third sub-feature and the second sub-feature by the second expert module comprises:

[0039] processing the third sub-feature, the fourth sub-feature and the second sub-feature by the second expert module.

[0040] In a possible implementation, the second computing unit is further deployed with a third expert module, which is different from the first expert module;

[0041] The method further includes: in another communication process with the second computing unit different from the one-time communication process, sending, to the second computing unit, a feature corresponding to the third expert module in the first features.

[0042] In a possible implementation, the first computing unit is deployed with a fourth expert module; and the process of sending the feature corresponding to the third expert module processing in the first features is performed by the first computing unit in parallel when processing a feature corresponding to the fourth expert module in the first features by the fourth expert module.

[0043] In a possible implementation, the first sub-feature is a segment of the feature corresponding to the first expert module in the first features;

[0044] The method further includes:

[0045] In another communication process different from the one-time communication process, another segment of the feature corresponding to the first expert module in the first features is sent to the second computing unit.

[0046] In a third aspect, the present application provides a data processing method, applied to a communication system, the communication system including a first computing unit and a second computing unit, the first computing unit being in communication with the second computing unit, and the second computing unit being deployed with a first expert module; the method including:

[0047] The first computing unit acquires first features, the first features being feature representations of a data sequence input to the first computing unit; a first sub-feature corresponding to the first expert module is identified from the first features; and in a one-time communication process with the second computing unit, a plurality of sub-features are sent to the second computing unit, wherein the plurality of sub-features sent to the second computing unit are all the first sub-feature corresponding to the first expert module.

[0048] In a possible implementation, the first computing unit is deployed with a second expert module; the method further includes:

[0049] The second computing unit acquires second features, the second features being feature representations of a data sequence input to the second computing unit;

[0050] The second computing unit identifies a second sub-feature corresponding to the second expert module from the second features;

[0051] The second computing unit sends, to the first computing unit, a plurality of sub-features in a one-time communication process with the first computing unit, wherein the plurality of sub-features sent to the first computing unit are all the second sub-feature corresponding to the second expert module.

[0052] In a fourth aspect, the present application provides a data processing apparatus, characterized in that, applied to a first computing unit, the first computing unit is in communication with a second computing unit, and the second computing unit is deployed with a first expert module; the apparatus comprises:

[0053] an acquisition module, configured to acquire a first feature, the first feature being a feature representation of a data sequence input to the first computing unit;

[0054] a gating module, configured to identify a first sub-feature corresponding to the first expert module from the first feature;

[0055] a transceiving module, configured to send a plurality of sub-features to the second computing unit in a communication process with the second computing unit, wherein the plurality of sub-features sent to the second computing unit are all the first sub-feature corresponding to the first expert module.

[0056] In a possible implementation, the communication process comprises AlltoAll communication or Allgather communication.

[0057] In a possible implementation, the first feature is specifically a feature representation of a data sequence input to the first computing unit in a batch.

[0058] In a possible implementation, the first computing unit is deployed with a second expert module, and the gating module is further configured to:

[0059] identify a third sub-feature corresponding to the second expert module from the first feature;

[0060] The apparatus further comprises:

[0061] a model processing module, configured to process the third sub-feature and the second sub-feature through the second expert module.

[0062] In a possible implementation, the first computing unit is deployed on a first node, and the transceiving module is further configured to acquire a fourth sub-feature corresponding to the second expert module sent by another computing unit on the first node;

[0063] the model processing module is specifically configured to:

[0064] process the third sub-feature, the fourth sub-feature and the second sub-feature through the second expert module.

[0065] In a possible implementation, the second computing unit is further deployed with a third expert module, and the third expert module is different from the first expert module;

[0066] The transceiving module is further configured to: in another communication process different from the one-time communication process, send, to the second computing unit, another sub-feature of the first feature corresponding to the first expert module.

[0067] In a possible implementation, the first computing unit is deployed with a fourth expert module; and the process of sending the feature of the first feature corresponding to the third expert module is performed in parallel by the first computing unit when processing the feature of the first feature corresponding to the fourth expert module by the fourth expert module.

[0068] In a possible implementation, the first sub-feature is a sub-feature of the feature of the first feature corresponding to the first expert module.

[0069] The transceiving module is further configured to:

[0070] In another communication process different from the one-time communication process, send, to the second computing unit, another sub-feature of the first feature corresponding to the first expert module.

[0071] In a fifth aspect, the present application provides a data processing apparatus, which can include a memory, a processor, and a bus system, wherein the memory is configured to store a program, and the processor is configured to execute the program in the memory to perform the method in the second aspect and any optional method thereof, or the method in the third aspect and any optional method thereof.

[0072] In a sixth aspect, the present application provides a computer readable storage medium, which stores a computer program, and when the computer program is run on a computer, the computer is caused to perform the method in the first aspect and any optional method thereof, or the method in the third aspect and any optional method thereof.

[0073] In a seventh aspect, the present application provides a computer program, and when the computer program is run on a computer, the computer is caused to perform the method in the first aspect and any optional method thereof, or the method in the third aspect and any optional method thereof.

[0074] In an eighth aspect, the present application provides a chip system, which includes a processor configured to support the functions involved in the above aspects, for example, sending or processing the data involved in the above methods; or, information. In a possible design, the chip system further includes a memory configured to save necessary program instructions and data for the execution device or the training device. The chip system can be composed of a chip, or can include a chip and other discrete devices. BRIEF DESCRIPTION OF DRAWINGS

[0075] FIG. 1 is a structural schematic diagram of an artificial intelligence main body framework;

[0076] Fig. 2 is a schematic diagram of an application system framework according to the present application;

[0077] Fig. 3 is a schematic diagram of an application system framework according to an embodiment of the present application;

[0078] Fig. 4 is a schematic diagram of an application system framework according to an embodiment of the present application;

[0079] Figs. 5A to 5K are schematic diagrams of some operations according to an embodiment of the present application;

[0080] Fig. 6 is a schematic diagram of a data processing method according to an embodiment of the present application;

[0081] Figs. 7A and 7B are schematic diagrams of network structures according to an embodiment of the present application;

[0082] Figs. 8A to 8C are schematic diagrams of a data processing method according to an embodiment of the present application;

[0083] Fig. 8D is a schematic diagram of attention calculation according to an embodiment of the present application;

[0084] Fig. 9 is a schematic diagram of a data processing apparatus according to an embodiment of the present application;

[0085] Fig. 10 is a schematic diagram of a chip according to an embodiment of the present application;

[0086] Figs. 11 to 13 are schematic diagrams of a computing unit according to an embodiment of the present application. DETAILED DESCRIPTION

[0087] The embodiments of the present application will be described below in conjunction with the accompanying drawings. The terms used in the embodiments of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0088] The embodiments of the present application will be described below in conjunction with the accompanying drawings. The skilled in the art can know that the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems as the technology develops and new scenarios appear.

[0089] The terms "first", "second", and the like in the specification and claims of the present application and the above-described accompanying drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the terms thus used can be interchanged under appropriate circumstances, and are merely used to distinguish the objects of the same attribute in the description of the embodiments of the present application. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus including a series of units does not have to be limited to those units, but can include other units not clearly listed or inherent to the process, method, product, or apparatus.

[0090] The terms "substantially," "about," and similar terms as used herein are taken to mean approximately in the sense of being close to as understood by one of ordinary skill in the art, unless otherwise noted. Further, the use of "may" when describing embodiments herein is meant to convey that one or more embodiments do not necessarily include that feature, an embodiment that does include that feature, and equivalent thereof. The terms "use," "using," and "used" are to be construed as synonymous with the term "utilize," "utilizing," and "utilized," respectively. Additionally, the term "exemplary" is intended to mean an example or an illustration.

[0091] In some scenarios, large-scale neural network models of neural networks can be deployed on distributed networks for computation, processing, to achieve applications such as large-scale parallel processing, distributed storage, elastic topology, high redundancy, and non-linear operation of cloud computing. Specifically, for example, from the initial AlexNet network, VGG network (visual geometry group network), to ResNet network (residual neural network) to GPT-3 (generative pre-trained transformer) model and discus model, the level of large-scale neural network models based on deep learning is deepening, the range is widening, and the parameter quantity is increasing. Through large-scale neural network models, a large amount of computing resources and data resources can be utilized to achieve multi-task concurrent processing and large-scale operation processing.

[0092] However, in the case of deploying a large-scale neural network model on a system for computing processing, as the model parameter scale and data increase rapidly, it is difficult to perform complete training, inference and other data processing on a single system. In the case of deploying a large-scale neural network model on a distributed system, although the scalability and computing efficiency are theoretically improved, since the large-scale neural network model actually needs to be deployed on multiple nodes, when multiple models on multiple nodes are processed in parallel, the communication overhead of data communication between nodes and model communication between models is significantly increased, resulting in reduced data transmission efficiency, reduced data processing efficiency, increased storage and computing resource consumption, increased application and maintenance costs and other problems. On the other hand, the current large-scale neural network model architecture only considers the case of being deployed in a DenseNet, does not consider the sparsity of the model (Sparsity), and also cannot support the model with a sparse architecture, that is, it cannot activate only a part of the large-scale neural network model architecture to achieve the computing processing of a given task, sample or label.

[0093] It should be understood that the nodes in the embodiments of the present application can be devices of the same granularity as the computing units, for example, both refer to chips or AI cards, etc. Alternatively, the computing units can be devices of a finer granularity than the nodes, for example, a node can include multiple computing units, specifically, a node can be a server and a computing unit can be a chip or an AI card included in the server, etc.

[0094] A mixture of experts (MoE) composed of one or more expert modules can be deployed on multiple nodes to build a large-scale neural network model. In the case of deploying multiple expert modules of the mixture of experts on the same layer on multiple nodes, the input data of each node is independent of each other and generally different. Further, the input data of each node is cut into data corresponding to the multiple expert modules respectively at each node, and each node sends the cut data to the corresponding other nodes. Thus, the expert modules on each node perform data processing on the set of data cut by the node itself and sent from other nodes, and after the expert modules on each node complete the data processing, the processed data are aggregated to generate the total output of the neural network model.

[0095] FIG. 1 shows a schematic diagram of an example system architecture 100 according to an embodiment of the present disclosure. As shown in FIG. 1, in the system architecture 100, there can be at least a node cluster 110, a node server 120, a client device 130, and a network 140. The node cluster 110, the node server 120, and the client device 130 in the system architecture 100 can be connected via the network 140. It should be understood that the system architecture 100 in FIG. 1 can include more or fewer functional devices, without being limited thereto.

[0096] In one example embodiment, the client device 130 sends data to be processed to the node cluster 110 and the node server 120 via the network 140 as input data for the node cluster 110. The node server 120 controls the node cluster 110 to process the data according to the data received from the client device 130. The node cluster 110 processes the data received from the client device 130 into a predetermined form by parallel processing, and feeds back to the node server 120 or returns to the client device 130. In some embodiments, only part of the structures in the system architecture 100 can participate in data processing. In one example embodiment, the node server 120 can directly provide data to be processed to the node cluster 110, and the client device 130 does not participate in data processing. In one example embodiment, only part of the nodes in the node cluster 110 can participate in data processing, and the other part of the nodes does not participate in data processing. It should be understood that the data processing operation between the node cluster 110, the node server 120, the client device 130, and the network 140 in the system architecture 100 is not limited thereto, and any operation method can be implemented.

[0097] In some embodiments, the node cluster 110 can include one or more nodes. For example, FIG. 1 exemplarily shows a case where the node cluster 110 includes multiple nodes (node 110-1, node 110-2, node 110-3, …, node 110-N). In the case of multiple nodes, the node cluster 110 can be distributed to constitute a neural network model, i.e., the neural network model is deployed in multiple nodes of the node cluster 110 to process data in parallel.

[0098] It should be understood that the node cluster 110 can include multiple nodes, each of which can include multiple computing units, or the node cluster 110 can include multiple computing units without an intermediate granularity device such as a node, for example, the node cluster 110 can be a server cluster, the server cluster can include multiple servers, each of which can include multiple computing units, for another example, the node cluster 110 is a server, and the server can include multiple computing units.

[0099] In some embodiments, the node server 120 can be configured to manage and control the data processing of the node cluster 110, to determine the data processing algorithm or manner of the node cluster 110 according to different data, or to perform neural network model-based training, inference, or other processing using the node cluster 110. In some embodiments, the node server 120 can be connected to all or part of the nodes of the node cluster 110, and can be distributed on all or part of the nodes of the node cluster 110.

[0100] In some embodiments, the client device 130 can be one or more any suitable mobile or non-mobile computing devices configured to provide data input or receive data feedback. The client device 130 can be equipped with data collection, processing, and output functions, such as implemented by input / output (I / O) devices such as microphones, cameras, video cameras, mice, keyboards, displays, and the like. The client device 130 can run various software applications, such as productivity or office support software, Web browsing software, camera software, software to support voice calls, video conferencing, and email, and the like. The computing device is connected to the network 140 via wired and / or wireless communication links to enable data communication with the node cluster 110 or the node server 120.

[0101] In some embodiments, the network 140 can be configured to enable communication among the various structures of the system architecture 100, including the node cluster 110, the node server 120, and the client device 130. For example, the network 140 can include, for example, a local area network (LAN), a wide area network (WAN), the Internet, a virtual LAN (VLAN), an enterprise LAN, a layer 3 virtual private network (VPN), an intranet, or any combination thereof. It should be understood that the various structures of the system architecture 100 can communicate with each other directly via the network 140, indirectly via any of the structures connected via the network 140, or not at all.

[0102] Herein, the term “input” refers to data sent to a device / unit / module / part, etc. Hereinafter, an example is described by taking image data as an example of input data of the neural network model of the node cluster 110. It should be understood that the input data includes at least one of image data, text data, voice data, classification data, and is not limited thereto, as long as it is a data type suitable for neural network model processing. According to embodiments of the present disclosure, model training, inference, processing, etc. can be effectively applied to various types of data such as structured data and unstructured data (e.g., image data, text data, voice data), etc.

[0103] In this document, the term “computing unit” represents a computing resource for data processing, which can be implemented by a graphics processing unit (GPU), a central processing unit (CPU), a tensor processing unit (TPU), a deep learning processing Unit (DPU), a neural network processing unit (NPU), a brain processing unit (BPU), etc. It should be understood that the type and implementation of the computing unit are not limited thereto, as long as it is a computing unit suitable for neural network model processing.

[0104] In this document, the term “parallel processing” represents that data processing is performed in batches or stages at independent asynchronous speeds, and the batched or staged data processing can overlap in time. For example, the execution of one batch of data processing has not ended in all nodes, while the execution of another batch of data processing has already started. It should be understood that parallel processing is not limited to what is listed herein, as long as it is data processing of a neural network model that can be understood by those skilled in the art, which is within the scope defined by parallel processing in embodiments of the present disclosure.

[0105] FIG. 2 shows a schematic diagram of an example node cluster 110 according to an embodiment of the present disclosure. For example, FIG. 2 exemplarily shows a case where the node cluster 110 as shown in FIG. 1 includes a plurality of nodes (node 210-1, node 210-2, node 210-3, …, node 210-N), where the neural network model 200 is deployed in the plurality of nodes (node 210-1, node 210-2, node 210-3, …, node 210-N), and the neural network model 200 includes P expert modules, where P is greater than or equal to 2. As an illustrative embodiment, the neural network model 200 can include 100 expert modules, which are distributedly deployed in the plurality of nodes (node 210-1, node 210-2, node 210-3, …, node 210-N), each of which can be deployed with a part of the 100 expert modules, and the expert modules included in each node collectively constitute the entirety of the expert modules of the neural network model 200. As shown in FIG. 2, in some embodiments, the nodes 210-1, 210-2, 210-3, …, 210-N are connected in sequence to perform data processing in a pipeline parallel manner. As an example implementation of the pipeline parallel manner, for example, the node 210-1 performs data processing on input data of the node cluster 110, and sends output data of the node 210-1 (i.e., intermediate data of the node cluster 110 obtained after processing by the node 210-1) to the node 210-2. Then, the node 210-2 performs data processing on the output data of the node 210-1, and sends output data of the node 210-2 (i.e., another intermediate data of the node cluster 110 obtained after processing by the node 210-2) to the node 210-3. In this way, the node 210-N performs data processing on output data of the last node connected thereto, and sends output data of the node 210-N (i.e., data obtained after processing by the node 210-N) as output data of the node cluster 110. It should be understood that the node connection manner and data sending manner of the plurality of nodes in the node cluster 110 are not limited thereto, and the pipeline parallel manner can be additionally or alternatively modified or extended based thereon.

[0106] FIG. 2 also illustrates the node structure in the node cluster 110. For example, the node 210-1 illustrates a case where one node includes one computing unit 20 and one computing unit 50, and the node 210-3 illustrates a case where one node includes one computing unit 20 and two computing units 50, where the computing unit 20 is a first computing unit based on expert parallelism, and the computing unit 50 is a second computing unit based on tensor parallelism. It should be understood that the number of the computing unit 20 and the computing unit 50 in each node can be arbitrary, each node can include only the computing unit 20, only the computing unit 50, both the computing unit 20 and the computing unit 50, or none of the computing unit 20 and the computing unit 50 in some cases.

[0107] FIG. 3 illustrates a schematic diagram of a computing unit (e.g., the first computing unit or the second computing unit in the embodiments of the present application) according to an embodiment of the present disclosure. For example, FIG. 3 exemplarily illustrates the computing unit 20 (i.e., the first computing unit) included in one node in the node cluster 110 as shown in FIGS. 1 and 2. It should be understood that although one node is exemplarily illustrated herein, the node structure illustrated herein can be applicable to any node in the node cluster 110, and in some embodiments, can also be applicable to other structures in the system architecture 100 (e.g., the node server 120, the client device 130, etc.).

[0108] As shown in FIG. 3, in some embodiments, each node includes one or more computing units, each of which can independently perform data processing in a physical manner. The node also includes one or more computing units, each of which can include one or more expert modules (e.g., a shard of expert modules) constituting a mixture of experts (MoE). Here, the mixture of experts can be implemented by one neural network or a combination of multiple neural networks, and each expert in the mixture of experts implements part or all of the neural network of the mixture of experts. In some embodiments, the mixture of experts can be deployed on one or more nodes or computing units. In some embodiments, each expert module can be used to implement a specific task and process a specific type of data in the input data of the node. In some embodiments, the node also includes one or more storage resources (not shown) for storing intermediate processing data of the node, etc.

[0109] In some embodiments, a node can include multiple computing units, each of which includes multiple expert modules E, wherein each expert module E can be a shard (i.e., a partial expert module) of a complete expert module, and the expert modules on different computing units (e.g., different computing units on the same node, or computing units on different nodes) perform data processing in an expert parallel manner.

[0110] As an example implementation of the expert parallel manner, for example, as shown in FIG. 3, a node includes computing unit 11 and computing unit 12 (described as computing resources in the figure). Computing unit 11 and computing unit 12 each independently perform data processing. For example, on computing unit 11, there are expert modules E11, E12, wherein E11 is a shard of expert module A, and E12 is a shard of expert module B, and on computing unit 12, there are expert modules E21, E22, wherein E21 is another shard of expert module A, and E22 is another shard of expert module B.

[0111] When data processing needs to be performed by expert module A, computing unit 11 can invoke E11 to perform data processing, and computing unit 12 can invoke E21 to perform data processing, and the computations of the two are parallel.

[0112] It should be understood that the number and relationship of the computing units, computing units, and expert modules E in this example are not limited and do not constitute a limitation on the embodiments of the present disclosure.

[0113] In some embodiments, each computing unit can include a gate unit. For example, as shown in FIG. 3, computing unit 11 includes gate unit 31, and computing unit 12 includes gate unit 32. The gate unit is used to route the input data for the computing unit to determine which expert module (e.g., expert module A or expert module B) to perform the data.

[0114] For example, input data can also need to be split and processed in parallel by different computing units, as shown in FIG. 4, which includes two nodes, each of which in turn includes four computing units. The input features can be split into 8 parts, each of which is processed by a computing unit on a node. In this case, after obtaining the input features, each computing unit can calculate which expert module is needed to process each token in the input features. If the expert module is deployed on the node where the computing unit is located, the data can be retained (and subsequently synchronized through communication between computing units within the node, such as through the Allgather communication process). If the expert module is deployed on another node (i.e., not on the node where the computing unit is located), the data can be synchronized through communication between computing units between nodes, such as through the AlltoAll communication process.

[0115] Next, several terms that will be involved in the embodiments of the present application are introduced:

[0116] 1), Reduce-scatter

[0117] This collective communication primitive combines the reduce and scatter operations. As shown in FIG. 5A, it performs a global reduce operation (such as sum, maximum, etc.) on an input array of data, and then distributes the results to each node.

[0118] 2), Allgather

[0119] Allgather is a many-to-many communication primitive, as shown in FIG. 5B. It allows each node to send its data to all other nodes and collect data from all other nodes. After the AllGather operation, each node holds the complete data set of all nodes.

[0120] 3), Allreduce

[0121] AllReduce is a collective communication primitive commonly used in distributed computing, and plays a key role in distributed training in the field of deep learning. This primitive is mainly used to synchronize data between multiple nodes or devices (such as NPUs), as shown in FIG. 5C. Typically, data on multiple nodes is aggregated and subjected to some reduction operation (such as sum, maximum, average, etc.), and then the results are broadcast back to all nodes. In the context of distributed training of large models, this primitive is mainly used for synchronization of activation values in tensor parallel forward propagation and backward propagation, and synchronization of gradient data in data parallel. It is not difficult to find that the AllReduce primitive is equivalent to the superposition of the Reduce-scatter and Allgather communication primitives.

[0122] 4), AlltoAll

[0123] AlltoAll operation allows each node to send its data to all other nodes and receive data from all other nodes. This is a many-to-many communication pattern that ensures each node can obtain the data of all other nodes. The collective communication primitive contains two phases: data distribution and data aggregation. In the data distribution phase, on each node, the input data is split into split_count blocks according to the specified split_dim. These blocks are sent to other nodes in a certain order (e.g., block index). For example, the i-th block can be sent to the i-th node. In the data aggregation phase, each node receives data blocks from all other nodes. The received data blocks are recombined into a complete Tensor according to the concat_dim. The Tensor on all nodes must have the same size and format to ensure correct data aggregation. As shown in FIG. 5D, AlltoAll completes the transpose operation of data among distributed nodes.

[0124] 5), Tensor Parallelism

[0125] Tensor Parallelism (TP) in distributed training of large models is a technique that divides model parameters into multiple tensors and allocates each tensor to a different computing unit (such as NPU) for calculation.

[0126] For the Transformer structure, tensor parallelism has specific applications in MLP layers and Attention layers. As shown on the right side of FIG. 5E, taking the MLP layer as an example, first, the parameter matrix W1 is cut by column, and then the parameter matrix W2 is cut by row to perform parallel calculation on two NPUs. In the forward propagation phase of training, according to the matrix operation rule, the function needs to perform Allreduce synchronization operation on the activation values calculated on different devices to ensure the correctness of forward calculation; similarly, in the backward propagation phase of training, the function f needs to perform Allreduce synchronization operation on the gradient of the activation to meet the correctness of matrix operation.

[0127] As shown on the left side of FIG. 5E, the Attention layer is similar to the MLP layer. First, the parameter matrices W Q , W K , and W V are cut by column, and then the matrix W LThe "row cut" is adopted, so the Allreduce synchronization operation is introduced in both the forward and backward stages of training. When tensor parallelism is adopted, the overall process of the calculation and communication of the Transformer Block is shown in FIG. 5F, 2 times of Allreduce in the forward direction, 2 times of Allreduce in the backward direction, and a total of 4 times of Allreduce.

[0128] 6) Megatron sequence parallelism

[0129] Megatron sequence parallelism (SP) is an important technology in the distributed training of large models, which is mainly used to solve the problem of memory occupation in large Transformer models.

[0130] As shown in FIG. 5G, taking the MLP module as an example, the sequence cut and the tensor parallelism of the model parameters are consistent. That is, the complete sequence is cut into TP parts and placed on TP NPU cards. In the process of forward propagation, since only part of the sequence is on each NPU card, the Allgather collective communication needs to be initiated in the TP domain to take the sequences on other cards back to the card, splice the complete sequence for operation. After the calculation is completed, the reduction operation required by the tensor parallelism needs to be completed, and the data is dispersed along the sequence dimension to each NPU card to restore to the "original" sequence state, which calls the Reduce-scatter collective communication. It is not difficult to understand that when the tensor parallelism is further superimposed on the Megatron sequence parallelism, the original Allreduce collective communication is split into Allgather and Reduce-scatter. As shown in FIG. 5G, in the forward propagation stage of model training, the function g represents the Allgather operation required by the Attention module and the MLP module calculation, and the function represents the Reduce-scatter operation; similarly, in the backward propagation stage, represents the Reduce-scatter operation, and the function g represents the Allgather operation. Normally, 2 times of Allgather and 2 times of Reduce-scatter in the forward direction, 2 times of Allgather and 2 times of Reduce-scatter in the backward direction, and a total of 4 times of Allgather and 4 times of Reduce-scatter.

[0131] 7) Expert parallelism

[0132] Expert parallelism (EP) in large model distributed training is a training strategy that allows the model to dynamically select a set of “experts” for computation based on the input data. This parallelization approach aims to optimize the use of computing resources by introducing sparsity, while maintaining or improving the performance of the model.

[0133] The input data is first passed through a gating network, which assigns a weight or probability to different experts based on the features of the input. Then, based on these weights, the model selectively activates one or more experts to process the input data. As shown in FIG. 5H, since different experts can be placed on different devices (such as NPUs), and each device can contain data belonging to all experts, from the perspective of each device, the data required by other experts on the current device needs to be sent to the corresponding device, and the data stored on other devices required by the current device expert needs to be taken back, which can be achieved by initiating AlltoAll collective communication. Each expert independently processes the data it receives and produces output. These outputs are then returned to the original device to be merged to produce the final output of the model, so here we need to initiate AlltoAll communication again. As shown in FIG. 5I, the function h represents the forward propagation phase of the model training before and after expert parallelism, and respectively represent dispatch and combine AlltoAll communication, and the reverse propagation phase is the same.

[0134] 8), Group-wise AlltoAll

[0135] As introduced above, the training of MoE sparse large model will introduce AlltoAll communication, thereby reducing the MFU of training. Group-wise AlltoAll is a technical idea for reducing the amount of AlltoAll communication. As shown in FIG. 5J, it is assumed that the model has 4 experts placed on 4 nodes for training, and each node has 4 NPUs for tensor parallelism. At this time, the data on each NPU card is consistent. Expert parallelism will initiate AlltoAll communication between the same number cards on different nodes. If no optimization is performed, the data of the same number card AlltoAll between nodes is the same, resulting in a large amount of redundant communication. Group-wise AlltoAll groups the NPU cards along the tensor parallelism, that is, the same number NPU cards on different nodes are in a group, and each group is responsible for the AlltoAll communication of the corresponding part of the data. Therefore, the communication amount is 1 / tp of the original AlltoAll communication. After this communication is completed, the NPU cards in the tensor parallelism domain of the node initiate Allgather communication again, thereby obtaining the completed data. This process can be organically combined with tensor parallelism and Megatron sequence parallelism. When tensor parallelism, Megatron sequence parallelism and expert parallelism are mixedly deployed, the forward propagation process of the model MLP part training is 1 / tp of dispatch AlltoAll of EP, Allgather of TP / Megatron SP, expert calculation, Reduce-scatter of TP / Megatron SP, 1 / tp of combine AlltoAll of EP.

[0136] In the embodiments of the present application, the first computing unit and the second computing unit can be the same number cards of different nodes.

[0137] 9), Expert packing

[0138] In addition, for expert parallelism, expert packing is a common deployment method. As shown in FIG. 5K, it is assumed that the model is deployed with 8-way DP, the model has 8 experts, denoted as Expert1-Expert8, and when each DP is placed with 4 experts, only two DPs are needed to place all the Expert parameters. At this time, the same number cards in the two nodes form the communication domain of AlltoAll. Since each NPU card is placed with 4 expert parameters, as shown in FIG. 5K, from the perspective of a single card, in the forward propagation stage of training, the communication and calculation operator execution process is:

[0139] 1 / tp dispatch AlltoAll of EP is executed on stream3

[0140] for (Expert1 ~ Expert4)

[0141] Allgather of TP / Megatron SP on Stream2

[0142] Expert computation on Stream1

[0143] Reduce-scatter of TP / Megatron SP on Stream2

[0144] 1 / tp combine AlltoAll of EP on Stream3.

[0145] Embodiments of the present application provide a data processing method. The data processing method of the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0146] Referring to FIG. 6, FIG. 6 is a flowchart of a data processing method according to an embodiment of the present application. As shown in FIG. 6, the data processing method according to an embodiment of the present application can include steps 601 to 603, which will be described in detail below.

[0147] 601. The first computing unit obtains a first feature, the first feature being a feature representation of a data sequence input to the first computing unit.

[0148] The first computing unit in the embodiments of the present application can belong to a communication system, the communication system including the first computing unit and a second computing unit, the first computing unit communicating with the second computing unit.

[0149] In one scenario, the communication system can be a set of computing units, and the computing unit can be chip-level, such as an NPU. In one scenario, the communication system can be a set of nodes, and each node contains a computing unit. The node can be a server, and the computing unit can be chip-level, such as an NPU.

[0150] The first feature can be a feature representation obtained by the first computing unit in the process of processing the input sequence by a machine learning model (e.g., a language model). For example, the first feature can be the input of a gating module, and the first feature can include a feature representation of each token in a plurality of tokens. The gating module can determine which expert module needs to be executed to perform the feature representation of the token by inputting the feature representation of each token.

[0151] The first feature can be a feature representation of all data sequences input to the first computing unit in a batch. For example, referring to the illustration in FIG. 4, the features that the entire system needs to process are sliced, and each feature slice can be processed by a computing unit. For the first computing unit, the complete feature slice that needs to be processed by the first computing unit can be the first feature in the embodiments of the present application.

[0152] The expert module in the embodiments of the present application can be part of a machine learning model, for example, it can be part of a Transformer Block (for example, a network in MLP for performing linear transformation), and of course it can also be other network types.

[0153] For example, the Transformer Block is mainly composed of two modules of Attention and MLP, as shown in FIG. 7A.

[0154] The Attention part plays a crucial role in the Transformer model, which allows the model to pay attention to other positions in the sequence when processing each position in the sequence. The working principle of the Attention mechanism can be roughly divided into the following steps:

[0155] 1. Calculate Query (Q), Key (K) and Value (V):

[0156] For each position in the input sequence, calculate its corresponding Q, K and V vectors. These vectors are usually obtained by multiplying the input embedding with the W Q , W K and W V weight matrices respectively. The Q vector represents the information that the current position needs to pay attention to, the K vector is used to match with Q, and the V vector contains the actual information content.

[0157] 2. Calculate the Attention weight:

[0158] The "attention" score for each position to the current position is obtained by computing the dot product of the Q vector with all K vectors, also known as scaled dot-product attention. The "dot product" operation measures the similarity or match between Q and K. Since the result of the dot product can be very large, especially when the input dimension is high, this can cause the softmax function to enter the saturation region when computing the attention weights. Therefore, the dot product result is usually scaled by a factor, which is typically the square root of the input dimension. The scaled dot product result is then input into the softmax function to obtain the attention weights for each Key with respect to the Query. These weights represent the model's attention to different positions when generating the output for the current position.

[0159] 3. Weighted Sum:

[0160] The attention weights are then weighted with the corresponding Value vectors to obtain the Attention output for the current position. This output is a weighted sum of all position information, allowing the model to attend to global information.

[0161] The MLP part in the Transformer model is mainly used to further transform the output of Attention with non-linear functions. Its working principle can be summarized as follows:

[0162] 1. Linear Transformation:

[0163] First, the output of Attention (usually a vector) is input into one or more linear layers. These linear layers perform linear transformation on the input through weight matrices and bias terms, extracting different features.

[0164] 2. Non-linear Activation:

[0165] After linear transformation, a non-linear activation function (such as ReLU) is usually applied to increase the non-linearity of the model. The non-linear activation function can map the output of linear transformation to a new space, allowing the model to learn more complex function relationships.

[0166] 3. Stack multiple layers:

[0167] In the Transformer model, the MLP part is usually composed of multiple such linear layers and non-linear activation functions stacked together. This stacking structure allows the model to learn more rich feature representations and improves the generalization ability of the model.

[0168] In summary, the Attention part in the Transformer model achieves global information attention by calculating attention weights, while the MLP part extracts more rich feature representations by nonlinear transformation of the Attention output through a multi-layer perceptron. These two parts together constitute the core components of the Transformer model, enabling the model to achieve significant performance improvement in natural language processing tasks.

[0169] The machine learning model in the embodiments of the present application can integrate multiple expert models, but only a few experts most relevant to a specific input are activated each time. "Sparse" here means that not all parameters are involved in the calculation when the model processes data, only the most relevant part of the parameters is activated. As shown in FIG. 7B, the MoE model usually extends the MLP / FFN (multi-layer perceptron / forward feed neural network) layer horizontally in network structure, and refines and organizes the parameters in the layer into multiple expert modules (experts). For a specific input, MoE dynamically selects which expert module(s) to activate through a gating module, thereby achieving sparse activation. Since MoE model only activates part of the parameters, it can greatly reduce the pre-training calculation cost and improve the inference performance. While maintaining the performance of the model, the consumption of computing resources is reduced.

[0170] 602、the first computing unit identifies a first sub-feature corresponding to the first expert module from the first feature;

[0171] The sub-feature corresponding to the expert module can be understood as a sub-feature that needs to be processed by the expert module. For example, the first sub-feature corresponding to the first expert module can be understood as: the first sub-feature is a feature that needs to be processed by the first expert module.

[0172] After obtaining the first feature, the first computing unit can determine the feature (e.g., the first sub-feature) in the first feature that needs to be processed by a specific expert module (e.g., the first expert module).

[0173] In order to speed up the calculation process, the first computing unit needs to split the first feature when processing the first feature by calling the network. First, the communication process (e.g., AlltoAll between nodes or Allgather between computing units) of a feature slice in the first feature is performed, and then the data is processed through the network (e.g., the expert module), and at the same time the communication process (e.g., AlltoAll between nodes or AlltoAll between computing units) of another feature slice of the first feature is performed through the network (e.g., the expert module). Since the data processing process through the network and the data communication process of other slices are executed in parallel, the entire data processing process can be accelerated.

[0174] In the prior art, when features are split, the features are split based on the address arrangement of the features in the storage space. For example, when the number of splits is 2, the features located in the first half of the address arrangement in the storage space can be taken as a feature slice, and the features located in the second half of the address arrangement in the storage space can be taken as a feature slice. When the features in the first half are processed, the expert modules required for processing the features of each token in the first half of the features can be determined, and then the features of the tokens that need to be processed by the expert modules deployed on other nodes or computing units are sent to the other nodes or computing units through a communication process (for example, AlltoAll between nodes or Allgather between computing units), which can cause the following problems:

[0175] Since the feature slicing is based on the address arrangement of the features in the storage space, each feature slice often contains the features of the tokens that need to be processed by each expert module in the entire expert module, for example, in the case shown in FIG. 4, each feature slice contains the features of the tokens that need to be processed by the expert module E1, the expert module E2, the expert module E3 and the expert module E4. In the communication process for a feature slice, the computing unit needs to communicate the features of the tokens processed by all the expert modules, for example, for the computing unit deployed with E11 and E21, the features of the tokens processed by the expert module E3 and the expert module E4 need to be sent to the computing unit on the node 12 in one communication process, and after the node 12 receives the features corresponding to the expert module E3 and the expert module E4, it needs to respectively perform the communication and calculation of the features corresponding to the expert module E3 and the expert module E4 (for example, first perform the Allgather and calculation of the features corresponding to E3, and then perform the Allgather and calculation of the features corresponding to E4), and each feature slice needs to repeatedly perform this process, and each communication transmits a small amount of data of the expert module (for example, first communicate a small part of the features corresponding to E3, then communicate a small part of the features corresponding to E4, then communicate a small part of the features corresponding to E3, and then communicate a small part of the features corresponding to E4), thereby causing waste of resources, and when processing data through the expert module, since the processing of the features on each feature slice needs to be performed in sequence, for a computing unit, when processing a feature slice, each expert module needs to be called in sequence for processing, and when processing another feature slice, each expert module needs to be called again for processing, which causes a large resource calling overhead, for example, in the case shown in FIG. 4, each feature slice contains the features of the tokens that need to be processed by the expert module E1, the expert module E2, the expert module E3 and the expert module E4, and taking the computing unit deployed with E11 and E21 as an example, for feature slice 1, the computing unit needs to first call E11 to execute the data on feature slice 1 that needs to be executed by E1, and then call E21 to execute the data on feature slice 1 that needs to be executed by E2, then, for feature slice 2, the computing unit needs to first call E11 to execute the data on feature slice 2 that needs to be executed by E1, and then call E21 to execute the data on feature slice 2 that needs to be executed by E2, which means that in the processing of the two feature slices, the expert module is called four times, and the resource calling overhead is large.

[0176] The idea of the embodiments of the present application is that when the features are sliced, it is not based on the address arrangement of the features on the storage space, but the features needed to be executed by each expert module are identified from the features, which is equivalent to each feature slice containing only the features needed to be executed by one expert module, and then only the features needed to be executed by the specific expert module are communicated in one communication process, and then the features are executed by the expert module. The advantage of this is that each feature slice contains the token features that need to be processed by a specific expert module, and each feature slice contains the token features that need to be sent to other nodes (or computing units) for processing by one expert module.For example, taking the case shown in FIG. 4 as an example, one feature slice contains the features of tokens that need to be processed by the expert module E1 and the expert module E3 (only contains the features of tokens that need to be processed by one expert module of the node 12, that is, the features of tokens that need to be processed by the expert module E3), and another feature slice contains the features of tokens that need to be processed by the expert module E2 and the expert module E4 (only contains the features of tokens that need to be processed by one expert module of the node 12, that is, the features of tokens that need to be processed by the expert module E4), in the communication process for one feature slice, the computing unit only needs to communicate the features of tokens that need to be processed by the specified expert module, for example, for the computing unit deployed with E11 and E21, it needs to send the features of tokens that need to be processed by the expert module E3 to the computing unit on the node 12 in one communication process, and does not send the features of tokens that need to be processed by the expert module E4 to the computing unit on the node 12, because the features that need to be processed by the specific expert module are identified in the features, and the features that need to be processed by the specific expert module are sent in one communication process, so that the bandwidth occupation in communication can be increased, and the waste of resources is reduced, and when the data is processed by the expert module, for one computing unit, only the specific expert module needs to be called for processing when processing one feature slice, and a new expert module can be called for processing when processing another feature slice, without the need to re-call all the expert modules when processing each feature slice, thereby reducing the resource calling overhead, for example, taking the case shown in FIG. 4 as an example, one feature slice (feature slice 1) contains the features of tokens that need to be processed by the expert module E1 and the expert module E3, and another feature slice (feature slice 2) contains the features of tokens that need to be processed by the expert module E2 and the expert module E4, taking the computing unit deployed with E11 and E21 as an example, for the feature slice 1, the computing unit only needs to call E11 to execute the data on the feature slice 1 that needs to be executed by E1, then, for the feature slice 2, the computing unit needs to call E21 to execute the data on the feature slice 2 that needs to be executed by E2, and then call E21 to execute the data on the feature slice 2 that needs to be executed by E2, which is equivalent to calling the expert module twice in the processing of the two feature slices, and the resource calling overhead is small.

[0177] For example, referring to FIG. 8A, FIG. 8A shows a schematic diagram of a feature slice, in which a total of 8 expert modules (e1 to e8) are included, e1-4 are deployed in node 1, e5-8 are deployed in node 2, e1 and e5 in the feature can be identified as a feature slice that needs to be processed, e2 and e6 in the feature can be identified as a feature slice that needs to be processed, e3 and e7 in the feature can be identified as a feature slice that needs to be processed, and e4 and e8 in the feature can be identified as a feature slice that needs to be processed. For the computing unit on node 1, in the first communication process, the features that need to be processed by e5 can be sent to the computing unit on node 2, the features that need to be processed by e1 are retained, and in addition, the features that need to be processed by e1 sent by the computing unit on node 2 can also be received.

[0178] 603、In the communication process with the second computing unit, a plurality of sub-features are sent to the second computing unit, wherein the plurality of sub-features sent to the second computing unit are all first sub-features corresponding to the first expert module.

[0179] In the communication process with the second computing unit, a plurality of sub-features are sent to the second computing unit, wherein the plurality of sub-features sent to the second computing unit are all first sub-features corresponding to the first expert module.

[0180] In one scenario, the communication system can be a computing unit set, and the computing unit can be a chip-level, such as an NPU. In this case, the above-mentioned “one communication process” can be an Allgather communication. In one scenario, the communication system can be a node set, and the node internally includes computing units. The node can be a server, and the computing unit can be a chip-level, such as an NPU. In this case, the above-mentioned “one communication process” can be an AlltoAll communication.

[0181] In one possible implementation, in the communication process with the second computing unit, the first sub-feature can be sent to the second computing unit, and the second sub-feature can not be sent to the second computing unit.

[0182] In one possible implementation, in addition to the first computing unit identifying the features that need to be executed by the first expert module deployed on the second computing unit, the second computing unit can also identify the features that need to be executed by the second expert module deployed on the first computing unit and send the features that need to be executed by the second expert module to the first computing unit in the communication process.

[0183] For example, the first computing unit is deployed with a second expert module; the second computing unit can obtain a second feature, which is a feature representation of a data sequence input to the second computing unit, and identify a second sub-feature from the second feature that needs to be processed by the second expert module.

[0184] In a possible implementation, the first computing unit can be deployed with a second expert module (referred to as expert module 1 for ease of description) and a third expert module (referreded to as expert module 2 for ease of description), and the second computing unit can be deployed with a first expert module (referred to as expert module 3 for ease of description) and a fourth expert module (referred to as expert module 4 for ease of description).

[0185] The first computing unit can identify a first sub-feature (referred to as sub-feature 1 for ease of description) from the first feature that needs to be processed by the expert module 1, and in addition, identify a sub-feature (referred to as sub-feature 2 for ease of description) from the first feature that needs to be processed by the expert module 2, and in addition, identify a sub-feature (referred to as sub-feature 3 for ease of description) from the first feature that needs to be processed by the expert module 3, and in addition, identify a sub-feature (referred to as sub-feature 4 for ease of description) from the first feature that needs to be processed by the expert module 4, and place the sub-feature 1, the sub-feature 2, the sub-feature 3, and the sub-feature 4 in a buffer, and to ensure that only the sub-feature 2 is sent to the second computing unit in a communication process (for example, AlltoAll), the sub-feature 1, the sub-feature 2, the sub-feature 3, and the sub-feature 4 can be placed in the buffer in the order of arrangement, and for AlltoAll, data in a first block space is placed locally, and data in a second block space is sent to the opposite end.

[0186] In a possible implementation, the first computing unit is deployed on a first node, and the first computing unit is further configured to, after sending the first sub-feature to the second computing unit, obtain, in a communication process (for example, Allgather), a fourth sub-feature that needs to be processed by the second expert module and that is sent by another computing unit on the first node; and the first computing unit can process the third sub-feature, the fourth sub-feature, and the second sub-feature by using the second expert module.

[0187] In a possible implementation, the second computing unit is further deployed with a third expert module, which is different from the first expert module; and the first computing unit can further send, in a communication process with the second computing unit, a feature in the first feature that needs to be processed by the third expert module to the second computing unit after sending the first sub-feature to the second computing unit.

[0188] In a possible implementation, the first computing unit is deployed with a fourth expert module; the process of sending the features in the first features corresponding to the processing of the third expert module is that the first computing unit is executed in parallel when the fourth expert module processes the features in the first features corresponding to the fourth expert module.

[0189] Specifically, the first computing unit can send the features requiring the processing of the first expert module to the second computing unit in a communication process with the second computing unit, and then, in order to accelerate the data execution process by implementing parallelism, the first computing unit can further communicate with the second computing unit in a next communication process while the first computing unit processes the data by using the expert module, for example, the first computing unit can send the features requiring the processing of the third expert module to the second computing unit. Parallelism (that is, mutual masking) of the computing and communication processes can accelerate the data processing process.

[0190] For example, referring to FIG. 8B, FIG. 8B shows a data processing process on a computing node (the applicable communication system can be as shown in FIG. 8A), in which the dispatch all2all communication performed on Stream3 sends the tokens belonging to Expert1 and Expert5 to NPU1 and NPU2 respectively; the Allgather communication performed on Stream2 collects the tokens belonging to Expert1 and Expert5 in the NPU in the tp domain respectively; once all the required tokens for the computation are available, the computation of Expert1 and Expert5 is started; while the computation is being performed, the communication of the tokens required by the next batch of experts (Expert2 and Expert6) can be initiated, to realize the computation and communication pipelining between experts, so as to achieve the effect of computation masking; after the communication of the tokens required by all the batches of experts is completed, the Reduce-scatter and combine all2all communication of TP / Megatron SP are initiated; the above process is repeated until the computation and communication process of all the experts are completed.

[0191] In addition, when each computing unit includes fewer expert modules (for example, only two), only the computation and communication pipelining between experts can result in a low proportion of masked communication. As shown in FIG. 8C. In order to improve the proportion of computation and communication masking, the computation and communication operators of each expert can be further split into multiple copies, and the communication and computation between different copies of the same expert have no dependency, and the similar scheduling as described above is used to achieve the multi-level pipelining of computation and communication between multiple experts and multiple copies. It should be noted that, compared with the scheme of only fine-grained splitting, the multi-level pipelining further utilizes the computation and communication pipelining between experts, to a certain extent, to alleviate the performance loss caused by too fine-grained splitting.

[0192] In this case, the first sub-feature identified in step 602 and step 603 can be a segment of the feature in the first feature that needs to be processed by the first expert module; the first computing unit can send another segment of the feature in the first feature that needs to be processed by the first expert module to the second computing unit in another communication process after the first communication process (for example, the second computing unit has completed the processing of the first sub-feature by the first expert module, or the second computing unit has not completed the processing of the first sub-feature by the first expert module). Of course, the number of segments of the feature that needs to be processed by the first expert module is not limited in the embodiments of the present application.

[0193] In a possible implementation, the first computing unit can further obtain a fifth sub-feature, the fifth sub-feature being obtained by fusing different sub-features through Allgather; the fifth sub-feature including features of a plurality of tokens; according to the fifth sub-feature, Key data and Query data of the plurality of tokens are obtained through linear transformation; the attention score between each two different tokens in the plurality of tokens and the attention score within each token in the plurality of tokens are calculated while maintaining the arrangement of the Key data and the Query data of the plurality of tokens in the memory, wherein the attention score between each two different tokens is obtained based on the Key data of one token and the Query data of the other token, and the attention score within each token is obtained based on the Key data and the Query data of the same token.

[0194] For the Attention part, the effect of calculation communication masking is achieved through fine-grained segmentation and operator scheduling. Since the calculation of the Attention part is sensitive to the memory arrangement of data, in order to ensure the correctness of the calculation, according to the number of data segmentation and combined with the Allgather / Reduce-scatter collective communication semantics, the necessary row and column conversion is performed on the incoming attention mask matrix, as shown in FIG. 8D, thereby avoiding the additional overhead caused by the memory rearrangement of data.

[0195] Referring to FIG. 9, FIG. 9 is a structural schematic of a data processing apparatus 900 provided by the embodiments of the present application, which is applied to a first computing unit, the first computing unit being in communication with a second computing unit, and the second computing unit being deployed with a first expert module; the apparatus comprises:

[0196] The obtaining module 901 is configured to obtain a first feature, the first feature being a feature representation of a data sequence input to the first computing unit;

[0197] The specific introduction of the obtaining module 901 can refer to step 601 in the above embodiment, and details are not described herein.

[0198] The gating module 902 is configured to identify, from the first feature, a first sub-feature corresponding to the first expert module.

[0199] The specific introduction of the gating module 902 can refer to step 602 in the above embodiment, and details are not described herein.

[0200] The transceiver module 903 is configured to send, in one communication process with the second computing unit, a plurality of sub-features to the second computing unit, wherein the plurality of sub-features sent to the second computing unit are all first sub-features corresponding to the first expert module.

[0201] The specific introduction of the transceiver module 903 can refer to step 603 in the above embodiment, and details are not described herein.

[0202] In a possible implementation, the communication process includes AlltoAll communication or Allgather communication.

[0203] In a possible implementation, the first feature is a feature representation of a data sequence input to the first computing unit in a batch.

[0204] In a possible implementation, the first computing unit is deployed with the second expert module and the gating module, and the gating module is further configured to:

[0205] identify, from the first feature, a third sub-feature corresponding to the second expert module;

[0206] The apparatus further includes:

[0207] The model processing module is configured to process the third sub-feature and the second sub-feature through the second expert module.

[0208] In a possible implementation, the first computing unit is deployed on a first node, and the transceiver module is further configured to obtain a fourth sub-feature corresponding to the second expert module sent by another computing unit on the first node;

[0209] The model processing module is specifically configured to:

[0210] process the third sub-feature, the fourth sub-feature, and the second sub-feature through the second expert module.

[0211] In a possible implementation, the second computing unit is further deployed with a third expert module, and the third expert module is different from the first expert module.

[0212] The transceiving module is further configured to: in another communication process different from the one-time communication process, send, to the second computing unit, another sub-feature of the first feature corresponding to the first expert module.

[0213] In a possible implementation, the first computing unit is deployed with a fourth expert module; and the process of sending the feature of the first feature corresponding to the third expert module is performed in parallel by the first computing unit when processing the feature of the first feature corresponding to the fourth expert module by the fourth expert module.

[0214] In a possible implementation, the first sub-feature is a sub-feature of the feature of the first feature corresponding to the first expert module.

[0215] The transceiving module is further configured to:

[0216] In another communication process different from the one-time communication process, send, to the second computing unit, another sub-feature of the first feature corresponding to the first expert module.

[0217] The embodiments of the present application further provide a computer program product, which, when running on a computer, causes the computer to perform the steps performed by the foregoing execution device, or causes the computer to perform the steps performed by the foregoing training device.

[0218] The embodiments of the present application further provide a computer readable storage medium, which stores a program for signal processing, and when running on a computer, causes the computer to perform the steps performed by the foregoing execution device, or causes the computer to perform the steps performed by the foregoing training device.

[0219] The execution device, the training device or the terminal device provided by the embodiments of the present application can be a chip, which includes a processing unit and a communication unit. The processing unit can be a processor, and the communication unit can be an input / output interface, a pin or a circuit, etc. The processing unit can execute computer execution instructions stored in a storage unit, so that the chip in the execution device performs the data processing method described in the foregoing embodiments, or so that the chip in the training device performs the data processing method described in the foregoing embodiments. Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc. The storage unit can also be a storage unit outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

[0220] Specifically, refer to FIG. 10, which is a structural schematic diagram of a chip provided by an embodiment of the present application. The chip can be a neural network processor NPU 1000, which is mounted on a host CPU as a coprocessor and is assigned tasks by the host CPU. The core part of the NPU is an operation circuit 1003, which extracts matrix data in a memory and performs multiplication operation under the control of a controller 1004.

[0221] In some implementations, the operation circuit 1003 internally includes a plurality of processing units (PEs). In some implementations, the operation circuit 1003 is a two-dimensional systolic array. The operation circuit 1003 can also be a one-dimensional systolic array or other electronic circuit capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 1003 is a general-purpose matrix processor.

[0222] For example, assume that there are an input matrix A, a weight matrix B, and an output matrix C. The operation circuit takes corresponding data of the matrix B from the weight memory 1002 and caches the data on each PE in the operation circuit. The operation circuit takes the matrix A data from the input memory 1001 and performs matrix operation with the matrix B, and the partial result or final result of the matrix is saved in an accumulator 1008.

[0223] The unified memory 1006 is used to store input data and output data. The weight data is transferred to the weight memory 1002 through a direct memory access controller (DMAC) 1005. The input data is also transferred to the unified memory 1006 through the DMAC.

[0224] The BIU is a bus interface unit 1010, which is used for interaction between the AXI bus and the DMAC and an instruction fetch buffer (IFB) 1009.

[0225] The bus interface unit 1010 (BIU) is used for the instruction fetch buffer 1009 to obtain instructions from an external memory, and is also used for the direct memory access controller 1005 to obtain original data of the input matrix A or the weight matrix B from the external memory.

[0226] The DMAC is mainly used to transfer input data in an external memory DDR to the unified memory 1006, or to transfer weight data to the weight memory 1002, or to transfer input data to the input memory 1001.

[0227] Vector computation unit 1007 includes multiple operation processing units that further process the output of operation circuit 1003 as needed, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, and the like. It is mainly used for non-convolution / full connection layer network computation in neural networks, such as Batch Normalization, pixel-level summation, up-sampling of feature planes, and the like.

[0228] In some implementations, vector computation unit 1007 can store the processed output vector to unified memory 1006. For example, vector computation unit 1007 can apply a linear function; or, a non-linear function to the output of operation circuit 1003, such as linear interpolation on the feature planes extracted by the convolution layer, and further, for example, accumulate the value vector to generate the activation value. In some implementations, vector computation unit 1007 generates normalized values, pixel-level summation values, or both. In some implementations, the processed output vector can be used as the activation input to operation circuit 1003, for example, for use in the subsequent layers in the neural network.

[0229] Controller 1004 is connected to instruction fetch buffer 1009 for storing instructions used by controller 1004;

[0230] Unified memory 1006, input memory 1001, weight memory 1002, and instruction fetch buffer 1009 are all on-chip memories. External memories are private to the NPU hardware architecture.

[0231] The application further provides a computing device 1000, which can be the computing unit (e.g., the first computing unit or the second computing unit) in the above embodiments. As shown in FIG. 11, the computing device 1000 includes a bus 102, a processor 104, a memory 106, and a communication interface 108. The processor 104, the memory 106, and the communication interface 108 communicate through the bus 102. The computing device 1000 can be a server or a terminal device. It should be understood that the number of processors and memories in the computing device 1000 is not limited in the application.

[0232] The bus 102 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one line is represented in FIG. 11, but it does not mean that there is only one bus or only one type of bus. The bus 102 can include a path for transmitting information between various components (e.g., the memory 106, the processor 104, the communication interface 108) of the computing device 1000.

[0233] The processor 104 can include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP), etc.

[0234] The memory 106 can include a volatile memory (e.g., a random access memory (RAM)) and a non-volatile memory (e.g., a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD)).

[0235] The memory 106 stores executable program codes, and the processor 104 executes the executable program codes to respectively implement the functions of the aforementioned acquisition module 901, the gating module 902, and the transceiving module 903, thereby implementing the data processing method. That is, the memory 106 has instructions for executing the data processing method.

[0236] The communication interface 108 uses a transceiving module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 1000 and other devices or communication networks.

[0237] The embodiments of the present application also provide a computing device cluster. The computing device cluster comprises at least one computing device. The computing device can be a server, for example, a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a notebook computer, or a smart phone.

[0238] As shown in FIG. 12, the computing device cluster comprises at least one computing device 1000. The memory 106 in one or more computing devices 1000 in the computing device cluster can store the same instructions for performing the data processing method.

[0239] In some possible implementations, the memory 106 in one or more computing devices 1000 in the computing device cluster can also respectively store partial instructions for performing the data processing method. In other words, the combination of one or more computing devices 1000 can collectively execute the instructions for performing the data processing method.

[0240] It should be noted that the memory 106 in different computing devices 1000 in the computing device cluster can store different instructions, respectively used for performing partial functions of the computing unit. That is, the instructions stored in the memory 106 in different computing devices 1000 can implement the functions of one or more of the acquisition module 901, the gating module 902, and the transceiving module 903.

[0241] In some possible implementations, one or more computing devices in the computing device cluster can be connected through a network. The network can be a wide area network or a local area network, etc. FIG. 13 shows one possible implementation. As shown in FIG. 13, two computing devices 1000A and 1000B are connected through a network. Specifically, the communication interface in each computing device is connected to the network. In this type of possible implementation, the memory 106 in the computing device 1000A stores instructions for performing the functions of the acquisition module 901. Meanwhile, the memory 106 in the computing device 1000B stores instructions for performing the functions of the gating module 902 and the transceiving module 903.

[0242] It should be understood that the functions of the computing device 1000A shown in FIG. 12 can also be completed by multiple computing devices 1000. Similarly, the functions of the computing device 1000B can also be completed by multiple computing devices 1000.

[0243] The embodiments of the present application further provide another computing device cluster. The connection relationship between the computing devices in the computing device cluster can be similar to the connection manners of the computing device clusters in FIG. 11 and FIG. 12. The difference is that the same instruction for performing the data processing method can be stored in the memory 106 of one or more computing devices 1000 in the computing device cluster.

[0244] In some possible implementation manners, part of the instruction for performing the data processing method can also be respectively stored in the memory 106 of one or more computing devices 1000 in the computing device cluster. In other words, the combination of one or more computing devices 1000 can collectively execute the instruction for performing the data processing method.

[0245] It should be noted that the memory 106 in different computing devices 1000 in the computing device cluster can store different instructions for performing part of the functions of the communication system. That is, the instructions stored in the memory 106 in different computing devices 1000 can implement the functions of one or more of the acquisition module 901, the gating module 902 and the transceiver module 903.

[0246] Wherein, the processor mentioned in any of the above can be a general central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of the above programs.

[0247] In addition, it should be noted that the apparatus embodiments described above are merely schematic, and the units illustrated as separate components can or can not be physically separated, and the components illustrated as units can or can not be physical units, that is, they can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments of the present application. In addition, the connection relationship between the modules in the apparatus embodiments provided by the present application indicates that there is a communication connection between them, which can be implemented as one or more communication buses or signal lines.

[0248] Those skilled in the art can clearly understand that the application can be implemented by means of software plus necessary universal hardware, of course, also can be implemented by special hardware including special integrated circuit, special CPU, special memory, special component, etc. Generally, the functions completed by computer program can be easily implemented by corresponding hardware, and the specific hardware structure for implementing the same function can also be various, such as analog circuit, digital circuit or special circuit, etc. However, for the application, the software program implementation is the better embodiment. Based on such understanding, the technical solutions of the application can be embodied in the form of software product, which is stored in a readable storage medium, such as the floppy disk, U disk, mobile hard disk, ROM, RAM, magnetic disc or optical disc of computer, etc., including a plurality of instructions for making a computer device (which can be a personal computer, training device or network device, etc.) execute the method of each embodiment of the application.

[0249] In the above embodiments, all or part can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part can be implemented in the form of a computer program product.

[0250] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, all or part of the processes or functions according to the embodiments of the application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer readable storage medium or transferred from one computer readable storage medium to another, for example, the computer instructions can be transmitted from one website, computer, training device or data center to another website, computer, training device or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer readable storage medium can be any available medium that the computer can store or the data storage device such as training device, data center, etc. integrated with one or more available media. The available medium can be magnetic medium (such as floppy disk, hard disk, magnetic tape), optical medium (such as DVD) or semiconductor medium (such as solid state disk (SSD)) etc.

Claims

1. A communication system, characterized by The communication system comprises a first computing unit and a second computing unit, the first computing unit is in communication with the second computing unit, and the second computing unit is deployed with a first expert module; The first computing unit is configured to obtain a first feature, the first feature being a feature representation of a data sequence input to the first computing unit; The first computing unit is configured to identify a first sub-feature corresponding to the first expert module from the first feature; The first computing unit is configured to send a plurality of sub-features to the second computing unit in a communication process with the second computing unit, wherein the plurality of sub-features sent to the second computing unit are all first sub-features corresponding to the first expert module.

2. The system of claim 1, wherein, The communication process comprises AlltoAll communication or Allgather communication.

3. The system of claim 1 or 2, wherein, The first feature is specifically a feature representation of a data sequence input to the first computing unit in a batch.

4. The system of any one of claims 1 to 3, wherein, The first computing unit is deployed with a second expert module; The second computing unit is configured to obtain a second feature, the second feature being a feature representation of a data sequence input to the second computing unit; The second computing unit is configured to identify a second sub-feature corresponding to the second expert module from the second feature; The second computing unit is configured to send a plurality of sub-features to the first computing unit in a communication process with the first computing unit, wherein the plurality of sub-features sent to the first computing unit are all second sub-features corresponding to the second expert module.

5. The system of claim 4, wherein, The first computing unit is further configured to identify a third sub-feature corresponding to the second expert module from the first feature; The third sub-feature and the second sub-feature are processed by the second expert module.

6. The system of claim 5, wherein, The first computing unit is deployed on a first node, and the first computing unit is further configured to obtain a fourth sub-feature corresponding to the second expert module sent by another computing unit on the first node; The first computing unit is specifically configured to process the third sub-feature, the fourth sub-feature, and the second sub-feature by the second expert module.

7. The system of any one of claims 1 to 6, wherein, The second computing unit is further deployed with a third expert module different from the first expert module; and the first computing unit is further configured to send a feature corresponding to the third expert module in the first feature to the second computing unit in another communication process different from the communication process with the second computing unit.

8. The system of claim 7, wherein, The first computing unit is deployed with a fourth expert module; and the process of sending the feature corresponding to the third expert module in the first feature is performed by the first computing unit in parallel with the process of processing a feature corresponding to the fourth expert module in the first feature by the fourth expert module.

9. The system of any of claims 1 to 8, wherein, The first sub-feature is a segment of a feature corresponding to the first expert module in the first feature; The first computing unit is further configured to: In another communication process different from the one communication process, another shard of the feature corresponding to the first expert module in the first feature is sent to the second computing unit.

10. The system of any one of claims 1 to 9, wherein, The first computing unit and the second computing unit are a graphics processing unit (GPU), a network processing unit (NPU), or a tensor processing unit (TPU), the first computing unit is deployed on a first node, and the second computing unit is deployed on a second node, and the first node and the second node are different servers. Or; The first computing unit and the second computing unit are a graphics processing unit (GPU), a network processing unit (NPU), or a tensor processing unit (TPU), and the first computing unit and the second computing unit are both deployed on a first node.

11. A data processing method, characterized by, Applied to a first computing unit, the first computing unit communicates with a second computing unit, and the second computing unit is deployed with a first expert module; the method comprises: Obtaining a first feature, the first feature being a feature representation of a data sequence input to the first computing unit; Identifying a first sub-feature corresponding to the first expert module from the first feature; In a communication process with the second computing unit, a plurality of sub-features are sent to the second computing unit, wherein the plurality of sub-features sent to the second computing unit are all first sub-features corresponding to the first expert module.

12. The method of claim 11, wherein, The communication process includes AlltoAll (AlltoAll) communication or Allgather (Allgather) communication.

13. The method according to claim 11 or 12, characterized in that, The first feature is specifically a feature representation of a data sequence input to the first computing unit in a batch.

14. The method according to any one of claims 11 to 13, characterized in that, The first computing unit is deployed with a second expert module, and the method further comprises: Identifying a third sub-feature corresponding to the second expert module from the first feature; Processing the third sub-feature and the second sub-feature through the second expert module.

15. The method of claim 14, wherein, The first computing unit is deployed on a first node, and the method further comprises: obtaining a fourth sub-feature corresponding to the second expert module sent by another computing unit on the first node; The processing of the third sub-feature and the second sub-feature through the second expert module comprises: Processing the third sub-feature, the fourth sub-feature, and the second sub-feature through the second expert module.

16. The method according to any one of claims 11 to 15, characterized in that, The second computing unit is also deployed with a third expert module, and the third expert module is different from the first expert module; The method further comprises: in another communication process different from the one communication process with the second computing unit, sending a feature corresponding to the third expert module in the first feature to the second computing unit.

17. The method of claim 16, wherein, The first computing unit is deployed with a fourth expert module; the process of sending the feature corresponding to the third expert module in the first feature is performed in parallel by the first computing unit when processing the feature corresponding to the fourth expert module in the first feature through the fourth expert module.

18. The method of any one of claims 11 to 17, wherein, The first sub-feature is a shard of the feature corresponding to the first expert module in the first feature; The method further comprises: In another communication process different from the one communication process, another shard of the first feature corresponding to the first feature of the first expert module is sent to the second computing unit.

19. The method of any one of claims 11 to 18, wherein, The first computing unit and the second computing unit are a graphics processing unit (GPU), a network processing unit (NPU), or a tensor processing unit (TPU), the first computing unit is deployed on a first node, and the second computing unit is deployed on a second node, and the first node and the second node are different servers. Or; The first computing unit and the second computing unit are a graphics processing unit (GPU), a network processing unit (NPU), or a tensor processing unit (TPU), and the first computing unit and the second computing unit are both deployed on a first node.

20. A data processing apparatus, characterized in that, Applied to a first computing unit, the first computing unit communicates with a second computing unit, and the second computing unit is deployed with a first expert module; the device comprises: An acquisition module for acquiring a first feature, the first feature being a feature representation of a data sequence input to the first computing unit; A gating module for identifying a first sub-feature corresponding to the first expert module from the first feature; A transceiver module for sending a plurality of sub-features to the second computing unit in a communication process with the second computing unit, wherein the plurality of sub-features sent to the second computing unit are all the first sub-feature corresponding to the first expert module.

21. The apparatus of claim 20, wherein, The communication process includes AlltoAll (AlltoAll) communication or Allgather (Allgather) communication.

22. A computer storage medium, comprising, The computer storage medium stores one or more instructions, which when executed by one or more computers, cause the one or more computers to perform the operations of the method of any one of claims 11-19.

23. A computer program product, characterised in that, Computer-readable instructions are included, which when executed on a computer device, cause the computer device to perform the method of any one of claims 11-19.

24. A system comprising at least one processor, at least one memory; the processor, the memory are connected through a communication bus and complete communication between each other; The at least one memory is used to store code; The at least one processor is used to execute the code to perform the method of any one of claims 11-19.

25. A chip, characterized by At least one processing unit and interface circuit are included, the interface circuit is used to provide program instructions or data for the at least one processing unit, and the at least one processing unit is used to execute the program instructions to implement the method of any one of claims 11-19.

Images