Model data processing method and device

Abstract

The invention provides a model data processing method, which is applied to a first processing node in a distributed system, and comprises the following steps: obtaining first input data and second input data, the first input data being input data of a first expert, and the second input data being input data of a second expert; calculating the first input data by using the parameters of the first expert to obtain a first result; loading parameters of a second expert through a second memory; and calculating the second input data by using the parameters of the second expert to obtain a second result. In the application, when the available space of the memory of the first processing node is not enough to store the parameters of the plurality of experts running on the processing node, the communication traffic of AllToAll can still be eliminated or even reduced by running all the experts on one first processing node, and the computing power utilization rate is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a method and apparatus for processing model data. Background Technology

[0002] In distributed training scenarios, Expert Parallel (EP) is a common training strategy. In EP, different experts run on different processing nodes. Each processing node receives one data stream, and an expert on one node may need data existing on other processing nodes during computation. From the perspective of a single processing node, it needs to use AllToAll communication to send the data needed by experts on other processing nodes from that node to those nodes, and receive the data stored on other processing nodes for computation by the expert on that node. After computation, AllToAll communication is used to send the processing results back to the original processing node, and it also receives the processing results from other processing nodes to generate the computation results for each data stream. This process requires AllToAll communication with a communication domain size equal to the total number of experts, resulting in significant traffic and reduced Model Flow Utilization (MFU).

[0003] Running multiple experts on a single processing node is an effective way to reduce or even eliminate all-to-all communication. From the perspective of a single processing node, when multiple experts run on a single node, there is no need for all-to-all communication between the experts on that node. The communication domain of the training task is smaller than the total number of experts, which reduces all-to-all communication and improves computational efficiency. However, there is a special case where all-to-all communication can even be eliminated when all experts are running on a single processing node. But if the available memory space on a processing node is insufficient to store the parameters of multiple experts running on that node, then this method cannot be used to reduce or even eliminate all-to-all communication. Summary of the Invention

[0004] This application provides a method and apparatus for processing model data. When the available memory space of the first processing node is insufficient to store the parameters of multiple experts running on the processing node, it is still possible to eliminate or reduce the AllToAll communication volume and improve the computing power utilization by running all experts on the first processing node.

[0005] In a first aspect, this application provides a method for processing model data, applied to a first processing node in a distributed system. The model is deployed in the distributed system, which includes a first processing node and a second processing node of different types. The first processing node includes a first memory storing parameters of a first expert in the model, and the second processing node includes a second memory storing parameters of a second expert in the model. The method includes: acquiring first input data and second input data, wherein the first input data is input data of the first expert and the second input data is input data of the second expert; calculating the first input data using the parameters of the first expert to obtain a first result; loading the parameters of the second expert through the second memory; and calculating the second input data using the parameters of the second expert to obtain a second result.

[0006] In the model data processing method provided in this application, when the available space of the memory (first memory) of the first processing node is insufficient to store multiple experts running on the first processing node, only the parameters of some experts are stored in the memory of the first processing node. The parameters of at least one other expert (second expert) running on the first processing node are stored in the memory of the second processing node. When it is necessary to use the parameters of the experts in the memory of the second processing node for calculation, the parameters of these experts are loaded from the memory of the second processing node to the memory of the first processing node so that the first processing node can use the parameters of these experts for calculation. Thus, even when the available space of the memory of the first processing node is insufficient to store multiple experts running on the first processing node, it is still possible to reduce or even eliminate the AllToAll communication volume and improve the computing power utilization by running multiple experts on the first processing node.

[0007] As one possible implementation, during the process of calculating the first input data using the parameters of the first expert to obtain the first result, the loading of the parameters of the second expert through the second memory is performed simultaneously; and / or

[0008] During the process of calculating the second input data using the parameters of the second expert to obtain the second result, the loading of the second expert's parameters through the second memory is performed simultaneously. In this way, the expert calculation and expert parameter loading processes are executed synchronously, thereby overlapping the time of expert calculation and expert parameter loading, further improving the utilization rate of computing power.

[0009] As one possible implementation, the distributed system further includes a third processing node of the same type as the first processing node. The third processing node stores parameters of a third expert in the model in its memory. The method further includes: acquiring third and fourth input data; the third and fourth input data originating from the third processing node; calculating the third input data using the parameters of the first expert to obtain a third result; calculating the fourth input data using the parameters of the second expert to obtain a fourth result; and sending the third and fourth results to the third processing node. This approach limits the scenario where the number of experts running on a first processing node is less than the total number of experts in the model. Even when the available memory space on a first processing node is insufficient to store multiple experts running on that first processing node (less than the total number of experts), it still allows for reducing AllToAll communication volume and improving computational efficiency by running multiple experts on a single first processing node.

[0010] As one possible implementation, during the process of calculating the third input data using the parameters of the first expert to obtain the third result, the loading of the second expert's parameters through the second memory is performed simultaneously; similarly, during the process of calculating the fourth input data using the parameters of the second expert to obtain the fourth result, the loading of the second expert's parameters through the second memory is performed simultaneously. In this way, the expert calculation and expert parameter loading processes are executed synchronously, thus overlapping the time of expert calculation and expert parameter loading, further improving the utilization rate of computing power.

[0011] As one possible implementation, during the process of calculating the first input data using the parameters of the first expert to obtain a first result or calculating the third input data using the parameters of the first expert to obtain a third result, the process of acquiring the third input data and the fourth input data or sending the third result and the fourth result to the third node is executed simultaneously; and / or

[0012] During the process of calculating the second input data using the parameters of the second expert to obtain the second result, or calculating the fourth input data using the parameters of the second expert to obtain the fourth result, the acquisition of the third and fourth input data, or the sending of the third and fourth results to the third node, are performed simultaneously. In this way, the expert calculation and AllToAll communication processes are executed synchronously, thus overlapping the time of expert calculation and AllToAll communication and further improving the utilization rate of computing power.

[0013] As one possible implementation, during the process of calculating the first input data using the parameters of the first expert to obtain a first result or calculating the third input data using the parameters of the first expert to obtain a third result, the loading of the parameters of the second expert through the second memory and the acquisition of the third and fourth input data are executed simultaneously; and / or

[0014] During the execution of either calculating the first input data using the parameters of the first expert to obtain a first result or calculating the third input data using the parameters of the first expert to obtain a third result, the simultaneous execution of loading the parameters of the second expert through the second memory and sending the third result and the fourth result to the third node; and / or

[0015] During the process of performing the calculation on the second input data using the parameters of the second expert to obtain a second result or the calculation on the fourth input data using the parameters of the second expert to obtain a fourth result, the loading of the parameters of the second expert through the second memory and the acquisition of the third and fourth input data are performed simultaneously; and / or

[0016] During the process of calculating the second input data using the parameters of the second expert to obtain a second result, or calculating the fourth input data using the parameters of the second expert to obtain a fourth result, the loading of the second expert's parameters through the second memory and the acquisition of the third and fourth input data are executed simultaneously. In this way, the processes of expert calculation, expert parameter loading, and AllToAll communication are executed synchronously, thereby overlapping the time of expert calculation, expert parameter loading, and AllToAll communication, further improving the utilization rate of computing power.

[0017] As one possible implementation, the distributed system further includes a third processing node and a fourth processing node. The third processing node is of the same type as the first processing node, while the fourth processing node is of a different type. The third processing node stores the parameters of the third expert in the model in its memory, and the fourth processing node stores the parameters of the fourth expert in the model in its memory. The method further includes: sending fifth and sixth input data to the third processing node; obtaining a fifth and a sixth result; the fifth result is the result calculated by the third processing node using the parameters of the third expert on the fifth input data; the sixth result is the result calculated by the third processing node using the parameters of the fourth expert on the sixth input data; the fifth and sixth results originate from the third processing node. In this approach, a scenario is further defined where the number of experts running on the first processing node is less than the total number of experts in the model. In this scenario, even when the available memory space on a first processing node is insufficient to store multiple experts running on that first processing node (less than the total number of experts), the communication volume of AllToAll can still be reduced and the computing power utilization improved by running multiple experts on a single first processing node.

[0018] As one possible implementation, during the process of performing the calculation on the first input data using the parameters of the first expert to obtain a first result or the calculation on the third input data using the parameters of the first expert to obtain a third result, the process of sending the fifth and sixth input data to the third processing node or obtaining the fifth and sixth results is performed simultaneously; and / or

[0019] During the process of performing the calculation on the second input data using the parameters of the second expert to obtain a second result, or performing the calculation on the fourth input data using the parameters of the second expert to obtain a fourth result, the process of sending the fifth and sixth input data to the third processing node or obtaining the fifth and sixth results is performed simultaneously; and / or

[0020] During the execution of either calculating the first input data using the parameters of the first expert to obtain a first result or calculating the third input data using the parameters of the first expert to obtain a third result, the simultaneous execution of loading the parameters of the second expert through the second memory and sending the fifth and sixth input data to the third processing node; and / or

[0021] During the execution of the calculation of the first input data using the parameters of the first expert to obtain a first result or the calculation of the third input data using the parameters of the first expert to obtain a third result, the loading of the parameters of the second expert through the second memory and the acquisition of the fifth and sixth results are executed simultaneously; and / or

[0022] During the process of performing the calculation on the second input data using the parameters of the second expert to obtain a second result, or performing the calculation on the fourth input data using the parameters of the second expert to obtain a fourth result, the loading of the parameters of the second expert through the second memory and the sending of the fifth and sixth input data to the third processing node are performed simultaneously; and / or

[0023] During the process of calculating the second input data using the parameters of the second expert to obtain the second result, or calculating the fourth input data using the parameters of the second expert to obtain the fourth result, the loading of the second expert's parameters through the second memory and the acquisition of the fifth and sixth results are executed simultaneously. In this way, the processes of expert calculation, expert parameter loading, and / or AllToAll communication are executed synchronously, thereby overlapping the time spent on expert calculation, expert parameter loading, and / or AllToAll communication, further improving the utilization rate of computing power.

[0024] As one possible implementation, the first processing node includes one or more of an NPU, GPU, TPU, and CPU, and the second processing node includes one or more of an NPU, GPU, TPU, and CPU.

[0025] As one possible implementation, the first memory includes HBM and / or DDR.

[0026] As one possible implementation, the second memory includes DDR and / or SSD.

[0027] As one possible implementation, the method is applied to the training or inference of the model.

[0028] As one possible implementation, when the available space in the first memory can only store the parameters of one expert, the total number of the first and second experts is determined based on the average estimated total consumption time of one expert and the average time of two AllToAll communication sessions of one expert. In this approach, the optimal number of experts that the first processing node can run is calculated to further improve the utilization of computing power.

[0029] As one possible implementation, assuming the available space in the first memory can store the parameters of at least two experts, the total number of the first and second experts is determined based on the average estimated total consumption time of one expert, the average computation time of one expert, the average time of one AllToAll communication, and the average time of two AllToAll communication sessions for one expert. In this approach, the optimal number of experts that the first processing node can run is calculated to further improve computing power utilization.

[0030] Secondly, this application provides a model data processing apparatus for executing the method in the first aspect or any possible implementation thereof. Specifically, the apparatus includes modules for executing the method in the first aspect or any possible implementation thereof.

[0031] Thirdly, this application provides a distributed system, including the apparatus described in the second aspect or any possible implementation of the second aspect.

[0032] Fourthly, this application provides a computing device, comprising: at least one memory for storing a program; at least one processor for executing the program stored in the memory; wherein, when the program stored in the memory is executed, the processor is configured to execute the method of the first aspect or a possible implementation thereof.

[0033] Fifthly, this application provides a computer storage medium having a computer program stored thereon, which, when run on a processor, causes the processor to perform the method of the first aspect or a possible implementation thereof.

[0034] In a sixth aspect, this application provides a computer program product that, when run on a processor, causes the processor to perform the method of the first aspect or a possible implementation thereof. Attached Figure Description

[0035] Figure 1 This is an example diagram of AllToAll communication;

[0036] Figure 2(a) is an example of expert parallelism;

[0037] Figure 2(b) is another example of expert parallelism;

[0038] Figure 3 This is a system architecture example diagram provided in this application;

[0039] Figure 4 This is a system architecture example diagram provided in this application;

[0040] Figure 5This is a flowchart of a model data processing method provided in this application;

[0041] Figure 6 This is an example diagram of a model data processing method provided in this application;

[0042] Figure 7 This is an example diagram of another model data processing method provided in this application;

[0043] Figure 8 This is another example diagram of the system architecture provided in this application;

[0044] Figure 9 This is a flowchart of another model data processing method provided in this application;

[0045] Figure 10 This is an example diagram of another model data processing method provided in this application;

[0046] Figure 11 This is an example diagram of another model data processing method provided in this application;

[0047] Figure 12 This is a flowchart of the deployment method for the model task provided in this application;

[0048] Figure 13 This is a functional block diagram of a model data processing device provided in this application;

[0049] Figure 14 This is a functional block diagram of another model data processing device provided in this application;

[0050] Figure 15 This is a schematic diagram of a computing device according to an embodiment of this application. Detailed Implementation

[0051] In this article, the term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The symbol " / " in this article indicates that the related objects are in an "or" relationship; for example, A / B means A or B.

[0052] The terms "first" and "second," etc., used in the specification and claims herein are used to distinguish different objects, not to describe a specific order of objects. For example, "first response message" and "second response message," etc., are used to distinguish different response messages, not to describe a specific order of response messages.

[0053] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0054] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, multiple elements means two or more elements, etc.

[0055] First, some of the technical terms involved in the embodiments of this application will be introduced.

[0056] 1. Distributed training

[0057] Distributed training involves breaking down a machine learning or deep learning training task into multiple subtasks and performing parallel training on multiple processing nodes. These processing nodes can be CPUs (Central Processing Units), GPUs (Graphics Processing Units), TPUs (Tensor Processing Units), or NPUs (Neural Processing Units), etc.

[0058] 2. AllToAll

[0059] This is a many-to-many communication model that ensures each processing node receives data from other processing nodes, comprising two phases: data distribution and data aggregation. In the data distribution phase, each processing node splits the input data into multiple data blocks according to a specified splitting dimension. These data blocks are then sent to other processing nodes in a certain order (e.g., block index), while retaining data blocks pointing to its own processing node. In the data aggregation phase, each processing node receives data blocks from other processing nodes.

[0060] like Figure 1 As shown, in a distributed system, taking the NPU as the processing unit of a processing node as an example, the system includes 4 NPUs, each containing 4 data blocks. After AllToAll communication, each NPU sends data blocks pointing to other NPUs to the corresponding NPU and receives data blocks sent from other NPUs, while keeping the data blocks pointing to itself unchanged, and so on after computation. For example, NPU1 contains 4 data blocks A1, B1, C1, and D1. After AllToAll communication, NPU0 contains 4 data blocks A1, A2, A3, and A4, where A2 comes from NPU2, A3 comes from NPU3, A4 comes from NPU4, and A1 is the data block retained by NPU1 itself. It should be noted that... Figure 1 The diagram shows the scenario where each NPU contains data blocks that require processing by all NPUs. In real-world scenarios, there might be cases where an NPU only contains data blocks that require processing by a subset of the NPUs. Figure 1 This is merely an example and is not intended to limit this application.

[0061] 3. Communication domain

[0062] A communication domain refers to a logical unit formed by devices or nodes connected by physical links in a communication network, used for information exchange and transmission. In this embodiment, a communication domain refers to the smallest logical unit in AllToAll communication, consisting of multiple processing nodes, used to transmit data required for expert calculations.

[0063] 4. Experts

[0064] In this application, "expert" refers to a submodule of the model. This submodule can be a small model, such as a feedforward neural network, a recurrent neural network, or a deep neural network, or it can be a complex network structure. In this application, a model includes multiple experts.

[0065] 5. Experts in parallel

[0066] Expert parallelism is a novel parallelization technique for optimizing the training and inference of deep learning models, often used in conjunction with sparse activation and sparse models. In expert parallelism, one or more experts are selectively activated by gating networks or routing to process the input data, thus achieving sparse activation. Each expert is a model, and all experts use the same model, but the parameters of each model are different. For example, experts might use feedforward neural network models, but the parameters of each feedforward neural network are distinct.

[0067] In the expert parallel strategy, different experts run on different processing nodes. Each processing node receives one data stream, and each data stream is unique. For a given processing node, the corresponding data stream may include data needed for calculations by experts on other processing nodes. AllToAll communication is required to send the corresponding data to the processing node of the expert in question, and after the calculation is completed, AllToAll communication is used to send the calculation result back to the original processing node.

[0068] As shown in Figure 2(a), as an example, the distributed system includes three processing nodes, namely NPU1, NPU2, and NPU3. Expert 1 runs on NPU1, expert 2 runs on NPU2, and expert 3 runs on NPU3. For NPU1, the corresponding data path includes data block 1, data block 2, and data block 3. Data block 1 points to expert 1, data block 2 points to expert 2, and data block 3 points to expert 3. Data block 2 is sent to NPU2 via AllToAll communication, and data block 3 is sent to NPU3. NPU1 also receives data block 4, which needs to be calculated by expert 1, from NPU2, and data block 7, which needs to be calculated by expert 1, from processing node 3. After expert 1 completes its calculation, NPU1 sends the processing result 4 of data block 4 to NPU2 and the processing result 7 of data block 7 to NPU3. NPU1 also receives the processing result 2 of data block 2 from NPU2 and the processing result 3 of data block 3 from NPU3. Based on processing result 1, processing result 2, and processing result 3, an output result is generated.

[0069] It should be noted that the gating network or routing determines which experts each data block in each data path points to (not shown in Figure 2(a)). Furthermore, the data blocks in each data path may point to all experts or only some experts. In the case where the data blocks in each data path point to only some experts, there may be cases where they do not point to the expert being processed by the processing node receiving that data path itself.

[0070] Figure 2(b) is another example of expert parallelism. In this example, there are 8 processing nodes, each with an NPU as its processing unit. Each NPU runs one expert, and the total time spent includes two AllToAll communications and the time for expert computation. In this embodiment, the scenario corresponding to this example is the baseline scenario.

[0071] 6. Synchronization

[0072] Synchronization refers to the execution of multiple tasks at a given time, where the start and end times of each task may be the same or different. In this embodiment, whether multiple tasks can be executed synchronously depends on the available memory space of the processing node of the running expert. If, at a certain time, the available memory space of the processing node does not allow for the synchronous execution of multiple tasks, then only one task is executed at that time, until a later time when the conditions for synchronous execution of multiple tasks are met, at which point the synchronous execution process is initiated.

[0073] The technical solution provided in this application will be described next.

[0074] Figure 3 This is a system architecture example diagram provided in this application. The positional relationships between the architecture, modules, etc. shown in the diagram do not constitute any limitation.

[0075] The system architecture provided in this application can be used for training and inference of models such as hybrid expert models and large language models. For example, the system architecture provided in this application can be applied to distributed frameworks such as Mindspore and Megatron-LM.

[0076] When using frameworks such as Mindspore and Megatron-LM for training or inference, the model execution process can be accelerated by using operator libraries such as CNAA, HCCL, CUDA, and NCCL. The system provided in this application can be applied to chip architectures such as Neural Processing Unit (NPU), Graphics Processing Unit (GPU), Tensor Processing Unit (TPU), or Central Processing Unit (CPU).

[0077] The model data processing method provided in this application can be used in combination with one or more parallel processing systems such as Tensor Parallelism (TP), Sequence Parallelism (SP), and Data Parallelism (DP), and applied to the system architecture provided in this application.

[0078] Figure 4 This application provides an example diagram of a system architecture. The system architecture includes a first processing node and a second processing node. Each first processing node receives one input data stream, and these input data streams are all different. The first processing node includes first memory, and the second processing node includes second memory. The first and second processing nodes are of different types, meaning the structures of their first and second memory streams are different. For a given first processing node, if the available space in its first memory is insufficient to store the parameters of N experts running on that node, the system stores the parameters of m experts out of the N experts running on the first processing node in the second memory of the second processing node, while storing only the parameters of the n experts in the first memory. Here, n + m = N, where n, m, and N are all positive integers, and N is the total number of experts. Figure 4 In this example, n=1, m=7, and N=8; this is merely an example and not a limitation of this application. In this embodiment, the expert whose parameters are stored in the first memory is the first expert, and the expert whose parameters are stored in the second memory is the second expert. Figure 4 In the table, e1 represents the parameters of the first expert, and e2-e8 represent the parameters of the seven second experts.

[0079] In the system architecture provided in this application embodiment, the number of first processing nodes and second processing nodes is arbitrary. The number of first processing nodes and second processing nodes can be the same or different. One second processing node can correspond to one first processing node or multiple first processing nodes, and multiple second processing nodes can also correspond to one first processing node. Figure 4 The quantities and correspondences shown are merely an example and are not intended to limit the embodiments of this application.

[0080] Each first processing node comprises a combination of any number of NPUs, GPUs, TPUs, and / or CPUs, each with independent data processing capabilities. The types and numbers of NPUs, GPUs, TPUs, and / or CPUs included in different first processing nodes can be the same or different. Each NPU, GPU, TPU, or CPU constituting a first processing node is connected to HBM and / or DDR, and the HBM and DDR connected to all NPUs, GPUs, TPUs, and CPUs constitute the first memory of that first processing node. Each second processing node comprises a combination of any number of NPUs, GPUs, TPUs, and / or CPUs, and the types and numbers of NPUs, GPUs, TPUs, and CPUs included in different second processing nodes can be the same or different. Each NPU, GPU, TPU, or CPU constituting a second processing node includes DDR and / or SSD, and the DDR and SSD connected to all NPUs, GPUs, TPUs, and CPUs constitute the second memory of that second processing node. In the embodiments of this application, the first memory and the second memory have different structures. For example, the first memory includes HBM and the second memory includes DDR. In this case, the structures of the first memory and the second memory are different. Or, for example, the first memory includes both HBM and DDR, and the second memory includes SSD. In this case, the structures of the first memory and the second memory are also different.

[0081] It should be noted that in the embodiments of this application, each processing node includes a processing unit and a storage unit. For example, for the first processing node, its processing unit is one or more of NPU, GPU, TPU and CPU, and its storage unit is a first memory, which includes HBM and / or DDR.

[0082] For the second processing node, its processing unit includes one or more of NPU, GPU, TPU and CPU, and the storage unit is the second memory, which includes DDR and / or SSD.

[0083] Using the system architecture provided in this application embodiment, when the available space of the first memory of a first processing node is insufficient to store all the experts running on that first processing node, the parameters of some experts can be stored in the second memory of a second processing node. This allows all the experts' parameters to be stored using both the first and second memory. When the first processing node needs to perform calculations using the experts' parameters in the second memory, these parameters are loaded from the second memory back to the first processing node. Therefore, even when the available space of the first processing node is insufficient to store the parameters of all the experts running on that first processing node, all experts can still be run on a single first processing node, eliminating AllToAll communication overhead and improving computational efficiency.

[0084] In the scenario corresponding to this system architecture, each first processing node can run all experts. Therefore, each input data only needs to perform calculations with the expert who receives the input data. After the calculation is completed and the calculation result is output, the next iteration process or the next prediction process for model training can be carried out without the need for AllToAll communication.

[0085] See Figure 5 This application provides a method for processing model data, wherein the model is deployed on... Figure 4 The system structure described above, the method is applied to Figure 4 The first processing node in the system architecture shown, for a single first processing node, the model data processing method includes the following steps:

[0086] S501, Obtain the first input data and the second input data;

[0087] The gating network or routing determines which experts the input data should be processed by. Input data typically requires processing by multiple experts. The expert whose parameters are stored in the first memory is the first expert, and the expert whose parameters are stored in the second memory is the second expert. The input data that the first expert needs to process is the first input data, and the input data that the second expert needs to process is the second input data. There can be one or more first and second input data sets. Both the first and second experts are experts within the model.

[0088] The input data may activate all experts or only some experts. If only some experts are activated, the unactivated experts will not participate in the subsequent steps S502-S504. That is to say, the first and second experts who execute the subsequent steps S502-S504 are both activated experts.

[0089] S502. Calculate the first input data using the parameters of the first expert to obtain the first result;

[0090] When performing expert calculations on the first input data, the system uses the parameters of the first expert to calculate the first input data and obtain the first result. As an example, the experts use feedforward neural network models, and the parameters of each expert are the parameters of their respective feedforward neural network models. When calculating the first input data based on the parameters of the first expert, the parameters of the feedforward neural network models are used to calculate the first input data.

[0091] When performing expert calculations, either a serial or parallel approach can be used. If there are multiple first input data points, then there are also multiple first experts participating in the calculations, and thus multiple first results are generated. When the first experts perform expert calculations, if a serial approach is used, the parameters of each first expert can be used to calculate the corresponding first input data according to a first order; if a parallel approach is used, the parameters of several first experts can be selected and calculated simultaneously according to a first order. The first order can be the order of the experts' serial numbers, weights, etc., and is not specifically limited here.

[0092] S503, Load the parameters of the second expert through the second memory;

[0093] Before performing expert calculations using the parameters of a second expert, these parameters need to be loaded from the second memory into the first memory. Since the first memory is connected to the first processing node and the second memory is connected to the second processing node, the loading of the second expert's parameters from the second memory into the first memory is achieved through communication between the first and second processing nodes. Specifically, this communication occurs between the processing units contained within the first and second processing nodes. As an example, the processing unit of the first processing node is an NPU, and the first memory is HBM; the processing unit of the second processing node is a CPU, and the second memory is DDR. The second expert's parameters are first transmitted from the DDR to the CPU, then transmitted from the CPU to the NPU via communication between the CPU and the NPU, and finally transmitted from the NPU to the HBM.

[0094] If there are multiple second experts, the parameters of each second expert can be loaded separately according to the second order, or the parameters of several second experts can be selected simultaneously according to the second order. The second order can be the order of the experts' serial numbers, or the weights, etc., without specific limitations here. The second order can be the same as or different from the first order.

[0095] S504. Calculate the second input data using the parameters of the second expert to obtain the second result.

[0096] When performing expert calculations on the second input data, the system uses the parameters of the second expert to calculate the second input data to obtain a second result. If there are multiple second input data points, then there are also multiple second experts participating in the calculation, and thus multiple second results. When the second expert performs the expert calculations, if a serial approach is used, the parameters of each first expert can be used to calculate the corresponding first input data according to the second order; if a parallel approach is used, the parameters of several second experts can be selected simultaneously for calculation according to the second order.

[0097] It should be noted that, in the embodiments of this application, steps S502 and S504 define the process of expert calculation, and step S503 defines the process of loading expert parameters.

[0098] After the expert calculation is completed, the first and second results are output, thus completing one iteration of the training or prediction process. The first and second results can be the direct processing results of the first and second input data, respectively, or other data generated during the calculation process. For example, in a training scenario, it can be the result of an iteration of expert parameters, which can be an activation value; no specific limitation is made here.

[0099] In this embodiment, when the available space of the memory (first memory) of the first processing node is insufficient to store all the experts running on the first processing node, only the parameters of some experts (first experts) are stored in the memory of the first processing node. The parameters of at least one other expert (second expert) running on the first processing node are stored in the memory (second memory) of the second processing node. When it is necessary to use the parameters of the experts in the memory of the second processing node for calculation, the parameters of these experts are loaded from the memory of the second processing node to the memory of the first processing node. Thus, even when the available space of the memory of a first processing node is insufficient to store the parameters of all the experts running on that processing node, it is still possible to eliminate the AllToAll communication traffic and improve the computing power utilization by running all the experts on the first processing node.

[0100] As an example, see Figure 6 The first processing node's processing unit is an NPU, and the second processing node's processing unit is a CPU. There are 8 NPUs and 8 CPUs in total, with one NPU connected to one CPU. Each first processing node receives one input data stream. The total number of experts is 8, and each NPU runs 8 experts. The first memory of the first processing node... Figure 6 The capacity (not shown) is the parameter of one expert. For a first processing node, the parameter e1 of expert E1 is stored in its first memory, and the corresponding second processing node's second memory (not shown) contains the parameter e1 of expert E1. Figure 6 (Not shown) stores the parameters e2-e8 of experts E2-E8. Among them, expert E1 is the first expert, experts E2-E8 are the second experts, and e1-e8 are the parameters of E1-E8 respectively.

[0101] For a single first processing node, the model data processing method provided in this application embodiment includes the following steps:

[0102] (1) Obtain the first input data pointing to expert E1 and the second input data pointing to experts E2-E8. The second input data consists of 7 items, each pointing to one of experts E2-E8.

[0103] (2) Calculate the first input data using the parameter e1 of expert E1 to obtain the first result;

[0104] (3) Load the parameter e2 of expert E2 from the second memory through communication between the CPU and NPU;

[0105] (4) Use the parameter e2 of expert E2 to calculate the second input data pointing to expert E2 to obtain a second result;

[0106] (5) After expert E2 has completed the calculation, steps (4)-(5) are executed sequentially for experts E3-E8 until all the second input data has been processed.

[0107] In this example, when the available memory space of the first processing node can only store the parameters of one expert, the parameter e1 of expert E1 is stored in the first memory of the first processing node, and the parameters e2-e8 of experts E2-E8 are stored in the second memory of the second processing node. When it is necessary to use the parameters e2-e8 of experts E2-E8 for calculation, the parameters e2-e8 of experts E2-E8 are loaded from the second memory to the first memory respectively, so that the NPU can use the parameters e2-e8 of experts E2-E8 for calculation. In this way, even when the available memory space of the first processing node can only store one expert parameter, it is still possible to eliminate AllToAll communication traffic and improve computing power utilization by running all experts E1-E8 on one first processing node.

[0108] In the scenario corresponding to this example, the time spent includes the time for loading expert parameters and the time for expert calculation. Compared with the benchmark scenario, this scenario has a time advantage since the time for loading expert parameters is less than the time for two AllToAll communications in the two benchmark scenarios.

[0109] Alternatively, one implementation is that, within the allowable memory space of a first processing node, step S503 can be executed synchronously during the execution of steps S502 and / or S504. That is, when performing expert calculations, the expert parameter loading process is executed synchronously, thereby overlapping the time of expert calculations and expert parameter loading.

[0110] In this implementation, the simultaneous execution of expert calculations and expert parameter loading overlaps the time of expert calculations and expert parameter loading, further improving the utilization rate of computing power.

[0111] As an example, see Figure 7 The first processing node's processing unit is an NPU, and the second processing node's processing unit is a CPU. There are 8 NPUs and 8 CPUs in total, with one NPU connected to one CPU. Each first processing node receives one input data stream. The total number of experts is 8, and each NPU runs 8 experts. The first memory of the first processing node... Figure 7 The available space (not shown) is for one expert's parameter. For a first processing node, the parameter e1 of expert E1 is stored in the first memory, and the parameter e1 of expert E1 is stored in the second memory of the corresponding second processing node. Figure 7 (Not shown) stores the parameters e2-e8 of experts E2-E8. Among them, expert E1 is the first expert, experts E2-E8 are the second experts, and e1-e8 are the parameters of E1-E8.

[0112] For a single first processing node, the model data processing method provided in this application embodiment includes the following steps:

[0113] (1) Obtain the first input data pointing to expert E1 and the second input data pointing to experts E2-E8, wherein the second input data consists of 7 items, each pointing to one of experts E2-E8;

[0114] (3) Calculate the first input data based on the parameter e1 of expert E1 to generate the first result;

[0115] (4) While expert E1 is performing calculations, the parameter e2 of expert E2 is loaded from the second memory synchronously through communication between the CPU and NPU.

[0116] Since the parameter e1 of expert E1 has been sent from the first memory to the NPU when expert E1 performs the calculation, the space of the first memory can store the parameter e2 of the next expert E2. Therefore, when expert E1 performs expert calculation, the loading process of the parameter e2 of expert E2 can be performed synchronously.

[0117] After the expert E1 finishes the calculation and the parameter e2 of the expert E2 has been loaded, start to execute the calculation of the second input data pointing to the expert E2 according to the parameter e2 of the expert E2 to generate a second result, and synchronously execute the communication between the CPU and the NPU to load the parameter e3 of the expert E3 from the second memory.

[0118] (6) After the expert E2 finishes the calculation, sequentially execute steps (4)-(5) for the experts E3-E8 until all the second input data is processed.

[0119] In the scenario corresponding to this example, since the available space of the first memory is the parameter of 1 expert, when executing the calculation of the previous expert, synchronously execute the parameter loading process of the next expert, so as to cover the time of expert parameter loading with the time of expert calculation. The total time spent only includes the time of expert calculation, which further improves the computing power utilization rate. Compared with the benchmark scenario, the time gain in the scenario corresponding to this example is the time of AllToAll communication.

[0120] Figure 8 Another system structure diagram provided by this application. In this system structure, it includes a first processing node, a second processing node, a third processing node, and a fourth processing node. Each first processing node and third processing node receives a path of input data, and the input data of each path is different. The first processing node includes a first memory, the second processing node includes a second memory, the third processing node includes a third memory, and the fourth processing node includes a fourth memory. For a first processing node, when the available space of the first memory is not enough to store the parameters of M experts running on the first processing node, the system stores the parameters of m experts out of the M experts that need to run on the first processing node in the second memory of the second processing node, and only stores the parameters of n experts in the first memory, where n + m = M, M < N, and n, m, M, and N are all positive integers, and N is the total number of experts in the system. It should be noted that n = 1, m = 3, M = 4, N = 8, which is only an example and not a limitation to this application. In this embodiment, the expert whose parameters are stored in the first memory is the first expert, and the expert whose parameters are stored in the second memory is the second expert. In Figure 8 e1 represents the parameter of the first expert, and e2-e4 represent the parameters of 3 second experts.

[0121] In the system architecture provided in this application, the number of first processing nodes, second processing nodes, third processing nodes, and fourth processing nodes is arbitrary. The number of first processing nodes, second processing nodes, third processing nodes, and fourth processing nodes can be the same or different. One second processing node can correspond to one first processing node, or multiple first processing nodes, and multiple second processing nodes can also correspond to one first processing node; one fourth processing node can correspond to one third processing node, or multiple third processing nodes, and multiple fourth processing nodes can also correspond to one third processing node. Figure 8 The number and correspondence of the first and second processing nodes, as well as the number and correspondence of the third and fourth processing nodes, are merely examples and are not intended to limit the embodiments of this application.

[0122] Each first processing node comprises a combination of any number of NPUs, GPUs, TPUs, and / or CPUs, each with independent data processing capabilities. The types and numbers of NPUs, GPUs, TPUs, and CPUs contained in different first processing nodes can be the same or different. Each NPU, GPU, TPU, or CPU constituting a first processing node is connected to HBM or DDR, which constitute the first memory of that first processing node. Each second processing node comprises a combination of any number of NPUs, GPUs, TPUs, and / or CPUs, and the types and numbers of NPUs, GPUs, TPUs, and CPUs contained in different second processing nodes can be the same or different. Each NPU, GPU, TPU, or CPU constituting a second processing node is connected to DDR or SSD, which constitute the second memory of that second processing node.

[0123] Each third processing node's processing unit is a combination of any number of NPUs, GPUs, TPUs, and / or CPUs, and each third processing node has independent data processing capabilities. The types and numbers of NPUs, GPUs, TPUs, and CPUs contained in different third processing nodes can be the same or different. Each NPU, GPU, TPU, or CPU constituting a third processing node is connected to HBM or DDR, which constitute the third memory of that third processing node. Each fourth processing node's processing unit is a combination of any number of NPUs, GPUs, TPUs, and / or CPUs, and the types and numbers of NPUs, GPUs, TPUs, and CPUs contained in different fourth processing nodes can be the same or different. Each NPU, GPU, TPU, or CPU constituting a fourth processing node is connected to DDR or SSD, which constitute the fourth memory of that fourth processing node. The third processing node and the first processing node are of the same type, and the fourth processing node and the second processing node are of the same type, meaning they have the same memory structure. In this embodiment, the expert whose parameters are stored in the third memory is the third expert, and the expert whose parameters are stored in the fourth memory is the fourth expert. Figure 8 In the table, e5 represents the parameters of the third expert, and e6-e8 represent the parameters of the three fourth experts.

[0124] It should be noted that, in the embodiments of this application, each processing node includes a processing unit and a storage unit. For example, for a first processing node, its processing unit is one or more of an NPU, GPU, TPU, and CPU, and its storage unit is a first memory, which includes HBM and / or DDR. For a second processing node, its processing unit includes one or more of an NPU, GPU, TPU, and CPU, and its storage unit is a second memory, which includes DDR and / or SSD. For a third processing node, its processing unit is one or more of an NPU, GPU, TPU, and CPU, and its storage unit is a third memory, which includes HBM and / or DDR. For a fourth processing node, its processing unit includes one or more of an NPU, GPU, TPU, and CPU, and its storage unit is a fourth memory, which includes DDR and / or SSD.

[0125] Using the system architecture provided in this application embodiment, when the available memory space of a first processing node is insufficient to store the parameters of multiple experts (less than the total number of experts) running on that first processing node, the parameters of some experts can be stored in the second memory of a second processing node. This allows both the first and second memory to store the parameters of multiple experts running on the first processing node. When a first processing node needs to perform calculations using the parameters of experts in the second memory, these parameters are loaded from the second memory into the first memory. Therefore, when the available memory space of a first processing node is insufficient to store the parameters of multiple experts running on that processing node, the communication volume of AllToAll can be reduced and the computing power utilization improved by running multiple experts on a single first processing node. The total number of experts running on a single first processing node is less than the total number of experts in the system.

[0126] In this system architecture scenario, since the total number of experts running in a single first processing node is less than the total number of experts in the system, AllToAll communication is required within the same communication domain to transmit the data needed for computation by the first processing node and to send the processing results back to the original first processing node. This same communication domain includes one first processing node and several third processing nodes. The first and third processing nodes are of the same type, meaning their memory structures are identical. Furthermore, the total number of experts within the same communication domain is N, and the parameters of each expert within the same communication domain are distinct.

[0127] In the scenario corresponding to this system architecture, each first processing node can run multiple experts. There is no need for all-to-all communication between the experts on the processing node. The size of its communication domain is smaller than the total number of experts, which can reduce the amount of all-to-all communication and improve the utilization of computing power.

[0128] See Figure 9 This application provides a method for processing model data, wherein the model is deployed on... Figure 8 The system structure described above, the method is applied to Figure 8 The first processing node in the system architecture shown, for a single first processing node, the model data processing method includes the following steps:

[0129] S901, Obtain the first input data and the second input data;

[0130] The gating network or routing determines which experts the input data is directed to. The expert whose parameters are stored in the first memory is the first expert, and the expert whose parameters are stored in the second memory is the second expert. The input data that needs to be processed by the first expert is the first input data, and the input data that needs to be processed by the second expert is the second input data.

[0131] The input data may activate all experts or only some experts. If only some experts are activated, the unactivated experts will not participate in the subsequent steps S902-S908. In other words, the first and second experts who execute the subsequent steps S902-S908 are both activated experts.

[0132] S902, Obtain the third input data and the fourth input data;

[0133] The third and fourth processing data are sent by the third processing node. The third processing data is the input data that needs to be processed by the first expert, and the fourth input data is the input data that needs to be processed by the second expert. There can be one or more third and fourth input data.

[0134] S903, Send the fifth input data and the sixth input data to the third processing node;

[0135] The fifth input data is the input data that needs to be processed by the third expert, and the sixth input data is the input data that needs to be processed by the fourth expert. The third expert is the expert whose parameters are stored in the third memory, and the fourth expert is the expert whose parameters are stored in the fourth memory. The logic of the third processing node in processing the fifth and sixth processing data is the same as that of the first processing node, and will not be elaborated here.

[0136] The fifth and sixth input data can be either one or multiple. Which third processing node the fifth and sixth input data are sent to is determined by the expert the third input data points to; that is, it is sent to the third processing node corresponding to the expert the fifth and sixth input data point to within the same communication domain. The fifth or sixth input data sent to a third processing node may point to one expert running on that third processing node, or it may point to multiple experts running on that third processing node. When it points to multiple experts running on other first processing nodes, the fifth or sixth input data includes multiple subsets, each subset corresponding to one of the experts.

[0137] It should be noted that if only the calculations performed by the expert running on the first processing node are considered, step S903 can be omitted. If S903 is omitted, calculations for the third and fourth input data are not required in steps S904 and S906.

[0138] In the embodiments of this application, the first expert, the second expert, the third expert, and the fourth expert are all experts in the model.

[0139] S904. Calculate the first input data and the third input data using the parameters of the first expert to obtain the first result and the third result;

[0140] The first result corresponds to the first input data, and the third result corresponds to the third input data. When performing calculations by the first expert, the first and third input data can be calculated separately, or the corresponding first and third input data can be calculated simultaneously using the parameters of the same first expert; no specific limitation is made here.

[0141] When performing expert calculations, either a serial or parallel approach can be used. If the total number of first and third input data is multiple, then the number of first experts participating in the calculation is also multiple. When the first experts perform expert calculations, if a serial approach is used, the parameters of each first expert can be used to calculate the corresponding first and third input data according to a first order; if a parallel approach is used, the parameters of several first experts can be selected and calculated simultaneously according to a first order. The first order can be the order of the experts' serial numbers, weights, etc., and is not specifically limited here.

[0142] S905, Load the parameters of the second expert through the second memory;

[0143] Before performing expert calculations using the parameters of a second expert, these parameters need to be loaded from the second memory into the first memory. Since the first memory is connected to the first processing node and the second memory is connected to the second processing node, the loading of the second expert's parameters from the second memory into the first memory is achieved through communication between the first and second processing nodes. Specifically, this communication occurs between the processing units contained within the first and second processing nodes. As an example, the processing unit of the first processing node is an NPU, and the first memory is HBM; the processing unit of the second processing node is a CPU, and the second memory is DDR. The second expert's parameters are first transmitted from the DDR to the CPU, then transmitted from the CPU to the NPU via communication between the CPU and the NPU, and finally transmitted from the NPU to the HBM.

[0144] If there are multiple second experts, the parameters of each second expert can be loaded separately according to the second order, or the parameters of several second experts can be selected simultaneously according to the second order. The second order can be the order of the experts' serial numbers, or the weights, etc., without specific limitations here. The second order can be the same as or different from the first order.

[0145] S906. Calculate the second input data and the fourth input data using the parameters of the second expert to obtain the second result and the fourth result;

[0146] The second result corresponds to the second input data, and the fourth result corresponds to the fourth input data. When performing calculations by the second expert, the second and fourth input data can be calculated separately, or the corresponding second and fourth input data can be calculated simultaneously using the parameters of the same second expert; no specific limitation is made here.

[0147] If the total number of second and fourth input data is multiple, then the number of second experts participating in the calculation is also multiple. When the second experts perform expert calculations, if a serial method is used, the parameters of each second expert can be used to calculate the corresponding second and fourth input data according to the second order; if a parallel method is used, the parameters of several second experts can be selected simultaneously according to the second order for calculation.

[0148] It should be noted that in this embodiment, steps S904 and S906 define the expert calculation process, and step S905 defines the expert parameter loading process. Steps S905-S906 can be executed on several second experts, and then S905-S906 can be executed on the remaining second experts.

[0149] S907. Send the third and fourth results to the third processing node;

[0150] Each third and fourth result is sent to its corresponding third processing node. Specifically, for each third or fourth result, the corresponding third or fourth result is sent to the same third processing node from which the third or fourth input data originated.

[0151] It should be noted that after several first experts and / or second experts have completed S904 or S906, the third processing result and / or the fourth result can be sent by executing S907. After the sending is completed, S902, S904-S906 can be executed again for the remaining second experts.

[0152] S908, Obtain the fifth and sixth results.

[0153] After the third processing node performs expert calculations on the fifth and sixth input data, it obtains the fifth and sixth results respectively, and the first processing node obtains the corresponding fifth and sixth results.

[0154] If only the calculations performed by the expert running on a first processing node are considered, step S909 may be omitted.

[0155] After the expert calculations are completed at the first processing node, the first result, the second result, the fifth result, and the sixth result are output, thus completing one iteration of the training process or the prediction process.

[0156] It should be noted that the first result, second result, third result, fourth result, fifth result, and sixth result can be the direct processing results of the first input data, second input data, third input data, fourth input data, fifth input data, and sixth input data, respectively, or other data generated during the calculation process. For example, in a training scenario, it can be the result of an iteration of relevant expert parameters. This iteration result can be an activation value, and no specific limitation is made here.

[0157] In this embodiment, when the available space of the memory (first memory) of the first processing node is insufficient to store multiple experts (less than the total number of experts) running on the first processing node, only the parameters of some experts (first experts) are stored in the memory of the first processing node. The parameters of at least one other expert (second expert) running on the first processing node are stored in the memory (second memory) of the second processing node. When it is necessary to use the parameters of the experts in the memory of the second processing node for calculation, the parameters of these experts are loaded from the memory of the second processing node to the memory of the first processing node so that the first processing node can use the parameters of these experts for calculation. Thus, when the available space of the memory of the first processing node is insufficient to store multiple experts (less than the total number of experts) running on the first processing node, the communication volume of AllToAll can be reduced and the computing power utilization can be improved by running multiple experts on the first processing node.

[0158] As an example, see Figure 10 The processing units of the first and third processing nodes are NPUs, while the processing units of the second and fourth processing nodes are CPUs. There are 8 NPUs and 8 CPUs in total, with one NPU connected to one CPU. Each first processing node receives one input data stream. The total number of experts is 8, with each NPU running 4 experts. Two NPUs running different experts constitute a communication domain, and NPUs within the same communication domain are represented by the same gray area. The first memory of the first processing node (… Figure 10 The available space (not shown) is the parameter of one expert. For two NPUs in the same communication domain, experts E1-E4 run in one NPU, and experts E5-E8 run in the other NPU. The NPU running experts E1-E4 is the processing unit of the first processing node, and the NPU running experts E5-E8 is the processing unit of the third processing node. The CPU connected to the first processing node is the processing unit of the second processing node, and the CPU connected to the third processing node is the processing unit of the fourth processing node. The parameter e1 of expert E1 is stored in the first memory of the first processing node, and the parameter e1 of expert E1 is stored in the second memory of the second processing node. Figure 10The parameters e2-e4 of storage experts E2-E4 are stored in the third memory of the third processing node (not shown). Figure 10 The parameter e5 of expert E5 is stored in the fourth memory of the fourth processing node (not shown). Figure 10 (Not shown) stores the parameters e6-e8 of experts E6-E8. Among them, expert E1 is the first expert, experts E2-E4 are the second experts, expert E5 is the third expert, and experts E6-E8 are the fourth experts.

[0159] For a single first processing node, the model data processing method provided in this application embodiment includes the following steps:

[0160] (1) Obtain the first input data pointing to expert E1 and the second input data pointing to experts E2-E4; wherein, the second input data consists of 3 items, each pointing to one of experts E2-E4.

[0161] (2) Obtain the third input data pointing to expert E1 and the fourth input data pointing to expert E2 sent by the CPU in the same communication domain;

[0162] (3) Send the fifth input data pointing to expert E5 and the sixth input data pointing to experts E6-E8 to the CPU in the same communication domain;

[0163] (4) Calculate the first input data and the third input data according to the parameter e1 of expert E1 to obtain a first result and a third result respectively;

[0164] (5) Load the parameter e2 of expert E2 from the second memory through communication between the CPU and NPU;

[0165] (6) Calculate the second and fourth input data pointing to expert E2 based on the parameter e2 of expert E2 to generate a second result and a fourth result;

[0166] (7) After expert E2 has finished its calculation, execute steps (5)-(7) sequentially for experts E3-E4 until all the second and fourth input data have been processed.

[0167] (8) Send all third and fourth results to the CPU in the same communication domain;

[0168] (9) Obtain the fifth and sixth results sent by the CPU in the same communication domain.

[0169] In this example, when the available memory space of the first processing node can only store the parameters of one expert, the parameter e1 of expert E1 is stored in the first memory of the first processing node NPU, and the parameters e2-e4 of experts E2-E4 are stored in the second memory connected to the second processing node CPU. When it is necessary to use the parameters e2-e4 of experts E2-E4 for calculation, the parameters e2-e4 of experts E2-E4 are loaded from the second memory to the first memory respectively, so that the NPU can use the parameters e2-e4 of experts E2-E4 for calculation. In this way, even when the available memory space of the NPU can only store the parameters of one expert, it is still possible to reduce the AllToAll communication volume and improve the computing power utilization by running multiple experts E1-E4 on one NPU.

[0170] In this example, AllToAll communication is reduced by running multiple experts in each NPU. In the scenario corresponding to this example, the total time spent includes expert parameter loading, expert computation, and the time for two AllToAll communications. Compared to the baseline scenario, this scenario demonstrates a time advantage when the sum of the time for expert parameter loading and the two AllToAll communications in the reduced communication domain is less than the time for the two AllToAll communications in the baseline scenario.

[0171] Optionally, as an implementation, within the allowable range of available memory space in the first processing node, step S905 can be executed synchronously during the execution of steps S904 and / or S906. That is, the expert calculation and expert parameter loading processes are executed synchronously, thereby overlapping the time of expert calculation and expert parameter loading and further improving the utilization rate of computing power.

[0172] Optionally, as an implementation, within the allowable memory space of a first processing node, steps S902, S903, S907, and / or S908 can be executed simultaneously during the execution of steps S904 and / or S906. That is, expert computation and AllToAll communication are executed synchronously, thereby overlapping the time of expert computation and AllToAll communication and further improving computing power utilization.

[0173] Optionally, as an implementation, within the allowable range of available memory space on the first processing node, steps S902, S903, S905, S907, and / or S908 can be executed synchronously during the execution of steps S904 and / or S906. That is, expert calculations, expert parameter loading, and AllToAll communication are executed synchronously, thereby overlapping the time of these processes and further improving computing power utilization.

[0174] As an example, see Figure 11 The processing units of the first and third processing nodes are NPUs, while the processing units of the second and fourth processing nodes are CPUs. There are 8 NPUs and 8 CPUs in total, with one NPU connected to one CPU. Each first processing node receives one input data stream. The total number of experts is 8, with each NPU running 4 experts. Two NPUs running different experts constitute a communication domain, and NPUs within the same communication domain are represented by the same gray area. The first memory of the first processing node (… Figure 10 The available space (not shown) is the parameter of one expert. For two NPUs in the same communication domain, experts E1-E4 run in one NPU, and experts E5-E8 run in the other NPU. The NPU running experts E1-E4 is the processing unit of the first processing node, and the NPU running experts E5-E8 is the processing unit of the third processing node. The CPU connected to the first processing node is the processing unit of the second processing node, and the CPU connected to the third processing node is the processing unit of the fourth processing node. The parameter e1 of expert E1 is stored in the first memory of the first processing node, and the parameter e1 of expert E1 is stored in the second memory of the second processing node. Figure 11 The parameters e2-e4 of storage experts E2-E4 are stored in the third memory of the third processing node (not shown). Figure 11 The parameter e5 of expert E5 is stored in the fourth memory of the fourth processing node (not shown). Figure 10 (Not shown) stores the parameters e6-e8 of experts E6-E8. Among them, expert E1 is the first expert, experts E2-E4 are the second experts, expert E5 is the third expert, and experts E6-E8 are the fourth experts.

[0175] For a single first processing node, the model data processing method provided in this application embodiment includes the following steps:

[0176] (1) Obtain the first input data pointing to expert E1 and the second input data pointing to experts E2-E4; wherein, the second input data consists of 3 items, each pointing to one of experts E2-E4.

[0177] (2) Receive the third input data A1 sent by the CPU in the same communication domain, which points to expert E1;

[0178] (3) Calculate the first input data and the third input data A1 based on the parameter e1 of expert E1 to obtain a first result and a fourth result a1 respectively;

[0179] (4) When expert E1 performs calculations, the process of loading parameter e2 of expert E2 from the second memory and receiving the fourth input data A2-A4 sent by the CPU in the same communication domain to experts E2-E4 is executed synchronously.

[0180] Since the parameter e1 of expert E1 has been sent from the first memory to the NPU when expert E1 performs the calculation, the available space in the first memory can accommodate the parameter e2 of the next expert E2. Therefore, when performing expert calculation on expert E1, the loading process of the parameter e2 of expert E2 can be performed synchronously.

[0181] (5) When expert E1 has finished calculating, expert E2 has finished loading its parameter e2, and the fourth input data A2 pointing to expert E2 has finished receiving, calculate the second input data and the fourth input data A2 according to expert E2's parameter e2 to obtain the second result and the fourth result a2 respectively.

[0182] (6) When expert E2 performs calculations, the process of loading parameter e3 of expert E3 from the second memory and receiving the fourth input data pointing to the remaining experts sent by the CPU are executed synchronously; and after the fifth input data A2-A4 pointing to experts E2-E4 is received, the process of sending the third result a1 and the fourth result a2-a4 are executed synchronously.

[0183] For example, when expert E2 is performing calculations, the fourth input data A4 has already been sent, then the fourth result a1 is sent.

[0184] (7) Perform steps (6)-(7) sequentially on experts E3-E4 until experts E1-E4 have completed their calculations.

[0185] (8) Send the fourth result a4 to the corresponding CPU.

[0186] In this example scenario, since the capacity of the first memory is equivalent to the parameters of one expert, the loading of parameters for the next expert and the AllToAll communication processes for other experts' data processing or other experts' calculation results can be executed simultaneously while the calculation of the previous expert is being performed. This allows the time for expert calculation, expert parameter loading, and AllToAll communication to overlap, further improving computing power utilization. Compared to the baseline scenario, the time gain in this example scenario is the AllToAll communication time of the baseline scenario minus the AllToAll communication time of one expert in the reduced communication domain.

[0187] As one implementation method, to further improve computing power utilization, in addition to executing multiple tasks simultaneously, the number of experts running on a single first processing node can be optimized. Figure 11For example, when the available space in the first memory can only store the parameters of one expert, and multiple tasks are already executed synchronously, the total time spent includes the time for two AllToAll communication sessions for one expert and the time for synchronous task execution. Therefore, before model deployment, when calculating the optimal number of experts running on each first processing node, the larger value is selected between the average estimated total consumption time of one expert and the average of the two AllToAll communication times of one expert. The solution is then found to maximize the time benefit of this larger value for the number of experts running on each first processing node. The number of experts running on the first processing node that maximizes the time benefit is the optimal number. Here, the average total consumption time of one expert is the ratio of the total system consumption time to the total number of experts, and the number of experts running on one first processing node is the total number of first and second experts.

[0188] As one implementation method, when the available space of the first memory of the first processing node can store the parameters of more than two experts, the expert loading time is significantly reduced compared to when the available space of the first memory can only store the parameters of one expert, since each first memory can store at least two expert parameters. Therefore, to further improve the utilization of computing power, when calculating the optimal number of experts running on each first processing node, given that multiple tasks are already executed simultaneously, the maximum value is selected from the average estimated total consumption time of an expert, the average computation time of an expert, the average time of one AllToAll communication, and the average time of two AllToAll communication of an expert. The optimal number of experts running on each first processing node is then calculated to maximize the time benefit of this maximum value. Here, the average total consumption time of an expert is the ratio of the total system consumption time to the total number of experts; the computation time of an expert is the computation time of an expert on the input data to be processed; and the number of experts running on a first processing node is the total number of first and second experts.

[0189] See Figure 12 This application provides a method for deploying a model task, including:

[0190] S1201. Determine the first deployment parameters based on the available space in the first memory;

[0191] The first memory is the storage unit of the first processing node, including HBM and / or DDR. The available space in the first memory is used to store the parameters of the experts in the model. The first processing node is the processing node in the distributed system used to run the experts. The distributed system includes first processing nodes and second processing nodes, with one second processing node connected to one first processing node. The storage unit of the second processing node is the second memory. When the available space in the first memory is insufficient to store the parameters of the experts running on the corresponding first processing node, the second memory is used to store some of the experts' parameters.

[0192] The first deployment parameter is the greater of the estimated average total time consumed by an expert and the average of two AllToAll communication times for an expert, or the greater of the estimated average total time consumed by an expert, the average computation time of an expert, the average of one AllToAll communication time, and the average of two AllToAll communication times for an expert. Since different models contain different numbers of experts and parameters, the corresponding expert computation time, AllToAll communication time, and expert parameter loading time will all differ. Therefore, for a given model, before determining the deployment method, it is necessary to estimate the expert computation time, AllToAll communication time, and expert parameter loading time for that model separately. Based on these estimates, the average total time consumed by an expert, the average of two AllToAll communication times for an expert, the average of one computation time and one AllToAll communication time for an expert, and the average of two AllToAll communication times for an expert are determined. The first deployment parameter is then determined based on these determined averages.

[0193] When the available space in the first memory can only store the parameters of one expert, the first deployment parameter is the larger of the estimated average total consumption time of one expert and the average of two AllToAll communication times of one expert. When the available space in the first memory can store the parameters of at least two experts, the first deployment parameter is the maximum of the estimated average total consumption time of one expert, the average of the calculation time of one expert, the average of one AllToAll communication time, and the average of two AllToAll communication times of one expert.

[0194] S1202. Determine the second deployment parameters based on the first deployment parameters;

[0195] The second deployment parameter is the number of experts running on the first processing node that maximizes the time benefit of the first deployment parameter. Those skilled in the art will understand that time benefit means minimizing the time consumed.

[0196] S1203. Determine the deployment method based on the available space of the first memory and the second deployment parameters;

[0197] If the value of the second deployment parameter is greater than the number of expert parameters that the available space of the first memory can store, the available storage space of the first memory is insufficient to run the parameters of the experts deployed on the first processing node, and the second memory of the second processing node needs to be used to store some of the parameters of the experts running on the first processing node.

[0198] If the value of the second deployment parameter is not greater than the number of expert parameters that the available space of the first memory can store, the available storage space of the first memory can store all the parameters of the experts running on the first processing node, without needing to use the second memory of the second processing node to store the parameters of the experts running on the first processing node.

[0199] The deployment method refers to deploying the task model based on a third deployment parameter and a fourth deployment parameter. The third deployment parameter is the number of expert parameters stored in the first memory, and the fourth deployment parameter is the number of expert parameters stored in the second memory. If the value of the second deployment parameter is greater than the number of expert parameters that the available space in the first memory can store, the third deployment parameter is the number of expert parameters that the available space in the first memory can store, and the fourth deployment parameter is the difference between the second and third deployment parameters. If the value of the second deployment parameter is not greater than the number of expert parameters that the available space in the first memory can store, the third deployment parameter is equal to the second deployment parameter, and the fourth deployment parameter is 0.

[0200] It should be noted that in this embodiment, it is assumed that the number of experts running on each first processing node is the same. Therefore, the determined deployment method only includes the third deployment parameter and the fourth deployment parameter, that is, the number of expert parameters stored in the first memory and the number of expert parameters stored in the second memory.

[0201] S1204. Deploy the model task according to the deployment method.

[0202] In this embodiment, the distributed system includes a first processing node and a second processing node. Each processing node includes a processing unit and a storage unit. For example, for the first processing node, its processing unit is one or more of an NPU, GPU, TPU, and CPU, and its storage unit is a first memory, which includes HBM and / or DDR. For the second processing node, its processing unit includes one or more of an NPU, GPU, TPU, and CPU, and its storage unit is a second memory, which includes DDR and / or SSD.

[0203] After determining the deployment method, the model tasks are deployed in the distributed system according to the deployment method.

[0204] In this embodiment, the model tasks are deployed with the goal of maximizing computing power utilization, which effectively utilizes system resources and improves the processing efficiency of the model tasks.

[0205] See Figure 13 This application provides a model data processing device 300, which is applied to... Figure 4 In the system architecture shown, a first processing node, for a single first processing node, includes:

[0206] The first acquisition module 1301 is used to acquire first input data and second input data;

[0207] The first calculation module 1302 is used to calculate the first input data using the parameters of the first expert to obtain the first result;

[0208] Loading module 1303 is used to load parameters from a second expert via a second memory.

[0209] The second calculation module 1304 is used to calculate the second input data using the parameters of the second expert to obtain the second result.

[0210] The implementation process of each module in the embodiments of this application is described in the following references. Figure 5 The details of each step of the method will not be described in detail here.

[0211] In this embodiment, when the memory capacity of a first processing node (first memory) is limited and insufficient to store all experts running on the first processing node, only the parameters of some experts (first experts) are stored in the memory of the first processing node. The parameters of at least one other expert (second expert) running on the first processing node are stored in the memory of the second processing node (second memory). When it is necessary to use the parameters of the experts in the memory of the second processing node for calculation, the parameters of these experts are loaded from the memory of the second processing node into the memory of the first processing node so that the first processing node can use these parameters of the experts for calculation. Thus, when the memory capacity of a first processing node is limited, the communication traffic of AllToAll can be eliminated by running all experts on a first processing node, thereby improving the utilization rate of computing power.

[0212] Optionally, as an implementation, within the allowable range of available memory space in the first processing node, the data processing apparatus of this application embodiment further includes a synchronization module for synchronously implementing the functions of the first calculation module 1302 and the loading module 1303, and / or synchronously implementing the functions of the second calculation module 1304 and the loading module 1303.

[0213] In this implementation, the simultaneous execution of expert calculations and expert parameter loading overlaps the time of expert calculations and expert parameter loading, thereby further improving the utilization rate of computing power.

[0214] See Figure 14 This application provides a model data processing device 400, which is applied to... Figure 8 In the system architecture shown, a first processing node, for a single first processing node, includes:

[0215] The first acquisition module 1401 is used to acquire first input data and second input data;

[0216] The second acquisition module 1402 is used to acquire the third input data and the fourth input data;

[0217] The first sending module 1403 is used to send the fifth input data and the sixth input data to the third processing node;

[0218] The first calculation module 1404 is used to calculate the first input data and the third input data using the parameters of the first expert to obtain the first result and the third result;

[0219] Loading module 1405 is used to load parameters from a second expert via a second memory.

[0220] The second calculation module 1406 is used to perform calculations on the second input data and the fourth input data using the parameters of the second expert to obtain the second result and the fifth result;

[0221] The second sending module 1407 is used to send the third result and the fourth result to the third processing node;

[0222] The third acquisition module 1408 is used to acquire the fifth and sixth results.

[0223] In this embodiment, when the available space of the memory (first memory) of the first processing node is insufficient to store multiple experts (less than the total number of experts) running on the first processing node, only the parameters of some experts (first experts) are stored in the memory of the first processing node. The parameters of at least one other expert (second expert) running on the first processing node are stored in the memory (second memory) of the second processing node. When it is necessary to use the parameters of the experts in the memory of the second processing node for calculation, the parameters of these experts are loaded from the memory of the second processing node to the memory of the first processing node so that the first processing node can use the parameters of these experts for calculation. Thus, when the available space of the memory of the first processing node is insufficient to store multiple experts (less than the total number of experts) running on the first processing node, the communication volume of AllToAll can be reduced and the computing power utilization can be improved by running multiple experts on the first processing node.

[0224] Optionally, as an implementation, within the allowable range of available memory space in the first processing node, the data processing apparatus of this application embodiment further includes a synchronization module for synchronously implementing the functions of the first calculation module 1404 and the loading module 1405, and / or synchronously implementing the functions of the second calculation module 1406 and the loading module 1405. In this implementation, by synchronously executing expert calculations and expert parameter loading, the time for expert calculations and expert parameter loading overlaps, further improving the utilization rate of computing power.

[0225] Optionally, as an implementation, within the allowable range of available memory space in the first processing node, the data processing apparatus of this application embodiment further includes a synchronization module, used to synchronously implement the functions of the second acquisition module 1402, the first sending module 1403, the first calculation module 1404, the second sending module 1407, and the third acquisition module 1408, and / or synchronously implement the functions of the second acquisition module 1402, the first sending module 1403, the second calculation module 1406, the second sending module 1407, and the third acquisition module 1408. In this implementation, by synchronously executing expert calculation and AllToAll communication, the time of expert calculation and AllToAll communication overlaps, further improving the utilization rate of computing power.

[0226] Optionally, as an implementation, within the allowable range of available memory space in the first processing node, the data processing apparatus of this application embodiment further includes a synchronization module, used to synchronously implement the functions of the second acquisition module 1402, the first sending module 1403, the first calculation module 1404, the loading module 1405, the second sending module 1407, and the third acquisition module 1408, and / or synchronously implement the functions of the second acquisition module 1402, the first sending module 1403, the loading module 1405, the second calculation module 1406, the second sending module 1407, and the third acquisition module 1408. In this implementation, by synchronously executing expert calculation, expert parameter loading, and AllToAll communication, the time of expert calculation and AllToAll communication overlaps, further improving the utilization rate of computing power.

[0227] This application provides a distributed system that may include a model data processing device 300 or a model data processing device 400.

[0228] Please see Figure 15 This is a schematic diagram of an embodiment of the computing device in this application.

[0229] The computing device provided in this embodiment can be a server or a dedicated data processing device, etc. The specific form of the device is not limited in this embodiment.

[0230] The computing device 1500 can vary considerably due to differences in configuration or performance, and may include one or more processors 1501 and a memory 1502, in which programs or data are stored. One or more processors 1501 constitute the processing unit in the above embodiments of this application, and one or more memories 1502 constitute the storage unit in the above embodiments of this application.

[0231] The memory 1502 can be volatile or non-volatile memory. Optionally, the processor 1501 is one or more central processing units (CPUs), graphics processing units (GPUs), or other dedicated processors, such as Ascend. The CPU can be a single-core CPU or a multi-core CPU. The processor 1501 can communicate with the memory 1502 and execute a series of instructions stored in the memory 1502 on the computing device 1500.

[0232] The computing device 1500 also includes one or more wired or wireless network interfaces 1503, such as Ethernet interfaces.

[0233] Optionally, although Figure 15As not shown in the diagram, the computing device 1500 may also include one or more power supplies; one or more input / output interfaces, which can be used to connect to a monitor, mouse, keyboard, touch screen device or sensing device, etc. The input / output interfaces are optional components and may or may not be present, and are not limited here.

[0234] In this embodiment, the memory 1502 in the computing device 1500 stores a computer program. The process executed by the processor 1501 when executing the computer program can refer to the method flow described in the foregoing method embodiment, and will not be repeated here.

[0235] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, they can be implemented in whole or in part in the form of a computer program product.

[0236] Computer program products include computer programs. When a computer program is loaded and executed on a computer, it produces, in whole or in part, the processes or functions according to the embodiments of this application. A computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. A computer program can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another, for example, from one website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means to another website, computer, server, or data center. A computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. Available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

[0237] The technical solutions provided in this application have been described in detail above. Specific examples have been used in this application to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method of processing model data, characterized by, A first processing node applied in a distributed system, the model is deployed in the distributed system, the distributed system includes the first processing node and a second processing node, the first processing node and the second processing node are different in type, the first processing node includes a first memory, the first memory stores parameters of a first expert in the model, the second processing node includes a second memory, the second memory stores parameters of a second expert in the model, and the method comprises: obtaining first input data and second input data, the first input data being input data of the first expert, and the second input data being input data of the second expert; calculating the first input data using the parameters of the first expert to obtain a first result; loading the parameters of the second expert through the second memory; calculating the second input data using the parameters of the second expert to obtain a second result.

2. The method of claim 1, wherein: during the execution of the calculation of the first input data using the parameters of the first expert to obtain a first result, the loading of the parameters of the second expert through the second memory is performed synchronously; and / or during the execution of the calculation of the second input data using the parameters of the second expert to obtain a second result, the loading of the parameters of the second expert through the second memory is performed synchronously.

3. The method of claim 1 or 2, wherein, The distributed system further includes a third processing node, the third processing node and the first processing node are of the same type, and the memory of the third processing node stores parameters of a third expert in the model, and the method further comprises: obtaining third input data and fourth input data; the third input data and the fourth input data come from the third processing node; calculating the third input data using the parameters of the first expert to obtain a third result; calculating the fourth input data using the parameters of the second expert to obtain a fourth result; sending the third result and the fourth result to the third processing node.

4. The method of claim 3, wherein: during the execution of the calculation of the third input data using the parameters of the first expert to obtain a third result, the loading of the parameters of the second expert through the second memory is performed synchronously; during the execution of the calculation of the fourth input data using the parameters of the second expert to obtain a fourth result, the loading of the parameters of the second expert through the second memory is performed synchronously.

5. The method of claim 3 or 4, wherein: during the execution of the calculation of the first input data using the parameters of the first expert to obtain a first result or the calculation of the third input data using the parameters of the first expert to obtain a third result, the obtaining of the third input data and the fourth input data or the sending of the third result and the fourth result to the third node is performed synchronously; and / or synchronously performing the loading the parameters of the second expert from the second memory and the obtaining the third input data and the fourth input data in the process of performing the calculating the second input data using the parameters of the second expert to obtain a second result or the calculating the fourth input data using the parameters of the second expert to obtain a fourth result; 6. The method of any one of claims 3-5, wherein: synchronously performing the loading the parameters of the second expert from the second memory and the obtaining the third input data and the fourth input data in the process of performing the calculating the first input data using the parameters of the first expert to obtain a first result or the calculating the third input data using the parameters of the first expert to obtain a third result; and / or synchronously performing the loading the parameters of the second expert from the second memory and the obtaining the third input data and the fourth input data in the process of performing the calculating the first input data using the parameters of the first expert to obtain a first result or the calculating the third input data using the parameters of the first expert to obtain a third result; and / or synchronously performing the loading the parameters of the second expert from the second memory and the obtaining the third input data and the fourth input data in the process of performing the calculating the second input data using the parameters of the second expert to obtain a second result or the calculating the fourth input data using the parameters of the second expert to obtain a fourth result; and / or synchronously performing the loading the parameters of the second expert from the second memory and the obtaining the third input data and the fourth input data in the process of performing the calculating the second input data using the parameters of the second expert to obtain a second result or the calculating the fourth input data using the parameters of the second expert to obtain a fourth result.

7. The method according to any one of claims 3 to 6, wherein, The distributed system further comprises a third processing node and a fourth processing node, the third processing node and the first processing node are of the same type, the fourth processing node and the first processing node are of different types, the memory of the third processing node stores parameters of a third expert in the model, the memory of the fourth processing node stores parameters of a fourth expert in the model, and the method further comprises: sending fifth input data and sixth input data to the third processing node; obtaining a fifth result and a sixth result; the fifth result is a result obtained by the third processing node calculating the fifth input data using the parameters of the third expert; the sixth result is a result obtained by the third processing node calculating the sixth input data using the parameters of the fourth expert; and the fifth result and the sixth result come from the third processing node.

8. The method of claim 7, wherein: synchronously performing the sending the fifth input data and the sixth input data to the third processing node or the obtaining the fifth result and the sixth result in the process of performing the calculation on the first input data using the parameters of the first expert to obtain the first result or the calculation on the third input data using the parameters of the first expert to obtain the third result; and / or synchronously performing the sending the fifth input data and the sixth input data to the third processing node or the obtaining the fifth result and the sixth result in the process of performing the calculation on the second input data using the parameters of the second expert to obtain the second result or the calculation on the fourth input data using the parameters of the second expert to obtain the fourth result; and / or synchronously performing the loading the parameters of the second expert through the second memory and the sending the fifth input data and the sixth input data to the third processing node in the process of performing the calculation on the first input data using the parameters of the first expert to obtain the first result or the calculation on the third input data using the parameters of the first expert to obtain the third result; and / or synchronously performing the loading the parameters of the second expert through the second memory and the obtaining the fifth result and the sixth result in the process of performing the calculation on the first input data using the parameters of the first expert to obtain the first result or the calculation on the third input data using the parameters of the first expert to obtain the third result; and / or synchronously performing the loading the parameters of the second expert through the second memory and the sending the fifth input data and the sixth input data to the third processing node in the process of performing the calculation on the second input data using the parameters of the second expert to obtain the second result or the calculation on the fourth input data using the parameters of the second expert to obtain the fourth result; and / or synchronously performing the loading the parameters of the second expert through the second memory and the obtaining the fifth result and the sixth result in the process of performing the calculation on the second input data using the parameters of the second expert to obtain the second result or the calculation on the fourth input data using the parameters of the second expert to obtain the fourth result.

9. The method according to any one of claims 1 to 8, wherein, The first processing node comprises one or more of NPU, GPU, TPU and CPU, and the second processing node comprises one or more of NPU, GPU, TPU and CPU.

10. The method of any one of claims 1-9, wherein, The first memory is HBM and / or DDR.

11. The method of any one of claims 1-10, wherein, The second memory is DDR and / or SSD.

12. The method of any one of claims 1-11, wherein, The method is applied to training or inference of a model.

13. The method of any one of claims 1-12, wherein, In the case that the available space of the first memory can only store the parameters of one expert, the total number of the first expert and the second expert is determined according to the average value of the estimated total consumption time of one expert and the average value of the two AllToAll communication times of one expert.

14. The method of any one of claims 1-12, wherein, In a case that the available space of the first memory is capable of storing parameters of at least two experts, the total number of the first expert and the second expert is determined according to an average value of total consumption time of one estimated expert, an average value of computation time of one expert and one AllToAll communication time, and an average value of two AllToAll communication times of one expert.

15. A model data processing apparatus, characterized by comprising: The application discloses a device applied to a first processing node in a distributed system, wherein the model is deployed in the distributed system, the distributed system comprises the first processing node and a second processing node, the first processing node and the second processing node are of different types, the first processing node comprises a first memory, the first memory stores parameters of a first expert in the model, the second processing node comprises a second memory, the second memory stores parameters of a second expert in the model, and the device comprises: a first obtaining module, configured to obtain first input data and second input data, wherein the first input data is input data of the first expert, and the second input data is input data of the second expert; a first calculating module, configured to calculate the first input data by using the parameters of the first expert to obtain a first result; a loading module, configured to load the parameters of the second expert through the second memory; a second calculating module, configured to calculate the second input data by using the parameters of the second expert to obtain a second result.

16. The device of claim 15, wherein: in the process of calculating the first input data by using the parameters of the first expert to obtain a first result, the loading of the parameters of the second expert through the second memory is synchronously implemented; and / or in the process of calculating the second input data by using the parameters of the second expert to obtain a second result, the loading of the parameters of the second expert through the second memory is synchronously implemented.

17. The apparatus of claim 15 or 16, wherein, The distributed system further comprises a third processing node, the third processing node and the first processing node are of the same type, and parameters of a third expert in the model are stored in a memory of the third processing node, and the device further comprises: a second obtaining module, configured to obtain third input data and fourth input data, wherein the third input data and the fourth input data are from the third processing node; the first calculating module is further configured to calculate the third input data by using the parameters of the first expert to obtain a third result; the second calculating module is further configured to calculate the fourth input data by using the parameters of the second expert to obtain a fourth result; a second sending module, configured to send the third result and the fourth result to the third processing node.

18. The device of claim 17, wherein: in the process of calculating the third input data by using the parameters of the first expert to obtain a third result, the loading of the parameters of the second expert through the second memory is synchronously implemented; In the process of implementing the calculation of the fourth input data using the parameters of the second expert to obtain a fourth result, the loading of the parameters of the second expert from the second memory is implemented synchronously.

19. The apparatus of claim 17 or 18, wherein: In the process of implementing the calculation of the first input data using the parameters of the first expert to obtain a first result or the calculation of the third input data using the parameters of the first expert to obtain a third result, the obtaining of the third input data and the fourth input data or the sending of the third result and the fourth result to the third node is implemented synchronously; and / or In the process of implementing the calculation of the second input data using the parameters of the second expert to obtain a second result or the calculation of the fourth input data using the parameters of the second expert to obtain a fourth result, the obtaining of the third input data and the fourth input data or the sending of the third result and the fourth result to the third node is implemented synchronously.

20. The apparatus of any of claims 17-19, wherein: In the process of implementing the calculation of the first input data using the parameters of the first expert to obtain a first result or the calculation of the third input data using the parameters of the first expert to obtain a third result, the loading of the parameters of the second expert from the second memory and the obtaining of the third input data and the fourth input data is implemented synchronously; and / or In the process of implementing the calculation of the first input data using the parameters of the first expert to obtain a first result or the calculation of the third input data using the parameters of the first expert to obtain a third result, the loading of the parameters of the second expert from the second memory and the sending of the third result and the fourth result to the third node is implemented synchronously; and / or In the process of implementing the calculation of the second input data using the parameters of the second expert to obtain a second result or the calculation of the fourth input data using the parameters of the second expert to obtain a fourth result, the loading of the parameters of the second expert from the second memory and the obtaining of the third input data and the fourth input data is implemented synchronously; and / or In the process of implementing the calculation of the second input data using the parameters of the second expert to obtain a second result or the calculation of the fourth input data using the parameters of the second expert to obtain a fourth result, the synchronous implementation of the loading of the parameters of the second expert from the second memory and the obtaining of the third input data and the fourth input data is implemented synchronously.

21. The apparatus of any one of claims 17-20, wherein, The distributed system further comprises a third processing node and a fourth processing node, the third processing node and the first processing node are of the same type, the fourth processing node and the first processing node are of different types, the parameters of a third expert in the model are stored in the memory of the third processing node, the parameters of a fourth expert in the model are stored in the memory of the fourth processing node; The first sending module is configured to send fifth input data and sixth input data to a third processing node; The third obtaining module is configured to obtain a fifth result and a sixth result; the fifth result is a result obtained by the third processing node using parameters of the third expert to calculate the fifth input data; The sixth result is a result obtained by the third processing node using parameters of the fourth expert to calculate the sixth input data; and the fifth result and the sixth result are from the third processing node.

22. The apparatus of claim 21, wherein: the sending of the fifth input data and the sixth input data to the third processing node or the obtaining of the fifth result and the sixth result is performed synchronously with the performing of the calculating of the first input data using the parameters of the first expert to obtain the first result or the calculating of the third input data using the parameters of the first expert to obtain the third result; and / or the sending of the fifth input data and the sixth input data to the third processing node or the obtaining of the fifth result and the sixth result is performed synchronously with the performing of the calculating of the second input data using the parameters of the second expert to obtain the second result or the calculating of the fourth input data using the parameters of the second expert to obtain the fourth result; and / or the synchronously performing of the loading of the parameters of the second expert from the second memory and the sending of the fifth input data and the sixth input data to the third processing node is performed synchronously with the performing of the calculating of the first input data using the parameters of the first expert to obtain the first result or the calculating of the third input data using the parameters of the first expert to obtain the third result; and / or the synchronously performing of the loading of the parameters of the second expert from the second memory and the obtaining of the fifth result and the sixth result is performed synchronously with the performing of the calculating of the first input data using the parameters of the first expert to obtain the first result or the calculating of the third input data using the parameters of the first expert to obtain the third result; and / or the synchronously performing of the loading of the parameters of the second expert from the second memory and the sending of the fifth input data and the sixth input data to the third processing node is performed synchronously with the performing of the calculating of the second input data using the parameters of the second expert to obtain the second result or the calculating of the fourth input data using the parameters of the second expert to obtain the fourth result; and / or the synchronously performing of the loading of the parameters of the second expert from the second memory and the obtaining of the fifth result and the sixth result is performed synchronously with the performing of the calculating of the second input data using the parameters of the second expert to obtain the second result or the calculating of the fourth input data using the parameters of the second expert to obtain the fourth result.

23. The apparatus of any one of claims 15-22, wherein, The first processing node comprises one or more of an NPU, a GPU, a TPU, and a CPU, and the second processing node comprises one or more of an NPU, a GPU, a TPU, and a CPU.

24. The apparatus of any one of claims 15-23, wherein, The first memory includes HBM and / or DDR.

25. The apparatus of any one of claims 15-24, wherein, The second memory includes DDR and / or SSD.

26. The apparatus of any one of claims 15-25, wherein, The method is applied to training or inference of a model.

27. The apparatus of any one of claims 15-26, wherein, In a case where the available space of the first memory can only store parameters of one expert, the total number of the first expert and the second expert is determined according to an average of the estimated total consumption time of one expert and an average of two AllToAll communication times of one expert.

28. The apparatus of any one of claims 15-26, wherein, In a case where the available space of the first memory can store parameters of at least two experts, the total number of the first expert and the second expert is determined according to an average of the estimated total consumption time of one expert, an average of the computation time and one AllToAll communication time of one expert, and an average of two AllToAll communication times of one expert.

29. A distributed system comprising the apparatus of any one of claims 15-28.

30. A computer device, comprising: comprising: at least one memory for storing a program; at least one processor for executing the program stored in the memory; wherein, when the program stored in the memory is executed, the processor is configured to perform the method of any one of claims 1-14.

31. A computer readable storage medium, the computer readable storage medium storing a computer program, when the computer program is run on a processor, causing the processor to perform the method of any one of claims 1-14.

32. A computer program product, characterised in that, when the computer program product is run on a processor, causing the processor to perform the method of any one of claims 1-14.

Images