Model data processing method and apparatus

Abstract

A model data processing method, applied to a first processing node in a distributed system. The method comprises: acquiring first input data and second input data (S501), wherein the first input data is input data of a first expert, and the second input data is input data of a second expert; using parameters of the first expert to calculate the first input data to obtain a first result (S502); loading parameters of the second expert by means of a second memory (S503); and using the parameters of the second expert to calculate the second input data to obtain a second result (S504). When the available space of a memory of the first processing node is insufficient to store parameters of a plurality of experts running on the processing node, the communication traffic of AllToAll can still be reduced or eliminated by running all experts on one first processing node, thereby increasing the utilization rate of computing power.

Description

Model data processing method and device

[0001] The present application claims priority to the Chinese patent application No. 202411393272.2, filed on September 30, 2024, and entitled "Model data processing method and device", the whole content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the field of artificial intelligence, and in particular to a model data processing method and device. BACKGROUND

[0003] In a distributed training scenario, expert parallel (EP) is a common training strategy. In the expert parallel training strategy, different experts run on different processing nodes, and each processing node receives one way of data. The experts in one processing node may need data existing in other processing nodes when calculating. From the perspective of a single processing node, the data required by the experts on other processing nodes needs to be sent from the processing node to other processing nodes through AllToAll communication, and the data stored on other processing nodes that is required by the experts on the processing node needs to be received. After the calculation is completed, the processing result is sent back to the original processing node through AllToAll communication, and the processing result belonging to the processing node sent back by other processing nodes is accepted, so as to produce the calculation result of each way of data. In this process, AllToAll communication with a communication domain size of the total number of experts is needed, which generates a large amount of communication and reduces the model FLOPS utilization (MFU).

[0004] Running multiple experts on one processing node is an effective way to reduce or even eliminate the AllToAll communication amount. From the perspective of a single processing node, in the case of running multiple experts on one processing node, AllToAll communication is not needed between the experts on the processing node, the size of the communication domain of the training task is less than the total number of experts, the AllToAll communication amount can be reduced, and the model FLOPS utilization can be improved. When all experts run on one processing node, AllToAll can even be eliminated. However, in the case where the available memory space of one processing node is not enough to store the parameters of multiple experts running on the processing node, it is not possible to reduce or even eliminate the AllToAll communication amount through this way. SUMMARY

[0005] The application provides a model data processing method and device. When the available memory space of a first processing node is insufficient to store parameters of multiple experts running on the processing node, the communication volume of AllToAll can still be eliminated or reduced by running all experts on the first processing node, and the computing power utilization rate is improved.

[0006] In a first aspect, the application provides a model data processing method applied to a first processing node in a distributed system, wherein the model is deployed in the distributed system, the distributed system includes the first processing node and a second processing node, the first processing node and the second processing node are different in type, the first processing node includes a first memory, the first memory stores parameters of a first expert in the model, the second processing node includes a second memory, the second memory stores parameters of a second expert in the model, and the method includes: obtaining first input data and second input data, the first input data is input data of the first expert, and the second input data is input data of the second expert; calculating the first input data using the parameters of the first expert to obtain a first result; loading the parameters of the second expert through the second memory; and calculating the second input data using the parameters of the second expert to obtain a second result.

[0007] In the model data processing method provided by the application, when the available memory space of a first processing node is insufficient to store multiple experts running on the first processing node, only parameters of part of the experts are stored in the memory of the first processing node, parameters of at least one other expert (second expert) running on the first processing node are stored in the memory of a second processing node, and when the parameters of the experts in the memory of the second processing node are needed for calculation, the parameters of the experts are loaded from the memory of the second processing node to the memory of the first processing node, so that the first processing node can perform calculation using the parameters of the experts. Therefore, when the available memory space of a first processing node is insufficient to store multiple experts running on the first processing node, the communication volume of AllToAll can still be reduced or even eliminated by running multiple experts on the first processing node, and the computing power utilization rate is improved.

[0008] As a possible implementation, in the process of performing the calculation of the first input data using the parameters of the first expert to obtain a first result, the loading of the parameters of the second expert through the second memory is performed synchronously; and / or

[0009] In the process of performing the calculation on the second input data using the parameters of the second expert to obtain a second result, the loading of the parameters of the second expert from the second memory is performed synchronously. In this way, the processes of expert calculation and expert parameter loading are performed synchronously, so that the times of expert calculation and expert parameter loading are overlapped with each other, and the utilization rate of computing power is further improved.

[0010] As a possible implementation manner, the distributed system further includes a third processing node, the third processing node and the first processing node are of the same type, and the parameters of a third expert in the model are stored in the memory of the third processing node. The method further includes: obtaining third input data and fourth input data; the third input data and the fourth input data come from the third processing node; performing calculation on the third input data using the parameters of the first expert to obtain a third result; performing calculation on the fourth input data using the parameters of the second expert to obtain a fourth result; and sending the third result and the fourth result to the third processing node. In this way, a scenario in which the number of experts running on a first processing node is less than the total number of experts in the model is defined. In the scenario, the memory available space of a first processing node is insufficient to store a plurality of experts (less than the total number of experts) running on the first processing node. However, the communication volume of AllToAll can still be reduced and the utilization rate of computing power can be improved by running a plurality of experts on a first processing node.

[0011] As a possible implementation manner, in the process of performing the calculation on the third input data using the parameters of the first expert to obtain a third result, the loading of the parameters of the second expert from the second memory is performed synchronously; and in the process of performing the calculation on the fourth input data using the parameters of the second expert to obtain a fourth result, the loading of the parameters of the second expert from the second memory is performed synchronously. In this way, the processes of expert calculation and expert parameter loading are performed synchronously, so that the times of expert calculation and expert parameter loading are overlapped with each other, and the utilization rate of computing power is further improved.

[0012] As a possible implementation manner, in the process of performing the calculation on the first input data using the parameters of the first expert to obtain a first result or the calculation on the third input data using the parameters of the first expert to obtain a third result, the obtaining of the third input data and the fourth input data or the sending of the third result and the fourth result to the third node is performed synchronously; and / or

[0013] In the process of performing the calculation using the parameters of the second expert on the second input data to obtain a second result or the calculation using the parameters of the second expert on the fourth input data to obtain a fourth result, the obtaining of the third input data and the fourth input data or the sending of the third result and the fourth result to the third node is synchronously performed. In this way, the processes of expert calculation and AllToAll communication are synchronously performed, so as to mutually cover the times of expert calculation and AllToAll communication, and further improve the utilization rate of computing power.

[0014] As a possible implementation manner, in the process of performing the calculation using the parameters of the first expert on the first input data to obtain a first result or the calculation using the parameters of the first expert on the third input data to obtain a third result, the loading of the parameters of the second expert through the second memory and the obtaining of the third input data and the fourth input data are synchronously performed; and / or

[0015] In the process of performing the calculation using the parameters of the first expert on the first input data to obtain a first result or the calculation using the parameters of the first expert on the third input data to obtain a third result, the loading of the parameters of the second expert through the second memory and the sending of the third result and the fourth result to the third node are synchronously performed; and / or

[0016] In the process of performing the calculation using the parameters of the second expert on the second input data to obtain a second result or the calculation using the parameters of the second expert on the fourth input data to obtain a fourth result, the loading of the parameters of the second expert through the second memory and the obtaining of the third input data and the fourth input data are synchronously performed; and / or

[0017] In the process of performing the calculation using the parameters of the second expert on the second input data to obtain a second result or the calculation using the parameters of the second expert on the fourth input data to obtain a fourth result, the loading of the parameters of the second expert through the second memory and the obtaining of the third input data and the fourth input data are synchronously performed. In this way, the processes of expert calculation, expert parameter loading and AllToAll communication are synchronously performed, so as to mutually cover the times of expert calculation, expert parameter loading and AllToAll communication, and further improve the utilization rate of computing power.

[0018] As a possible implementation manner, the distributed system further comprises a third processing node and a fourth processing node, the third processing node and the first processing node are of the same type, the fourth processing node and the first processing node are of different types, the third processing node has stored in the memory the parameters of the third expert in the model, the fourth processing node has stored in the memory the parameters of the fourth expert in the model, and the method further comprises: sending fifth input data and sixth input data to the third processing node; obtaining a fifth result and a sixth result; the fifth result is a result obtained by the third processing node using the parameters of the third expert to calculate the fifth input data; the sixth result is a result obtained by the third processing node using the parameters of the fourth expert to calculate the sixth input data; and the fifth result and the sixth result are from the third processing node. In this way, the number of experts running on a first processing node is further limited to be less than the total number of experts in the model, and in the case that the available memory space of a first processing node is insufficient to store multiple experts (less than the total number of experts) running on the first processing node, the communication volume of AllToAll can still be reduced and the computing power utilization rate can still be improved by running multiple experts on a first processing node.

[0019] As a possible implementation manner, in the process of performing the calculation using the parameters of the first expert on the first input data to obtain the first result or the calculation using the parameters of the first expert on the third input data to obtain the third result, the sending of the fifth input data and the sixth input data to the third processing node or the obtaining of the fifth result and the sixth result is performed synchronously; and / or

[0020] In the process of performing the calculation using the parameters of the second expert on the second input data to obtain the second result or the calculation using the parameters of the second expert on the fourth input data to obtain the fourth result, the sending of the fifth input data and the sixth input data to the third processing node or the obtaining of the fifth result and the sixth result is performed synchronously; and / or

[0021] In the process of performing the calculation using the parameters of the first expert on the first input data to obtain the first result or the calculation using the parameters of the first expert on the third input data to obtain the third result, the loading of the parameters of the second expert through the second memory and the sending of the fifth input data and the sixth input data to the third processing node are performed synchronously; and / or

[0022] In the process of performing the calculation of the first input data using the parameters of the first expert to obtain the first result or the calculation of the third input data using the parameters of the first expert to obtain the third result, the loading of the parameters of the second expert through the second memory and the obtaining of the fifth result and the sixth result are synchronously performed; and / or

[0023] In the process of performing the calculation of the second input data using the parameters of the second expert to obtain the second result or the calculation of the fourth input data using the parameters of the second expert to obtain the fourth result, the loading of the parameters of the second expert through the second memory and the sending of the fifth input data and the sixth input data to the third processing node are synchronously performed; and / or

[0024] In the process of performing the calculation of the second input data using the parameters of the second expert to obtain the second result or the calculation of the fourth input data using the parameters of the second expert to obtain the fourth result, the loading of the parameters of the second expert through the second memory and the obtaining of the fifth result and the sixth result are synchronously performed. In this way, the processes of expert calculation, expert parameter loading and / or AllToAll communication are synchronously performed, so as to overlap the time of expert calculation, expert parameter and / or loading AllToAll communication with each other, and further improve the utilization rate of computing power.

[0025] As a possible implementation manner, the first processing node comprises one or more of NPU, GPU, TPU and CPU, and the second processing node comprises one or more of NPU, GPU, TPU and CPU.

[0026] As a possible implementation manner, the first memory comprises HBM and / or DDR.

[0027] As a possible implementation manner, the second memory comprises DDR and / or SSD.

[0028] As a possible implementation manner, the method is applied to the training or inference of a model.

[0029] As a possible implementation manner, in the case that the available space of the first memory can only store the parameters of one expert, the total number of the first expert and the second expert is determined according to the average value of the estimated total consumption time of one expert and the average value of the twice AllToAll communication time of one expert. In this way, the optimal number of experts that can be run on the first processing node is calculated, so as to further improve the utilization rate of computing power.

[0030] As a possible implementation manner, in a case that the available space of the first memory can store parameters of at least two experts, the total number of the first expert and the second expert is determined according to an average value of an estimated total consumption time of one expert, an average value of the calculation time of one expert and the time of one AllToAll communication, and an average value of the time of two AllToAll communications of one expert. In this manner, the optimal number of experts that can be run on the first processing node is calculated to further improve the utilization of computing power.

[0031] In a second aspect, the present application provides a model data processing apparatus for executing the method in the first aspect or any possible implementation manner of the first aspect. Specifically, the apparatus includes a module for executing the method in the first aspect or any possible implementation manner of the first aspect.

[0032] In a third aspect, the present application provides a distributed system including the apparatus in the second aspect or any possible implementation manner of the second aspect.

[0033] In a fourth aspect, the present application provides a computing device including at least one memory for storing a program, and at least one processor for executing the program stored in the memory, wherein when the program stored in the memory is executed, the processor is configured to execute the method in the first aspect or any possible implementation manner of the first aspect.

[0034] In a fifth aspect, the present application provides a computer storage medium having a computer program stored thereon, when the computer program is executed on a processor, the processor is configured to execute the method in the first aspect or any possible implementation manner of the first aspect.

[0035] In a sixth aspect, the present application provides a computer program product, when the computer program product is executed on a processor, the processor is configured to execute the method in the first aspect or any possible implementation manner of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0036] FIG. 1 is an example diagram of AllToAll communication;

[0037] FIG. 2(a) is an example diagram of expert parallelism;

[0038] FIG. 2(b) is another example diagram of expert parallelism;

[0039] FIG. 3 is an example diagram of system architecture provided by the present application;

[0040] FIG. 4 is an example diagram of system architecture provided by the present application;

[0041] FIG. 5 is a flowchart of a model data processing method provided by the present application;

[0042] Figure 6 is an example diagram of a model data processing method provided in this application;

[0043] Figure 7 is an example of another model data processing method provided in this application;

[0044] Figure 8 is an example diagram of another system structure provided in this application;

[0045] Figure 9 is a flowchart of another model data processing method provided in this application;

[0046] Figure 10 is an example of another model data processing method provided in this application;

[0047] Figure 11 is an example diagram of another model data processing method provided in this application;

[0048] Figure 12 is a flowchart of the deployment method for the model task provided in this application;

[0049] Figure 13 is a functional block diagram of a model data processing device provided in this application;

[0050] Figure 14 is a functional block diagram of another model data processing device provided in this application;

[0051] Figure 15 is a schematic diagram of a computing device according to an embodiment of this application. Detailed Implementation

[0052] In this article, the term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The symbol " / " in this article indicates that the related objects are in an "or" relationship; for example, A / B means A or B.

[0053] The terms "first" and "second," etc., used in the specification and claims herein are used to distinguish different objects, not to describe a specific order of objects. For example, "first response message" and "second response message," etc., are used to distinguish different response messages, not to describe a specific order of response messages.

[0054] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0055] In the description of the embodiments of the present application, unless otherwise specified, the meaning of "a plurality of" is two or more, for example, a plurality of processing units means two or more processing units, and the like; a plurality of elements means two or more elements, and the like.

[0056] Firstly, some technical terms involved in the embodiments of the present application are introduced.

[0057] 1. Distributed training

[0058] Distributed training is to decompose a machine learning or deep learning training task into multiple sub-tasks and perform parallel training on multiple processing nodes. The processing nodes here can be CPU (Central Processing Unit), GPU (Graphics Processing Unit), TPU (Tensor Processing Unit) or NPU (Neural Network Processing Unit), etc.

[0059] 2. AllToAll

[0060] This is a many-to-many communication mode, which can ensure that each processing node can obtain the data of other processing nodes, including two stages of data distribution and data aggregation. In the data distribution stage, each processing node will split the input data into multiple data blocks according to the specified split dimension, and send the data blocks pointing to other processing nodes to other processing nodes in a certain order (such as block index), and keep the data blocks pointing to itself. In the data aggregation stage, each processing node will receive data blocks from other processing nodes.

[0061] As shown in FIG. 1, in a distributed system, taking the processing unit of the processing node as NPU for example, the system includes 4 NPU, each NPU contains 4 data blocks, after AllToAll communication, each NPU sends the data blocks pointing to other NPU to the corresponding NPU, and receives the data blocks sent from other NPU, keeps the data blocks pointing to itself unchanged, and calculates after completion. For example, NPU1 contains A1, B1, C1 and D1, after AllToAll communication, NPU0 contains A1, A2, A3 and A4, wherein A2 comes from NPU2, A3 comes from NPU3, A4 comes from NPU4, and A1 is the data block retained by NPU1 itself. It should be noted that the case shown in FIG. 1 is that each NPU contains data blocks that need to be processed by all NPU, in actual scenarios, there can be a case that one NPU only includes data blocks that need to be processed by part of NPU, and FIG. 1 is only an example and does not limit the present application.

[0062] 3. Communication domain

[0063] Communication domain refers to a logical unit formed by devices or nodes connected by physical links in a communication network, used to realize information exchange and transmission. In the embodiments of the present application, the communication domain refers to the minimum logical unit formed by multiple processing nodes for transmitting data required by expert computing in AllToAll communication.

[0064] 4、Expert

[0065] Expert in the present application refers to a sub-module of a model, which can be a small model itself, such as a feedforward neural network, a recurrent neural network or a deep neural network, etc., or a complex network structure. In the present application, a model includes multiple experts.

[0066] 5、Expert Parallelism

[0067] Expert Parallelism is a new parallelization technology for optimizing deep learning model training and inference, which is usually used in combination with Sparse Activation and Sparse Models. In the expert parallel technology, one or more experts are activated by a gating network or routing to process input data, thereby realizing sparse activation. Each expert is a model, and all experts use the same model, but the parameters of each model are different from each other, for example, experts use a feedforward neural network model, but the parameters of each feedforward neural network are different from each other.

[0068] In the strategy of expert parallelism, different experts run on different processing nodes, and each processing node receives one way of data, and each way of data is different from each other. For a processing node, the corresponding data may include data required by expert computing on other processing nodes, which needs to be sent to the processing node where the corresponding expert is located through AllToAll communication, and after the calculation is completed, the calculation result is sent back to the original processing node through AllToAll communication.

[0069] As shown in FIG. 2(a), as an example, the distributed system includes three processing nodes, the processing units of which are NPU1, NPU2 and NPU3 respectively, Expert 1 runs on NPU1, Expert 2 runs on NPU2, and Expert 3 runs on NPU3. For NPU1, the corresponding data of the lane includes data block 1, data block 2 and data block 3, data block 1 points to Expert 1, data block 2 points to Expert 2, and data block 3 points to Expert 3, data block 2 is sent to NPU2 and data block 3 is sent to NPU3 through AllToAll communication, and data block 4 which needs to be calculated by Expert 1 and data block 7 which needs to be calculated by Expert 1 are received from NPU2 and processing node 3 respectively, after Expert 1 finishes the calculation, NPU1 sends processing result 4 of data block 4 to NPU2 and processing result 7 of data block 7 to NPU3, and receives processing result 2 of data block 2 from NPU2 and processing result 3 of data block 3 from NPU3, and output result is generated according to processing result 1, processing result 2 and processing result 3.

[0070] It should be noted that the data blocks in each lane of data are determined to point to which experts by the gating network or routing (not shown in FIG. 2(a)). Moreover, the data blocks included in each lane of data can point to all experts or part of the experts. In the case that the data blocks included in each lane of data point to part of the experts, there can be a case that does not point to the expert running on the processing node itself receiving the lane of data.

[0071] FIG. 2(b) is another example diagram of expert parallelism, in which the number of processing nodes is 8, the processing unit of each processing node is NPU, and one expert runs in each NPU, and the total time spent includes twice AllToAll communication and expert calculation time. In the embodiments of the present application, the corresponding scenario of the example is the benchmark scenario.

[0072] 6. Synchronization

[0073] Synchronization refers to performing multiple tasks at a certain time, and the start time and end time of each task can be the same or different. In the embodiments of the present application, whether multiple tasks can be synchronously executed depends on the memory available space of the processing node running the expert, if the memory available space of the processing node does not allow synchronous execution of multiple tasks at a certain time, only one task is executed at the certain time, and the process of synchronous execution is started until a certain time later that meets the condition of synchronous execution of multiple tasks.

[0074] Next, the technical solutions provided by the present application are introduced.

[0075] FIG. 3 is an example diagram of the system architecture provided by the present application, and the positional relationship between the architectures and modules shown in the diagram does not constitute any limitation.

[0076] The system structure provided in the present application can be used for training and reasoning of mixed expert models, large language models and the like. For example, the system structure provided in the present application can be applied to Mindspore, Megatron-LM and the like distributed framework.

[0077] When training or reasoning is performed by using the framework such as Mindspore, Megatron-LM and the like, the model execution process can be accelerated based on using the operator library such as CNAA, HCCL, CUDA and NCCL. The system provided in the present application can be applied to chip architectures such as Neural Processing Unit (NPU), Graphics Processing Unit (GPU), Tensor Processing Unit (TPU) or Central Processing Unit (CPU).

[0078] The model data processing method provided in the embodiments of the present application can be used in combination with one or more of the parallel processing systems such as Tensor Parallelism (TP), Sequence Parallelism (SP) and Data Parallelism (DP), and applied to the system architecture provided in the present application.

[0079] FIG. 4 is an example diagram of a system structure provided in the present application. In the system structure, a first processing node and a second processing node are included. Each first processing node receives one way of input data, and each way of input data is different from each other. The first processing node includes a first memory, and the second processing node includes a second memory. The first processing node and the second processing node are different in type, which means that the first memory and the second memory are different in structure. For one first processing node, in the case that the available space of the first memory is not enough to store the parameters of N experts running on the first processing node, the system will store the parameters of m experts of the N experts running on the first processing node in the second memory of the second processing node, and only store the parameters of n experts in the first memory, where n+m=N, n, m and N are positive integers, and N is the total number of experts. In FIG. 4, n=1, m=7, and N=8, which is only an example and is not a limitation of the present application. In the present embodiment, the expert whose parameters are stored in the first memory is a first expert, and the expert whose parameters are stored in the second memory is a second expert. In FIG. 4, e1 represents the parameters of the first expert, and e2-e8 represent the parameters of the seven second experts.

[0080] In the system structure provided in the embodiments of the present application, the number of the first processing nodes and the number of the second processing nodes are both any number. The number of the first processing nodes and the number of the second processing nodes can be the same or different. One second processing node can correspond to one first processing node, or can correspond to a plurality of first processing nodes, and a plurality of second processing nodes can correspond to one first processing node. The number and the corresponding relationship in FIG. 4 are only an example and are not a limitation of the embodiments of the present application.

[0081] The processing unit of each first processing node is a combination of any number of NPUs, GPUs, TPUs and / or CPUs, and each first processing node has independent data processing capability. The types and numbers of NPUs, GPUs, TPUs and / or CPUs included in different first processing nodes can be the same or different. Each NPU, GPU, TPU or CPU constituting a first processing node is connected with an HBM and / or a DDR, and the HBM and the DDR connected by all NPUs, GPUs, TPUs and CPUs constitute the first memory of the first processing node. The processing unit of each second processing node is a combination of any number of NPUs, GPUs, TPUs and / or CPUs, and the types and numbers of NPUs, GPUs, TPUs and CPUs included in different second processing nodes can be the same or different. Each NPU, GPU, TPU or CPU constituting a second processing node includes a DDR and / or an SSD, and the DDR and the SSD connected by all NPUs, GPUs, TPUs and CPUs constitute the second memory of the second processing node. In the embodiments of the present application, the structures of the first memory and the second memory are different, for example, the first memory includes an HBM and the second memory includes a DDR, at this time, the structures of the first memory and the second memory are different, and for another example, the first memory includes an HBM and a DDR and the second memory includes an SSD, at this time, the structures of the first memory and the second memory are also different.

[0082] It should be noted that in the embodiments of the present application, each processing node includes a processing unit and a storage unit, for example, for the first processing node, the processing unit is one or more of an NPU, a GPU, a TPU and a CPU, and the storage unit is the first memory, and the first memory includes an HBM and / or a DDR.

[0083] For the second processing node, the processing unit includes one or more of an NPU, a GPU, a TPU and a CPU, and the storage unit is the second memory, and the second memory includes a DDR and / or an SSD.

[0084] With the system structure provided in the embodiments of the present application, in the case that the available space of the first memory of a first processing node is insufficient to store all the experts running on the first processing node, the parameters of part of the experts are stored in the second memory of the second processing node, so that the first memory and the second memory can be used to store the parameters of all the experts. When the first processing node needs to use the parameters of the experts in the second memory to perform calculation, the parameters of the experts are loaded from the second memory to the first processing node, so that in the case that the available space of the memory of a first processing node is insufficient to store all the parameters of the experts running on the first processing node, the communication amount of AllToAll can still be eliminated by running all the experts on a first processing node, and the utilization rate of computing power is improved.

[0085] In the scenario corresponding to the system structure, each first processing node can run all the experts, so each input data only needs to be calculated with the expert receiving the input data, and after the calculation is completed and the calculation result is output, the next iteration process for model training or the next prediction process is performed, without the need of AllToAll communication.

[0086] Referring to FIG. 5, the embodiments of the present application provide a model data processing method, the model is deployed in the system structure shown in FIG. 4, and the method is applied to a first processing node in the system structure shown in FIG. 4. For a single first processing node, the model data processing method includes the following steps:

[0087] S501, obtaining first input data and second input data;

[0088] The input data is determined to point to which experts by the gating network or routing, that is, which experts are needed to process the input data. The input data usually needs to be processed by multiple experts, the expert whose parameters are stored in the first memory is the first expert, the expert whose parameters are stored in the second memory is the second expert, the input data needed to be processed by the first expert is the first input data, and the input data needed to be processed by the second expert is the second input data. The number of the first input data and the second input data can be one or multiple. The first expert and the second expert are both experts in the model.

[0089] For the input data, all the experts can be activated, or part of the experts can be activated. If only part of the experts are activated, the experts that are not activated do not participate in the subsequent steps S502-S504, that is, the first expert and the second expert performing the subsequent steps S502-S504 are both activated experts.

[0090] S502, calculating the first input data using the parameters of the first expert to obtain a first result;

[0091] In the expert calculation on the first input data, the system calculates the first input data by using the parameters of the first experts to obtain the first results. As an example, the experts use feedforward neural network models, and the parameters of the respective experts are the parameters of the respective feedforward neural network models. In the calculation on the first input data according to the parameters of the first experts, the parameters of the feedforward neural network models are used to calculate the first input data.

[0092] In the execution of the expert calculation, a serial mode or a parallel mode can be used. If the number of the first input data is multiple, the number of the first experts participating in the calculation is also multiple, and the number of the first results generated is also multiple. In the execution of the expert calculation by the first experts, if the serial mode is used, the parameters of each first expert can be used to calculate the corresponding first input data according to the first order; if the parallel mode is used, the parameters of several first experts can be selected for calculation at the same time according to the first order. The first order can be the serial number order of the experts, or the weight, etc., which is not specifically limited here.

[0093] S503, loading the parameters of the second experts from the second memory to the first memory;

[0094] Before the expert calculation using the parameters of the second experts, the parameters of the second experts need to be loaded from the second memory to the first memory. Since the first memory is connected to the first processing node, and the second memory is connected to the second processing node, the loading of the parameters of the second experts from the second memory to the first memory is realized through the communication between the first processing node and the second processing node, specifically, through the communication between the processing units included in the first processing node and the second processing node. As an example, the processing unit of the first processing node is an NPU, the first memory is an HBM, the processing unit of the second processing node is a CPU, and the second memory is a DDR. The parameters of the second experts are first transmitted from the DDR to the CPU, then transmitted from the CPU to the NPU through the communication between the CPU and the NPU, and finally transmitted from the NPU to the HBM.

[0095] If the number of the second experts is multiple, the parameters of each second expert can be loaded according to the second order, or the parameters of several second experts can be selected at the same time according to the second order. The second order can be the serial number order of the experts, or the weight, etc., which is not specifically limited here. The second order can be the same as or different from the first order.

[0096] S504, calculating the second input data by using the parameters of the second experts to obtain the second results.

[0097] In the expert calculation on the second input data, the system uses the parameters of the second experts to calculate the second input data to obtain a second result. If the number of the second input data is multiple, the number of the second experts participating in the calculation is also multiple, and the number of the obtained second results is also multiple. In the expert calculation of the second experts, if a serial mode is used, the parameters of each first expert can be used to calculate the corresponding first input data according to the second order; if a parallel mode is used, the parameters of several second experts can be selected to calculate at the same time according to the second order.

[0098] It should be noted that in the embodiments of the present application, steps S502 and S504 define the process of expert calculation, and step S503 defines the process of loading expert parameters.

[0099] After the expert calculation is performed, the obtained first result and second result are output, thereby completing an iteration process of training or a prediction process. The first result and the second result can be direct processing results of the first input data and the second input data respectively, or can be other data generated in the calculation process, for example, in the training scenario, can be an iteration result of the expert parameters, the iteration result can be an activation value, which is not limited here.

[0100] In the embodiments of the present application, in the case that the available space of the memory (the first memory) of the first processing node is insufficient to store all the experts running on the first processing node, only the parameters of part of the experts (the first experts) are stored in the memory of the first processing node, and the parameters of other at least one expert (the second experts) running on the first processing node are stored in the memory (the second memory) of the second processing node. When the parameters of the experts in the memory of the second processing node are needed for calculation, the parameters of these experts are loaded from the memory of the second processing node to the memory of the first processing node, thereby when the available space of the memory of a first processing node is insufficient to store the parameters of all the experts running on the processing node, the communication amount of AllToAll can still be eliminated by running all the experts on a first processing node, and the computing power utilization rate is improved.

[0101] As an example, referring to FIG. 6, the processing unit of the first processing node is an NPU, the processing unit of the second processing node is a CPU, the number of the NPU and the CPU is 8, one NPU is connected with one CPU, and each first processing node receives one input data. The total number of experts is 8, each NPU runs 8 experts, and the capacity of the first memory (not shown in FIG. 6) of the first processing node is the parameters of one expert. For one first processing node, the parameters e1 of the expert E1 are stored in the first memory thereof, and the parameters e2-e8 of the experts E2-E8 are stored in the second memory (not shown in FIG. 6) of the corresponding second processing node. Among them, the expert E1 is the first expert, the experts E2-E8 are the second experts, and e1-e8 are the parameters of E1-E8, respectively.

[0102] For one first processing node, the model data processing method provided by the embodiment of the present application includes the following steps:

[0103] (1) obtaining the first input data pointing to the expert E1 and the second input data pointing to the experts E2-E8, wherein the second input data is 7 and points to one of the experts E2-E8.

[0104] (2) using the parameters e1 of the expert E1 to calculate the first input data to obtain a first result;

[0105] (3) loading the parameters e2 of the expert E2 from the second memory through the communication between the CPU and the NPU;

[0106] (4) using the parameters e2 of the expert E2 to calculate the second input data pointing to the expert E2 to obtain a second result;

[0107] (5) after the calculation of the expert E2, sequentially performing steps (4)-(5) on the experts E3-E8 in turn until all the second input data are processed.

[0108] In this example, in the case that the available memory space of the first processing node can only store the parameters of one expert, the parameters e1 of the expert E1 are stored in the first memory of the first processing node, and the parameters e2-e8 of the experts E2-E8 are stored in the second memory of the second processing node. When the parameters e2-e8 of the experts E2-E8 are needed for calculation, the parameters e2-e8 of the experts E2-E8 are loaded from the second memory to the first memory respectively, so that the NPU can use the parameters e2-e8 of the experts E2-E8 for calculation. Thus, in the case that the available memory space of the first processing node can only store the parameters of one expert, all the experts E1-E8 can still be run in one first processing node to eliminate the communication amount of AllToAll and improve the utilization rate of computing power.

[0109] In the corresponding scenario of this example, the time spent includes the time of expert parameter loading and expert calculation. Compared with the benchmark scenario, in the case where the time of expert parameter loading is less than the time of twice AllToAll communication in the two benchmark scenarios, the scenario has time benefits.

[0110] Optionally, in one implementation, in the process of executing steps S502 and / or S504, step S503 can be executed synchronously within the range of the available space of the memory of the first processing node, that is, the expert parameter loading process is executed synchronously when the expert calculation is executed, so as to overlap the time of expert calculation and expert parameter loading.

[0111] In this implementation, by synchronously executing the expert calculation and the expert parameter loading, the time of the expert calculation and the expert parameter loading is overlapped, and the utilization of computing power is further improved.

[0112] As an example, referring to FIG. 7, the processing unit of the first processing node is an NPU, the processing unit of the second processing node is a CPU, the number of the NPU and the CPU is 8, one NPU is connected with one CPU, and each first processing node receives one input data. The total number of experts is 8, each NPU runs 8 experts, and the available space of the first memory (not shown in FIG. 7) of the first processing node is the parameter of one expert. For one first processing node, the parameter e1 of the expert E1 is stored in the first memory, and the parameters e2-e8 of the experts E2-E8 are stored in the second memory (not shown in FIG. 7) of the corresponding second processing node. Among them, the expert E1 is the first expert, the experts E2-E8 are the second experts, and e1-e8 are the parameters of E1-E8.

[0113] For a single first processing node, the model data processing method provided by the embodiment of the application includes the following steps:

[0114] (1) obtaining first input data pointing to the expert E1 and second input data pointing to the experts E2-E8, wherein the second input data is 7 and points to one of the experts E2-E8;

[0115] (3) calculating the first input data according to the parameter e1 of the expert E1 to generate a first result;

[0116] (4) synchronously executing the loading of the parameter e2 of the expert E2 from the second memory through the communication between the CPU and the NPU while the expert E1 is executing the calculation;

[0117] Since the parameter e1 of the expert E1 has been sent from the first memory to the NPU when the expert E1 performs the calculation, the space of the first memory can store the parameter e2 of the next expert E2 at this time, so the loading process of the parameter e2 of the expert E2 can be performed synchronously when the expert calculation of the expert E1 is performed.

[0118] (5) After the expert E1 finishes the calculation and the parameter e2 of the expert E2 has been loaded, the calculation according to the parameter e2 of the expert E2 on the second input data directed to the expert E2 is performed to generate a second result, and the parameter e3 of the expert E3 is loaded from the second memory through the communication between the CPU and the NPU is performed synchronously;

[0119] (6) After the expert E2 finishes the calculation, the steps (4)-(5) are performed on the experts E3-E8 in turn until all the second input data are processed.

[0120] In the scenario corresponding to the example, since the available space of the first memory is the parameter of one expert, the parameter loading process of the next expert is performed synchronously when the calculation of the previous expert is performed, so that the time of the expert calculation covers the time of the expert parameter loading, and the total time spent only includes the time of the expert calculation, further improving the utilization rate of the computing power. Compared with the benchmark scenario, the time yield of the scenario corresponding to the example is the time of the AllToAll communication.

[0121] FIG. 8 is another system structure provided by the present application. In the system structure, a first processing node, a second processing node, a third processing node and a fourth processing node are included, each first processing node and third processing node receives one way of input data, each way of input data is different from each other. The first processing node includes a first memory, the second processing node includes a second memory, the third processing node includes a third memory, and the fourth processing node includes a fourth memory. For a first processing node, when the available space of the first memory is insufficient to store the parameters of M experts running on the first processing node, the system stores the parameters of m experts of the M experts running on the first processing node in the second memory of the second processing node, and only stores the parameters of n experts in the first memory, wherein n+m=M, M<N, n, m, M and N are positive integers, and N is the total number of experts in the system. It should be noted that n=1, m=3, M=4, and N=8, which is only an example and is not a limitation of the present application. In this embodiment, the expert whose parameter is stored in the first memory is the first expert, and the expert whose parameter is stored in the second memory is the second expert. In FIG. 8, e1 represents the parameter of the first expert, and e2-e4 represent the parameters of three second experts.

[0122] In the system structure provided in the application, the number of the first processing nodes, the second processing nodes, the third processing nodes and the fourth processing nodes is any multiple. The number of the first processing nodes, the second processing nodes, the third processing nodes and the fourth processing nodes can be the same or different. One second processing node can correspond to one first processing node or multiple first processing nodes, and multiple second processing nodes can correspond to one first processing node; one fourth processing node can correspond to one third processing node or multiple third processing nodes, and multiple fourth processing nodes can correspond to one third processing node. The number and corresponding relationship of the first processing nodes and the second processing nodes and the number and corresponding relationship of the third processing nodes and the fourth processing nodes in FIG. 8 are only an example and are not a limitation of the embodiments of the application.

[0123] The processing unit of each first processing node is a combination of any number of NPUs, GPUs, TPUs and / or CPUs, and each first processing node has independent data processing capability. The types and number of NPUs, GPUs, TPUs and CPUs contained in different first processing nodes can be the same or different. Each NPU, GPU, TPU or CPU constituting a first processing node is connected with an HBM or a DDR, and the HBM and the DDR constitute the first memory of the first processing node. The processing unit of each second processing node is a combination of any number of NPUs, GPUs, TPUs and / or CPUs, and the types and number of NPUs, GPUs, TPUs and CPUs contained in different second processing nodes can be the same or different. Each NPU, GPU, TPU or CPU constituting a second processing node is connected with a DDR or an SSD, and the DDR and the SSD constitute the second memory of the second processing node.

[0124] The processing unit of each third processing node is a combination of any number of NPUs, GPUs, TPUs and / or CPUs, each third processing node having independent data processing capability. The types and number of NPUs, GPUs, TPUs and CPUs contained in different third processing nodes can be the same or different. Each NPU, GPU, TPU or CPU constituting a third processing node is connected with an HBM or a DDR, which constitutes the third memory of the third processing node. The processing unit of each fourth processing node is a combination of any number of NPUs, GPUs, TPUs and / or CPUs, the types and number of NPUs, GPUs, TPUs and CPUs contained in different fourth processing nodes can be the same or different. Each NPU, GPU, TPU or CPU constituting a fourth processing node is connected with a DDR or an SSD, which constitutes the fourth memory of the fourth processing node. The third processing node and the first processing node are of the same type, and the fourth processing node and the second processing node are of the same type, the same type referring to the same memory structure. In this embodiment, the expert whose parameters are stored in the third memory is a third expert, and the expert whose parameters are stored in the fourth memory is a fourth expert. In FIG. 8, e5 represents the parameters of the third expert, and e6-e8 represent the parameters of three fourth experts.

[0125] It should be noted that in the embodiments of the present application, each processing node includes a processing unit and a storage unit. For example, for the first processing node, the processing unit is one or more of an NPU, a GPU, a TPU and a CPU, and the storage unit is the first memory, which includes an HBM and / or a DDR. For the second processing node, the processing unit includes one or more of an NPU, a GPU, a TPU and a CPU, and the storage unit is the second memory, which includes a DDR and / or an SSD. For the third processing node, the processing unit is one or more of an NPU, a GPU, a TPU and a CPU, and the storage unit is the third memory, which includes an HBM and / or a DDR. For the fourth processing node, the processing unit includes one or more of an NPU, a GPU, a TPU and a CPU, and the storage unit is the fourth memory, which includes a DDR and / or an SSD.

[0126] With the system structure provided in the embodiments of the present application, in the case that the available space of the first memory of a first processing node is insufficient to store a plurality of experts (less than the total number of experts) running on the first processing node, the parameters of part of the experts can be stored in the second memory of a second processing node, so that the first memory and the second memory can store the parameters of the plurality of experts running on the first processing node. When the first processing node needs to perform calculation by using the parameters of the experts in the second memory, the parameters of the experts are loaded from the second memory to the first memory, so that when the available space of the memory of a first processing node is insufficient to store the parameters of the plurality of experts running on the processing node, the communication amount of AllToAll can be reduced and the utilization rate of computing power can be improved by running a plurality of experts on a first processing node. The total number of experts running on a first processing node is less than the total number of experts in the system.

[0127] In the scenario corresponding to the system structure, since the total number of experts running in a first processing node is less than the total number of experts in the system, in the same communication domain, AllToAll communication is needed to deliver the data required for calculation of the first processing node, and AllToAll communication is needed to send the processing result to the original first processing node. The same communication domain includes a first processing node and a plurality of third processing nodes, the first processing node and the third processing nodes are of the same type, the same type means that the structures of the first memory and the third memory are the same, and the total number of experts in the same communication domain is N, and the parameters of each expert in the same communication domain are different.

[0128] In the scenario corresponding to the system structure, each first processing node can run a plurality of experts, and AllToAll communication is not needed between the experts on the processing node, the size of the communication domain is less than the total number of experts, the communication amount of AllToAll can be reduced, and the utilization rate of computing power can be improved.

[0129] Referring to FIG. 9, the embodiments of the present application provide a model data processing method, the model is deployed in the system structure shown in FIG. 8, the method is applied to a first processing node in the system structure shown in FIG. 8, and for a single first processing node, the model data processing method includes the following steps:

[0130] S901, obtaining first input data and second input data;

[0131] The data input is determined by the gating network or routing to point to which experts, the expert whose parameters are stored in the first memory is a first expert, the expert whose parameters are stored in the second memory is a second expert, the input data that needs to be processed by the first expert is first input data, and the input data that needs to be processed by the second expert is second input data.

[0132] For the input data, all experts can be activated or only part of the experts can be activated. If only part of the experts are activated, the experts that are not activated do not participate in the subsequent steps S902-S908, that is, the first expert and the second expert that execute the subsequent steps S902-S908 are both activated experts.

[0133] S902, obtain third input data and fourth input data;

[0134] The third processing data and the fourth processing data are sent by the third processing node, wherein the third processing data is the input data that needs to be processed by the first expert, and the fourth input data is the input data that needs to be processed by the second expert, and the number of the third input data and the fourth input data can be one or multiple.

[0135] S903, send fifth input data and sixth input data to the third processing node;

[0136] The fifth input data is the input data that needs to be processed by the third expert, and the sixth input data is the input data that needs to be processed by the fourth expert, the third expert is an expert whose parameters are stored in the third memory, and the fourth expert is an expert whose parameters are stored in the fourth memory, and the logic of the third processing node processing the fifth processing data and the sixth processing data is the same as that of the first processing node, which will not be described in detail here.

[0137] The number of the fifth input data and the sixth input data can be one or multiple, and the third processing node to which the fifth input data and the sixth input data are sent is determined by the expert pointed by the third input data, that is, the third processing node corresponding to the expert pointed by the fifth input data and the sixth input data in the same communication domain. The fifth input data or the sixth input data sent to one third processing node can point to one expert running in the third processing node, or can point to multiple experts running in the third processing node. When pointing to multiple experts running in the third processing node, the fifth input data or the sixth input data includes multiple subsets, and each subset corresponds to one expert.

[0138] It should be noted that in the case of only considering the calculation performed by the expert running in the first processing node, step S903 can not be executed, and in the case of not executing S903, the calculation for the third input data and the fourth input data in steps S904 and S906 is also not required.

[0139] In the embodiments of the present application, the first expert, the second expert, the third expert and the fourth expert are all experts of the model.

[0140] S904, calculate the first input data and the third input data using the parameters of the first expert to obtain the first result and the third result;

[0141] The first result corresponds to the first input data, and the third result corresponds to the third input data. In the calculation of the first expert, the first input data and the third input data can be calculated respectively, or the corresponding first input data and third input data can be calculated simultaneously using the parameters of the same first expert, which is not limited here.

[0142] In the execution of expert calculation, a serial mode or a parallel mode can be used. If the total number of the first input data and the third input data is multiple, the number of the first experts participating in the calculation is also multiple. In the execution of expert calculation by the first expert, if the serial mode is used, the corresponding first input data and third input data can be calculated using the parameters of each first expert according to the first order; if the parallel mode is used, the parameters of several first experts can be selected simultaneously for calculation according to the first order. The first order can be the order of the experts, or the weight, etc., which is not limited here.

[0143] S905, loading the parameters of the second expert from the second memory;

[0144] Before the expert calculation using the parameters of the second expert, the parameters of the second expert need to be loaded from the second memory to the first memory. Since the first memory is connected to the first processing node, and the second memory is connected to the second processing node, the loading of the parameters of the second expert from the second memory to the first memory is realized through the communication between the first processing node and the second processing node, specifically, through the communication between the processing units included in the first processing node and the second processing node. As an example, the processing unit of the first processing node is NPU, the first memory is HBM, the processing unit of the second processing node is CPU, and the second memory is DDR. The parameters of the second expert are first transmitted from the DDR to the CPU, then transmitted from the CPU to the NPU through the communication between the CPU and the NPU, and finally transmitted from the NPU to the HBM.

[0145] If the number of the second experts is multiple, the parameters of each second expert can be loaded according to the second order, or the parameters of several second experts can be selected simultaneously according to the second order. The second order can be the order of the experts, or the weight, etc., which is not limited here. The second order can be the same as or different from the first order.

[0146] S906, calculating the second input data and the fourth input data using the parameters of the second expert to obtain the second result and the fourth result;

[0147] The second result corresponds to the second input data, and the fourth result corresponds to the fourth input data. In the calculation of the second expert, the second input data and the fourth input data can be calculated respectively, or the corresponding second input data and the fourth input data can be calculated simultaneously by using the parameters of the same second expert, which is not limited here.

[0148] If the total number of the second input data and the fourth input data is multiple, the number of the second experts participating in the calculation is also multiple. In the calculation of the second expert, if a serial mode is used, the parameters of each second expert can be used to calculate the corresponding second input data and the fourth input data according to the second order; if a parallel mode is used, the parameters of several second experts can be selected simultaneously for calculation according to the second order.

[0149] It should be noted that in the embodiments of the present application, steps S904 and S906 define the process of expert calculation, and step S905 defines the process of expert parameter loading. After the steps S905-S906 are performed on the second experts, the steps S905-S906 can be performed again on the remaining second experts.

[0150] S907, sending the third result and the fourth result to the third processing node;

[0151] Each third result and fourth result is sent to the corresponding third processing node. Specifically, for each third result or fourth result, the corresponding third input data or fourth input data comes from which third processing node, and the corresponding third result or fourth result is sent to which third processing node.

[0152] It should be noted that after the steps S904 or S906 are performed on the first experts and / or the second experts, the third processing result and / or the fourth processing result can be sent by performing S907, and after the sending is completed, the steps S902, S904-S906 can be performed again on the remaining second experts.

[0153] S908, obtaining a fifth result and a sixth result.

[0154] After the expert calculation of the third processing node on the fifth input data and the sixth input data is completed, the fifth result and the sixth result are obtained respectively, and the first processing node obtains the corresponding fifth result and the sixth result.

[0155] In the case of only considering the calculation performed by the expert running on one first processing node, the step S909 can not be performed.

[0156] After the expert calculation of the first processing node is completed, the first result, the second result, the fifth result and the sixth result are output, so that one iteration process of training or a prediction process is completed.

[0157] It should be noted that the first result, the second result, the third result, the fourth result, the fifth result and the sixth result can be direct processing results of the first input data, the second input data, the third input data, the fourth input data, the fifth input data and the sixth input data respectively, or can be other data generated in the calculation process, for example, in the training scene, can be an iteration result of the expert parameters, and the iteration result can be an activation value, which is not limited herein.

[0158] In the embodiment of the present application, in the case that the available space of the memory (the first memory) of the first processing node is insufficient to store a plurality of experts (less than the total number of experts) running on the first processing node, only the parameters of part of the experts (the first experts) are stored in the memory of the first processing node, and the parameters of at least one other expert (the second expert) running on the first processing node are stored in the memory (the second memory) of the second processing node, and when the parameters of the experts in the memory of the second processing node are needed for calculation, the parameters of the experts are loaded from the memory of the second processing node to the memory of the first processing node, so that the first processing node performs calculation by using the parameters of the experts, thereby in the case that the available space of the memory of a first processing node is insufficient to store a plurality of experts (less than the total number of experts) running on the first processing node, the communication amount of AllToAll can be reduced by running a plurality of experts on a first processing node, and the utilization rate of computing power can be improved.

[0159] As an example, referring to FIG. 10, the processing units of the first processing node and the third processing node are NPUs, the processing units of the second processing node and the fourth processing node are CPUs, the number of the NPUs and the CPUs is 8, one NPU is connected with one CPU, and each first processing node receives one input data. The total number of experts is 8, each NPU runs 4 experts, two NPUs running different experts constitute a communication domain, and the NPUs in the same communication domain are represented by the same gray area. The available space of the first memory (not shown in FIG. 10) of the first processing node is the parameters of one expert, for two NPUs in the same communication domain, experts E1-E4 run in one NPU, and experts E5-E8 run in the other NPU, wherein the NPU running experts E1-E4 is the processing unit of the first processing node, the NPU running experts E5-E8 is the processing unit of the third processing node, the CPU connected with the first processing node is the processing unit of the second processing node, and the CPU connected with the third processing node is the processing unit of the fourth processing node. The parameters e1 of expert E1 are stored in the first memory of the first processing node, the parameters e2-e4 of experts E2-E4 are stored in the second memory (not shown in FIG. 10) of the second processing node, the parameters e5 of expert E5 are stored in the third memory (not shown in FIG. 10) of the third processing node, and the parameters e6-e8 of experts E6-E8 are stored in the fourth memory (not shown in FIG. 10) of the fourth processing node. Among them, expert E1 is the first expert, experts E2-E4 are the second experts, expert E5 is the third expert, and experts E6-E8 are the fourth experts.

[0160] For a single first processing node, the model data processing method provided by the embodiment of the present application includes the following steps:

[0161] (1) obtaining first input data pointing to expert E1 and second input data pointing to experts E2-E4; wherein the second input data is 3, and points to one of experts E2-E4.

[0162] (2) obtaining third input data pointing to expert E1 and fourth input data pointing to expert E2 sent by the CPU in the same communication domain;

[0163] (3) sending fifth input data pointing to expert E5 and sixth input data pointing to experts E6-E8 to the CPU in the same communication domain;

[0164] (4) calculating the first input data and the third input data according to the parameters e1 of expert E1, to obtain a first result and a third result, respectively;

[0165] (5) loading the parameters e2 of expert E2 from the second memory through communication between the CPU and the NPU;

[0166] (6) According to the parameter e2 of the expert E2, the second input data and the fourth input data directed to the expert E2 are calculated to generate a second result and a fourth result;

[0167] (7) After the calculation of the expert E2 is completed, steps (5)-(7) are sequentially executed for the experts E3-E4 respectively until all the second input data and the fourth input data are processed;

[0168] (8) The third result and the fourth result are sent to the CPU in the same communication domain;

[0169] (9) The fifth result and the sixth result sent by the CPU in the same communication domain are obtained.

[0170] In this example, in the case that the memory available space of the first processing node can only store the parameters of one expert, the parameters e1 of the expert E1 are stored in the first memory of the first processing node NPU, and the parameters e2-e4 of the experts E2-E4 are stored in the second memory connected to the CPU of the second processing node. When the parameters e2-e4 of the experts E2-E4 need to be used for calculation, the parameters e2-e4 of the experts E2-E4 are loaded from the second memory to the first memory respectively, so that the NPU can use the parameters e2-e4 of the experts E2-E4 for calculation. Therefore, in the case that the memory available space of the NPU can only store the parameters of one expert, the AllToAll communication amount can be reduced and the computing power utilization rate can be improved by running multiple experts E1-E4 in one NPU.

[0171] In this example, the AllToAll communication is reduced by running multiple experts in each NPU. In the corresponding scenario of this example, the total time spent includes the time of expert parameter loading, expert calculation and twice AllToAll communication. Compared with the benchmark scenario, when the total time of expert parameter loading and twice reduced AllToAll communication in the communication domain is less than the time of twice AllToAll communication in the benchmark scenario, there is a time benefit in this scenario.

[0172] Optionally, as an implementation manner, in the process of executing steps S904 and / or S906, step S905 can be executed synchronously within the range allowed by the memory available space of the first processing node, that is, the expert calculation and the expert parameter loading process are executed synchronously, so that the time of expert calculation and expert parameter loading is overlapped with each other, and the computing power utilization rate is further improved.

[0173] Optionally, as an implementation manner, in the range allowed by the available memory space of the first processing node, steps S902, S903, S907 and / or S908 can be executed synchronously in the process of executing steps S904 and / or S906. That is, the expert calculation and the AllToAll communication are executed synchronously, so as to overlap the time of the expert calculation and the AllToAll communication with each other, and further improve the utilization of computing power.

[0174] Optionally, as an implementation manner, in the range allowed by the available memory space of the first processing node, steps S902, S903, S905, S907 and / or S908 can be executed synchronously in the process of executing steps S904 and / or S906. That is, the expert calculation, the expert parameter loading and the AllToAll communication are executed synchronously, so as to overlap the time of the expert calculation, the expert parameter loading and the AllToAll communication with each other, and further improve the utilization of computing power.

[0175] As an example, referring to FIG. 11, the processing units of the first processing node and the third processing node are NPUs, the processing units of the second processing node and the fourth processing node are CPUs, the number of NPUs and CPUs is 8, one NPU is connected with one CPU, and each first processing node receives one input data. The total number of experts is 8, each NPU runs 4 experts, two NPUs running different experts constitute a communication domain, and the NPUs in the same communication domain are represented by the same gray area. The available space of the first memory (not shown in FIG. 10) of the first processing node is the parameters of one expert. For two NPUs in the same communication domain, one NPU runs experts E1-E4, and the other NPU runs experts E5-E8, wherein the NPU running experts E1-E4 is the processing unit of the first processing node, the NPU running experts E5-E8 is the processing unit of the third processing node, the CPU connected with the first processing node is the processing unit of the second processing node, and the CPU connected with the third processing node is the processing unit of the fourth processing node. The parameters e1 of expert E1 are stored in the first memory of the first processing node, the parameters e2-e4 of experts E2-E4 are stored in the second memory (not shown in FIG. 11) of the second processing node, the parameters e5 of expert E5 are stored in the third memory (not shown in FIG. 11) of the third processing node, and the parameters e6-e8 of experts E6-E8 are stored in the fourth memory (not shown in FIG. 10) of the fourth processing node. Among them, expert E1 is the first expert, experts E2-E4 are the second experts, expert E5 is the third expert, and experts E6-E8 are the fourth experts.

[0176] For a single first processing node, the model data processing method provided by the embodiment of the application includes the following steps:

[0177] (1) obtaining first input data pointing to the expert E1 and second input data pointing to the experts E2-E4; wherein the second input data is 3, and respectively points to one of the experts E2-E4.

[0178] (2) receiving third input data A1 pointing to the expert E1 sent by the CPU in the same communication domain;

[0179] (3) performing calculation on the first input data and the third input data A1 according to the parameter e1 of the expert E1, and respectively obtaining a first result and a fourth result a1;

[0180] (4) synchronously performing the process of loading the parameter e2 of the expert E2 from the second memory and receiving the fourth input data A2-A4 pointing to the experts E2-E4 sent by the CPU in the same communication domain while the expert E1 performs calculation;

[0181] Since the parameter e1 of the expert E1 has been sent from the first memory to the NPU while the expert E1 performs calculation, at this time, the available space of the first memory can accommodate the parameter e2 of the next expert E2, therefore, the loading process of the parameter e2 of the expert E2 can be synchronously performed while the expert calculation of the expert E1 is performed.

[0182] (5) when the expert E1 finishes calculation, the parameter e2 of the expert E2 is loaded, and the fourth input data A2 pointing to the expert E2 is received, performing calculation on the second input data and the fourth input data A2 according to the parameter e2 of the expert E2, and respectively obtaining a second result and a fourth result a2;

[0183] (6) synchronously performing the process of loading the parameter e3 of the expert E3 from the second memory and receiving the fourth input data sent by the CPU while the expert E2 performs calculation; and after the fifth input data A2-A4 pointing to the experts E2-E4 is received, synchronously sending the third result a1 and the fourth results a2-a4;

[0184] For example, when the expert E2 performs calculation, the fourth input data A4 has been sent, then the fourth result a1 starts to be sent.

[0185] (7) performing steps (6)-(7) on the experts E3-E4 in turn until the experts E1-E4 finish calculation.

[0186] (8) sending the fourth result a4 to the corresponding CPU.

[0187] In the scenario corresponding to the example, since the capacity of the first memory is 1 parameter of the expert, when the calculation of the previous expert is performed, the loading of the parameter of the next expert and the AllToAll communication process of the data needed to be processed by other experts or the calculation results of other experts can be performed synchronously, so as to overlap the time of expert calculation, expert parameter loading and AllToAll communication with each other, and further improve the utilization rate of computing power. Compared with the benchmark scenario, the time benefit of the scenario corresponding to the example is the AllToAll communication time of the benchmark scenario minus the AllToAll communication time of one expert in the reduced communication domain.

[0188] As an implementation manner, in order to further improve the utilization rate of computing power, in addition to synchronously performing multiple tasks, the number of experts running on one first processing node can also be optimized. Taking FIG. 11 as an example, when the available space of the first memory can only store one parameter of the expert, in the case where the synchronous execution of multiple tasks has been adopted, the total time consumed includes twice the AllToAll communication time of one expert and the time of synchronously executing tasks, therefore, before the model is deployed, the optimal number of experts running on each first processing node is calculated, the larger value is selected from the average value of the total consumption time of one expert and the average value of twice the AllToAll communication time of one expert, and the number of experts running on each first processing node that makes the time benefit of the larger value maximum is solved, so that the number of experts running on the first processing node that makes the time benefit maximum is the optimal number. Wherein, the average value of the total consumption time of one expert is the ratio of the total consumption time of the system to the total number of experts, and the number of experts running on one first processing node is the total number of the first expert and the second expert.

[0189] As an implementation, when the available space of the first memory of the first processing node can store parameters of more than two experts, since at least two parameters of experts can be stored in each first memory, the loading time of the experts is significantly reduced compared to the case where the available space of the first memory can only store parameters of one expert. Therefore, in order to further improve the utilization of computing power, in the case where the synchronous execution of multiple tasks has been adopted, when calculating the optimal number of experts running on each first processing node, the maximum value is selected from the average value of the total consumption time of one expert, the average value of the calculation time of one expert and the time of one AllToAll communication, and the average value of the time of two AllToAll communications of one expert, and the number of experts running on each first processing node that maximizes the time benefit of the maximum value is solved, so that the number of experts running on the first processing node that maximizes the time benefit is the optimal number. Wherein, the average value of the total consumption time of one expert is the ratio of the total consumption time of the system to the total number of experts, the calculation time of one expert is the calculation time of one expert on the input data to be processed, and the number of experts running on one first processing node is the total number of first experts and second experts.

[0190] Referring to FIG. 12, the embodiment of the present application provides a model task deployment method, comprising:

[0191] S1201, determining a first deployment parameter according to the available space of the first memory;

[0192] The first memory is a storage unit of the first processing node, including HBM and / or DDR, and the available space of the first memory is used to store parameters of experts in the model. The first processing node is a processing node for running experts in a distributed system. The distributed system includes first processing nodes and second processing nodes, one second processing node is connected with one first processing node, and the storage unit of the second processing node is a second memory. In the case where the available space of the first memory is insufficient to store the parameters of the experts running on the corresponding first processing node, the second memory needs to be used to store part of the parameters of the experts.

[0193] The first deployment parameter is the larger of the average of the estimated total consumption time of one expert and the average of the two AllToAll communication times of one expert, or the maximum of the average of the estimated total consumption time of one expert, the average of the computation time of one expert and the AllToAll communication time, and the average of the two AllToAll communication times of one expert. Since different models contain different numbers of experts and different parameters, the expert computation time, the AllToAll communication time and the expert parameter loading time are different. Therefore, for a model, the expert computation time, the AllToAll communication time and the expert parameter loading time of the model need to be estimated before the deployment mode of the model is determined, and the average of the total consumption time of one expert, the average of the two AllToAll communication times of one expert, the average of the computation time of one expert and the AllToAll communication time, and the two AllToAll communication times of one expert are determined according to the estimated values, and the first deployment parameter is determined according to the determined average values.

[0194] In the case that the available space of the first memory can only store the parameters of one expert, the first deployment parameter is the larger of the average of the estimated total consumption time of one expert and the average of the two AllToAll communication times of one expert. In the case that the available space of the first memory can store the parameters of at least two experts, the first deployment parameter is the maximum of the average of the estimated total consumption time of one expert, the average of the computation time of one expert and the AllToAll communication time, and the average of the two AllToAll communication times of one expert.

[0195] S1202, determining a second deployment parameter according to the first deployment parameter;

[0196] The second deployment parameter is the number of experts running on the first processing node that maximizes the time benefit of the first deployment parameter. Those skilled in the art should understand that the time benefit is the least consumption time.

[0197] S1203, determining a deployment mode according to the available space of the first memory and the second deployment parameter;

[0198] In the case that the value of the second deployment parameter is greater than the number of expert parameters that can be stored in the available space of the first memory, the available storage space of the first memory is insufficient to run the parameters of the experts deployed on the first processing node, and the second memory of the second processing node needs to be used to store part of the parameters of the experts running on the first processing node.

[0199] In a case where the value of the second deployment parameter is not greater than the number of the parameters of the experts that can be stored in the available space of the first memory, the available storage space of the first memory can store all the parameters of the experts running on the first processing node without the help of the second memory of the second processing node to store the parameters of the experts running on the first processing node.

[0200] The deployment manner refers to deploying the model task according to a third deployment parameter and a fourth deployment parameter, the third deployment parameter is the number of the parameters of the experts stored in the first memory, and the fourth deployment parameter is the number of the parameters of the experts stored in the second memory. In a case where the value of the second deployment parameter is greater than the number of the parameters of the experts that can be stored in the available space of the first memory, the third deployment parameter is the number of the parameters of the experts that can be stored in the available space of the first memory, and the fourth deployment parameter is the difference between the second deployment parameter and the third deployment parameter. In a case where the value of the second deployment parameter is not greater than the number of the parameters of the experts that can be stored in the available space of the first memory, the third deployment parameter is equal to the second deployment parameter, and the fourth deployment parameter is 0. It should be noted that in the embodiment, it is assumed that the number of the experts running on each first processing node is the same, and therefore, the determined deployment manner only includes the third deployment parameter and the fourth deployment parameter, that is, the number of the parameters of the experts stored in the first memory and the number of the parameters of the experts stored in the second memory.

[0201] S1204, deploying the model task according to the deployment manner.

[0202] In the embodiment of the present application, the distributed system includes a first processing node and a second processing node, each processing node includes a processing unit and a storage unit, for example, for the first processing node, the processing unit is one or more of NPU, GPU, TPU and CPU, and the storage unit is the first memory, and the first memory includes HBM and / or DDR. For the second processing node, the processing unit includes one or more of NPU, GPU, TPU and CPU, and the storage unit is the second memory, and the second memory includes DDR and / or SSD.

[0203] After the deployment manner is determined, the model task is deployed in the distributed system according to the deployment manner.

[0204] In the embodiment, the model task is deployed with the goal of maximizing the computing power utilization rate, which effectively utilizes the system resources and improves the processing efficiency of the model task.

[0205] Referring to FIG. 13, an embodiment of the present application provides a model data processing apparatus 300 applied to one first processing node in the system structure shown in FIG. 4, for a single first processing node, the apparatus includes:

[0206] The first obtaining module 1301 is configured to obtain first input data and second input data.

[0207] The first calculation module 1302 is configured to calculate the first input data using parameters of a first expert to obtain a first result.

[0208] The loading module 1303 is configured to load the parameters of the second expert through the second memory.

[0209] The second calculation module 1304 is configured to calculate the second input data using the parameters of the second expert to obtain a second result.

[0210] The implementation process of each module of the embodiment of the application is described in the related content of each step of the method shown in FIG. 5, and will not be described in detail here.

[0211] In the embodiment, in the case that the memory (the first memory) of a first processing node is limited in capacity and is not enough to store all the experts running on the first processing node, the parameters of only part of the experts (the first expert) are stored in the memory of the first processing node, the parameters of at least one other expert (the second expert) running on the first processing node are stored in the memory (the second memory) of a second processing node, and when it is necessary to use the parameters of the experts in the memory of the second processing node to perform calculation, the parameters of the experts are loaded from the memory of the second processing node to the memory of the first processing node, so that the first processing node uses the parameters of the experts to perform calculation, thereby in the case that the memory of a first processing node is limited in capacity, the communication amount of AllToAll can be eliminated by running all the experts on a first processing node, and the utilization rate of computing power is improved.

[0212] Optionally, as an implementation manner, within the range of available space of the memory of the first processing node, the data processing apparatus of the embodiment of the application further comprises a synchronization module, which is configured to synchronize the functions of the first calculation module 1302 and the loading module 1303, and / or synchronize the functions of the second calculation module 1304 and the loading module 1303.

[0213] In the implementation manner, the synchronization of expert calculation and expert parameter loading is performed, the time of expert calculation and expert parameter loading is overlapped with each other, and the utilization rate of computing power is further improved.

[0214] Referring to FIG. 14, the embodiment of the application provides a model data processing apparatus 400, which is applied to a first processing node in the system structure shown in FIG. 8, and for a single first processing node, the apparatus comprises:

[0215] The first obtaining module 1401 is configured to obtain first input data and second input data.

[0216] The second obtaining module 1402 is configured to obtain third input data and fourth input data.

[0217] The first sending module 1403 is configured to send fifth input data and sixth input data to the third processing node.

[0218] The first calculation module 1404 is configured to calculate the first input data and the third input data using the parameters of the first expert to obtain a first result and a third result.

[0219] The loading module 1405 is configured to load the parameters of the second expert through the second memory.

[0220] The second calculation module 1406 is configured to calculate the second input data and the fourth input data using the parameters of the second expert to obtain a second result and a fifth result.

[0221] The second sending module 1407 is configured to send the third result and the fourth result to the third processing node.

[0222] The third obtaining module 1408 is configured to obtain the fifth result and the sixth result.

[0223] In the embodiments of the present application, in the case that the available space of the memory (the first memory) of the first processing node is insufficient to store a plurality of experts (less than the total number of experts) running on the first processing node, the parameters of only part of the experts (the first expert) are stored in the memory of the first processing node, and the parameters of at least one other expert (the second expert) running on the first processing node are stored in the memory (the second memory) of the second processing node. When the parameters of the experts in the memory of the second processing node are needed for calculation, the parameters of these experts are loaded from the memory of the second processing node to the memory of the first processing node, so that the first processing node can perform calculation using the parameters of these experts. Thus, in the case that the available space of the memory of a first processing node is insufficient to store a plurality of experts (less than the total number of experts) running on the first processing node, the communication amount of AllToAll can be reduced and the utilization rate of computing power can be improved by running a plurality of experts on one first processing node.

[0224] Optionally, as an implementation manner, within the range allowed by the available space of the memory of the first processing node, the data processing apparatus of the embodiments of the present application further comprises a synchronization module configured to synchronize the functions of the first calculation module 1404 and the loading module 1405, and / or synchronize the functions of the second calculation module 1406 and the loading module 1405. In this implementation manner, the synchronization of expert calculation and expert parameter loading is performed, the time of expert calculation and expert parameter loading is overlapped, and the utilization rate of computing power is further improved.

[0225] Optionally, as an implementation form, within the range allowed by the available memory space of the first processing node, the data processing apparatus provided by the embodiment of the application further comprises a synchronization module, configured to synchronize the functions of the second obtaining module 1402, the first sending module 1403, the first calculation module 1404, the second sending module 1407 and the third obtaining module 1408, and / or synchronize the functions of the second obtaining module 1402, the first sending module 1403, the second calculation module 1406, the second sending module 1407 and the third obtaining module 1408. In this implementation form, the time of the expert calculation and the AllToAll communication is overlapped with each other through the synchronized execution of the expert calculation and the AllToAll communication, and the utilization rate of the computing power is further improved.

[0226] Optionally, as an implementation form, within the range allowed by the available memory space of the first processing node, the data processing apparatus provided by the embodiment of the application further comprises a synchronization module, configured to synchronize the functions of the second obtaining module 1402, the first sending module 1403, the first calculation module 1404, the second sending module 1407 and the third obtaining module 1408, and / or synchronize the functions of the second obtaining module 1402, the first sending module 1403, the second calculation module 1406, the second sending module 1407 and the third obtaining module 1408. In this implementation form, the time of the expert calculation and the AllToAll communication is overlapped with each other through the synchronized execution of the expert calculation, the expert parameter loading and the AllToAll communication, and the utilization rate of the computing power is further improved.

[0227] The embodiment of the application provides a distributed system, which can comprise the model data processing apparatus 300 or the model data processing apparatus 400.

[0228] Please refer to FIG. 15, which is a schematic diagram of an embodiment of a computing device in the embodiment of the application.

[0229] The computing device provided by the embodiment can be a server or a special-purpose data processing device, and the specific device form is not limited in the embodiment of the application.

[0230] The computing device 1500 can have great differences due to different configurations or performances, and can comprise one or more processors 1501 and memories 1502, wherein the memories 1502 store programs or data. The one or more processors 1501 constitute the processing unit in the above-mentioned embodiments of the application, and the one or more memories 1502 constitute the storage unit in the above-mentioned embodiments of the application.

[0231] The memory 1502 can be volatile memory or non-volatile memory. Optionally, the processor 1501 is one or more central processing units (CPUs), or graphics processing units (GPUs), or other special purpose processors, such as Ascend, etc. The CPU can be a single core CPU, or a multi-core CPU. The processor 1501 can communicate with the memory 1502 to execute a series of instructions in the memory 1502 on the computing device 1500.

[0232] The computing device 1500 further includes one or more wired or wireless network interfaces 1503, such as an Ethernet interface.

[0233] Optionally, although not shown in FIG. 15, the computing device 1500 can further include one or more power supplies, and one or more input / output interfaces, which can be used to connect a display, a mouse, a keyboard, a touch screen device, or a sensing device, etc. The input / output interface is an optional component, which can or can not exist, and is not limited herein.

[0234] The memory 1502 in the computing device 1500 in the embodiment stores a computer program, and the processor 1501 executes the flow performed when executing the computer program, which can refer to the method flow described in the foregoing method embodiments, and is not elaborated herein.

[0235] The foregoing embodiments can be implemented in whole or in part by software, hardware, firmware, or any combination thereof, and when implemented by software, can be implemented in whole or in part in the form of a computer program product.

[0236] The computer program product includes a computer program. When the computer program is loaded and executed on the computer, the flow or function according to the embodiments of the present application is generated wholly or partially. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer program can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, transmitted from one website site, computer, server or data center to another website site, computer, server or data center through wired (for example, coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (for example, infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that the computer can store or be integrated into a data storage device such as a server, data center, etc. containing one or more available media sets. The available media can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk (SSD)), etc.

[0237] The above describes the technical solutions provided by the present application in detail, and the principles and implementation manners of the present application are described by applying specific examples. The above examples are only used to help understand the method and core idea of the present application; meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation manner and application range will be changed, and the above description should not be understood as a limitation of the present application.

Claims

1. A method of processing model data, characterized by, A first processing node applied in a distributed system, the model is deployed in the distributed system, the distributed system includes the first processing node and a second processing node, the first processing node and the second processing node are different in type, the first processing node includes a first memory, the first memory stores parameters of a first expert in the model, the second processing node includes a second memory, the second memory stores parameters of a second expert in the model, and the method comprises: obtaining first input data and second input data, the first input data being input data of the first expert, and the second input data being input data of the second expert; calculating the first input data using the parameters of the first expert to obtain a first result; loading the parameters of the second expert through the second memory; calculating the second input data using the parameters of the second expert to obtain a second result.

2. The method of claim 1, wherein: during the execution of the calculation of the first input data using the parameters of the first expert to obtain a first result, the loading of the parameters of the second expert through the second memory is performed synchronously; and / or during the execution of the calculation of the second input data using the parameters of the second expert to obtain a second result, the loading of the parameters of the second expert through the second memory is performed synchronously.

3. The method of claim 1 or 2, wherein, The distributed system further includes a third processing node, the third processing node and the first processing node are of the same type, and the memory of the third processing node stores parameters of a third expert in the model, and the method further comprises: obtaining third input data and fourth input data; the third input data and the fourth input data come from the third processing node; calculating the third input data using the parameters of the first expert to obtain a third result; calculating the fourth input data using the parameters of the second expert to obtain a fourth result; sending the third result and the fourth result to the third processing node.

4. The method of claim 3, wherein: during the execution of the calculation of the third input data using the parameters of the first expert to obtain a third result, the loading of the parameters of the second expert through the second memory is performed synchronously; during the execution of the calculation of the fourth input data using the parameters of the second expert to obtain a fourth result, the loading of the parameters of the second expert through the second memory is performed synchronously.

5. The method of claim 3 or 4, wherein: during the execution of the calculation of the first input data using the parameters of the first expert to obtain a first result or the calculation of the third input data using the parameters of the first expert to obtain a third result, the obtaining of the third input data and the fourth input data or the sending of the third result and the fourth result to the third node is performed synchronously; and / or synchronously performing the loading the parameters of the second expert from the second memory and the obtaining the third input data and the fourth input data in the process of performing the calculating the second input data using the parameters of the second expert to obtain a second result or the calculating the fourth input data using the parameters of the second expert to obtain a fourth result; 6. The method of any one of claims 3-5, wherein: synchronously performing the loading the parameters of the second expert from the second memory and the obtaining the third input data and the fourth input data in the process of performing the calculating the first input data using the parameters of the first expert to obtain a first result or the calculating the third input data using the parameters of the first expert to obtain a third result; and / or synchronously performing the loading the parameters of the second expert from the second memory and the obtaining the third input data and the fourth input data in the process of performing the calculating the first input data using the parameters of the first expert to obtain a first result or the calculating the third input data using the parameters of the first expert to obtain a third result; and / or synchronously performing the loading the parameters of the second expert from the second memory and the obtaining the third input data and the fourth input data in the process of performing the calculating the second input data using the parameters of the second expert to obtain a second result or the calculating the fourth input data using the parameters of the second expert to obtain a fourth result; and / or synchronously performing the loading the parameters of the second expert from the second memory and the obtaining the third input data and the fourth input data in the process of performing the calculating the second input data using the parameters of the second expert to obtain a second result or the calculating the fourth input data using the parameters of the second expert to obtain a fourth result.

7. The method according to any one of claims 3 to 6, wherein, The distributed system further comprises a third processing node and a fourth processing node, the third processing node and the first processing node are of the same type, the fourth processing node and the first processing node are of different types, the memory of the third processing node stores parameters of a third expert in the model, the memory of the fourth processing node stores parameters of a fourth expert in the model, and the method further comprises: sending fifth input data and sixth input data to the third processing node; obtaining a fifth result and a sixth result; the fifth result is a result obtained by the third processing node calculating the fifth input data using the parameters of the third expert; the sixth result is a result obtained by the third processing node calculating the sixth input data using the parameters of the fourth expert; and the fifth result and the sixth result come from the third processing node.

8. The method of claim 7, wherein: synchronously performing the sending the fifth input data and the sixth input data to the third processing node or the obtaining the fifth result and the sixth result in the process of performing the calculation on the first input data using the parameters of the first expert to obtain the first result or the calculation on the third input data using the parameters of the first expert to obtain the third result; and / or synchronously performing the sending the fifth input data and the sixth input data to the third processing node or the obtaining the fifth result and the sixth result in the process of performing the calculation on the second input data using the parameters of the second expert to obtain the second result or the calculation on the fourth input data using the parameters of the second expert to obtain the fourth result; and / or synchronously performing the loading the parameters of the second expert through the second memory and the sending the fifth input data and the sixth input data to the third processing node in the process of performing the calculation on the first input data using the parameters of the first expert to obtain the first result or the calculation on the third input data using the parameters of the first expert to obtain the third result; and / or synchronously performing the loading the parameters of the second expert through the second memory and the obtaining the fifth result and the sixth result in the process of performing the calculation on the first input data using the parameters of the first expert to obtain the first result or the calculation on the third input data using the parameters of the first expert to obtain the third result; and / or synchronously performing the loading the parameters of the second expert through the second memory and the sending the fifth input data and the sixth input data to the third processing node in the process of performing the calculation on the second input data using the parameters of the second expert to obtain the second result or the calculation on the fourth input data using the parameters of the second expert to obtain the fourth result; and / or synchronously performing the loading the parameters of the second expert through the second memory and the obtaining the fifth result and the sixth result in the process of performing the calculation on the second input data using the parameters of the second expert to obtain the second result or the calculation on the fourth input data using the parameters of the second expert to obtain the fourth result.

9. The method according to any one of claims 1 to 8, wherein, The first processing node comprises one or more of NPU, GPU, TPU and CPU, and the second processing node comprises one or more of NPU, GPU, TPU and CPU.

10. The method of any one of claims 1-9, wherein, The first memory is HBM and / or DDR.

11. The method of any one of claims 1-10, wherein, The second memory is DDR and / or SSD.

12. The method of any one of claims 1-11, wherein, The method is applied to training or inference of a model.

13. The method of any one of claims 1-12, wherein, In the case that the available space of the first memory can only store the parameters of one expert, the total number of the first expert and the second expert is determined according to the average value of the estimated total consumption time of one expert and the average value of the two AllToAll communication times of one expert.

14. The method of any one of claims 1-12, wherein, In a case that the available space of the first memory is capable of storing parameters of at least two experts, the total number of the first expert and the second expert is determined according to an average value of total consumption time of one estimated expert, an average value of computation time of one expert and one AllToAll communication time, and an average value of two AllToAll communication times of one expert.

15. A model data processing apparatus, characterized by comprising: The application discloses a device applied to a first processing node in a distributed system, wherein the model is deployed in the distributed system, the distributed system comprises the first processing node and a second processing node, the first processing node and the second processing node are of different types, the first processing node comprises a first memory, the first memory stores parameters of a first expert in the model, the second processing node comprises a second memory, the second memory stores parameters of a second expert in the model, and the device comprises: a first obtaining module, configured to obtain first input data and second input data, wherein the first input data is input data of the first expert, and the second input data is input data of the second expert; a first calculating module, configured to calculate the first input data by using the parameters of the first expert to obtain a first result; a loading module, configured to load the parameters of the second expert through the second memory; a second calculating module, configured to calculate the second input data by using the parameters of the second expert to obtain a second result.

16. The device of claim 15, wherein: in the process of calculating the first input data by using the parameters of the first expert to obtain a first result, the loading of the parameters of the second expert through the second memory is synchronously implemented; and / or in the process of calculating the second input data by using the parameters of the second expert to obtain a second result, the loading of the parameters of the second expert through the second memory is synchronously implemented.

17. The apparatus of claim 15 or 16, wherein, The distributed system further comprises a third processing node, the third processing node and the first processing node are of the same type, parameters of a third expert in the model are stored in a memory of the third processing node, and the device further comprises: a second obtaining module, configured to obtain third input data and fourth input data, wherein the third input data and the fourth input data are from the third processing node; the first calculating module is further configured to calculate the third input data by using the parameters of the first expert to obtain a third result; the second calculating module is further configured to calculate the fourth input data by using the parameters of the second expert to obtain a fourth result; a second sending module, configured to send the third result and the fourth result to the third processing node.

18. The device of claim 17, wherein: in the process of calculating the third input data by using the parameters of the first expert to obtain a third result, the loading of the parameters of the second expert through the second memory is synchronously implemented; In the process of implementing the calculation of the fourth input data using the parameters of the second expert to obtain a fourth result, the loading of the parameters of the second expert from the second memory is implemented synchronously.

19. The apparatus of claim 17 or 18, wherein: In the process of implementing the calculation of the first input data using the parameters of the first expert to obtain a first result or the calculation of the third input data using the parameters of the first expert to obtain a third result, the obtaining of the third input data and the fourth input data or the sending of the third result and the fourth result to the third node is implemented synchronously; and / or In the process of implementing the calculation of the second input data using the parameters of the second expert to obtain a second result or the calculation of the fourth input data using the parameters of the second expert to obtain a fourth result, the obtaining of the third input data and the fourth input data or the sending of the third result and the fourth result to the third node is implemented synchronously.

20. The apparatus of any of claims 17-19, wherein: In the process of implementing the calculation of the first input data using the parameters of the first expert to obtain a first result or the calculation of the third input data using the parameters of the first expert to obtain a third result, the loading of the parameters of the second expert from the second memory and the obtaining of the third input data and the fourth input data is implemented synchronously; and / or In the process of implementing the calculation of the first input data using the parameters of the first expert to obtain a first result or the calculation of the third input data using the parameters of the first expert to obtain a third result, the loading of the parameters of the second expert from the second memory and the sending of the third result and the fourth result to the third node is implemented synchronously; and / or In the process of implementing the calculation of the second input data using the parameters of the second expert to obtain a second result or the calculation of the fourth input data using the parameters of the second expert to obtain a fourth result, the loading of the parameters of the second expert from the second memory and the obtaining of the third input data and the fourth input data is implemented synchronously; and / or In the process of implementing the calculation of the second input data using the parameters of the second expert to obtain a second result or the calculation of the fourth input data using the parameters of the second expert to obtain a fourth result, the synchronous implementation of the loading of the parameters of the second expert from the second memory and the obtaining of the third input data and the fourth input data is implemented synchronously.

21. The apparatus of any one of claims 17-20, wherein, The distributed system further comprises a third processing node and a fourth processing node, the third processing node and the first processing node are of the same type, the fourth processing node and the first processing node are of different types, the parameters of a third expert in the model are stored in the memory of the third processing node, the parameters of a fourth expert in the model are stored in the memory of the fourth processing node; The first sending module is configured to send fifth input data and sixth input data to a third processing node; The third obtaining module is configured to obtain a fifth result and a sixth result; the fifth result is a result obtained by the third processing node using parameters of the third expert to calculate the fifth input data; The sixth result is a result obtained by the third processing node using parameters of the fourth expert to calculate the sixth input data; and the fifth result and the sixth result are from the third processing node.

22. The apparatus of claim 21, wherein: the sending of the fifth input data and the sixth input data to the third processing node or the obtaining of the fifth result and the sixth result is performed synchronously with the performing of the calculating of the first input data using the parameters of the first expert to obtain the first result or the calculating of the third input data using the parameters of the first expert to obtain the third result; and / or the sending of the fifth input data and the sixth input data to the third processing node or the obtaining of the fifth result and the sixth result is performed synchronously with the performing of the calculating of the second input data using the parameters of the second expert to obtain the second result or the calculating of the fourth input data using the parameters of the second expert to obtain the fourth result; and / or the synchronously performing of the loading of the parameters of the second expert from the second memory and the sending of the fifth input data and the sixth input data to the third processing node is performed synchronously with the performing of the calculating of the first input data using the parameters of the first expert to obtain the first result or the calculating of the third input data using the parameters of the first expert to obtain the third result; and / or the synchronously performing of the loading of the parameters of the second expert from the second memory and the obtaining of the fifth result and the sixth result is performed synchronously with the performing of the calculating of the first input data using the parameters of the first expert to obtain the first result or the calculating of the third input data using the parameters of the first expert to obtain the third result; and / or the synchronously performing of the loading of the parameters of the second expert from the second memory and the sending of the fifth input data and the sixth input data to the third processing node is performed synchronously with the performing of the calculating of the second input data using the parameters of the second expert to obtain the second result or the calculating of the fourth input data using the parameters of the second expert to obtain the fourth result; and / or the synchronously performing of the loading of the parameters of the second expert from the second memory and the obtaining of the fifth result and the sixth result is performed synchronously with the performing of the calculating of the second input data using the parameters of the second expert to obtain the second result or the calculating of the fourth input data using the parameters of the second expert to obtain the fourth result.

23. The apparatus of any one of claims 15-22, wherein, The first processing node comprises one or more of an NPU, a GPU, a TPU, and a CPU, and the second processing node comprises one or more of an NPU, a GPU, a TPU, and a CPU.

24. The apparatus of any one of claims 15-23, wherein, The first memory includes HBM and / or DDR.

25. The apparatus of any one of claims 15-24, wherein, The second memory includes DDR and / or SSD.

26. The apparatus of any one of claims 15-25, wherein, The method is applied to training or inference of a model.

27. The apparatus of any one of claims 15-26, wherein, In a case where the available space of the first memory can only store parameters of one expert, the total number of the first expert and the second expert is determined according to an average of the estimated total consumption time of one expert and an average of two AllToAll communication times of one expert.

28. The apparatus of any one of claims 15-26, wherein, In a case where the available space of the first memory can store parameters of at least two experts, the total number of the first expert and the second expert is determined according to an average of the estimated total consumption time of one expert, an average of the computation time and one AllToAll communication time of one expert, and an average of two AllToAll communication times of one expert.

29. A distributed system comprising the apparatus of any one of claims 15-28.

30. A computer device, comprising: comprising: at least one memory for storing a program; at least one processor for executing the program stored in the memory; wherein, when the program stored in the memory is executed, the processor is configured to perform the method of any one of claims 1-14.

31. A computer readable storage medium, the computer readable storage medium storing a computer program, when the computer program is run on a processor, causing the processor to perform the method of any one of claims 1-14.

32. A computer program product, characterised in that, when the computer program product is run on a processor, causing the processor to perform the method of any one of claims 1-14.

Images