Automatic parallelization method and device for hybrid expert model, equipment and medium

Abstract

The invention provides an automatic parallelization method, device and equipment for a hybrid expert model, a medium and a program product, and relates to the technical field of computers, in particular to deep learning, artificial intelligence and large model technologies. The method comprises the following steps: acquiring a computational graph of a hybrid expert model; determining the process topology of the expert weight tensor of the expert model; the process is supported by computing equipment in the distributed system to run; splitting the global data tensor into sub-data tensors required by the corresponding expert model, and setting the process topology of the sub-data tensors to be the same as the process topology of the corresponding expert model; processing based on the input sub-data tensor and the expert weight tensor through an expert model to output a sub-result tensor; and determining a result tensor of the hybrid expert model according to the at least one sub-result tensor. According to the technical scheme provided by the embodiment of the invention, the limitation on the type of the expert model can be reduced, and the cross-process automatic parallel processing of the hybrid expert model is realized.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to deep learning, artificial intelligence and large model technology. Background Art

[0002] The Mixture of Experts (MoE) is a commonly used large model structure, generally composed of multiple expert models and a gating network. It aims to handle complex tasks by combining multiple "expert models". Each expert model is a neural network that focuses on solving a specific subtask or a specific aspect of the problem. The gating network is responsible for determining which expert model or models should take over the input data for processing. The output results after processing by the expert models are combined as the final output.

[0003] The MoE model has a huge number of parameters, and the distributed operation mode can utilize the computing and storage resources of multiple computing devices to achieve efficient operation. However, since the hierarchical architecture of each expert model in the MoE model may be the same or different, it is difficult to implement automatic parallel technology. If the tensor attributes are manually set, the workload is very large, which brings obstacles to the fast distributed operation of the MoE model. Summary of the invention

[0004] The present disclosure provides an automatic parallel method, apparatus, device, medium and program product for a hybrid expert model to optimize the automatic parallel technology of the hybrid expert model.

[0005] According to one aspect of the present disclosure, there is provided an automatic parallel method for a hybrid expert model, the method comprising:

[0006] Obtaining a computational graph of a hybrid expert model; wherein the hybrid expert model includes a gating network and at least two expert models;

[0007] Determine a process topology of an expert weight tensor of an expert model; wherein at least one process included in the process topology of the expert weight tensor is a partial process of a distributed system for running the hybrid expert model; and the process is supported and run by a computing device in the distributed system;

[0008] Splitting the global data tensor into sub-data tensors required by the corresponding expert model, and setting the process topology of the sub-data tensor to be the same as the process topology of the corresponding expert model; wherein the data in the sub-data tensor is the data required to be processed by the corresponding expert model;

[0009] Processing the input sub-data tensor and the expert weight tensor through the expert model to output a sub-result tensor;

[0010] A result tensor of the hybrid expert model is determined according to at least one result sub-tensor.

[0011] According to another aspect of the present disclosure, there is provided an automatic parallel device for a hybrid expert model, the device comprising:

[0012] A computational graph acquisition module, used to acquire a computational graph of a hybrid expert model; wherein the hybrid expert model includes a gating network and at least two expert models;

[0013] A weight tensor process determination module, used to determine the process topology of the expert weight tensor of the expert model; wherein at least one process included in the process topology of the expert weight tensor is a part of the distributed system for running the hybrid expert model; and the process is supported and run by a computing device in the distributed system;

[0014] A data tensor process splitting module is used to split the global data tensor into sub-data tensors required by the corresponding expert model, and set the process topology of the sub-data tensor to be the same as the process topology of the corresponding expert model; wherein the data in the sub-data tensor is the data required to be processed by the corresponding expert model;

[0015] A sub-result output module, used for processing based on the input sub-data tensor and the expert weight tensor through the expert model to output a sub-result tensor;

[0016] The result tensor determination module is used to determine the result tensor of the hybrid expert model according to at least one sub-result tensor.

[0017] Another aspect of the present disclosure further provides an electronic device, including:

[0018] at least one processor; and

[0019] a memory communicatively connected to the at least one processor; wherein,

[0020] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the automatic parallel method of the hybrid expert model provided in the embodiment of the present disclosure.

[0021] In another aspect of the embodiments of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to implement the automatic parallel method of the hybrid expert model provided by the embodiments of the present disclosure.

[0022] Another aspect of the embodiments of the present disclosure further provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the automatic parallel method of the hybrid expert model provided by the embodiments of the present disclosure.

[0023] The technical solution of the disclosed embodiment can reduce the restrictions on the expert model type and realize cross-process self-parallel processing of the hybrid expert model by unifying the process topology of the global data tensor and the expert weight tensor.

[0024] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0026] Figure 1 is a flow chart of an automatic parallel method of a hybrid expert model provided according to an embodiment of the present disclosure;

[0027] Figure 2 is a flow chart of an automatic parallel method of a hybrid expert model provided according to an embodiment of the present disclosure;

[0028] Figure 3 Schematic diagram of the conversion process of tensor data in an embodiment of the present disclosure;

[0029] Figure 4 A schematic diagram of the structure of an automatic parallel device for a hybrid expert model provided in an embodiment of the present disclosure;

[0030] Figure 5 A schematic block diagram of an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0031] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0032] First, the relevant technologies applicable to the technical solution of the present disclosure are introduced.

[0033] The distributed system that supports the distributed operation of the model is a computer cluster, which consists of multiple computers, each of which can be called a computing node. Each computing node generally includes multiple GPUs (graphics processing units) for performing model calculations respectively. GPUs can be called computing devices. One process can control the operation of one GPU. The operator tasks of the model can be assigned to each process and processed by the corresponding GPU.

[0034] The model calculation graph includes many types of operators that conform to the model operation logic, such as arithmetic operators, activation function operators, pooling operators, roll operators, normalization operators, splicing and splitting operators, and loop operators. In order to realize the transmission of tensors between different computing devices, various communication operators can also be included. The calculation graph represents the input and output flow of tensors between operators, and annotates the input tensors and output tensors of each operator with distributed attributes so that the calculation tasks of the operator can be processed in parallel on different computing devices. The input tensor includes the sample data to be processed by the model and its converted feature data, as well as the weight parameter matrix required for the operation of the operator. The output tensor processed by the operator can be used as the output result or as the input tensor of the next operator.

[0035] The same model and distributed system can be designed with different distributed operation modes as needed. Different distributed operation modes can be distinguished and determined by the distributed properties of tensors. The distributed properties of tensors include process mesh (referred to as mesh) and tensor placements, which provide a unified abstract representation for different distributed operation modes.

[0036] The process topology of a tensor is used to indicate in which processes the tensor is run, that is, which computing devices process it. Therefore, the process topology reflects the organization of different processes. The computing devices in a distributed system can be divided according to one or more dimensions, and tensors can also be divided according to the same dimensions to be assigned to corresponding processes. For example, a distributed system includes two processes supported by two computing devices, process 0 and process 1. If an input tensor of an operator needs to be input into two processes at the same time to be processed, the process topology of the tensor is expressed as [0, 1]. This indicates that the computing task of the operator is executed on two processes.

[0037] The tensor distribution of a tensor is used to indicate the distribution of the tensor on the specified process topology. There are generally three types of tensor distribution: Replicate, which means that the tensor remains in full state on all processes in the process topology; Shard (axis), which means that the tensor is split along the axis dimension and placed on different processes for processing. If the process topology of the tensor is process mesh = [0, 1], placements = Shard (0), it means that the two parts of the tensor after being split by row (0th dimension) are provided to process 0 and process 1 respectively; Partial, which means that each process only has part of the value of the tensor and needs to be restored to full data through the specified reduction operation. If placements = Partial (0), it means that after the tensor is split by row, the split tensor value is provided to the corresponding process, and the other parts of the tensor are padded with the value 0. The tensor matrix obtained by each process is still complete, but the value is not complete. The split dimension of the tensor and the process can be the same, which can be one dimension or two or more dimensions. For example, one dimension may be a row or a column, and two dimensions may include a matrix block divided by rows and columns. The embodiments of the present disclosure are not limited to this.

[0038] The distributed operation mode of the model is actually reflected in the distributed processing of operators, that is, the input tensors of any operator are divided into processes of different computing devices for execution, and each process performs a part of the calculation task of the operator. Because the calculation logic of each operator is different, there are requirements for the way the tensors are divided, and not any division can be distributed to different processes for correct calculation. At the same time, in order to ensure the correct calculation results of the entire model, after the operator performs distributed calculations, it may be necessary to insert some communication operations into the results, communicate the output tensors of multiple processes with each other, perform collection or aggregation and other operations, and then transmit them to subsequent operators for calculation. Therefore, by determining the process topology and tensor distribution of each tensor, the distributed operation mode of the calculation graph can be determined.

[0039] In the model calculation graph, if the attributes of all tensors need to be added manually, the workload is large, so generally users only need to mark the process topology and tensor distribution of some tensors of some operators, and then use automatic parallel technology. That is, according to the logic between operators represented by the calculation graph, the tensor attributes of other operators are deduced, and communication operators are automatically inserted, so as to realize the distributed operation of large models without considering the details such as distribution strategy and communication between different computing nodes.

[0040] The MoE model calculation graph includes multiple expert models. Each expert model can be seen as a collection of operators to realize its own model processing function. In addition to the operator set of the expert model, it also includes other functional operators such as the gated network operator set.

[0041] How to use automatic parallel technology to support the distributed operation of the MoE model is a difficult problem currently faced. The MoE model can deploy different expert models to different processes for operation, which is called the expert model parallel mode. If each expert model is deployed to a different process, it actually means setting the process topology of the input tensor of the expert model operator to one process, and the operator of the expert model is only processed in this process.

[0042] The complexity of tensor attribute labeling in this mode is that if the process topology and distribution of tensors are manually labeled, the workload of manual labeling is relatively large and the requirements on personnel skills are also relatively high; in automatic parallel technology, automatic parallelization can only be performed when the input tensors of the operator have the same process topology. Therefore, a feasible way to execute automatic parallel technology based on the computational graph of the MoE model includes:

[0043] Step 1: Put the weight parameters of all expert models into the same tensor, which can be called the complete weight tensor;

[0044] Step 2: Set the process topology and tensor distribution for the complete weight tensor, which means distributing the weight tensors of different expert models to different processes.

[0045] Step 3: Set the process topology and distribution properties of the input data tensor. The input data is generally global and should be provided to each expert model. Then, continue to run the hybrid expert model in parallel.

[0046] For any set of operators of an expert model, the process topology needs to be the same in order to perform automatic parallelism in the set of operators of the expert model and derive the tensor properties of subsequent operators. However, if the process topology of the input tensors of the set of operators of the expert model is different, it is difficult to perform automatic parallelism. For example, for any expert model, if the process topology of the input data tensor is all processes, the process topology of the weight tensor of the expert model must also be all processes, and the set of operators of the expert model can perform automatic parallelism. Then for the set of operators of the expert model, the process topology of the two input tensors is different.

[0047] However, the above technology needs to be improved in that in order to make the weight tensor of the input expert model operator set global, the weight parameters of all expert models need to be unified into a matrix for processing. However, if the architectures of the expert models are different or the parameter scales vary greatly, it is actually difficult to unify them into a global matrix. It can be seen that the above automatic parallel technology's requirement for the same process topology has imposed limitations on the hybrid expert model.

[0048] The technical solution of the disclosed embodiment provides an optimization technology for reducing the architectural restrictions on the expert model and performing automatic parallelization on the hybrid expert model based on the model architecture characteristics of the hybrid expert model.

[0049] Figure 1 It is a flowchart of an automatic parallel method of a hybrid expert model provided according to an embodiment of the present disclosure. The method is applicable to the case where the computation graph of the hybrid expert model is implemented based on the automatic parallel technology to realize the distributed operation, and the properties of the tensor are derived by the automatic parallel technology, so that each operator can be processed in parallel in the distributed system. The method can be executed by an automatic parallel device of the hybrid expert model, which can be implemented in the form of hardware and / or software, and can be configured in an electronic device with computing and storage capabilities, typically a distributed system that can support the distributed operation of the model, or an electronic device that controls the distributed operation. The method includes:

[0050] S110, obtaining a computation graph of a hybrid expert model; wherein the hybrid expert model includes a gating network and at least two expert models;

[0051] S120, determining a process topology of an expert weight tensor of an expert model; wherein at least one process included in the process topology of the expert weight tensor is a partial process of a distributed system for running the hybrid expert model; the process is supported and run by a computing device in the distributed system; as described above, a process is set in one-to-one correspondence with a computing device;

[0052] S130, splitting the global data tensor into sub-data tensors required by the corresponding expert model, and setting the process topology of the sub-data tensor to be the same as the process topology of the corresponding expert model; wherein the data in the sub-data tensor is the data required to be processed by the corresponding expert model;

[0053] S140, performing processing based on the input sub-data tensor and the expert weight tensor through the expert model to output a sub-result tensor;

[0054] S150. Determine a result tensor of the hybrid expert model according to at least one sub-result tensor.

[0055] As described in the above-mentioned related art, the calculation graph of the hybrid expert model can be obtained first. The hybrid expert model may include a gating network and at least two expert models, and may also include other functional modules as needed. The gating network is used to determine at least one expert model to be activated for this operation for the input data, and to determine the part of the input data that each expert model needs to process. For example, the gating network will calculate which expert models should process which input data. For the initial input data, the gating network may output a tensor of [E, C, H], where E represents the number of expert models, C represents the number of data to be processed by each expert model, and H is the vector size of each data. Therefore, by dividing according to the E dimension, the data required by each expert model can be divided into the operator set of the expert model for processing. For an expert model that does not need to process data, the value of the data in the corresponding area is usually 0, and it is deemed that the expert model will not be provided with data for processing. How the gating network determines which expert models should process the data, and the architecture of each expert model, is not limited in the embodiment of the present disclosure, and is determined by the specific model function and training purpose.

[0056] For the operator set of each expert model, the operation of the operator set requires at least the expert weight and the input data to be processed, that is, the expert weight tensor and the data tensor need to be obtained. In the embodiment of the present disclosure, the matrix dimension of the expert weight tensor of each expert model can be the same or different, which is determined by the architecture of the expert model itself. In the process of automatic parallelization, the expert weight tensor of at least one expert model is specifically provided with a process topology that only includes part of the process. Specifically, for all processes provided by the distributed system, the process topology of the expert weight tensor of at least one expert model only includes part of the process. Optionally, different process topologies can be set for different expert models. According to the operation requirements of the expert model, it is actually optional that the process topology of the expert weight tensor includes one or more processes of the distributed system, and the processes included in the process topologies of different expert weight tensors are partially the same or all the same. It can be seen that there can be multiple ways to deploy the expert model on the process, but there is no need to ensure that the weight tensor of each expert model is unified into a complete weight tensor, and the process topology of the complete weight tensor is set to include all processes. By removing this restriction, the architectures of various expert models can be the same or different, and the deployment of expert models on processes is more flexible. For example, each expert model has the conditions for parallel independent execution, so it is preferred that the process topology of each expert weight tensor includes a process, and the process topologies of each expert weight tensor are different. As a result, the distributed operation model of expert models in a distributed system has greater flexibility.

[0057] The process topology of the expert weight tensor of each expert model can determine the distributed operation mode. The setting method of the process topology can be set according to preset rules and model operation requirements. Optionally, determining the process topology of the expert weight tensor of the expert model may include: obtaining the process topology of the expert weight tensor of the manually labeled expert model. That is, it is allowed to manually label the process topology of the expert weight tensor, which strengthens the control of the model management personnel over the model operation mode. On the basis of manually defining the distribution method of the expert model on the process, the automatic parallel technology can still be used to run the model without manually labeling the attributes of other tensors on a large scale.

[0058] Although the expert weight tensor of the expert model can be a different process topology, for the sample data to be processed of the model, as input data, it is usually processed by a set of gated network operators in a global state, which can be called a global data tensor. After the gated network operator set processes the input data, a global data tensor is obtained, which is provided to each process for processing. In order to achieve automatic parallelization of the expert model, it is necessary to make the process topology of the global data tensor consistent with that of the expert weight tensor. Therefore, it is necessary to split the global data tensor into sub-data tensors required by the corresponding expert model, and set the process topology of the sub-data tensor to be the same as the process topology of the corresponding expert model. Among them, the data in the sub-data tensor is the data required to be processed by the corresponding expert model. Optionally, the relationship between the sub-data in the global data tensor and the corresponding expert model is determined by the gated network.

[0059] Then, the expert model is used to process the input sub-data tensor and the expert weight tensor. Since the process topology of the sub-data tensor and the expert weight tensor is the same, the automatic parallel technology can be executed to obtain the sub-result tensor output by the expert model. Finally, the final result tensor of the hybrid expert model is determined based on the sub-result tensors output by one or more expert models. For each sub-result tensor, the hybrid expert model can set other functional operators for post-processing to form the final result tensor.

[0060] The technical solution of the disclosed embodiment, in the process of executing the automatic parallel technology for the calculation graph of the hybrid expert model, realizes that the process topology of the expert weight tensor and the sub-data tensor of the expert model is the same, and the process topology does not have to be limited to all processes of the distributed system, so that the automatic parallelization of the expert model can be realized, and the consistency requirements for the architecture of each expert model can be reduced, and there is no need to manually annotate the attributes of a large number of tensors, so that the model function of the hybrid expert model is more flexible, and the automatic parallelization of cross-process topology is realized, and the parallel operation of expert models of different architectures can be supported, which brings the advantage of the amount of calculation for model training or derivation process. As a result, the computing device carrying the process can flexibly realize the distributed operation of the model of cross-process topology; the operation of the hybrid expert model can give full play to the hardware advantages of the distributed system, allowing each expert model to be deployed on one or more computing devices according to the needs of the model operation, without being restricted by the constraints of the automatic parallel technology, for example, expert models of different model architectures can be completely independently deployed on different computing devices. The technical solution of the disclosed embodiment can be widely applied to deep learning systems, distributed deep learning systems, machine learning platforms, and distributed training platforms, etc., to realize the automatic parallel operation of hybrid expert models.

[0061] Figure 2 It is a flowchart of an automatic parallel method of a hybrid expert model provided according to an embodiment of the present disclosure. Figure 3 Schematic diagram of the conversion process of tensor data in the embodiment of the present disclosure. Based on the above embodiment, this embodiment provides a specific implementation method for changing the process topology of the tensor, that is, splitting the global data tensor into sub-data tensors required by the corresponding expert model can be achieved by adjusting the tensor distribution of the sub-data as needed and then setting the process topology. The method includes:

[0062] S210, obtaining a computational graph of a hybrid expert model; wherein the hybrid expert model includes a gating network and at least two expert models;

[0063] S220, determining a process topology of an expert weight tensor of an expert model; wherein at least one process included in the process topology of the expert weight tensor is a partial process of a distributed system for running the hybrid expert model;

[0064] S231, according to the relationship between the sub-data in the global data tensor and the corresponding expert model, convert the tensor distribution of the global data tensor so that each sub-data can be segmented into the corresponding expert model for processing;

[0065] S232, from the global data tensor, obtaining the segmented sub-data to form a sub-data tensor, and setting the process topology of the sub-data tensor to be the same as the process topology of the expert weight tensor of the corresponding expert model; wherein the data in the sub-data tensor is the data required to be processed by the corresponding expert model;

[0066] S240, performing processing based on the input sub-data tensor and the expert weight tensor through the expert model to output a sub-result tensor;

[0067] S250. Determine a result tensor of the hybrid expert model according to at least one sub-result tensor.

[0068] Specifically, the process of splitting the global data tensor into sub-data tensors may include a tensor distribution conversion operation and a process topology adjustment operation. The tensor distribution conversion operation can divide the sub-data into processes corresponding to the expert model, and then form the divided sub-data into independent sub-data tensors, and set its process topology to be the same as the corresponding expert model process topology.

[0069] The following is a detailed description with reference to the accompanying drawings. Figure 3 As shown: the original process topology of the global data tensor includes all processes. Taking two processes, process 0 and process 1, as all processes, the original process topology of the global data tensor is process mesh = [0, 1]. The data in the global data tensor is provided to each process, and the original tensor distribution is a replication state, that is, placements = Replicate().

[0070] According to the relationship between the sub-data in the global data tensor and the corresponding expert model, the copy state of the tensor distribution of the global data tensor is converted into a split state, so that each sub-data can be split into the corresponding expert model for processing. Figure 3 As shown, assuming that after splitting by rows, the sub-data of the first row is provided to the expert model 0 supported by process 0, and the sub-data of the second row is provided to the expert model 1 supported by process 1, the tensor distribution of the global data tensor is set to the split state by rows, that is, placements = Shard (0). Preferably, the split dimension in the split state is the same as the process division dimension of the corresponding expert model. The split dimension can be split by rows or columns, or it can be split into local blocks by rows and columns at the same time. The embodiment of the present disclosure does not limit the split dimension, as long as the sub-data tensor can be split to the process of the corresponding expert model.

[0071] Although the tensor distribution state of the global data tensor has changed, it still needs to be split into independent sub-data tensors according to the split state. Figure 3As shown, based on the state of splitting by row, it is split into two independent sub-data tensors.

[0072] Furthermore, after being divided into sub-data tensors, the sub-data tensors are provided to the process of the corresponding expert model, so the process topology of the sub-data tensor is adjusted to be consistent with the process topology of the corresponding expert model. Figure 3 As shown, the process topology of sub-data tensor 0 for expert model 0 is mesh=[0], and the process topology of sub-data tensor 1 for expert model 1 is mesh=[1].

[0073] On this basis, the tensor distribution of the sub-data tensor can be further derived and updated to a copy state or a split state according to the process topology of the sub-data tensor. Specifically, the sub-data tensor is provided to the corresponding expert model, and the tensor distribution of the sub-data tensor is determined according to the process run by the expert model operator set. For example, if the expert model only runs in one process, the sub-data tensor is provided to one process, and its tensor distribution can be in a copy state. If the expert model runs in multiple processes, the sub-data tensor needs to be in a copy state or a split state so that it can be provided to multiple processes. The tensor distribution state of the sub-data tensor is determined by the running logic of the expert model operator set and is automatically derived and determined in the automatic parallel process.

[0074] The technical solution of the above embodiment realizes the splitting of the global data tensor through the tensor distribution conversion of the global data tensor and the process topology setting in a simple adjustment manner, ensures the consistency of the process topology of the sub-data tensor and the expert weight tensor, thereby ensuring the automatic parallel execution of the hybrid expert model.

[0075] Based on the above technical solution, the output sub-result tensor can be further processed. Preferably, determining the result tensor of the hybrid expert model according to at least one sub-result tensor includes:

[0076] Aggregating the sub-result tensors output by at least two expert models to form a global result tensor, and modifying the process topology of the global result tensor to be the same as the process topology of the global data tensor;

[0077] According to the original output process of each sub-result tensor in the global result tensor, the tensor distribution of the global result tensor is set to a segmentation state.

[0078] Usually, the process topology of the sub-result tensor output by each expert model is consistent with the process topology of the input sub-data tensor. However, the process topology of the sub-data tensor is inconsistent with the process topology of the global data tensor. This may affect the subsequent processing of the data results. That is, after the expert model outputs the result, the hybrid expert model may also perform other post-processing operations on the output results of each expert model. The specific post-processing operation function is not limited in the embodiment of the present disclosure, but the global unified post-processing operation of the hybrid expert model requires that the process topology of the result tensor is global. Therefore, it is preferred to restore the process topology of the sub-result tensor to be consistent with the process topology of the global data tensor.

[0079] Specifically, the specific means adopted in this embodiment are as follows: Figure 3 As shown, the two sub-result tensors are first aggregated to form a global result tensor. At this time, the process topology of the global result tensor is modified to be the same as the process topology of the global data tensor. For example, the process topology of sub-result tensor 0 output by expert model 0 is mesh=[0], and the process topology of sub-result tensor 1 output by expert model 1 is mesh=[1]; the process topology of the global result tensor formed after the two are merged is mesh=[0,1]. Since the original output processes of different sub-result tensors are different, the tensor distribution of the global result tensor is set to a shard state, that is, placements=Shard(0), to indicate the relationship between the sub-results and the corresponding expert models.

[0080] At this point, the global result tensor already has a global process topology, and can continue to be automatically parallelized by subsequent processing operators. Optionally, it also includes: according to the result output requirements of the hybrid expert model, changing the tensor distribution of the global result tensor from a split state to a copy state. If the subsequent processing operator of the hybrid expert model needs to provide the global result tensor to each process for processing, the tensor distribution of the global result tensor can be changed from a split state to a copy state. If the result output requirement of the hybrid expert model is to output the data of the sub-result tensors separately, the tensor distribution of the global result tensor can remain in a split state. The specific state of the tensor distribution can be flexibly determined by the output requirements of the hybrid expert model.

[0081] Figure 4 A schematic diagram of the structure of an automatic parallel device for a hybrid expert model provided in an embodiment of the present disclosure, the device comprising:

[0082] A computation graph acquisition module 410, used to acquire a computation graph of a hybrid expert model; wherein the hybrid expert model includes a gating network and at least two expert models;

[0083] The weight tensor process determination module 420 is used to determine the process topology of the expert weight tensor of the expert model; wherein at least one process included in the process topology of the expert weight tensor is a part of the distributed system for running the hybrid expert model; and the process is supported and run by a computing device in the distributed system;

[0084] The data tensor process splitting module 430 is used to split the global data tensor into sub-data tensors required by the corresponding expert model, and set the process topology of the sub-data tensor to be the same as the process topology of the corresponding expert model; wherein the data in the sub-data tensor is the data required to be processed by the corresponding expert model;

[0085] A sub-result output module 440, configured to process the input sub-data tensor and the expert weight tensor through an expert model to output a sub-result tensor;

[0086] The result tensor determination module 450 is used to determine the result tensor of the hybrid expert model according to at least one sub-result tensor.

[0087] The technical solution of the disclosed embodiment, in the process of executing automatic parallel technology on the calculation graph of the hybrid expert model, realizes that the process topology of the expert weight tensor and the sub-data tensor of the expert model is made the same, and the process topology does not have to be limited to all processes of the distributed system. This can not only realize the automatic parallelization of the expert model, but also reduce the consistency requirements for the architecture of each expert model. There is no need to manually label the attributes of a large number of tensors, making the model function of the hybrid expert model more flexible, realizing automatic parallelization across process topologies, and being able to support the parallel operation of expert models of different architectures, which brings advantages in computing power for both model training or derivation processes.

[0088] Based on the above technical solution, optionally, the process topology of the expert weight tensor in the device includes one or more processes of the distributed system, and the processes included in the process topologies of different expert weight tensors are partially or completely the same.

[0089] Optionally, the process topology of each expert weight tensor in the device includes a process, and the process topologies of the expert weight tensors are different from each other.

[0090] Optionally, the weight tensor process determination module in the device is specifically used to obtain the process topology of the expert weight tensor of the manually labeled expert model.

[0091] Optionally, the data tensor process splitting module in the device includes:

[0092] A sub-data segmentation unit, used for converting the tensor distribution of the global data tensor according to the relationship between the sub-data and the corresponding expert model in the global data tensor, so that each sub-data can be segmented to the corresponding expert model for processing;

[0093] The process setting unit is used to obtain the segmented sub-data from the global data tensor to form a sub-data tensor, and set the process topology of the sub-data tensor to be the same as the process topology of the expert weight tensor of the corresponding expert model.

[0094] Optionally, the neutron data segmentation unit of the device is specifically used for:

[0095] According to the relationship between the sub-data and the corresponding expert model in the global data tensor, the copy state of the tensor distribution of the global data tensor is converted into a segmentation state, so that each sub-data can be segmented to the corresponding expert model for processing.

[0096] Optionally, the data tensor process splitting module in the device further includes:

[0097] The tensor distribution updating unit is used to obtain the segmented sub-data to form a sub-data tensor, and then update the tensor distribution of the sub-data tensor to a copy state or a segmentation state according to the process topology of the sub-data tensor.

[0098] Optionally, the segmentation dimension in the segmentation state in the device is the same as the process division dimension of the corresponding expert model.

[0099] Optionally, the relationship between the sub-data in the global data tensor and the corresponding expert model in the device is determined by the gating network.

[0100] Optionally, the result tensor determination module in the device includes:

[0101] A sub-result aggregation unit, configured to aggregate sub-result tensors output by at least two expert models to form a global result tensor, and modify the process topology of the global result tensor to be the same as the process topology of the global data tensor;

[0102] The result distribution setting unit is used to set the tensor distribution of the global result tensor to a split state according to the original output process of each sub-result tensor in the global result tensor.

[0103] Optionally, the device further comprises:

[0104] The global result tensor distribution setting module is used to change the tensor distribution of the global result tensor from a segmentation state to a replication state according to the result output requirements of the hybrid expert model.

[0105] The automatic parallel device of the hybrid expert model provided in the embodiment of the present disclosure is used to implement the automatic parallel method of the hybrid expert model provided in the embodiment of the present disclosure, and has corresponding functional modules and beneficial effects.

[0106] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0107] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product. The electronic device is optionally used to control a distributed system or is a computer in a distributed system, and can control the operation of each computing device in the distributed system.

[0108] Figure 5 A schematic block diagram of an example electronic device 500 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0109] like Figure 5 As shown, the device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the device 500 can also be stored. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0110] A number of components in the device 500 are connected to the I / O interface 505, including: an input unit 506, such as a keyboard, a mouse, etc.; an output unit 507, such as various types of displays, speakers, etc.; a storage unit 508, such as a disk, an optical disk, etc.; and a communication unit 509, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 509 allows the device 500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0111] The computing unit 501 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 501 performs the various methods and processes described above, such as the automatic parallel method of the hybrid expert model. For example, in some embodiments, the automatic parallel method of the hybrid expert model may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 500 via the ROM 502 and / or the communication unit 509. When the computer program is loaded into the RAM 503 and executed by the computing unit 501, one or more steps of the automatic parallel method of the hybrid expert model described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to execute the automatic parallel method of the hybrid expert model in any other appropriate manner (eg, by means of firmware).

[0112] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0113] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0114] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0115] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0116] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0117] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services. The server may also be a server of a distributed system, or a server combined with a blockchain.

[0118] Artificial intelligence is a discipline that studies how to use computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.). It includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing; artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, machine learning / deep learning technology, big data processing technology, knowledge graph technology, and other major directions.

[0119] Cloud computing refers to a technology system that uses network access to elastically scalable shared physical or virtual resource pools. Resources can include servers, operating systems, networks, software, applications, and storage devices, and can be deployed and managed on demand and in a self-service manner. Cloud computing technology can provide efficient and powerful data processing capabilities for technical applications such as artificial intelligence and blockchain, as well as model training.

[0120] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution provided by this disclosure can be achieved, and this document does not limit this.

[0121] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. An automatic parallel method for a hybrid expert model, the method comprising: Obtaining a computational graph of a hybrid expert model; wherein the hybrid expert model includes a gating network and at least two expert models; Determine a process topology of an expert weight tensor of an expert model; wherein at least one process included in the process topology of the expert weight tensor is a partial process of a distributed system for running the hybrid expert model; and the process is supported and run by a computing device in the distributed system; Splitting the global data tensor into sub-data tensors required by the corresponding expert model, and setting the process topology of the sub-data tensor to be the same as the process topology of the corresponding expert model; wherein the data in the sub-data tensor is the data required to be processed by the corresponding expert model; Processing the input sub-data tensor and the expert weight tensor through the expert model to output a sub-result tensor; A result tensor of the hybrid expert model is determined according to at least one result sub-tensor.

2. The method according to claim 1, wherein: The process topology of the expert weight tensor includes one or more processes of the distributed system, and the processes included in the process topologies of different expert weight tensors are partially or completely the same.

3. The method according to claim 2, wherein: The process topology of each expert weight tensor includes a process, and the process topologies of the expert weight tensors are different from each other.

4. The method according to claim 1, wherein: The process topology for determining the expert weight tensor of the expert model includes: The process topology for obtaining the expert weight tensor of a manually labeled expert model.

5. The method according to claim 1, wherein: Splitting the global data tensor into sub-data tensors required by the corresponding expert model, and setting the process topology of the sub-data tensor to be the same as the process topology of the corresponding expert model includes: According to the relationship between the sub-data and the corresponding expert model in the global data tensor, the tensor distribution of the global data tensor is converted so that each sub-data can be segmented into the corresponding expert model for processing; From the global data tensor, the segmented sub-data are obtained to form a sub-data tensor, and the process topology of the sub-data tensor is set to be the same as the process topology of the expert weight tensor of the corresponding expert model.

6. The method according to claim 5, wherein: According to the relationship between the sub-data in the global data tensor and the corresponding expert model, converting the tensor distribution of the global data tensor so that each sub-data can be segmented into the corresponding expert model for processing includes: According to the relationship between the sub-data and the corresponding expert model in the global data tensor, the copy state of the tensor distribution of the global data tensor is converted into a segmentation state, so that each sub-data can be segmented to the corresponding expert model for processing.

7. The method according to claim 6, after obtaining the segmented sub-data to form a sub-data tensor, further comprises: According to the process topology of the sub-data tensor, the tensor distribution of the sub-data tensor is updated to a copy state or a split state.

8. The method according to claim 6, wherein: The segmentation dimension in the segmentation state is the same as the process division dimension of the corresponding expert model.

9. The method according to claim 1, wherein: The relationship between the sub-data in the global data tensor and the corresponding expert model is determined by the gating network.

10. The method according to claim 1, wherein: Determining the result tensor of the hybrid expert model according to at least one result sub-tensor comprises: Aggregating the sub-result tensors output by at least two expert models to form a global result tensor, and modifying the process topology of the global result tensor to be the same as the process topology of the global data tensor; According to the original output process of each sub-result tensor in the global result tensor, the tensor distribution of the global result tensor is set to a segmentation state.

11. The method according to claim 1 or 10, further comprising: According to the result output requirement of the hybrid expert model, the tensor distribution of the global result tensor is changed from a split state to a copy state.

12. An automatic parallel device for a hybrid expert model, the device comprising: A computational graph acquisition module, used to acquire a computational graph of a hybrid expert model; wherein the hybrid expert model includes a gating network and at least two expert models; A weight tensor process determination module, used to determine the process topology of the expert weight tensor of the expert model; wherein at least one process included in the process topology of the expert weight tensor is a part of the distributed system for running the hybrid expert model; and the process is supported and run by a computing device in the distributed system; A data tensor process splitting module is used to split the global data tensor into sub-data tensors required by the corresponding expert model, and set the process topology of the sub-data tensor to be the same as the process topology of the corresponding expert model; wherein the data in the sub-data tensor is the data required to be processed by the corresponding expert model; A sub-result output module, used for processing based on the input sub-data tensor and the expert weight tensor through the expert model to output a sub-result tensor; The result tensor determination module is used to determine the result tensor of the hybrid expert model according to at least one sub-result tensor.

13. The device according to claim 12, wherein: The process topology of the expert weight tensor includes one or more processes of the distributed system, and the processes included in the process topologies of different expert weight tensors are partially or completely the same.

14. The device according to claim 13, wherein: The process topology of each expert weight tensor includes a process, and the process topologies of the expert weight tensors are different from each other.

15. The device according to claim 12, wherein: The weight tensor process determination module is specifically used to obtain the process topology of the expert weight tensor of the manually labeled expert model.

16. The device according to claim 12, wherein: The data tensor process splitting module includes: A sub-data segmentation unit, used for converting the tensor distribution of the global data tensor according to the relationship between the sub-data and the corresponding expert model in the global data tensor, so that each sub-data can be segmented to the corresponding expert model for processing; The process setting unit is used to obtain the segmented sub-data from the global data tensor to form a sub-data tensor, and set the process topology of the sub-data tensor to be the same as the process topology of the expert weight tensor of the corresponding expert model.

17. The device according to claim 16, wherein: The sub-data segmentation unit is specifically used for: According to the relationship between the sub-data and the corresponding expert model in the global data tensor, the copy state of the tensor distribution of the global data tensor is converted into a segmentation state, so that each sub-data can be segmented to the corresponding expert model for processing.

18. The apparatus according to claim 17, wherein the data tensor process splitting module further comprises: The tensor distribution updating unit is used to obtain the segmented sub-data to form a sub-data tensor, and then update the tensor distribution of the sub-data tensor to a copy state or a segmentation state according to the process topology of the sub-data tensor.

19. The device according to claim 17, wherein: The segmentation dimension in the segmentation state is the same as the process division dimension of the corresponding expert model.

20. The device according to claim 12, wherein: The relationship between the sub-data in the global data tensor and the corresponding expert model is determined by the gating network.

21. The device according to claim 12, wherein: The result tensor determination module includes: A sub-result aggregation unit, configured to aggregate sub-result tensors output by at least two expert models to form a global result tensor, and modify the process topology of the global result tensor to be the same as the process topology of the global data tensor; The result distribution setting unit is used to set the tensor distribution of the global result tensor to a split state according to the original output process of each sub-result tensor in the global result tensor.

22. The apparatus according to claim 12 or 21, further comprising: The global result tensor distribution setting module is used to change the tensor distribution of the global result tensor from a segmentation state to a replication state according to the result output requirements of the hybrid expert model.

23. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the automatic parallel method for the hybrid expert model according to any one of claims 1 to 11.

24. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the automatic parallel method of the hybrid expert model according to any one of claims 1-11.

25. A computer program product comprising a computer program / instructions, which when executed by a processor implements the automatic parallel method of the hybrid expert model according to any one of claims 1-11.

Images