An automatic parallel method, device, equipment and medium for mixing expert models

Abstract

The present disclosure provides an automatic parallel method, device, equipment, medium and program product of mixed expert model, relates to the technical field of computers, in particular to deep learning, artificial intelligence and large model technology. The method comprises: acquiring a computation graph of a mixed expert model; determining a process topology of an expert weight tensor of an expert model; the process is supported by a computing device in the distributed system; splitting a global data tensor into sub-data tensors required by the corresponding expert model, and setting the process topology of the sub-data tensors to be the same as the process topology of the corresponding expert model; processing the input sub-data tensors and the expert weight tensor based on the expert model to output a sub-result tensor; and determining a result tensor of the mixed expert model according to at least one sub-result tensor. The technical scheme of the embodiment of the present disclosure can reduce the limitation on the type of expert model and realize self-parallel processing of the mixed expert model across processes.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, in particular to deep learning, artificial intelligence and large model technology. BACKGROUND

[0002] Mixture of Experts (MoE) is a commonly used large model structure, which generally consists of multiple expert models and a gating network, aiming to handle complex tasks by combining multiple "expert models", each of which is a neural network focusing on solving a specific sub-task or a certain aspect of a problem. The gating network is responsible for determining which expert model or models should take over the processing of the input data, and the output results of the expert models after processing are combined as the final output.

[0003] MoE model has a huge number of parameters, and adopts a distributed running mode to utilize the computing and storage resources of multiple computing devices for efficient operation. However, since the hierarchical architectures of the expert models in the MoE model may be the same or different, it is difficult to implement automatic parallel technology, and if manual setting of tensor attributes is used, the workload is very large, which poses an obstacle to the fast distributed running of the MoE model. SUMMARY

[0004] The present disclosure provides an automatic parallel method, device, equipment, medium and program product for a mixture of expert models, to optimize the automatic parallel technology for a mixture of expert models.

[0005] According to an aspect of the present disclosure, an automatic parallel method for a mixture of expert models is provided, the method comprising:

[0006] obtaining a computation graph of a mixture of expert models; wherein the mixture of expert models comprises a gating network and at least two expert models;

[0007] determining the process topology of the expert weight tensor of the expert model; wherein the processes included in at least one process topology of the expert weight tensor are part of the processes of a distributed system for running the mixture of expert models; the processes are supported by computing devices in the distributed system for running;

[0008] splitting a global data tensor into sub-data tensors required by the corresponding expert model, and setting the process topology of the sub-data tensors to be the same as the process topology of the corresponding expert model; wherein the data in the sub-data tensors is the data required for processing by the corresponding expert model;

[0009] processing by the expert model based on the input sub-data tensor and the expert weight tensor to output a sub-result tensor;

[0010] A result tensor of the hybrid expert model is determined according to the at least one sub-result tensor.

[0011] According to another aspect of the present disclosure, there is provided an automatic parallel device for a hybrid expert model, the device comprising:

[0012] A computation graph acquisition module, configured to acquire a computation graph of a hybrid expert model, wherein the hybrid expert model includes a gating network and at least two expert models;

[0013] a weight tensor process determination module, configured to determine a process topology of an expert weight tensor of an expert model; wherein at least one process included in the process topology of the expert weight tensor is a part of a distributed system for running the hybrid expert model; and the process is supported and run by a computing device in the distributed system;

[0014] A data tensor process splitting module is used to split the global data tensor into sub-data tensors required by the corresponding expert model, and set the process topology of the sub-data tensor to be the same as the process topology of the corresponding expert model; wherein the data in the sub-data tensor is the data required to be processed by the corresponding expert model;

[0015] A sub-result output module, configured to process the input sub-data tensor and the expert weight tensor through the expert model to output a sub-result tensor;

[0016] The result tensor determination module is used to determine the result tensor of the hybrid expert model according to at least one sub-result tensor.

[0017] Another aspect of the present disclosure provides an electronic device, including:

[0018] at least one processor; and

[0019] a memory communicatively connected to the at least one processor; wherein,

[0020] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to execute the automatic parallel method of the hybrid expert model provided by the embodiment of the present disclosure.

[0021] Another aspect of the embodiments of the present disclosure further provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to implement the automatic parallel method of the hybrid expert model provided by the embodiments of the present disclosure.

[0022] Another aspect of the embodiments of the present disclosure further provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the automatic parallel method of the hybrid expert model provided by the embodiments of the present disclosure.

[0023] The technical solution of the embodiment of the present disclosure can reduce the restrictions on expert model types and realize cross-process autonomous parallel processing of hybrid expert models by unifying the process topology of the global data tensor and the expert weight tensor.

[0024] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0026] Figure 1 is a flowchart of an automatic parallel method for a hybrid expert model provided according to an embodiment of the present disclosure;

[0027] Figure 2 is a flowchart of an automatic parallel method for a hybrid expert model provided according to an embodiment of the present disclosure;

[0028] Figure 3 Schematic diagram of the conversion process of tensor data in an embodiment of the present disclosure;

[0029] Figure 4 A schematic diagram of the structure of an automatic parallel device for a hybrid expert model provided in an embodiment of the present disclosure;

[0030] Figure 5 A schematic block diagram of an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0031] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0032] First, the relevant technologies applicable to the technical solution of the present disclosure are introduced.

[0033] The distributed system supporting the distributed operation of the model is a computer cluster consisting of multiple computers, each of which is called a compute node. Each compute node typically includes multiple GPUs (graphics processing units) for performing model calculations. GPUs are also called computing devices. A process controls the operation of a GPU. Model operator tasks can be assigned to each process and processed by the corresponding GPU.

[0034] The model computation graph includes many types of operators that conform to the model's operating logic, such as arithmetic operators, activation function operators, pooling operators, convolution operators, normalization operators, concatenation and splitting operators, and loop operators. Various communication operators can also be included to enable tensor transmission between different computing devices. The computation graph characterizes the input and output flow of tensors between operators, and annotates the input and output tensors of each operator with distributed attributes so that the operator's computational tasks can be processed in parallel on different computing devices. The input tensor includes the sample data to be processed by the model and its converted feature data, as well as the weight parameter matrix required for the operator to run. The output tensor processed by the operator can be used as the output result or as the input tensor of the next operator.

[0035] The same model and distributed system can be designed to operate in different distributed modes as needed. These modes can be distinguished and identified by the distributed properties of tensors. These properties include the process mesh (referred to as mesh) and placements, providing a unified abstract representation for different distributed operation modes.

[0036] The process topology of a tensor indicates which processes execute it, that is, which computing devices process it. Therefore, the process topology reflects the organization of different processes. Computing devices in a distributed system can be partitioned along one or more dimensions, and tensors can also be partitioned along the same dimensions to be assigned to corresponding processes. For example, a distributed system includes two processes, process 0 and process 1, supported by two computing devices. If an input tensor of an operator needs to be processed by both processes, the tensor's process topology is represented as [0, 1]. This indicates that the operator's computation is executed on both processes.

[0037] The tensor distribution of a tensor is used to represent the distribution of the tensor in a specified process topology. Tensor distribution generally includes three types: Replicate, which means that the tensor remains in its full state across all processes in the process topology; Shard (axis), which means that the tensor is split along the axis dimension and placed on different processes for processing. If the process topology of the tensor is process mesh = [0, 1] and placements = Shard (0), it means that the two parts of the tensor after being split by row (the 0th dimension) are provided to process 0 and process 1 respectively; Partial, which means that each process only has a partial value of the tensor and requires a specified reduction operation to restore it to full data. If placements = Partial (0), it means that after the tensor is split by row, the split tensor values ​​are provided to the corresponding process, and the rest of the tensor is padded with the value 0. The tensor matrix obtained by each process is still complete, but the values ​​are not complete. The split dimension of the tensor and the process can be the same, which can be one dimension, or two or more dimensions. For example, one dimension may be a row or a column, and two dimensions may include a matrix block divided by rows and columns. This embodiment of the present disclosure is not limited to this.

[0038] The distributed operation mode of a model is actually reflected in the distributed processing of operators. This means that the input tensors of any operator are split among processes on different computing devices for execution, with each process performing a portion of the operator's computational tasks. Because the computational logic of each operator is different, there are requirements for how the tensors are split; not all splits can be distributed to different processes for correct computation. Furthermore, to ensure the accuracy of the computational results of the entire model, operators may need to insert some communication operations into the results after performing distributed computations. This involves communicating the output tensors of multiple processes, performing operations such as collection or aggregation, and then transmitting them to subsequent operators for computation. Therefore, by determining the process topology and tensor distribution of each tensor, the distributed operation mode of the computation graph can be determined.

[0039] Manually adding attributes for all tensors in a model's computational graph would be a cumbersome task. Therefore, users typically only need to mark the process topology and tensor distribution for some operators and some tensors, and then use automatic parallelization. Specifically, the tensor attributes of other operators are inferred based on the inter-operator logic represented in the computational graph, and communication operators are automatically inserted, enabling distributed execution of large models without having to consider details such as distribution strategies and communication between different computing nodes.

[0040] The MoE model calculation graph includes multiple expert models. Each expert model can be regarded as a collection of operators to realize its own model processing function. In addition to the operator set of the expert model, it also includes other functional operators such as the gated network operator set.

[0041] A current challenge is how to use automatic parallelization technology to support the distributed operation of MoE models. MoE models can deploy different expert models to run on different processes, a process called expert model parallelism. Deploying each expert model to a different process essentially sets the process topology for the input tensors of the expert model operators to a single process, so that the expert model operators are processed only in that process.

[0042] The complexity of labeling tensor attributes in this mode lies in the following: if the process topology and distribution of tensors are manually labeled, the workload of manual labeling is relatively large and the requirements for personnel skills are also relatively high; in automatic parallelization technology, automatic parallelization can only be performed when the input tensors of the operator have the same process topology. Therefore, a feasible way to implement automatic parallelization technology based on the computation graph of the MoE model includes:

[0043] Step 1: Put the weight parameters of all expert models into the same tensor, which can be called the complete weight tensor;

[0044] Step 2: Set the process topology and tensor distribution for the complete weight tensor, which means distributing the weight tensors of different expert models to different processes.

[0045] Step 3: Set the process topology and distribution properties of the input data tensor. The input data is generally global and should be provided to each expert model. Then, continue to run the hybrid expert model in parallel.

[0046] For any expert model's operator set, the process topology must be identical to enable automatic parallelization within the expert model's operator set and to derive tensor properties for subsequent operators. However, if the process topologies of the input tensors of the expert model's operator set differ, automatic parallelization becomes difficult. For example, if the process topology of the input data tensor of any expert model is all processes, the process topology of the expert model's weight tensor must also be all processes for the expert model's operator set to be able to execute automatic parallelization. Therefore, for this expert model's operator set, the process topologies of the two input tensors are different.

[0047] However, the aforementioned technology needs improvement. To ensure that the weight tensors input to the expert model operator set are global, the weight parameters of all expert models must be unified into a single matrix for processing. However, if the expert models have different architectures or significantly different parameter sizes, unification into a single global matrix is ​​practically difficult. Therefore, the aforementioned automatic parallelization technology's requirement for identical process topologies places limitations on hybrid expert models.

[0048] The technical solution of the embodiment of the present disclosure provides an optimization technology that reduces the architectural restrictions on the expert model and can perform automatic parallel optimization on the hybrid expert model based on the model architecture characteristics of the hybrid expert model.

[0049] Figure 1 This is a flowchart of an automatic parallel method for a hybrid expert model provided according to an embodiment of the present disclosure. The method is applicable to the case where distributed operation is achieved based on the automatic parallel technology for the computation graph of the hybrid expert model, and the properties of the tensor are derived by the automatic parallel technology, so that each operator can be processed in parallel in the distributed system. The method can be executed using an automatic parallel device for the hybrid expert model, which can be implemented in the form of hardware and / or software and can be configured in an electronic device with computing and storage capabilities, typically a distributed system that can support the distributed operation of the model, or an electronic device that controls the distributed operation. The method includes:

[0050] S110, obtaining a computation graph of a hybrid expert model; wherein the hybrid expert model includes a gating network and at least two expert models;

[0051] S120: Determine a process topology of an expert weight tensor of an expert model; wherein at least one process included in the process topology of the expert weight tensor is a process that is part of a distributed system for running the hybrid expert model; the process is supported and run by a computing device in the distributed system; as described above, processes and computing devices are set in a one-to-one correspondence;

[0052] S130: Split the global data tensor into sub-data tensors required by the corresponding expert model, and set the process topology of the sub-data tensors to be the same as the process topology of the corresponding expert model; wherein the data in the sub-data tensors is the data required to be processed by the corresponding expert model;

[0053] S140, processing the input sub-data tensor and the expert weight tensor by the expert model to output a sub-result tensor;

[0054] S150. Determine a result tensor of the hybrid expert model according to at least one sub-result tensor.

[0055] As introduced in the foregoing related art, the computational graph of the hybrid expert model can be acquired first. The hybrid expert model can include a gating network and at least two expert models, and can also include other functional modules as needed. The gating network is used to determine at least one expert model to be activated for the input data in this run, and determine the part of the input data to be processed by each expert model. For example, the gating network calculates which expert models should process which input data. For the input data at the beginning, the gating network can output a tensor of [E, C, H], where E represents the number of expert models, C represents the number of data to be processed by each expert model, and H is the vector size of each data. Therefore, according to the E dimension, the data required by each expert model can be split to the operator set of the expert model for processing. For the expert model that does not need to process data, the value of the data in the corresponding area is usually 0, which is regarded as the expert model will not be provided with data for processing. How the gating network determines which expert model to process the data and the architecture of each expert model is not limited in the embodiments of the present disclosure, and is determined by the specific model function and training purpose.

[0056] For each operator set of the expert model, the operator set needs at least the expert weight and the input data to be processed, that is, the expert weight tensor and the data tensor need to be acquired. In the embodiments of the present disclosure, the matrix dimensions of the expert weight tensor of each expert model can be the same or different, which is determined by the architecture of the expert model itself. In the process of automatic parallelism, the process topology of the expert weight tensor of at least one expert model is set to include only part of the processes. Specifically, for all processes provided by the distributed system, the process topology of the expert weight tensor of at least one expert model only includes part of the processes. Optionally, different process topologies can be set for different expert models. According to the running requirements of the expert model, the process topology of the expert weight tensor actually can optionally include one or more processes of the distributed system, and the processes included in the process topologies of different expert weight tensors are partially the same or all the same. It can be seen that the deployment mode of the expert model on the process can be various, but it is not necessary to ensure that the weight tensors of each expert model are unified into a complete weight tensor, and the process topology of the complete weight tensor includes all the processes. Without this limitation, the architecture of each expert model can be the same or different, and the deployment of the expert model on the process is more flexible. For example, each expert model has the condition of parallel independent execution, so it is preferred that the process topology of each expert weight tensor includes one process, and the process topologies of the expert weight tensors are different from each other. Thus, the distributed running model of the expert model in the distributed system has greater flexibility.

[0057] The process topology of the expert weight tensor for each expert model determines the distributed operation mode. The process topology can be set according to pre-set rules and model operation requirements. Optionally, determining the process topology of the expert weight tensor for an expert model may include obtaining the manually annotated process topology of the expert weight tensor for the expert model. This allows for manual annotation of the process topology of the expert weight tensor, strengthening model management personnel's control over the model's operation mode. Even with the manually specified distribution of expert models across processes, automatic parallelization techniques can still be used to run the model, eliminating the need for extensive manual annotation of other tensor attributes.

[0058] Although the expert weight tensor of the expert model can be of different process topologies, for the sample data to be processed of the model, as input data, it is usually processed by a set of gated network operators in a global state, which can be called a global data tensor. After the gated network operator set processes the input data, a global data tensor is obtained, which is provided to each process for processing. In order to achieve automatic parallelization of the expert model, it is necessary to make the process topology of the global data tensor consistent with that of the expert weight tensor. Therefore, it is necessary to split the global data tensor into sub-data tensors required by the corresponding expert model, and set the process topology of the sub-data tensor to be the same as the process topology of the corresponding expert model. Among them, the data in the sub-data tensor is the data required to be processed by the corresponding expert model. Optionally, the relationship between the sub-data in the global data tensor and the corresponding expert model is determined by the gating network.

[0059] The expert model then processes the input sub-data tensors and expert weight tensors. Because the sub-data tensors and expert weight tensors share the same process topology, automatic parallelization techniques can be implemented to obtain the sub-result tensors output by the expert model. Finally, the hybrid expert model determines the final result tensor based on the sub-result tensors output by one or more expert models. The hybrid expert model can apply additional functional operators to each sub-result tensor for post-processing, thereby forming the final result tensor.

[0060] The technical solution of the embodiment of the present disclosure, in the process of executing the automatic parallelization technology for the computation graph of the hybrid expert model, realizes that the process topology of the expert weight tensor and the sub-data tensor of the expert model is the same, and the process topology does not have to be limited to all processes of the distributed system. This can not only realize the automatic parallelization of the expert model, but also reduce the consistency requirements for the architecture of each expert model. It also does not need to manually label the attributes of a large number of tensors, making the model function of the hybrid expert model more flexible, realizing automatic parallelization across process topologies, and supporting the parallel operation of expert models with different architectures, which brings computational advantages to both model training and derivation processes. As a result, it also enables the computing device that hosts the process to flexibly realize the distributed operation of the model across process topologies; the operation of the hybrid expert model can give full play to the hardware advantages of the distributed system, allowing each expert model to be deployed on one or more computing devices according to the needs of the model operation, without being restricted by the constraints of the automatic parallelization technology. For example, expert models with different model architectures can be deployed completely independently on different computing devices. The technical solution of the embodiment of the present disclosure can be widely applied to scenarios such as deep learning systems, distributed deep learning systems, machine learning platforms, and distributed training platforms to realize the automatic parallel operation of hybrid expert models.

[0061] Figure 2 This is a flowchart of an automatic parallel method for a hybrid expert model provided according to an embodiment of the present disclosure. Figure 3 This is a schematic diagram of the tensor data conversion process in the embodiment of the present disclosure. This embodiment, based on the previous embodiment, provides a specific implementation method for changing the process topology of a tensor. Specifically, the global data tensor is split into sub-data tensors required by the corresponding expert model. This can be achieved by adjusting the tensor distribution of the sub-data as needed and then setting the process topology. The method includes:

[0062] S210, obtaining a computational graph of a hybrid expert model; wherein the hybrid expert model includes a gating network and at least two expert models;

[0063] S220: Determine a process topology of an expert weight tensor of an expert model; wherein at least one process included in the process topology of the expert weight tensor is a partial process of a distributed system for running the hybrid expert model;

[0064] S231, converting the tensor distribution of the global data tensor according to the relationship between the sub-data in the global data tensor and the corresponding expert model, so that each sub-data can be segmented and processed by the corresponding expert model;

[0065] S232. Obtain the segmented sub-data from the global data tensor to form a sub-data tensor, and set the process topology of the sub-data tensor to be the same as the process topology of the expert weight tensor of the corresponding expert model; wherein the data in the sub-data tensor is the data required to be processed by the corresponding expert model;

[0066] S240, processing the input sub-data tensor and the expert weight tensor by the expert model to output a sub-result tensor;

[0067] S250. Determine a result tensor of the hybrid expert model according to at least one sub-result tensor.

[0068] Specifically, the process of splitting the global data tensor into sub-data tensors can include a tensor distribution transformation and a process topology adjustment. The tensor distribution transformation can allocate the sub-data to the processes corresponding to the expert model. The split sub-data are then formed into independent sub-data tensors, and their process topology is set to be the same as the corresponding expert model process topology.

[0069] The following is a detailed description with reference to the accompanying drawings. Figure 3 As shown, the original process topology of the global data tensor includes all processes. For example, two processes, process 0 and process 1, are used as the entire process. The original process topology of the global data tensor is process mesh = [0, 1]. The data in the global data tensor is provided to each process, so the original tensor distribution is in a replicated state, that is, placements = Replicate().

[0070] According to the relationship between the sub-data in the global data tensor and the corresponding expert model, the copy state of the tensor distribution of the global data tensor is converted into a split state, so that each sub-data can be split into the corresponding expert model for processing. Figure 3 As shown, assuming that after splitting by rows, the sub-data of the first row portion is provided to the expert model 0 supported by process 0, and the sub-data of the second row portion is provided to the expert model 1 supported by process 1, the tensor distribution of the global data tensor is set to the split state by rows, that is, placements = Shard(0). Preferably, the split dimension in the split state is the same as the process division dimension of the corresponding expert model. The split dimension can be split by rows or by columns, or it can be split into local blocks by rows and columns at the same time. The embodiment of the present disclosure does not limit the split dimension, as long as the sub-data tensor can be split to the process of the corresponding expert model.

[0071] Although the tensor distribution state of the global data tensor has changed, it still needs to be split into independent sub-data tensors according to the split state. Figure 3As shown, based on the state of splitting by row, the splitting is into two independent sub-data tensors.

[0072] Further, after splitting into a sub-data tensor, the sub-data tensor is provided to the process corresponding to the expert model, so the process topology of the sub-data tensor is adjusted again to be consistent with the process topology of the corresponding expert model. As shown, the process topology of the sub-data tensor 0 of the expert model 0 is mesh=[0], and the process topology of the sub-data tensor 1 of the expert model 1 is mesh=[1]. Figure 3

[0073] On this basis, the tensor distribution of the sub-data tensor can be further derived and updated to a replication state or a split state according to the process topology of the sub-data tensor. Specifically, the sub-data tensor is provided to the corresponding expert model, and the tensor distribution of the sub-data tensor is determined according to the process running by the operator set of the expert model. For example, if the expert model only runs on one process, the sub-data tensor is provided to one process, and the tensor distribution of the sub-data tensor can be in a replication state. If the expert model runs on multiple processes, the sub-data tensor needs to be in a replication state or a split state, so as to be able to provide multiple processes. The tensor distribution state of the sub-data tensor is determined by the running logic of the operator set of the expert model, and is automatically derived and determined in the automatic parallel process.

[0074] The technical solutions of the above embodiments achieve splitting of the global data tensor, ensure the consistency of the process topology of the sub-data tensor and the expert weight tensor, and thus guarantee the automatic parallel execution of the hybrid expert model, by tensor distribution conversion of the global data tensor and process topology setting, in a simple adjustment manner.

[0075] On the basis of the above technical solutions, the output sub-result tensor can be further processed. Preferably, according to at least one sub-result tensor, the result tensor of the hybrid expert model is determined to include:

[0076] The sub-result tensors output by at least two expert models are aggregated to form a global result tensor, and the process topology of the global result tensor is modified to be the same as the process topology of the global data tensor;

[0077] According to the original output process of each sub-result tensor in the global result tensor, the tensor distribution of the global result tensor is set to a split state.

[0078] ​Typically, the sub-result tensor output by each expert model is consistent with the process topology of the input sub-data tensor. However, the process topology of the sub-data tensor is inconsistent with the process topology of the global data tensor. This may affect the subsequent processing of the data results. That is, after the expert model outputs the result, the hybrid expert model may also perform other post-processing operations on the output results of each expert model. The specific post-processing operation functions are not limited in the embodiments of this disclosure, but the global unified post-processing operation of the hybrid expert model requires the process topology of the result tensor to be global. Therefore, it is preferred to restore the process topology of the sub-result tensor to be consistent with the process topology of the global data tensor.

[0079] Specifically, the specific means adopted in this embodiment are as follows: Figure 3 As shown, the two sub-result tensors are first aggregated to form a global result tensor. At this time, the process topology of the global result tensor is modified to be the same as the process topology of the global data tensor. For example, the process topology of sub-result tensor 0 output by expert model 0 is mesh = [0], and the process topology of sub-result tensor 1 output by expert model 1 is mesh = [1]. The process topology of the global result tensor formed after the two are merged is mesh = [0, 1]. Since the original output processes of different sub-result tensors are different, the tensor distribution of the global result tensor is set to a shard state, that is, placements = Shard (0), to indicate the relationship between the sub-results and the corresponding expert models.

[0080] At this point, the global result tensor already has a global process topology and can continue to be automatically parallelized by subsequent processing operators. Optionally, it also includes: according to the result output requirements of the hybrid expert model, changing the tensor distribution of the global result tensor from a split state to a copy state. If the subsequent processing operator of the hybrid expert model needs to provide the global result tensor to each process for processing, the tensor distribution of the global result tensor can be changed from a split state to a copy state. If the result output requirement of the hybrid expert model is to output the data of the sub-result tensors separately, the tensor distribution of the global result tensor can remain in a split state. The specific state of the tensor distribution can be flexibly determined by the output requirements of the hybrid expert model.

[0081] Figure 4 This is a schematic diagram of the structure of an automatic parallel device for a hybrid expert model provided in an embodiment of the present disclosure, the device comprising:

[0082] A computation graph acquisition module 410 is configured to acquire a computation graph of a hybrid expert model, wherein the hybrid expert model includes a gating network and at least two expert models;

[0083] The weight tensor process determination module 420 is configured to determine a process topology of an expert weight tensor of an expert model; wherein at least one process included in the process topology of the expert weight tensor is a process that is part of a distributed system for running the hybrid expert model; and the process is supported and run by a computing device in the distributed system;

[0084] The data tensor process splitting module 430 is used to split the global data tensor into sub-data tensors required by the corresponding expert model, and set the process topology of the sub-data tensors to be the same as the process topology of the corresponding expert model; wherein the data in the sub-data tensors is the data required to be processed by the corresponding expert model;

[0085] A sub-result output module 440 is configured to process the input sub-data tensor and the expert weight tensor through the expert model to output a sub-result tensor;

[0086] The result tensor determination module 450 is configured to determine the result tensor of the hybrid expert model according to at least one sub-result tensor.

[0087] The technical solution of the disclosed embodiment, in the process of executing automatic parallel technology on the calculation graph of the hybrid expert model, realizes that the process topology of the expert weight tensor and the sub-data tensor of the expert model is made the same, and the process topology does not have to be limited to all processes of the distributed system. This can not only realize the automatic parallelization of the expert model, but also reduce the consistency requirements of each expert model architecture. There is no need to manually label the attributes of a large number of tensors, making the model function of the hybrid expert model more flexible, realizing automatic parallelization across process topologies, and being able to support the parallel operation of expert models of different architectures, which brings advantages in computational complexity for both model training or derivation processes.

[0088] Based on the above technical solution, optionally, the process topology of the expert weight tensor in the device includes one or more processes of the distributed system, and the processes included in the process topologies of different expert weight tensors are partially or completely the same.

[0089] Optionally, the process topology of each expert weight tensor in the device includes a process, and the process topologies of the expert weight tensors are different from each other.

[0090] Optionally, the weight tensor process determination module in the device is specifically used to obtain the process topology of the expert weight tensor of the manually labeled expert model.

[0091] Optionally, the data tensor process splitting module in the device includes:

[0092] a sub-data segmentation unit, configured to convert the tensor distribution of the global data tensor according to the relationship between the sub-data in the global data tensor and the corresponding expert model, so that each sub-data can be segmented into the corresponding expert model for processing;

[0093] The process setting unit is used to obtain the segmented sub-data from the global data tensor to form a sub-data tensor, and set the process topology of the sub-data tensor to be the same as the process topology of the expert weight tensor of the corresponding expert model.

[0094] Optionally, the neutron data segmentation unit of the device is specifically used to:

[0095] According to the relationship between the sub-data in the global data tensor and the corresponding expert model, the copy state of the tensor distribution of the global data tensor is converted into a split state, so that each sub-data can be split into the corresponding expert model for processing.

[0096] Optionally, the data tensor process splitting module in the device further includes:

[0097] The tensor distribution updating unit is used to obtain the segmented sub-data to form a sub-data tensor, and then update the tensor distribution of the sub-data tensor to a copy state or a segmentation state according to the process topology of the sub-data tensor.

[0098] Optionally, the segmentation dimension in the segmentation state in the device is the same as the process division dimension of the corresponding expert model.

[0099] Optionally, the relationship between the sub-data in the global data tensor and the corresponding expert model in the device is determined by the gating network.

[0100] Optionally, the result tensor determination module in the device includes:

[0101] a sub-result aggregation unit, configured to aggregate the sub-result tensors output by at least two expert models to form a global result tensor, and modify the process topology of the global result tensor to be the same as the process topology of the global data tensor;

[0102] The result distribution setting unit is used to set the tensor distribution of the global result tensor to a split state according to the original output process of each sub-result tensor in the global result tensor.

[0103] Optionally, the device further includes:

[0104] The global result tensor distribution setting module is used to change the tensor distribution of the global result tensor from a split state to a copy state according to the result output requirements of the hybrid expert model.

[0105] The automatic parallel device of the hybrid expert model provided in the embodiment of the present disclosure is used to implement the automatic parallel method of the hybrid expert model provided in the embodiment of the present disclosure, and has corresponding functional modules and beneficial effects.

[0106] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0107] According to an embodiment of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium, and a computer program product. The electronic device is optionally used to control a distributed system or is a computer in a distributed system, capable of controlling the operation of each computing device in the distributed system.

[0108] Figure 5 A schematic block diagram of an example electronic device 500 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0109] like Figure 5 As shown, the device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. Various programs and data required for the operation of the device 500 can also be stored in the RAM 503. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0110] Various components in device 500 are connected to I / O interface 505, including: an input unit 506, such as a keyboard, mouse, etc.; an output unit 507, such as various types of displays, speakers, etc.; a storage unit 508, such as a magnetic disk, optical disk, etc.; and a communication unit 509, such as a network card, modem, wireless communication transceiver, etc. The communication unit 509 allows device 500 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0111] The computing unit 501 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above, such as the automatic parallelization method for the hybrid expert model. For example, in some embodiments, the automatic parallelization method for the hybrid expert model can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 508. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 500 via the ROM 502 and / or the communication unit 509. When the computer program is loaded into the RAM 503 and executed by the computing unit 501, one or more steps of the automatic parallelization method for the hybrid expert model described above can be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to execute the automatic parallel method of the hybrid expert model in any other appropriate manner (for example, by means of firmware).

[0112] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0113] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0114] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0115] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0116] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0117] A computer system may include a client and a server. The client and server are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services. The server may also be a server in a distributed system or a server integrated with blockchain.

[0118] Artificial intelligence (AI) is the study of how computers can simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It encompasses both hardware and software technologies. AI hardware technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, and big data processing. AI software technologies primarily encompass computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graphs.

[0119] Cloud computing refers to a technology system that provides network access to elastically scalable shared pools of physical or virtual resources. These resources can include servers, operating systems, networks, software, applications, and storage devices, and can be deployed and managed on-demand in a self-service manner. Cloud computing technology provides efficient and powerful data processing capabilities for the application of technologies such as artificial intelligence and blockchain, as well as for model training.

[0120] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions provided by this disclosure can be achieved. This is not a limitation herein.

[0121] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. An automatic parallel method for a hybrid expert model, the method comprising: Obtaining a computational graph of a hybrid expert model; wherein the hybrid expert model includes a gating network and at least two expert models; Determining a process topology of an expert weight tensor of an expert model; wherein at least one process included in the process topology of the expert weight tensor is a part of a distributed system for running the hybrid expert model; and the process is supported and run by a computing device in the distributed system; According to the relationship between the sub-data in the global data tensor and the corresponding expert model, the tensor distribution of the global data tensor is converted so that each sub-data can be segmented and processed by the corresponding expert model; Obtaining the segmented sub-data from the global data tensor to form a sub-data tensor, and setting the process topology of the sub-data tensor to be the same as the process topology of the expert weight tensor of the corresponding expert model; wherein the data in the sub-data tensor is the data required to be processed by the corresponding expert model; The expert model processes the input sub-data tensor and the expert weight tensor to output a sub-result tensor; A result tensor of the hybrid expert model is determined according to at least one sub-result tensor.

2. The method according to claim 1, wherein The process topology of the expert weight tensor includes one or more processes of the distributed system, and the processes included in the process topologies of different expert weight tensors are partially or completely the same.

3. The method according to claim 2, wherein: The process topology of each expert weight tensor includes a process, and the process topologies of the expert weight tensors are different from each other.

4. The method according to claim 1, wherein The process topology for determining the expert weight tensor of the expert model includes: The process topology for obtaining the expert weight tensor of a manually labeled expert model.

5. The method according to claim 1, wherein According to the relationship between the sub-data in the global data tensor and the corresponding expert model, converting the tensor distribution of the global data tensor so that each sub-data can be segmented into the corresponding expert model for processing includes: According to the relationship between the sub-data in the global data tensor and the corresponding expert model, the copy state of the tensor distribution of the global data tensor is converted into a split state, so that each sub-data can be split into the corresponding expert model for processing.

6. The method according to claim 5, after obtaining the segmented sub-data to form a sub-data tensor, further comprising: According to the process topology of the sub-data tensor, the tensor distribution of the sub-data tensor is updated to a copy state or a split state.

7. The method according to claim 5, wherein: The segmentation dimension in the segmentation state is the same as the process division dimension of the corresponding expert model.

8. The method according to claim 1, wherein The relationship between the sub-data in the global data tensor and the corresponding expert model is determined by the gating network.

9. The method according to claim 1, wherein Determining a result tensor of the hybrid expert model according to at least one sub-result tensor includes: Aggregating the sub-result tensors output by at least two expert models to form a global result tensor, and modifying the process topology of the global result tensor to be the same as the process topology of the global data tensor; According to the original output process of each sub-result tensor in the global result tensor, the tensor distribution of the global result tensor is set to a split state.

10. The method according to claim 9, further comprising: According to the result output requirement of the hybrid expert model, the tensor distribution of the global result tensor is changed from a split state to a copy state.

11. An automatic parallel device for a hybrid expert model, comprising: A computation graph acquisition module, configured to acquire a computation graph of a hybrid expert model, wherein the hybrid expert model includes a gating network and at least two expert models; a weight tensor process determination module, configured to determine a process topology of an expert weight tensor of an expert model; wherein at least one process included in the process topology of the expert weight tensor is a part of a distributed system for running the hybrid expert model; and the process is supported and run by a computing device in the distributed system; A data tensor process splitting module is used to split the global data tensor into sub-data tensors required by the corresponding expert model, and set the process topology of the sub-data tensor to be the same as the process topology of the corresponding expert model; wherein the data in the sub-data tensor is the data required to be processed by the corresponding expert model; A sub-result output module, configured to process the input sub-data tensor and the expert weight tensor through the expert model to output a sub-result tensor; A result tensor determination module, configured to determine a result tensor of the hybrid expert model according to at least one sub-result tensor; Among them, the data tensor process splitting module includes: a sub-data segmentation unit, configured to convert the tensor distribution of the global data tensor according to the relationship between the sub-data in the global data tensor and the corresponding expert model, so that each sub-data can be segmented into the corresponding expert model for processing; The process setting unit is used to obtain the segmented sub-data from the global data tensor to form a sub-data tensor, and set the process topology of the sub-data tensor to be the same as the process topology of the expert weight tensor of the corresponding expert model.

12. The device according to claim 11, wherein The process topology of the expert weight tensor includes one or more processes of the distributed system, and the processes included in the process topologies of different expert weight tensors are partially or completely the same.

13. The device according to claim 12, wherein The process topology of each expert weight tensor includes a process, and the process topologies of the expert weight tensors are different from each other.

14. The device according to claim 11, wherein The weight tensor process determination module is specifically used to obtain the process topology of the expert weight tensor of the manually labeled expert model.

15. The device according to claim 11, wherein The sub-data segmentation unit is specifically used for: According to the relationship between the sub-data in the global data tensor and the corresponding expert model, the copy state of the tensor distribution of the global data tensor is converted into a split state, so that each sub-data can be split into the corresponding expert model for processing.

16. The apparatus according to claim 15, wherein the data tensor process splitting module further comprises: The tensor distribution updating unit is used to obtain the segmented sub-data to form a sub-data tensor, and then update the tensor distribution of the sub-data tensor to a copy state or a segmentation state according to the process topology of the sub-data tensor.

17. The device according to claim 15, wherein The segmentation dimension in the segmentation state is the same as the process division dimension of the corresponding expert model.

18. The device according to claim 11, wherein The relationship between the sub-data in the global data tensor and the corresponding expert model is determined by the gating network.

19. The device according to claim 11, wherein The result tensor determination module includes: a sub-result aggregation unit, configured to aggregate the sub-result tensors output by at least two expert models to form a global result tensor, and modify the process topology of the global result tensor to be the same as the process topology of the global data tensor; The result distribution setting unit is used to set the tensor distribution of the global result tensor to a split state according to the original output process of each sub-result tensor in the global result tensor.

20. The apparatus according to claim 19, further comprising: The global result tensor distribution setting module is used to change the tensor distribution of the global result tensor from a split state to a copy state according to the result output requirements of the hybrid expert model.

21. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the automatic parallel method for the hybrid expert model according to any one of claims 1 to 10.

22. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to enable the computer to execute the automatic parallel method of the hybrid expert model according to any one of claims 1-10.

23. A computer program product comprising a computer program / instruction, which, when executed by a processor, implements the automatic parallel method of the hybrid expert model according to any one of claims 1 to 10.

Images