Cloud management platform-based model training method and cloud management platform

Abstract

The present application discloses a cloud management platform-based model training method and a cloud management platform, which can satisfy model training requirements of tenants. The method in the present application comprises: when a tenant requires model training, the tenant can send, to a model training interface provided by a cloud management platform, a model training task, a first training policy needing to be followed when a plurality of clusters jointly execute the model training task, and a second training policy needing to be followed when a single cluster executes the model training task; then, on the basis of the first training policy and the second training policy, the cloud management platform can divide the model training task into a first subtask needing to be executed by a first cluster and a second subtask needing to be executed by a second cluster; and then, the cloud management platform creates the first cluster and the second cluster in an infrastructure, and notifies the first cluster to execute the first subtask and notifies the second cluster to execute the second subtask, thereby completing the model training task formulated by the tenant.

Description

A model training method based on a cloud management platform and the cloud management platform

[0001] The present application claims priority to the Chinese Patent Application No. 202410804374.2, filed on June 20, 2024, entitled "A Model Training Method Based on a Cloud Management Platform and the Cloud Management Platform", and to the Chinese Patent Application No. 202411098708.5, filed on August 9, 2024, entitled "A Model Training Method Based on a Cloud Management Platform and the Cloud Management Platform", both of which are incorporated herein by reference in their entirety. TECHNICAL FIELD

[0002] Embodiments of the present application relate to the field of cloud technology, and in particular to a model training method based on a cloud management platform and the cloud management platform. BACKGROUND

[0003] With the rapid development of cloud technology, more and more tenants choose to use the clusters provided by cloud vendors to complete the training of neural network models. Since a cluster is usually composed of multiple computing nodes, each computing node has a certain specification of resources, so the cluster has sufficient resources and performance to successfully complete the tenant's model training task.

[0004] In related technologies, when a tenant needs to train a certain model, the cloud management platform can create a dedicated cluster for the tenant according to the tenant's requirements, and notify the cluster to train the to-be-trained model according to a series of requirements set by the tenant, thereby obtaining a trained model, and return the trained model to the tenant for use, thereby meeting the tenant's model training needs.

[0005] However, when the scale of the model to be trained by the tenant is large, the performance of a single cluster is limited, and it cannot carry the training task of a large-scale neural network model. Therefore, how to complete model training across clusters has become a technical problem to be solved. SUMMARY

[0006] Embodiments of the present application provide a model training method based on a cloud management platform and the cloud management platform, which can call several clusters with sufficient performance to carry the tenant's model training task regardless of the size of the tenant's model training task, thereby meeting the tenant's model training needs.

[0007] A first aspect of embodiments of the present application provides a model training method based on a cloud management platform, and a cloud management platform for implementing the method can manage the infrastructure providing cloud services for tenants. The method comprises:

[0008] When the tenant has a model training requirement, the cloud management platform can provide the tenant with a model training interface. Then, the tenant can input a model training task set by the tenant, a first training strategy that needs to be followed by multiple clusters to jointly execute the model training task, and a second training strategy that needs to be followed by a single cluster to execute the model training task into the model training interface. In this way, the cloud management platform can receive the model training task, the first training strategy, and the second training strategy sent by the tenant through the model training interface.

[0009] After obtaining the model training task, the first training strategy, and the second training strategy, the cloud management platform can split the model training task into a first subtask to be executed by a first cluster and a second subtask to be executed by a second cluster according to the indications of the first training strategy and the second training strategy, wherein the first cluster is set in a site located at a first geographic location, and the second cluster is set in a site located at a second geographic location, i.e., the first cluster and the second cluster are located in different sites.

[0010] After obtaining the first subtask and the second subtask, the cloud management platform can create the first cluster and the second cluster in the infrastructure, and schedule the first subtask to the first cluster and the second subtask to the second cluster, so that the first cluster executes the first subtask and the second cluster executes the second subtask, thereby completing the model training task.

[0011] As can be seen from the above method, the cloud management platform can split the model training task set by the tenant into subtasks to be executed by clusters according to the cross-cluster training strategy (i.e., the first training strategy) and the intra-cluster training strategy (i.e., the second training strategy) set by the tenant, and notify the clusters to execute the corresponding subtasks, thereby realizing the joint completion of the model training task by the clusters and realizing the cross-cluster execution of the model training. As can be seen, no matter how large the model training task is, the cloud management platform can call several clusters with sufficient performance to carry the model training task of the tenant, thereby meeting the model training requirement of the tenant.

[0012] In a possible implementation, since the model training task is used to instruct training of the to-be-trained model, when the first training strategy is in a data parallel manner and the second training strategy is in a model parallel manner, the first subtask assigned to the first cluster can be used to instruct the first cluster to train the to-be-trained model in an internal model parallel manner, and the first subtask can also be used to instruct the first cluster to train the to-be-trained model in a data parallel manner in combination with the second cluster. The second subtask assigned to the second cluster can be used to instruct the second cluster to train the to-be-trained model in an internal model parallel manner, and the second subtask can also be used to instruct the second cluster to train the to-be-trained model in a data parallel manner in combination with the first cluster.

[0013] In a possible implementation, since the model training task is used to instruct training of the to-be-trained model, when the first training strategy is in a model parallel manner and the second training strategy is in a data parallel manner, the first subtask assigned to the first cluster can be used to instruct the first cluster to train the to-be-trained model in an internal data parallel manner, and the first subtask can also be used to instruct the first cluster to train the to-be-trained model in a model parallel manner in combination with the second cluster. The second subtask assigned to the second cluster can be used to instruct the second cluster to train the to-be-trained model in an internal data parallel manner, and the second subtask can also be used to instruct the second cluster to train the to-be-trained model in a model parallel manner in combination with the first cluster.

[0014] In a possible implementation, since the model training task is used to instruct training of the to-be-trained model, when the first training strategy and the second training strategy are both in an expert parallel manner, the first subtask assigned to the first cluster can be used to instruct the first cluster to train the to-be-trained model in an internal expert parallel manner, and the first subtask can also be used to instruct the first cluster to train the to-be-trained model in an expert parallel manner in combination with the second cluster. The second subtask assigned to the second cluster can be used to instruct the second cluster to train the to-be-trained model in an internal expert parallel manner, and the second subtask can also be used to instruct the second cluster to train the to-be-trained model in an expert parallel manner in combination with the first cluster.

[0015] In a possible implementation, the model training task is used to instruct training of a to-be-trained model based on the first data set and the second data set, the to-be-trained model including a first sub-model and a second sub-model, and the cloud management platform notifying the first cluster to perform the first sub-task and notifying the second cluster to perform the second sub-task includes: the cloud management platform notifying the first cluster to sequentially process the first data set through the first sub-model and the second sub-model to obtain a first processing result, and updating parameters of the first sub-model and parameters of the second sub-model based on the first processing result to obtain a first model; the cloud management platform notifying the second cluster to sequentially process the second data set through the first sub-model and the second sub-model to obtain a second processing result, and updating the parameters of the first sub-model and the parameters of the second sub-model based on the second processing result to obtain a second model; and the cloud management platform notifying the first cluster and the second cluster to aggregate the first model and the second model to obtain a third model. In the foregoing implementation, when the first sub-task is performed, the first cluster can sequentially process the first data set through the first sub-model and the second sub-model split from the to-be-trained model to obtain the first processing result, and update the parameters of the first sub-model and the parameters of the second sub-model based on the first processing result to obtain the first model, and send the first model to the second cluster. When the second sub-task is performed, the second cluster can sequentially process the second data set through the first sub-model and the second sub-model to obtain the second processing result, and update the parameters of the first sub-model and the parameters of the second sub-model based on the second processing result to obtain the second model, and send the second model to the first cluster. Subsequently, the first cluster can aggregate the first model and the second model with the second cluster to obtain the third model. Based on the third model, the first cluster and the second cluster can finally obtain the trained model, thereby meeting the model training requirement of the tenant.

[0016] In a possible implementation, the model training task is used to instruct training of a to-be-trained model based on a data set, the to-be-trained model includes a first sub-model and a second sub-model, the data set includes a first data set and a second data set, and the cloud management platform instructing the first cluster to perform the first sub-task and the second cluster to perform the second sub-task includes: the cloud management platform instructing the first cluster to process the first data set and the second data set respectively by using the first sub-model to obtain a third processing result and a fourth processing result; the cloud management platform instructing the first cluster to compress the third processing result and the fourth processing result, and sending the compressed third processing result and the compressed fourth processing result to the second cluster; and the cloud management platform instructing the second cluster to process the compressed third processing result and the compressed fourth processing result respectively by using the second sub-model to obtain a fifth processing result and a sixth processing result, and the fifth processing result and the sixth processing result are used for updating parameters of the first sub-model and parameters of the second sub-model by the first cluster and the second cluster respectively to obtain a fourth model. In the foregoing implementation, when the first sub-task is performed, the first cluster can process the first data set and the second data set respectively by using the first sub-model included in the to-be-trained model, and accordingly obtain the third processing result and the fourth processing result, and then quantize and compress the third processing result and the fourth processing result to obtain the compressed third processing result and the compressed fourth processing result, and send the compressed third processing result and the compressed fourth processing result to the second cluster. When the second sub-task is performed, the second cluster can process the compressed third processing result and the compressed fourth processing result respectively by using the second sub-model included in the to-be-trained model, and accordingly obtain the fifth processing result and the sixth processing result, and the fifth processing result and the sixth processing result can be used for updating the parameters of the first sub-model and the parameters of the second sub-model by the first cluster and the second cluster to obtain the fourth model. Based on the fourth model, the first cluster and the second cluster can finally obtain a trained model, thereby meeting the model training requirement of the tenant.

[0017] In a possible implementation, the model training task is used to instruct training of a to-be-trained model based on a data set, the to-be-trained model comprising a first routing network, a first expert network and a second expert network, and the cloud management platform notifying the first cluster to perform the first subtask and notifying the second cluster to perform the second subtask comprises: the cloud management platform notifying the first cluster to sequentially process the data set through the first routing network and the first expert network to obtain a seventh processing result, and updating parameters of the first routing network and parameters of the first expert network based on the seventh processing result to obtain a third routing network and a third expert network; the cloud management platform notifying the second cluster to sequentially process the data set through the second routing network and the second expert network to obtain an eighth processing result, and updating parameters of the second routing network and parameters of the second expert network based on the eighth processing result to obtain a fourth routing network and a fourth expert network, the second routing network being obtained by copying the first routing network; and the cloud management platform notifying the first cluster and the second cluster to aggregate the third routing network and the fourth routing network to obtain a fifth routing network. In the foregoing implementation, when the first subtask is performed, the first cluster can first process the data set through the first routing network and the first expert network comprised in the to-be-trained model to obtain the seventh processing result, and update the parameters of the first routing network and the parameters of the first expert network based on the seventh processing result, to correspondingly obtain the third routing network and the third expert network. When the second subtask is performed, the second cluster can first process the data set through the second routing network and the second expert network comprised in the to-be-trained model to obtain the eighth processing result, and update the parameters of the second routing network and the parameters of the second expert network based on the eighth processing result, to obtain the fourth routing network and the fourth expert network. Then, the first cluster can aggregate the third routing network and the fourth routing network in conjunction with the second cluster to obtain the fifth routing network. Based on the fifth routing network, the third expert network and the fourth expert network, the first cluster and the second cluster can finally obtain a trained model, thereby meeting the model training requirement of the tenant.

[0018] In a possible implementation, the first cluster comprises a plurality of first computing nodes, and the plurality of first computing nodes are used to perform the first subtask. The first computing node comprises any one of the following: a physical server, a virtual machine, a container, a micro virtual machine and a bare metal server.

[0019] In a possible implementation, the site comprises any one of the following: a region, an availability zone, a data center, a computer room and a rack.

[0020] A second aspect of the embodiments of the present application provides a cloud management platform, the cloud management platform being configured to manage an infrastructure providing cloud services, the cloud management platform comprising: a receiving module configured to receive, through a model training interface, a model training task sent by a tenant, a first training strategy to be followed by multiple clusters when jointly performing the model training task, and a second training strategy to be followed by a single cluster when performing the model training task; a splitting module configured to split, based on the first training strategy and the second training strategy, the model training task into a first subtask to be performed by a first cluster and a second subtask to be performed by a second cluster, the first cluster and the second cluster being located in different sites; and an executing module configured to create the first cluster and the second cluster in the infrastructure, and notify the first cluster to perform the first subtask and notify the second cluster to perform the second subtask.

[0021] In a possible implementation, the model training task is configured to instruct training of a to-be-trained model, the first training strategy is a data parallel manner, the second training strategy is a model parallel manner, the first subtask is configured to instruct the first cluster to train the to-be-trained model based on the model parallel manner and train the to-be-trained model based on the data parallel manner with the second cluster, and the second subtask is configured to instruct the second cluster to train the to-be-trained model based on the model parallel manner and train the to-be-trained model based on the data parallel manner with the first cluster.

[0022] In a possible implementation, the model training task is configured to instruct training of a to-be-trained model, the first training strategy is a model parallel manner, the second training strategy is a data parallel manner, the first subtask is configured to instruct the first cluster to train the to-be-trained model based on the data parallel manner and train the to-be-trained model based on the model parallel manner with the second cluster, and the second subtask is configured to instruct the second cluster to train the to-be-trained model based on the data parallel manner and train the to-be-trained model based on the model parallel manner with the first cluster.

[0023] In a possible implementation, the model training task is configured to instruct training of a to-be-trained model, the first training strategy and the second training strategy are both expert parallel manners, the first subtask is configured to instruct the first cluster to train the to-be-trained model based on the expert parallel manner and train the to-be-trained model based on the expert parallel manner with the second cluster, and the second subtask is configured to instruct the second cluster to train the to-be-trained model based on the expert parallel manner and train the to-be-trained model based on the expert parallel manner with the first cluster.

[0024] In a possible implementation, the model training task is configured to instruct training of a to-be-trained model based on a first data set and a second data set, the to-be-trained model comprising a first sub-model and a second sub-model, and the execution module is configured to: instruct the first cluster to sequentially process the first data set through the first sub-model and the second sub-model to obtain a first processing result, and update parameters of the first sub-model and parameters of the second sub-model based on the first processing result to obtain a first model; instruct the second cluster to sequentially process the second data set through the first sub-model and the second sub-model to obtain a second processing result, and update the parameters of the first sub-model and the parameters of the second sub-model based on the second processing result to obtain a second model; and instruct the first cluster and the second cluster to aggregate the first model and the second model to obtain a third model.

[0025] In a possible implementation, the model training task is configured to instruct training of a to-be-trained model based on a data set, the to-be-trained model comprising a first sub-model and a second sub-model, and the data set comprising a first data set and a second data set, and the execution module is configured to: instruct the first cluster to process the first data set and the second data set through the first sub-model respectively to obtain a third processing result and a fourth processing result; instruct the first cluster to compress the third processing result and the fourth processing result, and send the compressed third processing result and the compressed fourth processing result to the second cluster; and instruct the second cluster to process the compressed third processing result and the compressed fourth processing result through the second sub-model to obtain a fifth processing result and a sixth processing result, the fifth processing result and the sixth processing result being used for updating parameters of the first sub-model and parameters of the second sub-model by the first cluster and the second cluster respectively to obtain a fourth model.

[0026] In a possible implementation, the model training task is configured to instruct training of a to-be-trained model based on a data set, the to-be-trained model comprising a first routing network, a first expert network, and a second expert network, and the execution module is configured to: instruct the first cluster to sequentially process the data set through the first routing network and the first expert network to obtain a seventh processing result, and update parameters of the first routing network and parameters of the first expert network based on the seventh processing result to obtain a third routing network and a third expert network; instruct the second cluster to sequentially process the data set through a second routing network and the second expert network to obtain an eighth processing result, and update parameters of the second routing network and parameters of the second expert network based on the eighth processing result to obtain a fourth routing network and a fourth expert network, the second routing network being obtained by copying the first routing network; and instruct the first cluster and the second cluster to aggregate the third routing network and the fourth routing network to obtain a fifth routing network.

[0027] In a possible implementation, the first cluster includes a plurality of first computing nodes, and the plurality of first computing nodes are configured to execute the first subtask. The first computing node includes any one of the following: a physical server, a virtual machine, a container, a micro virtual machine, and a bare metal server.

[0028] In a possible implementation, the site includes any one of the following: a region, an availability zone, a data center, a computer room, and a rack.

[0029] A third aspect of the embodiments of the present application provides a computing device cluster, including at least one computing device, each computing device including a processor and a memory: the memory is configured to store instructions; and the processor is configured to execute the instructions to cause the computing device cluster to perform the method in the first aspect or any possible implementation of the first aspect.

[0030] A fourth aspect of the embodiments of the present application provides a computer storage medium, which stores one or more instructions, and the instructions, when executed by one or more computers, cause the one or more computers to implement the method in the first aspect or any possible implementation of the first aspect.

[0031] A fifth aspect of the embodiments of the present application provides a computer program product, which stores instructions, and the instructions, when executed by a computer, cause the computer to implement the method in the first aspect or any possible implementation of the first aspect.

[0032] In the embodiments of the present application, when the tenant has a model training requirement, the tenant can send a model training task, a first training strategy to be followed by multiple clusters for jointly executing the model training task, and a second training strategy to be followed by a single cluster for executing the model training task to a model training interface provided by the cloud management platform. Then, the cloud management platform can divide the model training task into a first subtask to be executed by a first cluster and a second subtask to be executed by a second cluster based on the first training strategy and the second training strategy. Subsequently, the cloud management platform creates the first cluster and the second cluster in the infrastructure and causes the first cluster to execute the first subtask and causes the second cluster to execute the second subtask, thereby completing the model training task formulated by the tenant. In the foregoing process, the cloud management platform can split the model training task formulated by the tenant into subtasks to be executed by clusters (i.e., the first subtask and the second subtask) according to the cross-cluster training strategy (i.e., the first training strategy) and the intra-cluster training strategy (i.e., the second training strategy) set by the tenant, and causes the clusters to execute the corresponding subtasks, thereby realizing joint completion of the model training task of the tenant by the clusters and realizing cross-cluster execution of the model training. As can be seen, regardless of the size of the model training task, the cloud management platform can invoke clusters with sufficient performance to carry the model training task of the tenant, thereby meeting the model training requirement of the tenant. BRIEF DESCRIPTION OF DRAWINGS

[0033] FIG. 1 is a structural schematic diagram of a cloud service system provided by an embodiment of the present application;

[0034] FIG. 2 is another structural schematic diagram of a cloud service system provided by an embodiment of the present application;

[0035] FIG. 3 is a flow schematic diagram of a model training method based on a cloud management platform provided by an embodiment of the present application;

[0036] FIG. 4 is another structural schematic diagram of a cloud service system provided by an embodiment of the present application;

[0037] FIG. 5 is another structural schematic diagram of a cloud service system provided by an embodiment of the present application;

[0038] FIG. 6 is another structural schematic diagram of a cloud service system provided by an embodiment of the present application;

[0039] FIG. 7 is a structural schematic diagram of a cloud management platform provided by an embodiment of the present application;

[0040] FIG. 8 is a structural schematic diagram of a computing device provided by an embodiment of the present application;

[0041] FIG. 9 is a structural schematic diagram of a cluster of computing devices provided by an embodiment of the present application;

[0042] FIG. 10 is a schematic diagram of a computer device in a computer cluster connected through a network according to an embodiment of the present application. DETAILED DESCRIPTION

[0043] The embodiments of the present application provide a model training method based on a cloud management platform and the cloud management platform. No matter how large the scale of a model training task of a tenant is, a number of clusters with sufficient performance can be invoked to carry the model training task of the tenant, so as to meet the model training requirement of the tenant.

[0044] The terms "first", "second", etc. in the description and claims of the present application and the above drawings are used to distinguish similar objects, and do not necessarily have to describe a specific order or sequence. It should be understood that the terms thus used can be interchanged under appropriate circumstances, and this is merely a distinguishing way adopted in the description of the embodiments of the present application for the objects with the same attributes in the description. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, so that the processes, methods, systems, products or devices containing a series of units do not have to be limited to those units, but can include other units not clearly listed or inherent to these processes, methods, products or devices.

[0045] With the rapid development of cloud technology, more and more tenants choose to use the clusters provided by cloud vendors to complete the training of neural network models. Since a cluster is usually composed of multiple computing nodes, each computing node has a certain specification of resources, so the cluster has sufficient resources and performance to successfully complete the model training task of the tenant.

[0046] In the related art, when a tenant needs to train a certain model, the cloud management platform can create a dedicated cluster for the tenant according to the tenant's requirements, and notify the cluster to train the to-be-trained model according to a series of requirements set by the tenant, so as to obtain the trained model, and return the trained model to the tenant for use, thereby meeting the model training requirement of the tenant.

[0047] However, when the scale of the model to be trained by the tenant is large, the performance of a single cluster is limited, and it cannot carry the training task of a large-scale neural network model. Therefore, how to implement model training across clusters has become a technical problem to be solved.

[0048] Further, when model training across clusters is needed, a large amount of communication is often involved between clusters when performing model training, and how to reduce the communication amount between clusters also becomes a difficulty in implementing model training across clusters.

[0049] Further, when model training across clusters is needed, how to determine the specification of resources required by each cluster also becomes another difficulty in implementing model training across clusters.

[0050] To solve the above problems, the embodiment of the present application provides a model training method based on a cloud management platform. The method can be implemented through a cloud service system. FIG. 1 is a structural schematic diagram of the cloud service system provided by the embodiment of the present application. As shown in FIG. 1, the cloud service system includes infrastructures that can provide cloud services and a cloud management platform that manages the infrastructures. The cloud management platform and the infrastructures are introduced respectively as follows:

[0051] (1) The cloud management platform can comprehensively manage the infrastructures in the entire cloud service system (for example, in the infrastructures, a plurality of clusters serving a tenant are created according to the instructions of the tenant, each cluster of the tenant can include a plurality of computing nodes, which are used to train a to-be-trained model of the tenant to obtain a trained model and return the trained model to the tenant, thereby meeting the model training requirements of the tenant, and the like). The cloud management platform can also be open to tenants outside the cloud service system and respond to their requests. For example, the cloud management platform can provide various interfaces such as a login interface and a model training interface, so as to be accessed by a client (for example, a terminal device used by the tenant or a browser on the terminal device, and the like) of the tenant. Among them, the cloud management platform can authenticate the client of the tenant through the login interface, and allow the client of the tenant to log in to the cloud management platform after the authentication is successful. For another example, the cloud management platform can also allow the client of the tenant to send, through the model training interface, model training information such as a model training task of the tenant for a to-be-trained model, a cross-cluster training strategy required for a plurality of clusters to jointly execute the model training task, and an intra-cluster training strategy required for a single cluster to execute the model training task. After obtaining the model training information, the cloud management platform can divide the model training task into a plurality of sub-tasks required to be executed by the clusters based on the cross-cluster training strategy and the intra-cluster training strategy. The plurality of clusters are the clusters allocated by the cloud management platform to the tenant for model training. Then, the cloud management platform can create the plurality of clusters, and allocate the sub-tasks required to be executed by the plurality of clusters to the plurality of clusters, so that the plurality of clusters execute the sub-tasks required to be executed by the plurality of clusters, thereby completing the model training task of the tenant.

[0052] Specifically, as shown in FIG. 2 (FIG. 2 is another structural schematic diagram of the cloud service system provided by the embodiment of the present application), the cloud management platform can include a model controller and a resource scheduler. The model controller and the resource scheduler are introduced respectively as follows:

[0053] (1.1) The model controller can receive, through the model training interface, a model training task sent by the tenant, a cross-cluster training strategy to be followed by multiple clusters for jointly executing the model training task, and an intra-cluster training strategy to be followed by a single cluster for executing the model training task, and the like model training information. The model training task can include a model to be trained by the tenant (e.g., a perception model, a feedforward neural network model, a convolutional neural network model, a long short-term memory network model, a generative adversarial network model, a mixture of experts (MOE) model, and the like), at least one dataset (each dataset can include multiple data), and a training completion condition (e.g., the model training process can include multiple rounds, when the number of rounds of model training reaches a certain threshold, when the loss obtained in the model training process converges, and the like). The cross-cluster training strategy can be any one of the following: a data-parallel (training) mode, a model-parallel mode (which can include a pipeline-parallel mode and a tensor-parallel mode, and the like), and an expert-parallel mode, and the like. The intra-cluster training strategy can also be any one of the following: a data-parallel mode, a model-parallel mode, and an expert-parallel mode, and the like.

[0054] The model controller can also divide the model training task into sub-tasks to be executed by a plurality of clusters serving the tenant based on the cross-cluster training strategy and the intra-cluster training strategy. It should be noted that for any one of the plurality of clusters, the cloud management platform can usually assign multiple sub-tasks to the cluster. It should be noted that the multiple sub-tasks to be executed by the cluster have strong affinity, indicating that the multiple sub-tasks to be executed by the cluster must be scheduled within the site where the cluster is located. Of course, the multiple sub-tasks to be executed by the cluster and the multiple sub-tasks to be executed by other clusters can have weak affinity, indicating that the multiple sub-tasks to be executed by the cluster and the multiple sub-tasks to be executed by other clusters are usually scheduled in different sites because the cluster and the remaining clusters are located in these different sites.

[0055] It is worth noting that for any one of the plurality of clusters, the multiple sub-tasks to be executed by the cluster include training tasks and communication tasks. The training tasks to be executed by the cluster are usually completed within the cluster, and the communication tasks to be completed by the cluster need to be completed by the cluster and the remaining clusters. Therefore, the model training task formulated by the tenant is split into training tasks to be completed by each cluster and communication tasks, the training tasks are completed within each cluster, and the communication tasks are completed between clusters, so the model training task can be implemented across clusters and within clusters.

[0056] It is also worth noting that in the above example, the model controller can divide the model training task into a plurality of sub-tasks to be executed by each of the clusters according to the training strategy. In another possible implementation, the tenant can also divide the model training task formulated by the tenant into a plurality of sub-tasks, and provide the model controller with the sub-tasks and the affinity between the sub-tasks, so that the model controller can assign the sub-tasks to the clusters according to the strength of the affinity, so that a plurality of sub-tasks to be executed by each of the clusters can also be obtained.

[0057] (1.2) The resource scheduler can obtain the sub-tasks to be executed by the clusters from the model controller. Then, the resource scheduler can determine the specifications of the resources required by the sub-tasks to be executed by the clusters, which is equivalent to determining the specifications of the resources required by the clusters. Next, the resource scheduler can select resources that meet the specifications from each resource pool, and create the clusters on the selected resources. Subsequently, the resource scheduler can assign and schedule the sub-tasks to be executed by the clusters to the clusters, so that the clusters execute their respective sub-tasks, thereby completing the model training task of the tenant.

[0058] It is worth noting that since a sub-task is usually executed by a computing node in a cluster, when creating any of the clusters, the resource scheduler can cause the cluster to contain a plurality of computing nodes, and the number of the computing nodes of the cluster is usually equal to the number of the sub-tasks to be executed by the cluster. Since the computing nodes of the cluster correspond one-to-one to the sub-tasks to be executed by the cluster, for any of the computing nodes of the cluster, the specification of the resource occupied by the computing node is equal to the specification of the resource required by the sub-task to be executed by the computing node (i.e., the sub-task corresponding to the computing node).

[0059] It is also worth noting that when scheduling the sub-tasks to be executed by the clusters, the resource scheduler usually needs to schedule all the sub-tasks to the corresponding clusters, and will not only schedule part of the sub-tasks. In addition, if the model training task of the tenant can tolerate a certain degree of asynchrony, the resource scheduler can first schedule part of the sub-tasks to the corresponding clusters, and then schedule another part of the sub-tasks to the corresponding clusters. The scheduling time difference between the two parts of sub-tasks can be determined by the resource scheduler according to the degree of asynchrony that the model training task can tolerate. In addition, when a cluster or a computing node in a cluster fails, the resource scheduler can schedule the sub-tasks executed by the failed cluster or computing node to a standby cluster or computing node for continuous execution, to ensure the smooth completion of the model training task.

[0060] (2) The infrastructure comprises a plurality of physical resource pools, which comprise a computing resource pool, a storage resource pool, a network resource pool, and the like. The computing resource pool can comprise a central processing unit (CPU) pool, a graphics processing unit (GPU) pool, a neural processing unit (NPU) pool, and the like, the storage resource pool can comprise a memory pool and a hard disk pool, and the like, and the network resource pool can comprise a network card pool, and the like. It should be noted that for any resource pool, the resource pool can comprise a plurality of physical resources of different specifications, for example, the CPU pool can comprise a plurality of CPUs of different specifications (e.g., core quantity, thread quantity, and the like), the memory pool can comprise a plurality of memories of different specifications (e.g., capacity, and the like), the network card pool can comprise a plurality of network cards of different specifications (e.g., bandwidth, and the like), and the like.

[0061] It should be noted that when the resource scheduler determines the specifications of the resources required by the clusters, the resource scheduler can select resources that meet the specifications from the plurality of resource pools, and create the clusters on the selected resources, each cluster can comprise a plurality of computing nodes (each computing node can have a certain specification of resources, i.e., each computing node has resources comprising a certain specification of computing resources, storage resources, and network resources), so that the resource scheduler can schedule the plurality of sub-tasks required to be executed by the clusters to the plurality of computing nodes comprised by the clusters for execution, thereby completing the entire model training task of the tenant.

[0062] Further, for the clusters serving the tenant, each cluster can comprise a plurality of computing nodes, which can be presented in various forms, for example, the computing nodes can be physical servers (comprising a certain specification of computing resources, storage resources, and network resources, and the like) selected by the cloud management platform in the infrastructure, for another example, the computing nodes can also be bare metal servers (comprising a certain specification of computing resources, storage resources, and network resources, and the like) selected by the cloud management platform in the infrastructure. For another example, the computing nodes can also be virtual machines (VMs) created by the cloud management platform in the physical servers or bare metal servers through virtualization technology, for another example, the computing nodes can also be containers (docker) created by the cloud management platform in the physical servers or bare metal servers through virtualization technology, for another example, the computing nodes can also be micro virtual machines (microVM) created by the cloud management platform in the physical servers or bare metal servers through virtualization technology, and the like.

[0063] Further, for the several clusters serving the tenant, the several clusters are usually deployed in different sites, and the sites can be presented in various forms, for example, the sites can be regions in the infrastructure, for another example, the sites can be availability zones in the infrastructure, for another example, the sites can be data centers (DCs) in the infrastructure, for another example, the sites can be rooms in the infrastructure, for another example, the sites can be cabinets in the infrastructure, and the like.

[0064] Based on the cloud service system described above, when the tenant has a model training requirement, the tenant can send, to a model training interface provided by the cloud management platform, a model training task, a cross-cluster training strategy that the several clusters need to follow to jointly execute the model training task, and an intra-cluster training strategy that a single cluster needs to follow to execute the model training task. Then, the cloud management platform can divide the model training task into subtasks that the several clusters need to execute based on the cross-cluster training strategy and the intra-cluster training strategy. Subsequently, the cloud management platform creates the several clusters in the infrastructure and notifies the several clusters to execute the subtasks that the several clusters need to execute, thereby completing the model training task formulated by the tenant. In the foregoing process, the cloud management platform can split the model training task formulated by the tenant into subtasks that the several clusters need to execute according to the cross-cluster training strategy and the intra-cluster training strategy set by the tenant, and notify the several clusters to execute the corresponding subtasks, thereby realizing that the several clusters jointly complete the model training task of the tenant, and realizing cross-cluster execution of the model training. As can be seen, regardless of the size of the model training task, the cloud management platform can invoke the several clusters with sufficient performance to carry the model training task of the tenant, thereby meeting the model training requirement of the tenant. In order to further understand the workflow of the cloud management platform, the process is further introduced below in combination with FIG. 3, which is a flowchart of a model training method based on a cloud management platform provided by an embodiment of the present application. The method can be implemented by the cloud service system shown in FIG. 1 or FIG. 2, which includes an infrastructure that can provide cloud services for tenants and a cloud management platform that manages the infrastructure. The method includes:

[0065] 301. The cloud management platform receives, through a model training interface, a model training task sent by a tenant, a first training strategy that the several clusters need to follow to jointly execute the model training task, and a second training strategy that a single cluster needs to follow to execute the model training task.

[0066] In this embodiment, when the tenant has a model training requirement, the cloud management platform can provide a model training interface (e.g., a model training column of a tenant interface, etc.) to the client of the tenant. Then, the tenant can input, through the client used by the tenant, a model training task set by the tenant, a first training strategy (i.e., the aforementioned cross-cluster training strategy) required for multiple clusters to jointly execute the model training task, and a second training strategy (i.e., the aforementioned intra-cluster training strategy) required for a single cluster to execute the model training task. In this way, the cloud management platform can receive, through the model training interface, the model training task, the first training strategy, and the second training strategy sent by the tenant through the client.

[0067] Specifically, the model training task formulated by the tenant can include a to-be-trained model specified by the tenant, a data set for the to-be-trained model, and a training completion condition for the to-be-trained model, so that the model training task can be used to instruct training of the to-be-trained model based on the data set. The data set can be presented in various ways, for example, the tenant can only provide one data set to the cloud management platform, and the cloud management platform can select to retain the data set or divide the data set into two part data sets based on subsequent requirements, one part data set being a first data set and the other part data set being a second data set. For another example, the tenant can also provide multiple data sets to the cloud management platform, the multiple data sets including the first data set and the second data set, the first data set being one of the multiple data sets, and the second data set being another of the multiple data sets. The training completion condition can also be presented in various ways, for example, since the training process for the to-be-trained model can include multiple rounds, the training completion condition can refer to the number of rounds of the training process reaching a certain preset value. For another example, in each round of the training process for the to-be-trained model, since a loss is calculated in each round to update the parameters of the model, the training completion condition can refer to the loss reaching convergence, etc.

[0068] More specifically, the first training strategy can be any one of a data parallel (training) mode, a model parallel (training) mode, and an expert parallel (training) mode, and similarly, the second training strategy can also be any one of the data parallel (training) mode, the model parallel (training) mode, and the expert parallel (training) mode. Based on this, the combination of the first training strategy and the second training strategy can be presented in various ways, for example, the first training strategy is the data parallel mode, and the second training strategy is the model parallel mode. For another example, the first training strategy is the model parallel mode, and the second training strategy is the data parallel mode. For another example, the first training strategy and the second training strategy are both the expert parallel mode, etc. The specific presentation mode of the combination can be set by the tenant, which is not limited here.

[0069] 302、The cloud management platform splits the model training task into a first sub-task to be executed by the first cluster and a second sub-task to be executed by the second cluster based on the first training strategy and the second training strategy, the first cluster and the second cluster being located in different sites.

[0070] 303、The cloud management platform creates the first cluster and the second cluster in the infrastructure, and notifies the first cluster to execute the first sub-task and the second cluster to execute the second sub-task.

[0071] After obtaining the model training task, the first training strategy and the second training strategy, the cloud management platform can split the model training task into a first sub-task to be executed by the first cluster and a second sub-task to be executed by the second cluster according to the instructions of the first training strategy and the second training strategy. The first cluster is a cluster serving a tenant, and the second cluster is another cluster serving the tenant, and the first cluster is arranged in a site located in a first geographic location, and the second cluster is arranged in a site located in a second geographic location, the first geographic location and the second geographic location being different geographic locations, that is, the first cluster and the second cluster are located in different sites.

[0072] After obtaining the first sub-task and the second sub-task, the cloud management platform can create the first cluster and the second cluster in the infrastructure, and schedule the first sub-task to the first cluster and the second sub-task to the second cluster, so that the first cluster executes the first sub-task and the second cluster executes the second sub-task, thereby completing the model training task to meet the model training needs of the tenant.

[0073] Specifically, the cloud management platform can split the model training task in the following multiple ways:

[0074] (1) Since the model training task is used to instruct training of a to-be-trained model, when the first training strategy is a data parallel manner and the second training strategy is a model parallel manner, the cloud management platform can split the model training task into a first sub-task assigned to the first cluster and a second sub-task assigned to the second cluster based on the two training manners, wherein the first sub-task can be used to instruct the first cluster to train the to-be-trained model based on a model parallel manner internally, and the first sub-task can also be used to instruct the first cluster to train the to-be-trained model based on a data parallel manner jointly with the second cluster. Correspondingly, the second sub-task can be used to instruct the second cluster to train the to-be-trained model based on a model parallel manner internally, and the second sub-task can also be used to instruct the second cluster to train the to-be-trained model based on a data parallel manner jointly with the first cluster.

[0075] Further, assuming that the model training task comprises a tenant-specified model to be trained, a tenant-specified training completion condition, a first data set provided by the tenant directly to the cloud management platform, and a second data set, the model training task can be used to instruct training of the model to be trained based on the first data set and the second data set. Since the first training strategy is in a data parallel manner and the second training strategy is in a model parallel manner, the cloud management platform splits the model training task based on the two manners, and a first sub-task allocated to the first cluster can comprise the first training task and a first communication task, and a second sub-task allocated to the second cluster can comprise a second training task and a second communication task.

[0076] The first training task can comprise a first sub-model split from the model to be trained (the model to be trained comprises a plurality of network layers, and the first sub-model is a part of the network layers included in the model to be trained), a second sub-model split from the model to be trained (the second sub-model is another part of the network layers included in the model to be trained), the training completion condition, the first data set, and a first training instruction. The first communication task can comprise a first communication instruction. The second training task can also comprise the first sub-model, the second sub-model, the training completion condition, the second data set, and a second training instruction. The second communication task can comprise a second communication instruction.

[0077] After obtaining the first training task, the first communication task, the second training task, and the second communication task, the cloud management platform can determine a first specification of resources required by the first training task, a second specification of resources required by the first communication task, a third specification of resources required by the second training task, and a fourth specification of resources required by the second communication task, and select, in a plurality of resource pools of the infrastructure, a first resource satisfying the first specification, a second resource satisfying the second specification, a third resource satisfying the third specification, and a fourth resource satisfying the fourth specification, and then create a first computing node on the first resource, a second computing node on the second resource, a third computing node on the third resource, and a fourth computing node on the fourth resource. Then, the first computing node and the second computing node constitute the first cluster, and the third computing node and the fourth computing node constitute the second cluster, wherein the first computing node and the second computing node are disposed in a site located at a first geographic location, and the third computing node and the fourth computing node are disposed in a site located at a second geographic location.

[0078] After creating the first computing node, the second computing node, the third computing node, and the fourth computing node, i.e., after creating the first cluster and the second cluster, the cloud management platform schedules the first training task to the first computing node of the first cluster, schedules the first communication task to the second computing node of the first cluster, schedules the second training task to the third computing node of the second cluster, and schedules the second communication task to the fourth computing node of the second cluster.

[0079] Then, when performing the first training task, since the training process for the first sub-model and the second sub-model includes multiple rounds (e.g., one thousand rounds, etc.), according to the indication of the first training instruction, when the first computing node performs a first round, the first computing node can sequentially process some data in the first data set through the first sub-model and the second sub-model to obtain a first processing result, and calculate a first loss based on the first processing result and the first real result, so as to update the parameters of the first sub-model and the parameters of the second sub-model by using the first loss to obtain the first sub-model after first updating the parameters and the second sub-model after first updating the parameters, and then the first computing node performs a second round,..., until the first computing node performs a certain round (e.g., the tenth round, etc.), the first sub-model after multiple (e.g., ten times, etc.) updating the parameters and the second sub-model after multiple (e.g., ten times, etc.) updating the parameters obtained in this round are the first model, and the first model is sent to the second computing node.

[0080] When performing the second training task, according to the indication of the second training instruction, when the third computing node performs a first round, the first computing node can sequentially process some data in the second data set through the first sub-model and the second sub-model to obtain a second processing result, and calculate a second loss based on the second processing result and the second real result, so as to update the parameters of the first sub-model and the parameters of the second sub-model by using the second loss to obtain the first sub-model after first updating the parameters and the second sub-model after first updating the parameters, and then the third computing node performs a second round,..., until the third computing node performs a certain round (e.g., the tenth round, etc.), the first sub-model after multiple (e.g., ten times, etc.) updating the parameters and the second sub-model after multiple (e.g., ten times, etc.) updating the parameters obtained in this round are the second model, and the second model is sent to the fourth computing node.

[0081] When performing the first communication task, according to the indication of the first communication instruction, the second computing node can send the parameters of the first model to the fourth computing node, receive the parameters of the second model sent by the fourth computing node, and perform weighted summation (equivalent to aggregating the first model and the second model) on the parameters of the first model and the parameters of the second model to obtain a first aggregated model, that is, the third model.

[0082] In performing the second communication task, the fourth computing node can send the parameters of the second model to the third computing node, receive the parameters of the first model sent by the second computing node, and perform a weighted summation of the parameters of the second model and the parameters of the first model (equivalent to aggregating the first model and the second model) to obtain a first-aggregated model, i.e., a third model.

[0083] Then, the first computing node, the second computing node, the third computing node, and the fourth computing node can cyclically perform the above steps until a training completion condition is met (for example, one thousand rounds of execution, or the first loss and the second loss reach convergence, etc.), and the last-aggregated model of the second computing node and the fourth computing node is the final model of the training, which can be returned to the cloud management platform and then to the tenant.

[0084] For example, as shown in FIG. 4 (which is another structure schematic diagram of a cloud service system provided by an embodiment of the present application), when the tenant needs to train a convolutional neural network (CNN) to be trained, the tenant can log in to the cloud management platform, and the cloud management platform can provide a tenant interface, which includes a model training column. The tenant can input a model training task, a cross-cluster training strategy, and an intra-cluster training strategy into the model training column. The model training task includes the CNN to be trained (i.e., the aforementioned model to be trained), a training completion condition (1000 rounds of training are completed), a data set 1, and a data set 2. The cross-cluster training strategy is a data parallel manner, and the intra-cluster training strategy is a model parallel manner.

[0085] Then, the cloud management platform can divide the model training task into eight sub-tasks based on the cross-cluster training strategy and the intra-cluster training strategy, including training task 1 to training task 3 (i.e., the aforementioned first training task), training task 4 to training task 6 (i.e., the aforementioned second training task), communication task 1 (i.e., the aforementioned first communication task), and communication task 2 (i.e., the aforementioned second communication task).

[0086] Next, the cloud management platform can create cluster 1 and cluster 2. Cluster 1 includes computing nodes 1 to 4 (including the aforementioned first computing node and the second computing node), and cluster 2 includes computing nodes 5 to 8 (including the aforementioned third computing node and the fourth computing node). Cluster 1 is arranged at a site located at geographic location 1, and cluster 2 is arranged at a site located at geographic location 2.

[0087] Then, the cloud management platform can schedule training task 1 to training task 3 to computing nodes 1 to 3, schedule communication task 1 to computing node 4, schedule training task 4 to 6 to computing nodes 5 to 7, and schedule communication task 2 to computing node 8.

[0088] Subsequently, since the training task 1 to the training task 3 contain the sub-model 1 (i.e., the first sub-model), the sub-model 2 and the sub-model 3 (i.e., the second sub-model), the data set 1 and the training completion condition of the CNN to be trained which are split by network layers, the sub-model 1 of the CNN is deployed on the computing node 1, the sub-model 2 of the CNN is deployed on the computing node 2, and the sub-model 3 of the CNN is deployed on the computing node 3. Based on this, the computing node 1, the computing node 2 and the computing node 3 train the sub-model 1 to the sub-model 3 by using the data in the data set 1, and it is assumed that 10 rounds of training are performed, the result of the 10th round of training is the model 1, and the model 1 is provided to the computing node 4.

[0089] Similarly, since the training task 4 to the training task 6 contain the sub-model 1, the sub-model 2, the sub-model 3, the data set 2 and the training completion condition of the CNN, the sub-model 1 of the CNN is deployed on the computing node 5, the sub-model 2 of the CNN is deployed on the computing node 6, and the sub-model 3 of the CNN is deployed on the computing node 7. Based on this, the computing node 5, the computing node 6 and the computing node 7 train the sub-model 1 to the sub-model 3 by using the data in the data set 2, and it is assumed that 10 rounds of training are performed, the result of the 10th round of training is the model 2, and the model 2 is provided to the computing node 8.

[0090] Based on the communication task 1 and the communication task 2, the computing node 4 and the computing node 8 can aggregate the model 1 and the model 2 (i.e., the computing node 4 and the computing node 8 exchange the parameters of the model 1 and the parameters of the model 2, and the parameters of the two models are weighted and summed, and the parameters after the weighted summation are the parameters of the model obtained by the first aggregation), thereby obtaining the result of the first aggregation, i.e., the model 3. In this way, the computing node 1 to the computing node 8 can continuously perform the foregoing operations until the training completion condition is met (for example, 1000 rounds of training are performed), and the result obtained by the last aggregation of the computing node 4 and the computing node 8 can be used as the CNN after the training is completed.

[0091] (2) Since the model training task is used to instruct to train the to-be-trained model, when the first training strategy is the model parallel manner and the second training strategy is the data parallel manner, the cloud management platform can split the model training task into a first subtask assigned to the first cluster and a second subtask assigned to the second cluster based on the two training manners, wherein the first subtask can be used to instruct the first cluster to train the to-be-trained model based on the data parallel manner internally, and the first subtask can also be used to instruct the first cluster to train the to-be-trained model based on the model parallel manner jointly with the second cluster. Correspondingly, the second subtask can be used to instruct the second cluster to train the to-be-trained model based on the data parallel manner internally, and the second subtask can also be used to instruct the second cluster to train the to-be-trained model based on the model parallel manner jointly with the first cluster.

[0092] Further, assuming that the model training task includes a to-be-trained model specified by a tenant, a training completion condition specified by the tenant, and a data set provided by the tenant directly to the cloud management platform, the model training task can be used to instruct to train the to-be-trained model based on the data set, and the data set can include a first data set and a second data set (which can be directly provided by the tenant or divided by the cloud management platform, which is not limited here). Since the first training strategy is the data parallel manner and the second training strategy is the model parallel manner, the cloud management platform splits the model training task based on the two manners, and the first subtask assigned to the first cluster can include a first training task and a first communication task, and the second subtask assigned to the second cluster can include a second training task and a second communication task.

[0093] Wherein, the first training task can include a first sub-model split from the to-be-trained model (the to-be-trained model includes multiple network layers, and the first sub-model is a part of the network layers included in the to-be-trained model), the first data set, the second data set, and the first training instruction. The first communication task can include the first communication instruction. The second training task can include a second sub-model split from the to-be-trained model (the second sub-model is another part of the network layers of the to-be-trained model), the training completion condition, and the second training instruction. The second communication task can include the second communication instruction.

[0094] After obtaining the first training task, the first communication task, the second training task, and the second communication task, the cloud management platform can determine a first specification of resources required by the first training task, a second specification of resources required by the first communication task, a third specification of resources required by the second training task, and a fourth specification of resources required by the second communication task, and select, in the plurality of resource pools of the infrastructure, a first resource satisfying the first specification, a second resource satisfying the second specification, a third resource satisfying the third specification, and a fourth resource satisfying the fourth specification, and then create a first computing node on the first resource, a second computing node on the second resource, a third computing node on the third resource, and a fourth computing node on the fourth resource. Then, the first computing node and the second computing node constitute a first cluster, and the third computing node and the fourth computing node constitute a second cluster, wherein the first computing node and the second computing node are arranged in a site located at a first geographic location, and the third computing node and the fourth computing node are arranged in a site located at a second geographic location.

[0095] After creating the first computing node, the second computing node, the third computing node, and the fourth computing node, i.e., after creating the first cluster and the second cluster, the cloud management platform schedules the first training task to the first computing node of the first cluster, schedules the first communication task to the second computing node of the first cluster, schedules the second training task to the third computing node of the second cluster, and schedules the second communication task to the fourth computing node of the second cluster.

[0096] Then, since the training process for the first sub-model and the second sub-model includes multiple rounds (e.g., one thousand rounds, etc.), in the first round, when performing the first training task, the first computing node can process, according to the indication of the first training instruction, a certain data in the first data set and a certain data in the second data set through the first sub-model respectively, thereby obtaining a third processing result and a fourth processing result respectively, and send the third processing result and the fourth processing result to the second computing node.

[0097] When performing the first communication task, according to the indication of the first communication instruction, the second computing node can quantize and compress the third processing result and the fourth processing result, thereby obtaining a compressed third processing result and a compressed fourth processing result, and send the compressed third processing result and the compressed fourth processing result to the fourth computing node.

[0098] When performing the second communication task, according to the indication of the second communication instruction, the fourth computing node can send the compressed third processing result and the compressed fourth processing result to the third computing node.

[0099] When performing the second training task, the third computing node can process the compressed third processing result and the compressed fourth processing result respectively according to the indication of the second training instruction, so as to obtain a fifth processing result and a sixth processing result, and obtain a third loss based on the fifth processing result, the third real result, the sixth processing result and the fourth real result. The third loss can be used by the first computing node and the third computing node to update the parameters of the first sub-model and the parameters of the second sub-model, so as to obtain the first sub-model with first-time updated parameters and the second sub-model with first-time updated parameters, that is, the fourth model. At this time, the first round is completed.

[0100] The first computing node, the second computing node, the third computing node and the fourth computing node can cyclically execute the above steps, that is, the second round and subsequent rounds are executed continuously until the training completion condition is met (for example, one thousand rounds are executed, or the third loss reaches convergence, etc.), and the last updated parameters of the first sub-model and the last updated parameters of the second sub-model obtained by the first computing node and the third computing node, that is, the final model of the completed training, can be returned to the cloud management platform, and then returned to the tenant for use.

[0101] For example, as shown in FIG. 5 (FIG. 5 is another structure schematic diagram of a cloud service system provided by an embodiment of the present application), when the tenant needs to train the CNN to be trained, the tenant can log in to the cloud management platform, and the cloud management platform can provide a tenant interface, which includes a model training column. The tenant can input a model training task, a cross-cluster training strategy and an intra-cluster training strategy into the model training column. The model training task includes a CNN to be trained, a training completion condition (1000 rounds of training are completed) and a data set. The cross-cluster training strategy is a data parallel manner, and the intra-cluster training strategy is a model parallel manner. The cloud management platform can divide the data set into three parts, that is, a data set 1 and a data set 2.

[0102] Then, the cloud management platform can divide the model training task into eight sub-tasks based on the cross-cluster training strategy and the intra-cluster training strategy, including training task 1 to training task 2 (that is, the first training task described above), training task 3 to training task 6 (that is, the second training task described above), communication task 1 (that is, the first communication task described above) and communication task 2 (that is, the second communication task described above).

[0103] Next, the cloud management platform can create cluster 1 and cluster 2. Cluster 1 includes computing nodes 1 to 3 (including the first computing node and the second computing node described above), and cluster 2 includes computing nodes 4 to 8 (including the third computing node and the fourth computing node described above). Cluster 1 is arranged at a site located at geographic location 1, and cluster 2 is arranged at a site located at geographic location 2.

[0104] Then, the cloud management platform can schedule training task 1 to training task 2 to computing nodes 1 to 2, schedule communication task 1 to computing node 3, schedule training task 3 to 6 to computing nodes 4 to 7, and schedule communication task 2 to computing node 8.

[0105] Subsequently, since training task 1 to training task 2 contain sub-model 1 of CNN to be trained which is split by network layer, data set 1 and data set 2, sub-model 1-1 of CNN is deployed on computing node 1, and sub-model 1-2 of CNN is deployed on computing node 2. Based on this, computing node 1 can input sub-model 1-1 with data in data set 1 to obtain processing result 1 (i.e., the third processing result described above), and computing node 2 can input sub-model 1-2 with data in data set 2 to obtain processing result 2 (i.e., the fourth processing result described above), and provide processing result 1 and processing result 2 to computing node 3.

[0106] Computing node 3 can quantize and compress processing result 1 and processing result 2, and send compressed processing result 1 and compressed processing result 2 to computing node 8, and computing node 8 can provide compressed processing result 1 and compressed processing result 2 to computing node 4 and computing node 5, respectively.

[0107] Since training task 3 to training task 6 contain sub-model 2, sub-model 3 (sub-model 2 contains sub-model 2-1 and sub-model 2-2 copied from sub-model 2, and sub-model 3 contains sub-model 3-1 and sub-model 3-2 copied from sub-model 3, i.e., the second sub-model described above) of CNN to be trained and training completion condition, sub-model 2-1 of CNN is deployed on computing node 4, sub-model 2-2 of CNN is deployed on computing node 5, sub-model 3-1 of CNN is deployed on computing node 6, and sub-model 3-2 of CNN is deployed on computing node 7. Based on this, computing node 4 can input compressed processing result 1 to sub-model 2-1, computing node 5 can input compressed processing result 2 to sub-model 2-2, computing node 6 can input the output of sub-model 2-1 to sub-model 3-1, and computing node 7 can input the output of sub-model 2-2 to sub-model 3-2, the output of sub-model 3-1 is processing result 3 (i.e., the fifth processing result described above), and the output of sub-model 3-2 is processing result 4 (i.e., the sixth processing result described above).

[0108] Based on the processing result 3 and the processing result 4, the computing node 1 to the computing node 2, the computing node 4 to the computing node 7 can update the parameters of the sub-model 1-1 to the parameters of the sub-model 3-2 respectively, so as to obtain the first updated parameter sub-model 1-1 to the first updated parameter sub-model 3-2. At this time, the first round of training is completed. Similarly, the computing node 1 to the computing node 8 can continuously perform the foregoing operations until the training completion condition is met (for example, 1000 rounds of training are performed), and after obtaining the last updated parameter sub-model 1-1 to the last updated parameter sub-model 3-2, the corresponding aggregation can be performed, so as to obtain the trained CNN.

[0109] (3) Since the model training task is used to instruct to train the to-be-trained model, when the first training strategy and the second training strategy are both expert parallel modes, the cloud management platform can split the model training task into a first subtask assigned to the first cluster and a second subtask assigned to the second cluster based on the two training modes, wherein the first subtask can be used to instruct the first cluster to train the to-be-trained model based on the expert parallel mode inside the first cluster, and the first subtask can also be used to instruct the first cluster to train the to-be-trained model based on the expert parallel mode together with the second cluster. Correspondingly, the second subtask can be used to instruct the second cluster to train the to-be-trained model based on the expert parallel mode inside the second cluster, and the second subtask can also be used to instruct the second cluster to train the to-be-trained model based on the expert parallel mode together with the first cluster.

[0110] Further, assuming that the model training task contains a tenant-specified to-be-trained model, a tenant-specified training completion condition, and a data set directly provided by the tenant to the cloud management platform, the model training task can be used to instruct to train the to-be-trained model based on the data set. Since the first training strategy and the second training strategy are both expert parallel modes, the cloud management platform splits the model training task based on the mode, and the first subtask assigned to the first cluster can contain a first training task and a first communication task, and the second subtask assigned to the second cluster can contain a second training task and a second communication task.

[0111] The first training task can contain a first expert network contained in the training model, a training completion condition, the data set, and a first training instruction. The first communication task can contain a first routing network contained in the to-be-trained model, the data set, and a first communication instruction. The second training task can also contain a second expert network contained in the training model, a training completion condition, the data set, and a second training instruction. The second communication task can contain a second routing network (copied from the first routing network), the data set, and a second communication instruction.

[0112] After obtaining the first training task, the first communication task, the second training task, and the second communication task, the cloud management platform can determine a first specification of resources required by the first training task, a second specification of resources required by the first communication task, a third specification of resources required by the second training task, and a fourth specification of resources required by the second communication task, and select a first resource satisfying the first specification, a second resource satisfying the second specification, a third resource satisfying the third specification, and a fourth resource satisfying the fourth specification from a plurality of resource pools of the infrastructure, and then create a first computing node on the first resource, a second computing node on the second resource, a third computing node on the third resource, and a fourth computing node on the fourth resource. Then, the first computing node and the second computing node constitute a first cluster, and the third computing node and the fourth computing node constitute a second cluster, wherein the first computing node and the second computing node are arranged in a site located at a first geographic location, and the third computing node and the fourth computing node are arranged in a site located at a second geographic location.

[0113] After creating the first computing node, the second computing node, the third computing node, and the fourth computing node, i.e., after creating the first cluster and the second cluster, the cloud management platform schedules the first training task to the first computing node of the first cluster, schedules the first communication task to the second computing node of the first cluster, schedules the second training task to the third computing node of the second cluster, and schedules the second communication task to the fourth computing node of the second cluster.

[0114] Then, when executing the first training task and the first communication task, since the training process for the to-be-trained model includes multiple rounds (e.g., one thousand rounds, etc.), according to the first training instruction and the first communication instruction, when performing a first round, the second computing node can first process a certain data in the data set through the first routing network to obtain an output of the first routing network and send the output to the first computing node. The first computing node can further process the output of the first routing network through the first expert network to obtain a seventh processing result. Then, the first computing node can obtain a fourth loss based on the seventh processing result and the fifth real result, and the fourth loss can be used for the second computing node and the first computing node to update parameters of the first routing network and parameters of the first expert network, respectively, to obtain the first routing network after the first update of the parameters and the first expert network after the first update of the parameters, and then the second computing node and the first computing node perform a second round,..., until the second computing node and the first computing node perform a certain round (e.g., the tenth round, etc.). The first routing network after multiple (e.g., ten times, etc.) updates of the parameters in this round is a third routing network, and the first expert network after multiple (e.g., ten times, etc.) updates of the parameters in this round is a third expert network.

[0115] In performing the second training task and the second communication task, in the first round, the fourth computing node can process a certain data in the data set through the second routing network to obtain an output of the second routing network and send the output to the third computing node according to the indication of the second training instruction and the second communication instruction. The third computing node can further process the output of the second routing network through the second expert network to obtain an eighth processing result. Then, the third computing node can obtain a fifth loss based on the eighth processing result and the sixth real result, and the fifth loss can be used for updating the parameters of the second routing network and the parameters of the second expert network of the fourth computing node and the third computing node respectively to obtain the second routing network after the first update of the parameters and the second expert network after the first update of the parameters. Then, the fourth computing node and the third computing node perform a second round, and so on, until the fourth computing node and the third computing node perform a certain round (for example, the tenth round, etc.). The second routing network after multiple (for example, ten times, etc.) updates of the parameters in the round is the fourth routing network, and the second expert network after multiple (for example, ten times, etc.) updates of the parameters in the round is the fourth expert network.

[0116] Then, according to the indication of the first communication instruction, the second computing node can send the parameters of the third routing network to the fourth computing node, receive the parameters of the fourth routing network sent by the fourth computing node, and perform weighted summation on the parameters of the third routing network and the parameters of the fourth routing network (equivalent to aggregating the third routing network and the fourth routing network) to obtain the first aggregated routing network, that is, the fifth routing network.

[0117] According to the indication of the second communication instruction, the fourth computing node can send the parameters of the fourth routing network to the third computing node, receive the parameters of the third routing network sent by the second computing node, and perform weighted summation on the parameters of the fourth routing network and the parameters of the third routing network to obtain the first aggregated routing network, that is, the fifth routing network.

[0118] Then, the first computing node, the second computing node, the third computing node and the fourth computing node can perform the above steps cyclically until a training completion condition (for example, one thousand rounds are performed, or the fourth loss and the fifth loss converge, etc.) is met. The routing network aggregated by the second computing node and the fourth computing node for the last time and the last round updated expert network obtained by the first computing node and the third computing node are the final model after training, which can be returned to the cloud management platform and then returned to the tenant for use.

[0119] For example, as shown in FIG. 6 (which is another structural schematic diagram of the cloud service system provided in the embodiments of the present application), when the tenant needs to train the MOE to be trained, the tenant can log in to the cloud management platform, and the cloud management platform can provide a tenant interface, which includes a model training column. The tenant can input a model training task, a cross-cluster training strategy, and an intra-cluster training strategy into the model training column. The model training task includes the MOE to be trained (i.e., the aforementioned model to be trained), a training completion condition (1000 rounds of training are completed), a data set, the cross-cluster training strategy, and the intra-cluster training strategy, which is an expert parallel mode.

[0120] Then, the cloud management platform can divide the model training task into six sub-tasks, including training task 1 to training task 2 (i.e., the aforementioned first training task), training task 3 to training task 4 (i.e., the aforementioned second training task), communication task 1 (i.e., the aforementioned first communication task), and communication task 2 (i.e., the aforementioned second communication task), based on the cross-cluster training strategy and the intra-cluster training strategy.

[0121] Next, the cloud management platform can create cluster 1 and cluster 2. Cluster 1 includes computing nodes 1 to 3 (including the aforementioned first computing node and the second computing node), and cluster 2 includes computing nodes 4 to 6 (including the aforementioned third computing node and the fourth computing node). Cluster 1 is arranged at a site located in geographic location 1, and cluster 2 is arranged at a site located in geographic location 2.

[0122] Then, the cloud management platform can schedule training task 1 to training task 2 to computing nodes 1 to 2, schedule communication task 1 to computing node 3, schedule training task 3 to 4 to computing nodes 4 to 5, and schedule communication task 3 to computing node 6.

[0123] Subsequently, since communication task 1 includes the routing network 1 (i.e., the aforementioned first routing network) of the MOE to be trained and the data set, and training task 1 to training task 2 includes the expert network 1 and the expert network 2 (i.e., the aforementioned first expert network) of the MOE to be trained, the data set, and the training completion condition, the expert network 1 is deployed on computing node 1, the expert network 2 is deployed on computing node 2, and the routing network 1 is deployed on computing node 3. Based on this, computing nodes 1, 2, and 3 train the routing network 1, the expert network 1, and the expert network 2 using the data in the data set, and assume that 10 rounds of training are performed, obtaining the routing network 1 after 10 times of parameter updates (i.e., the aforementioned third routing network), the expert network 1 after 10 times of parameter updates, and the expert network 2 after 10 times of parameter updates (which are the aforementioned third expert network).

[0124] Similarly, since the communication task 2 comprises the routing network 2 (i.e., the second routing network as mentioned above) and the data set, the training task 3 to the training task 4 comprises the expert network 3 and the expert network 4 (i.e., the second expert network as mentioned above) of the MOE to be trained, the data set and the training completion condition, the expert network 3 is deployed on the computing node 4, the expert network 4 is deployed on the computing node 5, and the routing network 2 is deployed on the computing node 6. Based on this, the computing node 4, the computing node 5 and the computing node 6 train the routing network 2, the expert network 3 and the expert network 4 by using the data in the data set, and suppose that the training is performed for 10 rounds, and the routing network 2 after 10 times of parameter update (i.e., the fourth routing network as mentioned above), the expert network 3 after 10 times of parameter update and the expert network 4 after 10 times of parameter update (i.e., the fourth expert network as mentioned above) are obtained.

[0125] Then, the computing node 3 and the computing node 6 can aggregate the routing network 1 after 10 times of parameter update and the routing network 2 after 10 times of parameter update (i.e., exchange the parameters of the two networks, and perform weighted summation on the parameters of the two networks, and the parameters after the weighted summation are the parameters of the routing network obtained by the first aggregation), so as to obtain the routing network obtained by the first aggregation (i.e., the fifth routing network as mentioned above). Similarly, the computing node 1 to the computing node 6 can continuously perform the above-mentioned operation until the training completion condition is met (for example, the training is performed for 1000 rounds), and the routing network obtained by the last aggregation of the computing node 3 and the computing node 6 and the 1000 times of parameter update of the expert network obtained by the remaining four computing nodes in the 1000th round can be regarded as the MOE after the training is completed.

[0126] In the embodiments of this application, when a tenant has a model training requirement, the tenant can send a model training task, a first training strategy to be followed by multiple clusters for jointly executing the model training task, and a second training strategy to be followed by a single cluster for executing the model training task to a model training interface provided by a cloud management platform. Then, the cloud management platform can divide the model training task into a first subtask to be executed by a first cluster and a second subtask to be executed by a second cluster based on the first training strategy and the second training strategy. Subsequently, the cloud management platform creates the first cluster and the second cluster in the infrastructure and notifies the first cluster to execute the first subtask and the second cluster to execute the second subtask, thereby completing the model training task formulated by the tenant. In the foregoing process, the cloud management platform can split the model training task formulated by the tenant into subtasks to be executed by a plurality of clusters (i.e., the first cluster and the second cluster) according to the cross-cluster training strategy (i.e., the first training strategy) and the intra-cluster training strategy (i.e., the second training strategy) set by the tenant, and notify the plurality of clusters to execute the corresponding subtasks, thereby realizing the joint completion of the model training task of the tenant by the plurality of clusters and realizing the cross-cluster execution of the model training. As can be seen, regardless of the size of the model training task, the cloud management platform can invoke a plurality of clusters with sufficient performance to carry the model training task of the tenant, thereby meeting the model training requirement of the tenant.

[0127] Further, in the embodiments of this application, the cloud management platform can reduce the communication volume between the plurality of clusters in various ways, for example, by performing a certain number of rounds of model training within the cluster before aggregating the model, to reduce the number of rounds of communication, and for example, by quantizing and compressing the data processing results output by the model to reduce the communication volume required in each round, and the like, which is conducive to the implementation of cross-cluster model training.

[0128] Further, in the embodiments of this application, after splitting the model training task into a plurality of subtasks, the cloud management platform can determine the specifications of the resources required by each subtask, and then determine the specifications of the resources required by each cluster to which a certain number of subtasks are allocated, thereby realizing accurate allocation and scheduling of cluster resources, which is conducive to the implementation of cross-cluster model training.

[0129] The above is a detailed description of the model training method based on the cloud management platform provided by the embodiments of this application. The cloud management platform provided by the embodiments of this application will be introduced below. FIG. 7 is a structural schematic diagram of a cloud management platform provided by the embodiments of this application, as shown in FIG. 7, the cloud management platform is used to manage an infrastructure providing cloud services, and the cloud management platform comprises:

[0130] The receiving module 701 is configured to receive, through a model training interface, a model training task sent by a tenant, a first training strategy that needs to be followed by multiple clusters for jointly performing the model training task, and a second training strategy that needs to be followed by a single cluster for performing the model training task; for example, the receiving module 701 is configured to implement step 301 in the embodiment shown in FIG. 3.

[0131] The splitting module 702 is configured to split, based on the first training strategy and the second training strategy, the model training task into a first subtask that needs to be performed by a first cluster and a second subtask that needs to be performed by a second cluster, the first cluster and the second cluster being located in different sites; for example, the splitting module 702 is configured to implement step 302 in the embodiment shown in FIG. 3.

[0132] The performing module 703 is configured to create the first cluster and the second cluster in the infrastructure, and notify the first cluster to perform the first subtask and notify the second cluster to perform the second subtask. For example, the performing module 703 is configured to implement step 303 in the embodiment shown in FIG. 3.

[0133] In a possible implementation, the model training task is used to instruct training of a to-be-trained model, the first training strategy is a data parallel manner, the second training strategy is a model parallel manner, the first subtask is used to instruct the first cluster to train the to-be-trained model based on the model parallel manner and train the to-be-trained model based on the data parallel manner with the second cluster, and the second subtask is used to instruct the second cluster to train the to-be-trained model based on the model parallel manner and train the to-be-trained model based on the data parallel manner with the first cluster.

[0134] In a possible implementation, the model training task is used to instruct training of a to-be-trained model, the first training strategy is a model parallel manner, the second training strategy is a data parallel manner, the first subtask is used to instruct the first cluster to train the to-be-trained model based on the data parallel manner and train the to-be-trained model based on the model parallel manner with the second cluster, and the second subtask is used to instruct the second cluster to train the to-be-trained model based on the data parallel manner and train the to-be-trained model based on the model parallel manner with the first cluster.

[0135] In a possible implementation, the model training task is used to instruct training of a to-be-trained model, the first training strategy and the second training strategy are both expert parallel manners, the first subtask is used to instruct the first cluster to train the to-be-trained model based on the expert parallel manner and train the to-be-trained model based on the expert parallel manner with the second cluster, and the second subtask is used to instruct the second cluster to train the to-be-trained model based on the expert parallel manner and train the to-be-trained model based on the expert parallel manner with the first cluster.

[0136] In a possible implementation, the model training task is configured to instruct training of a to-be-trained model based on a first data set and a second data set, the to-be-trained model comprising a first sub-model and a second sub-model, and the execution module is configured to: instruct the first cluster to sequentially process the first data set through the first sub-model and the second sub-model to obtain a first processing result, and update parameters of the first sub-model and parameters of the second sub-model based on the first processing result to obtain a first model; instruct the second cluster to sequentially process the second data set through the first sub-model and the second sub-model to obtain a second processing result, and update the parameters of the first sub-model and the parameters of the second sub-model based on the second processing result to obtain a second model; and instruct the first cluster and the second cluster to aggregate the first model and the second model to obtain a third model.

[0137] In a possible implementation, the model training task is configured to instruct training of a to-be-trained model based on a data set, the to-be-trained model comprising a first sub-model and a second sub-model, and the data set comprising a first data set and a second data set, and the execution module is configured to: instruct the first cluster to process the first data set and the second data set through the first sub-model respectively to obtain a third processing result and a fourth processing result; instruct the first cluster to compress the third processing result and the fourth processing result, and send the compressed third processing result and the compressed fourth processing result to the second cluster; and instruct the second cluster to process the compressed third processing result and the compressed fourth processing result through the second sub-model to obtain a fifth processing result and a sixth processing result, the fifth processing result and the sixth processing result being used for updating parameters of the first sub-model and parameters of the second sub-model by the first cluster and the second cluster respectively to obtain a fourth model.

[0138] In a possible implementation, the model training task is configured to instruct training of a to-be-trained model based on a data set, the to-be-trained model comprising a first routing network, a first expert network, and a second expert network, and the execution module is configured to: instruct the first cluster to sequentially process the data set through the first routing network and the first expert network to obtain a seventh processing result, and update parameters of the first routing network and parameters of the first expert network based on the seventh processing result to obtain a third routing network and a third expert network; instruct the second cluster to sequentially process the data set through a second routing network and the second expert network to obtain an eighth processing result, and update parameters of the second routing network and parameters of the second expert network based on the eighth processing result to obtain a fourth routing network and a fourth expert network, the second routing network being obtained by copying the first routing network; and instruct the first cluster and the second cluster to aggregate the third routing network and the fourth routing network to obtain a fifth routing network.

[0139] In a possible implementation, the first cluster includes a plurality of first computing nodes, and the plurality of first computing nodes are configured to execute the first subtask. The first computing node includes any one of the following: a physical server, a virtual machine, a container, a micro virtual machine, and a bare metal server.

[0140] In a possible implementation, the site includes any one of the following: a region, an availability zone, a data center, a machine room, and a rack.

[0141] It should be noted that the information interaction and implementation process between the modules / units of the apparatus described above are based on the same concept as the method embodiments of the present application, and the technical effects brought by the method embodiments of the present application are the same. For specific content, refer to the description of the method embodiments described above.

[0142] Please refer to FIG. 8, which is a structural schematic diagram of a computing device provided by an embodiment of the present application. As shown in FIG. 8, the computing device 800 (which can be used to present the cloud management platform described above) includes a processor 801, a memory 802, a communication interface 803, and a bus 804. The processor 801, the memory 802, and the communication interface 803 are coupled through a bus (not shown in the figure). The memory 802 stores instructions, and when the instructions in the memory 802 are executed, the computing device 800 executes the method performed by the cloud management platform in the method embodiments described above.

[0143] The computing device 800 can be one or more integrated circuits configured to implement the above method, such as one or more application specific integrated circuits (ASICs), or one or more digital signal processors (DSPs), or one or more field programmable gate arrays (FPGAs), or a combination of at least two of these integrated circuit forms. For another example, when the units in the apparatus can be implemented in the form of a processing element scheduler, the processing element can be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call a program. For another example, these units can be integrated together in the form of a system-on-a-chip (SOC).

[0144] The processor 801 can be a central processing unit (CPU), and can also be other general-purpose processors, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor can be a microprocessor or any conventional processor.

[0145] The memory 802 can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memory. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically EPROM (EEPROM) or a flash memory. The volatile memory can be a random access memory (RAM) used as an external cache. By way of example but not limitation, many forms of RAM can be used, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced SDRAM (ESDRAM), synchlink DRAM (SLDRAM) and direct rambus RAM (DR RAM).

[0146] The memory 802 stores executable program code, and the processor 801 executes the executable program code to respectively implement the functions of the aforementioned receiving module, splitting module, and executing module and the like, thereby implementing the cloud management platform-based model training method described above. That is, the memory 802 has instructions for executing the cloud management platform-based model training method described above.

[0147] The communication interface 803 uses a transceiving module such as, but not limited to, a network interface card, a transceiver, to enable communication between the computing device 800 and other devices or communication networks.

[0148] The bus 804 can include, in addition to a data bus, a power bus, a control bus, and a state signal bus, etc. The bus can be a peripheral component interconnect express (PCIe) bus, or an extended industry standard architecture (EISA) bus, a unified bus (Ubus or UB), a compute express link (CXL), a cache coherent interconnect for accelerators (CCIX), etc. The bus can be divided into an address bus, a data bus, a control bus, etc.

[0149] Referring to FIG. 9, FIG. 9 is a structural schematic diagram of a computing device cluster provided by an embodiment of the present application. As shown in FIG. 9, the computing device cluster 900 includes at least one computing device 800.

[0150] As shown in FIG. 9, the computing device cluster 900 includes at least one computing device 800. The memory 802 in one or more computing devices 800 in the computing device cluster 900 can have the same instructions for performing the cloud management platform-based model training method described above.

[0151] In some possible implementation manners, the memory 802 in one or more computing devices 800 in the computing device cluster 900 can also respectively have partial instructions for performing the cloud management platform-based model training method described above. In other words, the combination of one or more computing devices 800 can collectively perform the cloud management platform-based model training method described above.

[0152] It should be noted that the memory 802 in different computing devices 800 in the computing device cluster 900 can store different instructions, respectively, for performing part of the functions of the cloud management platform described above. That is, the instructions stored in the memory 802 in different computing devices 800 can implement the functions of one or more of the receiving module, the splitting module, the executing module, and the like.

[0153] In some possible implementation manners, one or more computing devices 800 in the computing device cluster 900 can be connected through a network. The network can be a wide area network or a local area network, etc.

[0154] Referring to FIG. 10, FIG. 10 is a schematic diagram of connection of computer devices in a computer cluster provided by an embodiment of the present application through a network. As shown in FIG. 10, two computer devices 800A and 800B are connected through a network. Specifically, the communication interface in each computer device is connected to the network.

[0155] In a possible implementation, the memory in the computer device 800A stores instructions for performing the functions of the receiving module and other modules. Meanwhile, the memory in the computer device 800B stores instructions for performing the functions of the splitting module, the execution module and other modules.

[0156] It should be understood that the functions of the computer device 800A shown in FIG. 10 can also be completed by multiple computer devices. Similarly, the functions of the computer device 800B can also be completed by multiple computer devices.

[0157] The embodiments of the present application also relate to a computer storage medium, which stores a program for signal processing, and when the program is run on a computer, the computer executes the steps performed by the cloud management platform in the embodiment shown in FIG. 3.

[0158] The embodiments of the present application also relate to a computer program product, which stores instructions, and when the instructions are executed by a computer, the computer executes the steps performed by the cloud management platform in the embodiment shown in FIG. 3.

[0159] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described system, device and unit can refer to the corresponding processes in the foregoing method embodiments, which will not be described herein.

[0160] In several embodiments provided by the present application, it should be understood that the disclosed system, device and method can be implemented by other manners. For example, the above-described device embodiments are merely schematic, for example, the division of the units is merely a logical function division, and actual implementation can have another division manner, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0161] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0162] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware, or in the form of a software functional unit.

[0163] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part of the prior art that contributes to the technical solutions or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, read-only memory), a random access memory (RAM, random access memory), a magnetic disk or an optical disk, and various program code storage media.

Claims

1. A model training method based on a cloud management platform, characterized in that, The cloud management platform is used to manage the infrastructure that provides cloud services, and the method includes: The cloud management platform receives model training tasks sent by tenants through the model training interface. The first training strategy that multiple clusters must follow when executing the model training task together, and the second training strategy that a single cluster must follow when executing the model training task. Based on the first training strategy and the second training strategy, the cloud management platform splits the model training task into a first sub-task to be executed by a first cluster and a second sub-task to be executed by a second cluster, wherein the first cluster and the second cluster are located in different sites; The cloud management platform creates the first cluster and the second cluster in the infrastructure, and notifies the first cluster to execute the first sub-task and notifies the second cluster to execute the second sub-task.

2. The method according to claim 1, characterized in that, The model training task is used to instruct the training of the model to be trained. The first training strategy is data parallelism, and the second training strategy is model parallelism. The first subtask is used to instruct the first cluster to train the model to be trained based on the model parallelism, and the second cluster to train the model to be trained based on the data parallelism. The second subtask is used to instruct the second cluster to train the model to be trained based on the model parallelism, and the first cluster to train the model to be trained based on the data parallelism.

3. The method according to claim 1, characterized in that, The model training task is used to instruct the training of the model to be trained. The first training strategy is model parallelism, and the second training strategy is data parallelism. The first subtask is used to instruct the first cluster to train the model to be trained based on the data parallelism, and the second cluster to train the model to be trained based on the model parallelism. The second subtask is used to instruct the second cluster to train the model to be trained based on the data parallelism, and the first cluster to train the model to be trained based on the model parallelism.

4. The method according to claim 1, characterized in that, The model training task is used to instruct the training of the model to be trained. Both the first training strategy and the second training strategy are expert parallel methods. The first subtask is used to instruct the first cluster to train the model to be trained based on the expert parallel method, and the second cluster to train the model to be trained based on the expert parallel method. The second subtask is used to instruct the second cluster to train the model to be trained based on the expert parallel method, and the first cluster to train the model to be trained based on the expert parallel method.

5. The method according to claim 2, characterized in that, The model training task is used to instruct the training of a model to be trained based on a first dataset and a second dataset. The model to be trained includes a first sub-model and a second sub-model. The cloud management platform notifies the first cluster to execute the first sub-task and notifies the second cluster to execute the second sub-task, including: The cloud management platform notifies the first cluster to process the first dataset sequentially through the first sub-model and the second sub-model to obtain the first processing result, and updates the parameters of the first sub-model and the second sub-model based on the first processing result to obtain the first model; The cloud management platform notifies the second cluster to process the second dataset sequentially through the first sub-model and the second sub-model to obtain the second processing result, and updates the parameters of the first sub-model and the second sub-model based on the second processing result to obtain the second model; The cloud management platform notifies the first cluster and the second cluster to aggregate the first model and the second model to obtain the third model.

6. The method according to claim 3, characterized in that, The model training task is used to instruct the training of a model to be trained based on a dataset. The model to be trained includes a first sub-model and a second sub-model. The dataset includes a first dataset and a second dataset. The cloud management platform notifies the first cluster to execute the first sub-task and notifies the second cluster to execute the second sub-task, including: The cloud management platform notifies the first cluster to process the first dataset and the second dataset respectively through the first sub-model to obtain the third processing result and the fourth processing result; The cloud management platform notifies the first cluster to compress the third processing result and the fourth processing result, and sends the compressed third processing result and the compressed fourth processing result to the second cluster; The cloud management platform notifies the second cluster to process the compressed third processing result and the compressed fourth processing result through the second sub-model to obtain the fifth processing result and the sixth processing result. The fifth processing result and the sixth processing result are used by the first cluster and the second cluster to update the parameters of the first sub-model and the second sub-model respectively to obtain the fourth model.

7. The method according to claim 4, characterized in that, The model training task is used to instruct the training of a model to be trained based on a dataset. The model to be trained includes a first routing network, a first expert network, and a second expert network. The cloud management platform notifying the first cluster to execute the first sub-task and notifying the second cluster to execute the second sub-task includes: The cloud management platform notifies the first cluster to process the dataset sequentially through the first routing network and the first expert network to obtain a seventh processing result, and updates the parameters of the first routing network and the first expert network based on the seventh processing result to obtain a third routing network and a third expert network. The cloud management platform notifies the second cluster to process the dataset sequentially through the second routing network and the second expert network to obtain an eighth processing result, and updates the parameters of the second routing network and the second expert network based on the eighth processing result to obtain a fourth routing network and a fourth expert network. The second routing network is obtained by replicating the first routing network. The cloud management platform notifies the first cluster and the second cluster to aggregate the third routing network and the fourth routing network to obtain the fifth routing network.

8. The method according to any one of claims 1 to 7, characterized in that, The first cluster includes multiple first computing nodes, which are used to execute the first subtask. The first computing nodes include any one of the following: physical servers, virtual machines, containers, micro virtual machines, and bare metal servers.

9. The method according to any one of claims 1 to 8, characterized in that, The site includes any of the following: region, availability zone, data center, server room, and rack.

10. A cloud management platform, characterized in that, The cloud management platform is used to manage the infrastructure that provides cloud services, and the cloud management platform includes: The receiving module is used to receive model training tasks sent by tenants through the model training interface, a first training strategy that multiple clusters should follow when jointly executing the model training task, and a second training strategy that a single cluster should follow when executing the model training task. The splitting module is used to split the model training task into a first sub-task to be executed by a first cluster and a second sub-task to be executed by a second cluster based on the first training strategy and the second training strategy, wherein the first cluster and the second cluster are located in different sites; An execution module is configured to create the first cluster and the second cluster in the infrastructure, and to notify the first cluster to execute the first subtask and notify the second cluster to execute the second subtask.

11. The cloud management platform according to claim 10, characterized in that, The model training task is used to instruct the training of the model to be trained. The first training strategy is data parallelism, and the second training strategy is model parallelism. The first subtask is used to instruct the first cluster to train the model to be trained based on the model parallelism, and the second cluster to train the model to be trained based on the data parallelism. The second subtask is used to instruct the second cluster to train the model to be trained based on the model parallelism, and the first cluster to train the model to be trained based on the data parallelism.

12. The cloud management platform according to claim 10, characterized in that, The model training task is used to instruct the training of the model to be trained. The first training strategy is model parallelism, and the second training strategy is data parallelism. The first subtask is used to instruct the first cluster to train the model to be trained based on the data parallelism, and the second cluster to train the model to be trained based on the model parallelism. The second subtask is used to instruct the second cluster to train the model to be trained based on the data parallelism, and the first cluster to train the model to be trained based on the model parallelism.

13. The cloud management platform according to claim 10, characterized in that, The model training task is used to instruct the training of the model to be trained. Both the first training strategy and the second training strategy are expert parallel methods. The first subtask is used to instruct the first cluster to train the model to be trained based on the expert parallel method, and the second cluster to train the model to be trained based on the expert parallel method. The second subtask is used to instruct the second cluster to train the model to be trained based on the expert parallel method, and the first cluster to train the model to be trained based on the expert parallel method.

14. The cloud management platform according to claim 11, characterized in that, The model training task is used to instruct the training of the model to be trained based on a first dataset and a second dataset. The model to be trained includes a first sub-model and a second sub-model. The execution module is used to: The first cluster is notified to process the first dataset sequentially through the first sub-model and the second sub-model to obtain the first processing result, and to update the parameters of the first sub-model and the second sub-model based on the first processing result to obtain the first model; The second cluster is notified to process the second dataset sequentially through the first sub-model and the second sub-model to obtain the second processing result, and to update the parameters of the first sub-model and the second sub-model based on the second processing result to obtain the second model; The first cluster and the second cluster are notified to aggregate the first model and the second model to obtain the third model.

15. The cloud management platform according to claim 12, characterized in that, The model training task is used to instruct the training of a model to be trained based on a dataset. The model to be trained includes a first sub-model and a second sub-model, and the dataset includes a first dataset and a second dataset. The execution module is used to: The first cluster is notified to process the first dataset and the second dataset respectively through the first sub-model to obtain the third processing result and the fourth processing result; The first cluster is notified to compress the third processing result and the fourth processing result, and the compressed third processing result and the compressed fourth processing result are sent to the second cluster. The second cluster is notified to process the compressed third processing result and the compressed fourth processing result through the second sub-model to obtain the fifth processing result and the sixth processing result. The fifth processing result and the sixth processing result are used by the first cluster and the second cluster to update the parameters of the first sub-model and the second sub-model respectively to obtain the fourth model.

16. The cloud management platform according to claim 13, characterized in that, The model training task is used to instruct the training of a model to be trained based on a dataset. The model to be trained includes a first routing network, a first expert network, and a second expert network. The execution module is used for: The first cluster is notified to process the dataset sequentially through the first routing network and the first expert network to obtain a seventh processing result, and the parameters of the first routing network and the first expert network are updated based on the seventh processing result to obtain a third routing network and a third expert network. The second cluster is notified to process the dataset sequentially through the second routing network and the second expert network to obtain an eighth processing result. Based on the eighth processing result, the parameters of the second routing network and the second expert network are updated to obtain a fourth routing network and a fourth expert network. The second routing network is obtained by replicating the first routing network. The first cluster and the second cluster are notified to aggregate the third routing network and the fourth routing network to obtain the fifth routing network.

17. The cloud management platform according to any one of claims 10 to 16, characterized in that, The first cluster includes multiple first computing nodes, which are used to execute the first subtask. The first computing nodes include any one of the following: physical servers, virtual machines, containers, micro virtual machines, and bare metal servers.

18. The cloud management platform according to any one of claims 10 to 17, characterized in that, The site includes any of the following: region, availability zone, data center, server room, and rack.

19. A computing device cluster, characterized in that, The computing device cluster includes at least one computing device, each computing device including a processor and memory: The memory is used to store instructions; The processor is configured to, according to the instructions, cause the computing device cluster to perform the method of any one of claims 1 to 9.

20. A computer storage medium, characterized in that, The computer storage medium stores one or more instructions that, when executed by one or more computers, cause the one or more computers to perform the method of any one of claims 1 to 9.

21. A computer program product, characterized in that, The computer program product stores instructions that, when executed by a computer, cause the computer to perform the method described in any one of claims 1 to 9.

Images