Model training method based on cloud management platform, and cloud management platform

Abstract

Disclosed in the present application are a model training method based on a cloud management platform, and a cloud management platform, which can efficiently complete model training. The method in the present application comprises: when a tenant has a model training requirement, the tenant sending to a cloud management platform model training tasks which are set by the tenant for a plurality of computing nodes; next, the cloud management platform formulating training strategies required to be followed by the plurality of computing nodes for executing model training conditions; then, on the basis of the model training tasks and the training strategies, the cloud management platform determining a plurality of communication domains formed by the plurality of computing nodes, and the communication traffic generated by each communication domain for executing a model training task; subsequently, the cloud management platform adjusting the communication relationship between computing nodes in each communication domain on the basis of the communication traffic, so as to obtain a plurality of adjusted communication domains; and finally, the cloud management platform creating the plurality of adjusted communication domains, and enabling the plurality of adjusted communication domains to complete the model training tasks.

Description

A model training method based on cloud management platform and cloud management platform

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on April 3, 2024, with application number 202410405792.4, and with the invention name “A model training method based on a cloud management platform and a cloud management platform”, and claims priority to the Chinese patent application filed with the State Intellectual Property Office on June 28, 2024, with application number 202410865874.7, and with the invention name “A model training method based on a cloud management platform and a cloud management platform”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The embodiments of the present application relate to the field of cloud technology, and in particular to a model training method based on a cloud management platform and a cloud management platform. Background Art

[0003] With the development of artificial intelligence (AI) technology, the scale of neural network models is growing. As the parameters and training data of neural network models are growing exponentially, cloud vendors provide tenants with distributed training clusters to complete the training of tenants' neural network models, thereby meeting tenants' model training needs.

[0004] In related technologies, based on tenants' model training needs, cloud providers can plan distributed training clusters for tenants. These distributed training clusters typically consist of multiple communication domains, each containing several compute nodes. For any of these multiple communication domains, the compute nodes within the domain have a certain communication relationship. The cloud provider can adjust the communication relationship between the compute nodes within the domain based on the compute node's priority, thereby optimizing the domain. This ultimately results in multiple optimized communication domains, which can be created to complete model training for tenants.

[0005] In the above process, cloud vendors only consider the priority of computing nodes when optimizing the communication domain. The factors considered are relatively simple, which cannot achieve the best optimization effect on the communication domain. As a result, the overall communication efficiency within the distributed training cluster is poor, and model training cannot be completed efficiently. Summary of the Invention

[0006] The embodiments of the present application provide a model training method and a cloud management platform based on a cloud management platform, which can improve the overall communication efficiency within a distributed training cluster and thereby efficiently complete model training.

[0007] A first aspect of an embodiment of the present application provides a model training method based on a cloud management platform. The cloud management platform used to implement the method can manage infrastructure that provides cloud services. The method includes:

[0008] When a tenant has model training needs, the cloud management platform can provide the tenant with a model training interface, into which the tenant can input the model training tasks they have set for multiple computing nodes. In this way, the cloud management platform can receive the model training tasks sent by the tenant through the model training interface, which are required to be executed by multiple computing nodes.

[0009] After obtaining the model training tasks that need to be performed by multiple computing nodes, the cloud management platform can generate training strategies that the multiple computing nodes need to follow to complete the model training tasks.

[0010] After obtaining the training strategies that multiple computing nodes need to follow to complete the model training tasks, the cloud management platform can obtain multiple communication domains divided by multiple computing nodes based on the model training tasks and training strategies. For any one of these multiple communication domains, the communication domain may include several computing nodes among the multiple computing nodes, and the several computing nodes included in the communication domain have a certain communication relationship.

[0011] After obtaining the communication volume generated by each communication domain executing the model training task, the cloud management platform can adjust the communication relationship between the computing nodes of each communication domain based on this communication volume, thereby obtaining multiple adjusted communication domains. For any one of the multiple adjusted communication domains, the several computing nodes contained in the adjusted communication domain have an adjusted communication relationship.

[0012] After determining the adjusted multiple communication domains, the cloud management platform can create the adjusted multiple communication domains in the infrastructure and enable the adjusted multiple communication domains to complete the model training tasks, thereby meeting the tenant's model training needs.

[0013] The above method demonstrates that when a tenant has a need for model training, they can send the model training tasks they have set for multiple compute nodes to the cloud management platform. The cloud management platform can then formulate a training strategy for these multiple compute nodes to follow in order to execute the model training conditions. Based on the model training tasks and the training strategy, the cloud management platform can then determine the multiple communication domains constructed by these multiple compute nodes and the amount of communication generated by each communication domain performing the model training task. Subsequently, based on the amount of communication generated by each communication domain performing the model training task, the cloud management platform can adjust the communication relationships between the compute nodes in each communication domain, thereby obtaining multiple adjusted communication domains. Finally, the cloud management platform can create the multiple adjusted communication domains and instruct them to complete the model training task. It can be seen from this that when the cloud management platform optimizes multiple communication domains, that is, when adjusting the communication relationship between computing nodes in each communication domain, it not only considers the training strategies that multiple communication domains need to follow to perform model training tasks, but also considers the communication volume generated by multiple communication domains performing model training tasks. Therefore, the factors considered by the cloud management platform are relatively comprehensive, and can achieve high-quality optimization effects on multiple communication domains, thereby improving the overall communication efficiency within the distributed training cluster, and thus efficiently completing model training.

[0014] In one possible implementation, a model training task includes the number of compute nodes, the types of compute nodes, and the parameters of the models to be trained by the compute nodes. In this implementation, when creating a model training task, tenants can define key parameters such as the number of compute nodes required to complete the model training task, the specific types of compute nodes, and the parameters of the models to be trained by the compute nodes.

[0015] In one possible implementation, the training strategy includes at least one of the following: pipeline parallelism, data parallelism, tensor parallelism, and sequence parallelism. In the aforementioned implementation, the training strategy that multiple computing nodes must follow to complete the model training task may include one or more of the various parallel strategies, such as pipeline parallelism, data parallelism, tensor parallelism, and sequence parallelism.

[0016] In one possible implementation, the communication relationship includes at least one of the following: the identifiers of the computing nodes in each communication domain, the physical locations of the computing nodes in each communication domain, and the communication order between the computing nodes in each communication domain. In the aforementioned implementation, for any one of the multiple communication domains indicated by the task topology, the communication relationship between the computing nodes in the communication domain may include one or more of the following information: the identifiers of all computing nodes in the communication domain, the physical locations of all computing nodes in the communication domain, and the communication order between the computing nodes in the communication domain.

[0017] In one possible implementation, the cloud management platform generates a training strategy that multiple computing nodes need to follow to complete a model training task, including: the cloud management platform obtains multiple candidate training strategies based on the number of multiple computing nodes, the types of multiple computing nodes, and the parameters of the models to be trained that the multiple computing nodes need to train; the cloud management platform selects the training strategy that the multiple computing nodes need to follow to complete the model training task from the multiple candidate training strategies. In the aforementioned implementation, after obtaining the model training task that the multiple computing nodes need to perform, the cloud management platform can extract information such as the number of multiple computing nodes, the types of multiple computing nodes, and the parameters of the models to be trained that the multiple computing nodes need to train from the model training task, and perform a series of processing on this information to generate multiple candidate training strategies. Then, the cloud management platform can select a candidate training strategy from the multiple candidate training strategies as the training strategy that the multiple computing nodes need to follow to complete the model training task.

[0018] In one possible implementation, the cloud management platform determines the multiple communication domains constructed by the multiple computing nodes based on the model training task and the training strategy, including: the cloud management platform determines the multiple communication domains constructed by the multiple computing nodes based on the number of the multiple computing nodes, the types of the multiple computing nodes, the parameters of the models to be trained that the multiple computing nodes need to train, and the training strategy. In the aforementioned implementation, after obtaining the training strategy that the multiple computing nodes need to follow to complete the model training task, the cloud management platform can use the information such as the number of the multiple computing nodes, the types of the multiple computing nodes, the parameters of the models to be trained that the multiple computing nodes need to train, and the training strategy that the multiple computing nodes need to follow to complete the model training task as a reference to divide the multiple computing nodes, thereby successfully obtaining multiple communication domains.

[0019] In one possible implementation, the cloud management platform determines the amount of communication generated by each communication domain performing the model training task based on the model training task and the training strategy, including: the cloud management platform determines the basic amount of communication generated by each communication domain performing the model training task based on the parameters of the to-be-trained model that the multiple computing nodes need to train and the training strategy. In the aforementioned implementation, after determining the multiple communication domains divided by the multiple computing nodes, the cloud management platform can perform a series of calculations using the parameters of the to-be-trained model that the multiple computing nodes need to train and the training strategy that the multiple computing nodes need to follow when performing the model training task, thereby accurately predicting the basic amount of communication generated by each of the multiple communication domains performing the model training task.

[0020] In one possible implementation, the cloud management platform adjusts the communication relationship between the computing nodes in each communication domain based on the communication volume, and the multiple communication domains obtained after adjustment include: the cloud management platform adjusts the physical location of the computing nodes in each communication domain based on the basic communication volume, and obtains the multiple communication domains after the initial adjustment; the cloud management platform selects a communication algorithm for the multiple communication domains after the initial adjustment; the cloud management platform determines the actual communication volume generated by each communication domain after the initial adjustment to perform the model training task based on the communication algorithm; the cloud management platform adjusts the communication order between the computing nodes in each communication domain after the initial adjustment based on the actual communication volume, and obtains the multiple communication domains after the re-adjustment. In the aforementioned implementation, after obtaining the basic communication volume generated by the multiple communication domains performing the model training task, for any one of the multiple communication domains indicated by the task topology, the cloud management platform can adjust the physical location of the computing nodes in the communication domain based on the basic communication volume generated by the communication domain performing the model training task, so that the communication domain after the initial adjustment can be obtained. The cloud management platform can also perform similar operations on the remaining communication domains in the multiple communication domains, excluding the specified communication domain. Ultimately, the cloud management platform can obtain multiple communication domains after initial adjustment. After obtaining the multiple communication domains after initial adjustment, the cloud management platform can select a communication algorithm for the multiple communication domains after initial adjustment. Then, for any of the communication domains after initial adjustment, the cloud management platform can predict the actual communication volume generated by executing a model training task in the communication domain after initial adjustment based on the communication algorithm selected for the communication domain after initial adjustment. Then, based on the actual communication volume generated by executing a model training task in the communication domain after initial adjustment, the cloud management platform can adjust the communication order between the computing nodes in the communication domain after initial adjustment, thereby obtaining the re-adjusted communication domain. The cloud management platform can also perform similar operations on the remaining communication domains in the multiple communication domains, excluding the specified communication domain. Ultimately, the cloud management platform can successfully obtain the multiple communication domains after initial adjustment.

[0021] The second aspect of an embodiment of the present application provides a cloud management platform, which is used to manage the infrastructure for providing cloud services. The cloud management platform includes: a receiving module for receiving model training tasks to be performed by multiple computing nodes sent by tenants; a generation module for generating training strategies to be followed by multiple computing nodes to complete the model training tasks; a determination module for determining, based on the model training tasks and the training strategies, multiple communication domains constructed by multiple computing nodes, and the communication volume generated by each communication domain performing the model training tasks, each communication domain including at least two computing nodes from the multiple computing nodes; an adjustment module for adjusting the communication relationship between the computing nodes in each communication domain based on the communication volume to obtain multiple adjusted communication domains; a creation module for creating the adjusted multiple communication domains in the infrastructure, and enabling the adjusted multiple communication domains to complete the model training tasks.

[0022] In one possible implementation, the model training task includes the number of computing nodes, the types of computing nodes, and the parameters of the to-be-trained models that the computing nodes need to train.

[0023] In one possible implementation, the training strategy includes at least one of the following: a pipeline parallel strategy, a data parallel strategy, a tensor parallel strategy, and a sequence parallel strategy.

[0024] In one possible implementation, the communication relationship includes at least one of the following: an identifier of a computing node in each communication domain, a physical location of the computing nodes in each communication domain, and a communication sequence between computing nodes in each communication domain.

[0025] In one possible implementation, a generation module is used to: obtain multiple candidate training strategies based on the number of multiple computing nodes, the types of multiple computing nodes, and the parameters of the to-be-trained models that the multiple computing nodes need to train; and select, from the multiple candidate training strategies, a training strategy that the multiple computing nodes need to follow to complete the model training task.

[0026] In one possible implementation, a determination module is used to determine multiple communication domains constructed by multiple computing nodes based on the number of multiple computing nodes, the types of multiple computing nodes, the parameters of the to-be-trained models that the multiple computing nodes need to train, and the training strategy.

[0027] In one possible implementation, a determination module is used to determine the basic communication volume generated by each communication domain performing a model training task based on parameters of a to-be-trained model that needs to be trained by multiple computing nodes and a training strategy.

[0028] In one possible implementation, the adjustment module is used to: adjust the physical positions of the computing nodes of each communication domain based on the basic communication volume to obtain multiple communication domains after the initial adjustment; select a communication algorithm for the multiple communication domains after the initial adjustment; determine the actual communication volume generated by each communication domain after the initial adjustment performing the model training task based on the communication algorithm; and adjust the communication order between the computing nodes of each communication domain after the initial adjustment based on the actual communication volume to obtain multiple communication domains after the re-adjustment.

[0029] A third aspect of an embodiment of the present application provides a computing device cluster, which includes at least one computing device, each computing device including a processor and a memory: the memory is used to store instructions; the processor is used to enable the computing device cluster to execute the method described in the first aspect or any possible implementation method of the first aspect according to the instructions.

[0030] A fourth aspect of an embodiment of the present application provides a computer storage medium storing one or more instructions, which, when executed by one or more computers, enables the one or more computers to implement the method described in the first aspect or any possible implementation method of the first aspect.

[0031] A fifth aspect of the embodiments of the present application provides a computer program product, which stores instructions. When the instructions are executed by a computer, the computer implements the method described in the first aspect or any possible implementation method of the first aspect.

[0032] In an embodiment of the present application, when a tenant has a need for model training, the tenant can send the model training task set by the tenant for multiple computing nodes to the cloud management platform. Then, the cloud management platform can formulate the training strategy that these multiple computing nodes need to follow to execute the model training conditions. Then, based on the model training task and the training strategy, the cloud management platform can determine the multiple communication domains constructed by these multiple computing nodes, as well as the communication volume generated by each communication domain executing the model training task. Subsequently, the cloud management platform can adjust the communication relationship between the computing nodes of each communication domain based on the communication volume generated by each communication domain executing the model training task, thereby obtaining multiple adjusted communication domains. Finally, the cloud management platform can create multiple adjusted communication domains and enable the adjusted multiple communication domains to complete the model training task. It can be seen from this that when the cloud management platform optimizes multiple communication domains, that is, when adjusting the communication relationship between computing nodes in each communication domain, it not only considers the training strategies that multiple communication domains need to follow to perform model training tasks, but also considers the communication volume generated by multiple communication domains performing model training tasks. Therefore, the factors considered by the cloud management platform are relatively comprehensive, and can achieve high-quality optimization effects on multiple communication domains, thereby improving the overall communication efficiency within the distributed training cluster, and thus efficiently completing model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] FIG1 is a schematic diagram of the structure of a cloud service system provided in an embodiment of the present application;

[0034] FIG2 is a schematic diagram of the structure of a cloud management platform provided in an embodiment of the present application;

[0035] FIG3 is a flow chart of a model training method based on a cloud management platform provided in an embodiment of the present application;

[0036] FIG4 is another structural diagram of the cloud management platform provided in an embodiment of the present application;

[0037] FIG5 is another structural diagram of the cloud management platform provided in an embodiment of the present application;

[0038] FIG6 is a schematic diagram of a structure of a computing device provided in an embodiment of the present application;

[0039] FIG7 is a schematic diagram of a structure of a computing device cluster provided in an embodiment of the present application;

[0040] FIG8 is a schematic diagram of computer devices in a computer cluster provided by an embodiment of the present application being connected via a network. DETAILED DESCRIPTION

[0041] The embodiments of the present application provide a model training method and a cloud management platform based on a cloud management platform, which can improve the overall communication efficiency within a distributed training cluster and thereby efficiently complete model training.

[0042] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, and this is merely a way of distinguishing the objects of the same attributes when describing them in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0043] With the development of AI technology, the scale of neural network models is growing. As the parameters and training data of neural network models are growing exponentially, cloud vendors provide tenants with distributed training clusters to complete the training of tenants' neural network models, thereby meeting tenants' model training needs.

[0044] In related technologies, based on the tenants' model training needs, cloud vendors can plan distributed training clusters for tenants. These distributed training clusters typically include multiple communication domains, each of which contains several computing nodes. For any of the multiple communication domains, the computing nodes in the communication domain have a certain communication relationship. The cloud vendor can adjust the communication relationship between the computing nodes in the communication domain according to the priority of the computing nodes, thereby obtaining an optimized communication domain. In this way, the cloud vendor can ultimately obtain multiple optimized communication domains and create multiple optimized communication domains, which is equivalent to creating an optimized distributed training cluster to complete model training for the tenants.

[0045] In the above process, when cloud vendors optimize the communication domain, that is, when adjusting the communication relationship between computing nodes in the communication domain, they only consider the priority of the computing nodes. The factors considered are relatively simple. This cannot achieve the best optimization effect on the communication domain, resulting in poor overall communication efficiency within the distributed training cluster and inability to efficiently complete model training.

[0046] To address the above issues, the present application provides a model training method based on a cloud management platform. This method can be implemented through a cloud service system. Figure 1 is a schematic diagram of the structure of the cloud service system provided by the present application. As shown in Figure 1, the cloud service system includes an infrastructure that can provide cloud services and a cloud management platform that manages these infrastructures. The cloud management platform and infrastructure are introduced separately below:

[0047] The cloud management platform can coordinate and manage the infrastructure of the entire cloud service system (for example, within the infrastructure, it can plan and create distributed training clusters dedicated to tenants according to the tenant's instructions to complete the tenant's model training tasks, thereby meeting the tenant's model training needs). The cloud management platform can also be open to tenants outside the cloud service system and respond to their requests. For example, the cloud management platform can provide various interfaces such as login interfaces and model training interfaces for access by tenant clients (for example, the terminal device used by the tenant or the browser on the terminal device, etc.). Among them, the cloud management platform can authenticate the tenant's client through the login interface, and after successful authentication, the tenant's client can be allowed to log in to the cloud management platform. For example, the cloud management platform can also allow the tenant's client to send the model training tasks set by the tenant for multiple computing nodes to the cloud management platform through the model training interface, that is, the model training tasks that these multiple computing nodes need to complete. The cloud management platform can then formulate the training strategies that these multiple computing nodes need to follow to execute the model training conditions. Then, based on the model training tasks and training strategies, the cloud management platform can plan multiple communication domains constructed by these multiple computing nodes and predict the communication volume generated by each communication domain performing the model training tasks. Subsequently, based on the communication volume generated by each communication domain performing the model training tasks, the cloud management platform can adjust (i.e., optimize) the communication relationship between the computing nodes in each communication domain (i.e., the internal structure of the communication domain), thereby obtaining multiple adjusted communication domains. Finally, the cloud management platform can create multiple adjusted communication domains in the infrastructure and enable these multiple adjusted communication domains to complete the model training tasks, thereby meeting the tenants' model training needs.

[0048] The infrastructure can include a large number of physical servers equipped with hardware for model training. This hardware can serve as or deploy multiple compute nodes required by tenants. For the multiple compute nodes selected by the tenant, these compute nodes can form multiple communication domains according to the planning of the cloud management platform. These multiple communication domains constitute a distributed training cluster serving the tenant. Within these multiple communication domains, each communication domain contains at least two of the multiple compute nodes. It is understood that while compute nodes within the same communication domain can communicate, compute nodes in different communication domains cannot communicate. For any one of the multiple communication domains, the compute nodes within that domain have certain communication relationships. The communication relationships between the compute nodes within that domain can be adjusted by the cloud management platform to optimize the internal structure of that domain, resulting in an adjusted communication domain. Similarly, for the remaining communication domains, the communication relationships between the compute nodes within those domains can also be adjusted by the cloud management platform, ultimately resulting in multiple adjusted communication domains. It's worth noting that the adjustment rounds for multiple communication domains can be one or multiple rounds. Therefore, the cloud management platform can use the multiple communication domains obtained from the final round of adjustment as the optimal multiple communication domains, that is, the optimal distributed training cluster. Therefore, the cloud management platform can create the optimal distributed training cluster within the infrastructure to execute model training tasks.

[0049] Furthermore, as shown in Figure 2 (Figure 2 is a structural diagram of the cloud management platform provided by an embodiment of the present application), the cloud management platform may include a platform control component and a network service component, and the network service component may include a topology perception module, a structure optimization module and a path planning module. Among them, after the platform control component receives the model training task set by the tenant for multiple computing nodes, it can provide the key parameters in the model training task to the topology perception module. The topology perception module can generate a training strategy that multiple computing nodes need to follow to perform the model training task based on the key parameters in the model training task, and then plan a task topology (ranktable) based on the key parameters and training strategy in the model training task. The task topology is used to indicate multiple communication domains composed of multiple computing nodes. The structure optimization module can predict the communication volume generated by each communication domain performing the model training task, and based on these communication volumes, adjust the content of the task topology, that is, adjust the communication relationship between the computing nodes of each communication domain, thereby obtaining an adjusted task topology, and the adjusted task topology is used to indicate the adjusted multiple communication domains, and provide the adjusted task topology to the platform control component. Based on the adjusted task topology, the path planning module constructs communication paths between the compute nodes in each adjusted communication domain and provides these communication paths to the platform control component. The platform control component then selects hardware within the infrastructure for creating the adjusted multiple communication domains based on the adjusted task topology and connects these hardware along the communication paths to successfully create the adjusted multiple communication domains.

[0050] Furthermore, for multiple computing nodes serving tenants, the computing nodes can be cloud instances in the infrastructure, and these cloud instances can be presented in a variety of ways. For example, these cloud instances can be physical servers selected by the cloud management platform. For another example, these cloud instances can be processors of physical servers selected by the cloud management platform. For another example, these cloud instances can be bare metal servers selected by the cloud management platform. For another example, these cloud instances can be virtual machines (VMs) created by the cloud management platform on physical servers through virtualization technology. For another example, these cloud instances can also be containers (docker) created by the cloud management platform on physical servers through virtualization technology. For another example, these cloud instances can also be micro virtual machines (microVMs) created by the cloud management platform on physical servers through virtualization technology, and so on.

[0051] Furthermore, for a tenant's multiple computing nodes, multiple computing nodes and storage nodes can be deployed in the same site or different sites. The site can be presented in various forms. For example, the site can be a region in the infrastructure, or an availability zone in the infrastructure, or a data center (DC) in the infrastructure, or a room in the infrastructure, or a cabinet in the infrastructure, or a super node in the infrastructure, and so on.

[0052] Based on the aforementioned cloud service system, after receiving a model training task set by a tenant for multiple compute nodes, the cloud management platform can formulate a training strategy for these multiple compute nodes to follow in executing the model training conditions. Then, based on the model training task and the training strategy, the cloud management platform can determine the multiple communication domains constructed by these multiple compute nodes and obtain the communication volume generated by each communication domain during the execution of the model training task. Subsequently, the cloud management platform can use this communication volume to adjust the communication relationships between the compute nodes in each communication domain, thereby obtaining multiple adjusted communication domains. Finally, the cloud management platform can create the multiple adjusted communication domains in the infrastructure and enable these multiple adjusted communication domains to complete the model training task. It can be seen that when the cloud management platform optimizes multiple communication domains, that is, when adjusting the communication relationship between the computing nodes in each communication domain, it not only considers the training strategy that multiple communication domains need to follow to perform model training tasks (because the communication volume generated by multiple communication domains performing model training tasks comes from the training strategy), but also considers the communication volume generated by multiple communication domains performing model training tasks. Therefore, the factors considered by the cloud management platform are relatively comprehensive, and a high-quality optimization effect can be achieved for multiple communication domains, thereby improving the overall communication efficiency within the distributed training cluster, and then efficiently completing the model training. In order to further understand the aforementioned process, the process is further introduced below in conjunction with Figure 3. Figure 3 is a flow chart of a model training method based on a cloud management platform provided in an embodiment of the present application. As shown in Figure 3, the method is implemented by a cloud service system as shown in Figure 1. The cloud service system includes an infrastructure that provides cloud services to tenants and a cloud management platform that manages these infrastructures. The method includes:

[0053] 301. The cloud management platform receives model training tasks sent by tenants for multiple computing nodes to execute.

[0054] In this embodiment, when a tenant has a model training requirement, the cloud management platform can provide a model training interface to the tenant's client (for example, a model training task input field on the tenant interface, etc.). The tenant can then formulate the model training tasks required to be executed by multiple computing nodes and input the tenant-defined model training tasks into the model training interface through its client. In this way, the cloud management platform can receive the model training tasks required to be executed by multiple computing nodes sent by the tenant through the client through the model training interface.

[0055] Specifically, the model training tasks that the tenant specifies for the multiple computing nodes to be executed may include the following information: (1) When specifying a model training task, the tenant may define the number of computing nodes required to complete the model training task. Therefore, the tenant may specify that the model training task includes the number of computing nodes that execute the model training task. (2) At the same time, the tenant may also define the specific types of the multiple computing nodes that complete the model training task. For example, the tenant may select that the multiple computing nodes are all processors of a certain specification in a physical server, or that the tenant may select that the multiple computing nodes are all virtual machines of a certain specification, and so on. Therefore, the tenant may specify that the model training task includes the types of the multiple computing nodes that execute the model training task. (3) At the same time, the tenant can also define the parameters of the model to be trained (i.e., the model to be trained that multiple computing nodes need to train) for the model training task. These parameters may include the input data volume of the model to be trained (batch size), the dimension of the hidden layer in the model to be trained (hidden size), the total number of layers of the model to be trained (layer number), and the maximum input length of the model to be trained (sequence length), etc. Therefore, the tenant can make the model training task include the parameters of the model to be trained that multiple computing nodes need to train, etc.

[0056] For example, as shown in Figure 4 (Figure 4 is another structural diagram of the cloud management platform provided by an embodiment of the present application), when a tenant has a model training requirement, the tenant can send a model training task formulated by the tenant to the model training interface provided by the cloud management platform. The model training task may include the number of processors required for the task, the type of processor, and the model parameters of the model to be trained, etc. For example, the number of processors required is 16, the type of processor required is a graphics processing unit (GPU), and the model parameters of the model to be trained are batch_size: 200, hidden_size: 20, layer_num: 10, seq_length: 100, etc.

[0057] After receiving the model training task, the platform control component can extract key parameters such as the number of processors required for the task, the type of processors, and the model parameters of the model to be trained from the model training task, and send these key parameters to the network service component.

[0058] 302. The cloud management platform generates a training strategy for multiple computing nodes to complete the model training task.

[0059] After obtaining the model training tasks that need to be performed by multiple computing nodes, the cloud management platform can generate the training strategies that multiple computing nodes need to follow to complete the model training tasks (that is, the parallelism splitting strategy that multiple computing nodes need to follow to complete the model training tasks). This training strategy is the training method used by multiple computing nodes when training the training model.

[0060] Specifically, the training strategies that multiple computing nodes need to follow to complete the model training task may include one or more parallel strategies such as pipeline parallelism (PP) strategy, data parallelism (DP) strategy, tensor parallelism (TP) strategy, and sequence parallelism (SP) strategy.

[0061] Still as in the above example, after obtaining the key parameters of the model training task, the topology awareness module in the network service component can generate the training strategy that 16 GPUs need to follow to execute the model training task based on these parameters. The training strategy includes the PP strategy and the DP strategy, where PP=4 in the PP strategy (indicating that the model to be trained is divided into 4 parts and deployed on 4 GPUs respectively), and DP=2 in the DP strategy (indicating that the training data of the model to be trained can be divided into 2 parts and input into the model to be trained in units of parts).

[0062] More specifically, the cloud management platform can obtain the training strategies that multiple computing nodes must follow to complete model training tasks in the following ways:

[0063] After obtaining the model training tasks required of multiple computing nodes, the cloud management platform can extract information such as the number of computing nodes, the types of computing nodes, and the parameters of the models to be trained by the multiple computing nodes from the model training tasks. The cloud management platform then performs a series of processing on this information to generate multiple candidate training strategies. The cloud management platform can then randomly select one of the candidate training strategies as the training strategy that the multiple computing nodes must follow to complete the model training tasks.

[0064] As in the previous example, after obtaining the key parameters for the model training task, the topology-aware module in the network service component processes these parameters to obtain four candidate training strategies. Candidate training strategy 1 includes the PP strategy and the DP strategy; candidate training strategy 2 includes the PP strategy, the DP strategy, and the TP strategy; candidate training strategy 3 includes the PP strategy, the DP strategy, and the SP strategy; and candidate training strategy 4 includes the PP strategy, the DP strategy, the TP strategy, and the SP strategy. The topology-aware module then selects candidate training strategy 1 as the training strategy for the 16 GPUs to follow when executing the model training task.

[0065] It should be understood that in this embodiment, the training strategy is only generated by the cloud management platform for schematic introduction. In actual applications, the training strategy can also be provided by the tenant to the cloud management platform. That is, when the tenant sends the model training tasks that need to be completed by multiple computing nodes to the cloud management platform, it can also send the training strategies that need to be followed by multiple computing nodes to execute the model training tasks to the cloud management platform at the same time. There is no restriction here.

[0066] 303. The cloud management platform determines, based on the model training task and the training strategy, multiple communication domains constructed by multiple computing nodes, and the communication volume generated by each communication domain executing the model training task, wherein each communication domain includes at least two computing nodes from the multiple computing nodes.

[0067] After obtaining the training strategies that multiple computing nodes need to follow to complete the model training task, the cloud management platform can perform a topology search on the multiple computing nodes based on the model training task and the training strategy, thereby obtaining a preliminary task topology. The task topology is used to indicate the multiple communication domains obtained by dividing the multiple computing nodes. For any one of the multiple communication domains, the communication domain may include several computing nodes among the multiple computing nodes, and the several computing nodes included in the communication domain have a certain communication relationship, which is recorded in the task topology.

[0068] After obtaining the task topology, the cloud management platform can also evaluate and predict each communication domain indicated by the task topology based on the model training task and training strategy, thereby obtaining the communication volume generated by each communication domain when executing the model training task.

[0069] Specifically, the cloud management platform can obtain the task topology in the following ways:

[0070] After obtaining the training strategies that multiple computing nodes must follow to complete the model training task, the cloud management platform can use information such as the number of multiple computing nodes, the types of multiple computing nodes, the parameters of the models to be trained that the multiple computing nodes must train, and the training strategies that the multiple computing nodes must follow to complete the model training task as a reference to perform a topological search on the multiple computing nodes, thereby obtaining a preliminary task topology. The task topology is used to indicate the multiple communication domains divided by the multiple computing nodes. It should be noted that after the cloud management platform determines the multiple communication domains based on the key parameters of the model training task and the training strategy, the training strategy is also the strategy that these multiple communication domains must follow to subsequently execute the model training task, and will not be further described.

[0071] More specifically, for any one of the multiple communication domains indicated by the task topology, the communication relationship between the computing nodes in the communication domain may include: (1) the unique identifiers of all computing nodes in the communication domain. It should be noted that the cloud management platform can determine in advance the cluster composed of all physical servers in the infrastructure. Since each physical server in the cluster can serve as or contain hardware that carries computing nodes, the computing nodes in the communication domain can be planned by the cloud management platform to be deployed on different hardware of a physical server or hardware of different physical servers. Therefore, the cloud management platform can generate unique identifiers for all these computing nodes in the cluster and record them in the task topology. (2) the physical locations of all computing nodes in the communication domain. It should be noted that since the cloud management platform can plan to deploy each computing node in the communication domain on hardware at different locations of the same physical server in the cluster or hardware of physical servers at different locations, the cloud management platform can determine that these computing nodes have unique physical locations (which can also be understood as network addresses) in the cluster. (3) Information such as the communication sequence between the computing nodes of the communication domain. It should be noted that since the cloud management platform has determined that the communication domain contains several computing nodes, the cloud management platform can plan the communication sequence between these computing nodes. For example, one of these computing nodes is used as the communication starting node in the communication domain, and another computing node is used as the next node of the communication starting node, and so on, until the last computing node among these computing nodes is used as the communication ending node in the communication domain.

[0072] As in the above example, after obtaining the training strategy, the topology awareness module can use the key parameters of the model training task and the training strategy as a reference to perform a topology search for the 16 GPUs, thereby obtaining a preliminary task topology. The task topology is used to indicate the four communication domains obtained by dividing the 16 GPUs, where communication domain 1 includes GPU1 to GPU4, communication domain 2 includes GPU5 to GPU8, communication domain 3 includes GPU9 to GPU12, and communication domain 4 includes GPU13 to GPU16. It is worth noting that the task topology can record the communication relationship between GPU1 to GPU4 in communication domain 1, which can be presented as:

[0073] In the above communication relationship, "server_list" refers to communication domain 1. "server_id": "server_10.155.111.140" is the network address of a physical server in a supernode (the ID of this supernode is 0) selected by the cloud management platform for the four GPUs in communication domain 1, indicating that the topology awareness module selects the four GPUs on this physical server as GPUs 1 to 4. "device_id": "1" indicates that GPU 1 has a global ID of 1, "device_id": "2" indicates that GPU 2 has a global ID of 2, "device_id": "3" indicates that GPU 3 has a global ID of 3, and "device_id": "4" indicates that GPU 4 has a global ID of 4. These IDs can be customized by the topology awareness module. "device_ip": "192.1.27.6" is the global network address of GPU1, "device_ip": "192.2.27.6" is the global network address of GPU2, "device_ip": "192.3.27.6" is the global network address of GPU3, and "device_ip": "192.4.27.6" is the global network address of GPU4. Since the topology awareness module selects the four GPUs on the physical server, the network addresses of these four GPUs are also the network addresses of GPU1 through GPU4. "rank_id": "1" indicates that GPU1 is the starting node in communication domain 1, "rank_id": "2" indicates that GPU2 is the second node in communication domain 1, "rank_id": "3" indicates that GPU3 is the third node in communication domain 1, and "rank_id": "4" indicates that GPU4 is the final node in communication domain 1. This indicates that during model training, the topology awareness module specifies that the data flow goes from GPU1 through GPU2, GPU3, and finally GPU4.

[0074] Similarly, the task topology can record the communication relationship between GPU5 to GPU8 in communication domain 2, the communication relationship between GPU9 to GPU12 in communication domain 3, and the communication relationship between GPU13 to GPU16 in communication domain 4, which will not be expanded here.

[0075] More specifically, the cloud management platform can determine the communication volume generated by each communication domain executing the model training task in the following ways:

[0076] After obtaining the preliminary task topology, the cloud management platform can extract the parameters of the models to be trained that need to be trained by multiple computing nodes from the model training task, and use the parameters of the models to be trained that need to be trained by multiple computing nodes and the training strategies that need to be followed by multiple computing nodes to perform model training tasks to perform a series of calculations, thereby predicting the basic communication volume generated by each communication domain executing the model training task in the multiple communication domains constructed by multiple computing nodes indicated by the task topology.

[0077] As in the previous example, after obtaining the preliminary task topology, the topology awareness module can send the task topology, the key parameters of the model training task, and the training strategy to the structure optimization module. The structure optimization module can then use the key parameters of the model training task (for example, the batch size of the model to be trained) and the training strategy to calculate and predict the basic communication volume generated when executing the model training task in communication domains 1, 2, 3, and 4 indicated by the task topology.

[0078] 304. The cloud management platform adjusts the communication relationship between the computing nodes in each communication domain based on the communication volume to obtain multiple adjusted communication domains.

[0079] After obtaining the communication volume generated by each communication domain executing the model training task, the cloud management platform can adjust the content of the preliminary task topology based on this communication volume, that is, adjust the communication relationship between the computing nodes of each communication domain recorded in the task topology, thereby obtaining the adjusted task topology. The adjusted task topology is used to indicate the adjusted multiple communication domains. For any adjusted communication domain among the adjusted multiple communication domains, the several computing nodes contained in the adjusted communication domain have an adjusted communication relationship, and the adjusted communication relationship is recorded in the adjusted task topology.

[0080] Specifically, the cloud management platform can obtain the adjusted task topology in the following ways:

[0081] After obtaining the basic communication volume generated by multiple communication domains executing model training tasks, for any one of the multiple communication domains indicated by the task topology, the cloud management platform can adjust the physical location of the computing nodes in the communication domain in the task topology based on the basic communication volume generated by the communication domain executing the model training task (for example, the multiple computing nodes contained in the communication domain can be divided into multiple communication pairs according to the rules of whether they will communicate. The cloud management platform can deploy the communication pairs with large communication volume in the same site and the communication pairs with small communication volume in different sites. Since these multiple communication pairs were originally deployed in the same site, this In this way, deploying communication pairs with small communication volume in different sites will change the physical positions of the computing nodes in these communication pairs), so the communication domain after the initial adjustment can be obtained. The computing nodes in the communication domain after the initial adjustment have the communication relationship after the initial adjustment, and the communication relationship includes the identification of the computing nodes in the communication domain after the initial adjustment (compared with the original communication relationship, it has not changed), the physical positions of the computing nodes in the communication domain after the initial adjustment (compared with the original communication relationship, it has changed), and the communication order between the computing nodes in the communication domain after the initial adjustment (compared with the original communication relationship, it has not changed).

[0082] For the remaining communication domains in multiple communication domains except this communication domain, the cloud management platform can also perform similar operations on the remaining communication domains, so the cloud management platform can finally obtain the task topology after the initial adjustment. The task topology after the initial adjustment is used to indicate the multiple communication domains after the initial adjustment, and records the communication relationship after the initial adjustment between the computing nodes of each communication domain after the initial adjustment.

[0083] After obtaining the initially adjusted task topology, the cloud management platform selects a communication algorithm for the initially adjusted multiple communication domains indicated by the initially adjusted task topology. Then, for any of the initially adjusted communication domains, the cloud management platform can predict the actual communication volume generated by the initially adjusted communication domain when executing the model training task based on the communication algorithm selected for the initially adjusted communication domain. Then, based on the actual communication volume generated by the initially adjusted communication domain when executing the model training task, the cloud management platform can adjust the communication order between the computing nodes in the initially adjusted communication domain, thereby obtaining a re-adjusted communication domain. The computing nodes in the re-adjusted communication domain have a re-adjusted communication relationship, which includes the identifiers of the computing nodes in the re-adjusted communication domain (which remain unchanged compared to the initially adjusted communication relationship), the physical locations of the computing nodes in the re-adjusted communication domain (which remain unchanged compared to the initially adjusted communication relationship), and the communication order between the computing nodes in the re-adjusted communication domain (which has changed compared to the initially adjusted communication relationship).

[0084] For the remaining communication domains in multiple communication domains except this communication domain, the cloud management platform can also perform similar operations on the remaining communication domains, so the cloud management platform can finally obtain the readjusted task topology. The readjusted task topology is used to indicate the readjusted multiple communication domains and record the readjusted communication relationship between the computing nodes of each communication domain after the readjustment.

[0085] As in the above example, after obtaining the basic communication volume generated by communication domains 1, 2, 3, and 4 when executing the model training task, in the preliminary task topology, for communication domain 1, since the basic communication volume generated by communication domain 1 when executing the model training task indicates that the basic communication volume between GPU1 and GPU2 and the basic communication volume between GPU2 and GPU3 are both large, while the basic communication volume between GPU3 and GPU4 is small, the structure optimization module does not adjust the physical locations of GPU1 to GPU3, but migrates GPU4 from supernode 0 to a physical server of supernode 1. Therefore, the global network address of GPU4 is adjusted from "device_ip":"192.4.27.6" to "device_ip":"192.4.28.6", and the initially adjusted communication domain 1 can be obtained.

[0086] Similarly, the structure optimization module can also perform similar operations on communication domains 2 to 4 in the preliminary task topology, so that the task topology after the initial adjustment can be obtained, which is used to indicate the communication domain 1 after the initial adjustment to the communication domain 4 after the initial adjustment, and records the communication relationship between GPU1 to GPU4 in the communication domain 1 after the initial adjustment,..., and the communication relationship between GPU13 to GPU16 in the communication domain 4 after the initial adjustment.

[0087] Then, the structure optimization module can select a communication algorithm (e.g., the Ring algorithm, the HD algorithm, etc.) for the initially adjusted communication domains 1 to 4. Specifically, the structure optimization module can select the Ring algorithm for the initially adjusted communication domain 1. The Ring algorithm causes GPUs 1 to 4 in the initially adjusted communication domain 1 to form four communication pairs, namely [GPU 1, GPU 2], [GPU 2, GPU 3], [GPU 3, GPU 4], and [GPU 4, GPU 1]. Based on the Ring algorithm, the structure optimization module can predict the actual communication volume generated when the initially adjusted communication domain 1 performs the model training task, and adjust the predicted communication order between GPUs 1 to GPU 4 in the initially adjusted communication domain 1 based on the actual communication volume. For example, the ranking "rank_id": "1" of GPU 1 can be adjusted to "rank_id": "2", and the ranking "rank_id": "2" of GPU 2 can be adjusted to "rank_id": "1", and so on. Thus, the re-adjusted communication domain 1 can be obtained.

[0088] Similarly, the structure optimization module can also perform similar operations on the initially adjusted communication domain 2 to the initially adjusted communication domain 4 in the task topology after the initial adjustment, so that a readjusted task topology can be obtained, which is used to indicate the readjusted communication domain 1 to the readjusted communication domain 4, and records the communication relationship between GPU1 to GPU4 in the readjusted communication domain 1,..., and the communication relationship between GPU13 to GPU16 in the readjusted communication domain 4.

[0089] It should be understood that in this embodiment, only one round of adjustment of the cloud management platform for multiple communication domains is schematically introduced. In actual applications, the cloud management platform can also perform multiple rounds of adjustments on multiple communication domains. That is to say, after obtaining the task topology after adjustment, that is, after obtaining the task topology obtained in the first round of adjustment, the cloud management platform can generate a new training strategy for multiple computing nodes to execute model training tasks based on the task topology obtained in the first round of adjustment, that is, execute steps 302 to 304 again, and these steps can be executed repeatedly for multiple rounds until the final round of adjusted task topology is obtained, which is used to indicate the multiple communication domains obtained in the final round of adjustment, and then execute step 305.

[0090] 305. The cloud management platform creates multiple adjusted communication domains in the infrastructure and enables the multiple adjusted communication domains to complete the model training task.

[0091] After obtaining the adjusted task topology, the cloud management platform can determine multiple adjusted communication domains based on the adjusted task topology. For each adjusted communication domain, the cloud management platform can create communication paths between the compute nodes in each adjusted communication domain. Therefore, the cloud management platform can create multiple adjusted communication domains in the infrastructure, i.e., optimized distributed training clusters, based on the communication relationships and communication paths between the compute nodes in each adjusted communication domain. Therefore, the cloud management platform can use the adjusted multiple communication domains to complete the tenant's model training tasks, i.e., train the to-be-trained model into the target model, thereby meeting the tenant's model training needs.

[0092] Continuing with the above example, after obtaining the readjusted task topology, the structure optimization module can provide the readjusted task topology to the path planning module. The path planning module can then determine the readjusted communication domains 1 to 4 based on the readjusted task topology. For the readjusted communication domain 1, the path planning module can combine information such as the actual communication volume generated by executing the model training task in the readjusted communication domain 1, the communication algorithm, and the training strategy to construct the communication paths between GPU1 and GPU2, the communication path between GPU2 and GPU3, the communication path between GPU3 and GPU4, and the communication path between GPU4 and GPU1 in the readjusted communication domain 1. When constructing these communication paths, the path planning module will strive to ensure that these communication paths do not conflict. For example, assuming that communication between GPU1 and GPU4 is accomplished through the same switch, the path planning module will ensure that communication between GPU1 and GPU2 is accomplished through port 1 of the switch, communication between GPU2 and GPU3 is accomplished through port 2 of the switch, communication between GPU3 and GPU4 is accomplished through port 3 of the switch, communication between GPU4 and GPU1 is accomplished through port 4 of the switch, and so on.

[0093] Similarly, the path planning module can construct the communication paths between the GPUs in the readjusted communication domain 2 to the communication domain 4, and provide these communication paths and the readjusted task topology to the platform control component. Therefore, the platform control component can create the readjusted communication domain 1 to the communication domain 4 in the infrastructure based on these communication paths and the instructions of the readjusted task topology, that is, successfully create an optimized distributed training cluster to control the distributed training cluster to complete the tenant's model training tasks.

[0094] In an embodiment of the present application, when a tenant has a need for model training, the tenant can send the model training task set by the tenant for multiple computing nodes to the cloud management platform. Then, the cloud management platform can formulate the training strategy that these multiple computing nodes need to follow to execute the model training conditions. Then, based on the model training task and the training strategy, the cloud management platform can determine the multiple communication domains constructed by these multiple computing nodes, as well as the communication volume generated by each communication domain executing the model training task. Subsequently, the cloud management platform can adjust the communication relationship between the computing nodes of each communication domain based on the communication volume generated by each communication domain executing the model training task, thereby obtaining multiple adjusted communication domains. Finally, the cloud management platform can create multiple adjusted communication domains and enable the adjusted multiple communication domains to complete the model training task. It can be seen from this that when the cloud management platform optimizes multiple communication domains, that is, when adjusting the communication relationship between computing nodes in each communication domain, it not only considers the training strategies that multiple communication domains need to follow to perform model training tasks, but also considers the communication volume generated by multiple communication domains performing model training tasks. Therefore, the factors considered by the cloud management platform are relatively comprehensive, and can achieve high-quality optimization effects on multiple communication domains, thereby improving the overall communication efficiency within the distributed training cluster, and thus efficiently completing model training.

[0095] Furthermore, in an embodiment of the present application, the cloud management platform can perform multiple rounds of iterative adjustments (optimizations) for multiple communication domains. During this iterative process, the cloud management platform will also continuously adjust the training strategies that multiple communication domains need to follow to perform model training tasks. Therefore, the cloud management platform can achieve coordinated optimization of training strategies and multiple communication domains to avoid the occurrence of local optimal situations, thereby achieving global optimization for distributed training clusters.

[0096] Furthermore, in an embodiment of the present application, after completing the adjustment of multiple communication domains, the cloud management platform can also realize path planning between computing nodes within the communication domain, and within the same communication domain, the cloud management platform will try its best to reduce conflicts in the communication paths of different communication pairs (i.e., different computing node groups), which can reduce traffic congestion within the distributed training cluster, thereby improving communication efficiency.

[0097] The above is a detailed description of the model training method based on the cloud management platform provided in the embodiment of the present application. The cloud management platform provided in the embodiment of the present application will be introduced below. FIG5 is another structural diagram of the cloud management platform provided in the embodiment of the present application. As shown in FIG5, the cloud management platform is used to manage the infrastructure for providing cloud services. The cloud management platform includes:

[0098] Receiving module 501, used to receive model training tasks to be executed by multiple computing nodes sent by tenants;

[0099] A generation module 502 is used to generate a training strategy that multiple computing nodes need to follow to complete the model training task;

[0100] A determination module 503 is configured to determine, based on the model training task and the training strategy, multiple communication domains constructed by the multiple computing nodes, and the communication volume generated by each communication domain performing the model training task, each communication domain including at least two computing nodes from the multiple computing nodes;

[0101] An adjustment module 504 is configured to adjust the communication relationship between the computing nodes in each communication domain based on the communication volume to obtain a plurality of adjusted communication domains;

[0102] The creation module 505 is used to create multiple adjusted communication domains in the infrastructure and enable the multiple adjusted communication domains to complete the model training task.

[0103] In one possible implementation, the model training task includes the number of computing nodes, the types of computing nodes, and the parameters of the to-be-trained models that the computing nodes need to train.

[0104] In one possible implementation, the training strategy includes at least one of the following: a pipeline parallel strategy, a data parallel strategy, a tensor parallel strategy, and a sequence parallel strategy.

[0105] In one possible implementation, the communication relationship includes at least one of the following: an identifier of a computing node in each communication domain, a physical location of the computing nodes in each communication domain, and a communication sequence between computing nodes in each communication domain.

[0106] In one possible implementation, the generation module 502 is used to: obtain multiple candidate training strategies based on the number of multiple computing nodes, the types of multiple computing nodes, and the parameters of the to-be-trained models that the multiple computing nodes need to train; and select, from the multiple candidate training strategies, the training strategies that the multiple computing nodes need to follow to complete the model training task.

[0107] In one possible implementation, the determination module 503 is used to determine multiple communication domains constructed by multiple computing nodes based on the number of multiple computing nodes, the types of multiple computing nodes, the parameters of the to-be-trained models that the multiple computing nodes need to train, and the training strategy.

[0108] In one possible implementation, the determination module 503 is used to determine the basic communication volume generated by each communication domain performing the model training task based on the parameters of the to-be-trained model that needs to be trained by multiple computing nodes and the training strategy.

[0109] In one possible implementation, the adjustment module 504 is used to: adjust the physical position of the computing nodes of each communication domain based on the basic communication volume to obtain multiple communication domains after the initial adjustment; select a communication algorithm for the multiple communication domains after the initial adjustment; determine the actual communication volume generated by each communication domain after the initial adjustment performing the model training task based on the communication algorithm; and adjust the communication order between the computing nodes of each communication domain after the initial adjustment based on the actual communication volume to obtain multiple communication domains after the re-adjustment.

[0110] It should be noted that the information interaction, implementation process, etc. between the modules / units of the above-mentioned device are based on the same concept as the method embodiment of the present application, and the technical effects they bring are the same as those of the method embodiment of the present application. For specific contents, please refer to the description in the method embodiment shown above in the embodiment of the present application, and no further details will be given here.

[0111] Please refer to Figure 6, which is a schematic diagram of the structure of a computing device provided in an embodiment of the present application. As shown in Figure 6, the computing device 600 (which can be used to present the aforementioned cloud management platform) includes: a processor 601, a memory 602, a communication interface 603, and a bus 604. The processor 601, the memory 602, and the communication interface 603 are coupled via a bus (not labeled in the figure). The memory 602 stores instructions. When the execution instructions in the memory 602 are executed, the computing device 600 executes the method executed by the cloud management platform in the above method embodiment.

[0112] The computing device 600 may be one or more integrated circuits configured to implement the above method, such as one or more application specific integrated circuits (ASICs), one or more digital signal processors (DSPs), one or more field programmable gate arrays (FPGAs), or a combination of at least two of these integrated circuit forms. For example, when a unit in the apparatus can be implemented in the form of a processing element scheduler, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call a program. For example, these units may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0113] The processor 601 may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0114] Memory 602 may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Non-volatile memory may be read-only memory (ROM), programmable ROM (PROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0115] Memory 602 stores executable program code, and processor 601 executes the executable program code to implement the functions of the aforementioned receiving module, generating module, determining module, adjusting module, and creating module, thereby implementing the aforementioned cloud management platform-based model training method. In other words, memory 602 stores instructions for executing the aforementioned cloud management platform-based model training method.

[0116] The communication interface 603 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 600 and other devices or a communication network.

[0117] In addition to the data bus, bus 604 may also include a power bus, a control bus, and a status signal bus. The bus may be a Peripheral Component Interconnect Express (PCIe) bus, an Extended Industry Standard Architecture (EISA) bus, a unified bus (Ubus or UB), a Compute Express Link (CXL), or a Cache Coherent Interconnect for Accelerators (CCIX). Buses can be categorized as address buses, data buses, and control buses.

[0118] Please refer to Figure 7 , which is a schematic diagram of the structure of a computing device cluster provided in an embodiment of the present application. As shown in Figure 7 , the computing device cluster 700 includes at least one computing device 600 .

[0119] As shown in Figure 7, the computing device cluster 700 includes at least one computing device 600. The memory 602 in one or more computing devices 600 in the computing device cluster 700 may store the same instructions for executing the above-mentioned cloud management platform-based model training method.

[0120] In some possible implementations, the memory 602 of one or more computing devices 600 in the computing device cluster 700 may also store partial instructions for executing the aforementioned cloud management platform-based model training method. In other words, the combination of one or more computing devices 600 can jointly execute the aforementioned cloud management platform-based model training method.

[0121] It should be noted that the memory 602 in different computing devices 600 in the computing device cluster 700 may store different instructions, each for executing a portion of the functions of the aforementioned cloud management platform. In other words, the instructions stored in the memory 602 in different computing devices 600 may implement the functions of one or more modules such as the receiving module, the generating module, the determining module, the adjusting module, and the creating module.

[0122] In some possible implementations, one or more computing devices 600 in the computing device cluster 700 may be connected via a network, which may be a wide area network or a local area network.

[0123] Please refer to Figure 8, which is a schematic diagram of computer devices in a computer cluster 800 provided in an embodiment of the present application being connected via a network. As shown in Figure 8, two computing devices 600A and 600B are connected via a network. Specifically, each computing device is connected to the network via a communication interface within the computing device.

[0124] In one possible implementation, the memory of computing device 600A stores instructions for executing functions of a receiving module, etc. Meanwhile, the memory of computing device 600B stores instructions for executing functions of a generating module, a determining module, an adjusting module, and a creating module, etc.

[0125] It should be understood that the functions of the computing device 600A shown in Figure 8 may also be completed by multiple computing devices. Similarly, the functions of the computing device 600B may also be completed by multiple computing devices.

[0126] An embodiment of the present application also relates to a computer storage medium, in which a program for signal processing is stored. When the computer storage medium is run on a computer, the computer executes the steps executed by the cloud management platform in the embodiment shown in Figure 3.

[0127] An embodiment of the present application also relates to a computer program product, which stores instructions that, when executed by a computer, enable the computer to execute the steps performed by the cloud management platform in the embodiment shown in FIG3 .

[0128] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0129] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0130] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0131] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0132] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

Claims

1. A model training method based on a cloud management platform, characterized in that: The cloud management platform is used to manage the infrastructure for providing cloud services, and the method includes: The cloud management platform receives model training tasks to be executed by multiple computing nodes sent by tenants; The cloud management platform generates a training strategy that the multiple computing nodes need to follow to complete the model training task; The cloud management platform determines, based on the model training task and the training strategy, a plurality of communication domains constructed by the plurality of computing nodes, and a communication volume generated by each communication domain executing the model training task, wherein each communication domain includes at least two computing nodes from the plurality of computing nodes; The cloud management platform adjusts the communication relationship between the computing nodes of each communication domain based on the communication volume to obtain a plurality of adjusted communication domains; The cloud management platform creates the adjusted multiple communication domains in the infrastructure and enables the adjusted multiple communication domains to complete the model training task.

2. The method according to claim 1, characterized in that The model training task includes the number of the multiple computing nodes, the types of the multiple computing nodes, and the parameters of the to-be-trained model that the multiple computing nodes need to train.

3. The method according to claim 1 or 2, characterized in that The training strategy includes at least one of the following: a pipeline parallel strategy, a data parallel strategy, a tensor parallel strategy, and a sequence parallel strategy.

4. The method according to any one of claims 1 to 3, characterized in that The communication relationship includes at least one of the following: an identifier of the computing node of each communication domain, a physical location of the computing node of each communication domain, and a communication sequence between the computing nodes of each communication domain.

5. The method according to any one of claims 1 to 4, characterized in that The cloud management platform generates a training strategy for the multiple computing nodes to follow to complete the model training task, including: The cloud management platform obtains a plurality of candidate training strategies based on the number of the plurality of computing nodes, the types of the plurality of computing nodes, and the parameters of the to-be-trained models to be trained for the plurality of computing nodes; The cloud management platform selects a training strategy from the multiple candidate training strategies that the multiple computing nodes need to follow to complete the model training task.

6. The method according to any one of claims 1 to 5, characterized in that The cloud management platform determines, based on the model training task and the training strategy, that the plurality of communication domains constructed by the plurality of computing nodes include: The cloud management platform determines multiple communication domains constructed by the multiple computing nodes based on the number of the multiple computing nodes, the types of the multiple computing nodes, the parameters of the to-be-trained models that the multiple computing nodes need to train, and the training strategy.

7. The method according to any one of claims 1 to 5, characterized in that The cloud management platform determines, based on the model training task and the training strategy, the communication volume generated by each communication domain executing the model training task, including: The cloud management platform determines the basic communication volume generated by each communication domain executing the model training task based on the parameters of the to-be-trained model that needs to be trained by the multiple computing nodes and the training strategy.

8. The method according to claim 7, characterized in that The cloud management platform adjusts the communication relationship between the computing nodes in each communication domain based on the communication volume, and the adjusted communication domains include: The cloud management platform adjusts the physical location of the computing nodes of each communication domain based on the basic communication volume to obtain a plurality of communication domains after initial adjustment; The cloud management platform selects a communication algorithm for the plurality of communication domains after the initial adjustment; The cloud management platform determines, based on the communication algorithm, the actual communication volume generated by each communication domain after the initial adjustment when executing the model training task; The cloud management platform adjusts the communication sequence between the computing nodes of each communication domain after the initial adjustment based on the actual communication volume to obtain a plurality of communication domains after re-adjustment.

9. A cloud management platform, characterized in that: The cloud management platform is used to manage the infrastructure for providing cloud services, and the cloud management platform includes: The receiving module is used to receive model training tasks sent by tenants and executed by multiple computing nodes; A generation module, configured to generate a training strategy for the plurality of computing nodes to follow in completing the model training task; A determination module is configured to determine, based on the model training task and the training strategy, a plurality of communication domains constructed by the plurality of computing nodes, and a communication volume generated by each communication domain executing the model training task, wherein each communication domain includes at least two computing nodes from the plurality of computing nodes; an adjustment module, configured to adjust the communication relationship between the computing nodes of each communication domain based on the communication volume to obtain a plurality of adjusted communication domains; A creation module is used to create the adjusted multiple communication domains in the infrastructure and enable the adjusted multiple communication domains to complete the model training task.

10. The cloud management platform according to claim 9, characterized in that: The model training task includes the number of the multiple computing nodes, the types of the multiple computing nodes, and the parameters of the to-be-trained model that the multiple computing nodes need to train.

11. The cloud management platform according to claim 9 or 10, characterized in that: The training strategy includes at least one of the following: a pipeline parallel strategy, a data parallel strategy, a tensor parallel strategy, and a sequence parallel strategy.

12. The cloud management platform according to any one of claims 9 to 11, characterized in that: The communication relationship includes at least one of the following: an identifier of the computing node of each communication domain, a physical location of the computing node of each communication domain, and a communication sequence between the computing nodes of each communication domain.

13. The cloud management platform according to any one of claims 9 to 12, characterized in that: The generating module is used to: Acquire multiple candidate training strategies based on the number of the multiple computing nodes, the types of the multiple computing nodes, and the parameters of the to-be-trained models that need to be trained for the multiple computing nodes; Among the multiple candidate training strategies, select a training strategy that the multiple computing nodes need to follow to complete the model training task.

14. The cloud management platform according to any one of claims 9 to 13, characterized in that: The determination module is used to determine multiple communication domains constructed by the multiple computing nodes based on the number of the multiple computing nodes, the types of the multiple computing nodes, the parameters of the to-be-trained models that the multiple computing nodes need to train, and the training strategy.

15. The cloud management platform according to any one of claims 9 to 14, characterized in that: The determination module is used to determine the basic communication volume generated by each communication domain executing the model training task based on the parameters of the to-be-trained model that needs to be trained by the multiple computing nodes and the training strategy.

16. The cloud management platform according to claim 15, characterized in that: The adjustment module is used to: Adjusting the physical position of the computing node of each communication domain based on the basic communication volume to obtain a plurality of communication domains after initial adjustment; selecting a communication algorithm for the plurality of communication domains after the initial adjustment; Based on the communication algorithm, determining the actual communication volume generated by each communication domain after the initial adjustment when performing the model training task; Based on the actual communication volume, the communication order between the computing nodes of each communication domain after the initial adjustment is adjusted to obtain a plurality of communication domains after re-adjustment.

17. A computing device cluster, characterized in that: The computing device cluster includes at least one computing device, each computing device including a processor and a memory: The memory is used to store instructions; The processor is configured to cause the computing device cluster to execute the method according to any one of claims 1 to 8 according to the instructions.

18. A computer storage medium, characterized in that The computer storage medium stores one or more instructions, which, when executed by one or more computers, enable the one or more computers to implement the method of any one of claims 1 to 8.

19. A computer program product, characterized in that The computer program product stores instructions, which, when executed by a computer, enable the computer to implement the method according to any one of claims 1 to 8.

Images