Training method and device of AI model, computing device and storage medium

Abstract

The application provides a training method and device of an AI model, a computing device and a storage medium, and belongs to the technical field of artificial intelligence. The method is applied to an AI platform, the AI platform is associated with a computing resource pool, the computing resource pool includes computing nodes for model training, and the method comprises the following steps: providing a training configuration interface to a user, the training configuration interface comprises multiple training modes selected by the user, each training mode represents a distribution strategy of the computing nodes required for training an initial AI model, at least one training task is generated according to the selection of the user on the training configuration interface, the at least one training task is executed to train the initial AI model, an AI model is obtained, and the obtained AI model is downloaded or used by the user. According to the application, distributed training can be more flexibly performed.

Description

[0001] This application claims priority to Chinese Patent Application No. 202010926721.0, filed on September 7, 2020, entitled "A Method and System for Elastic Training", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of artificial intelligence (AI) technology, and in particular to a training method, apparatus, computing device, and storage medium for an AI model. Background Technology

[0003] With the development of artificial intelligence technology, AI models, represented by deep learning, are widely used in various fields, such as image classification, object detection, and natural language processing. Training the initial AI model is a crucial process. Training refers to inputting data from the training dataset into the initial AI model, having it perform calculations, updating its parameters based on the calculation results, and ultimately obtaining an AI model with certain capabilities (e.g., image classification, object detection, natural language recognition).

[0004] Since the training process is complex and requires huge computing resources, using multiple computing nodes to perform distributed training on the initial AI model has become an effective way to meet training efficiency requirements. However, how to perform distributed training more flexibly to balance the user's training needs and resource utilization needs is an urgent problem to be solved. Summary of the Invention

[0005] This application provides a training method, apparatus, computing device, and storage medium for AI models, enabling more flexible distributed training.

[0006] In a first aspect, this application provides a method for training an AI model. This method is applied to an AI platform associated with a computing resource pool, which includes computing nodes for model training. The method includes: providing a training configuration interface to a user, wherein the training configuration interface includes multiple training modes for the user to select, each training mode representing an allocation strategy for computing nodes required to train an initial AI model; generating at least one training task based on the user's selection in the training configuration interface; executing the at least one training task to train the initial AI model to obtain an AI model, which is then available for download or use by the user.

[0007] The solution presented in this application provides users with the ability to select training modes. Users can choose a suitable training mode to generate at least one training task, instead of using conventional distributed training. This allows distributed training to be executed flexibly, thereby balancing the user's training needs and resource utilization.

[0008] In one possible implementation, multiple training modes include a first mode and / or a second mode. The first mode indicates that the number of training tasks is automatically adjusted during the training of the initial AI model, and the second mode indicates that different training tasks share the resources of the same computing node. Multiple training modes may also include a third mode, which represents the regular mode, indicating that distributed training is performed using preset or pre-selected computing nodes.

[0009] The scheme presented in this application can be categorized into two modes: a first mode, also known as a performance mode or turbo mode, and a second mode, also known as a sharing mode or economic mode. The first mode automatically adjusts the number of training tasks within a training job during the initial AI model training process. The second mode allows different training tasks to share the resources of the same computing node. These different training tasks can belong to the same training job or different training jobs. Thus, when using at least the first mode, the number of training tasks can be dynamically adjusted to accelerate training; when using at least the second mode, training resources can be shared with other training jobs, improving resource utilization.

[0010] In one possible implementation, the at least one training task runs on a container, and the method further includes: providing the user with state information of the training process during the training of the initial AI model, wherein the state information includes at least one of the following: the number of containers executing the training task, the resource usage of each container, the number of computing nodes executing the training task, and the resource usage of the computing nodes executing the training task.

[0011] The scheme presented in this application involves at least one training task running on a container, each container containing a complete runtime environment: a training task, all dependencies required to execute that training task, etc. During the training of the initial AI model, the AI ​​platform can also provide users with status information about the training process. This allows users to more intuitively understand the training process.

[0012] In one possible implementation, multiple training modes include a first mode and a second mode, and at least one training task is generated based on the user's selection in the training configuration interface, including: generating at least one training task based on the first mode and the second mode selected by the user in the training configuration interface.

[0013] The solution presented in this application includes multiple training modes, including a first mode and a second mode. The AI ​​platform can generate at least one training task based on the first mode and the second mode selected by the user in the training configuration interface. Thus, by using both the first and second modes simultaneously, the training speed is accelerated because the first mode can dynamically adjust the number of training tasks, and the second mode allows sharing of computing node resources with other training tasks, improving resource utilization.

[0014] In one possible implementation, when the user selects the first mode in the training configuration interface, the training configuration interface also allows the user to input or select the number of containers that can run training tasks; based on the user's selection in the training configuration interface, at least one training task is generated, including: generating at least one training task based on the training mode selected by the user in the training configuration interface and the number of containers that can run training tasks input or selected by the user.

[0015] The solution presented in this application allows the user to input or select the number of containers capable of running training tasks when the user selects the first mode in the training configuration interface. The AI ​​platform can then generate at least one training task based on the training mode and the number of containers selected by the user in the training configuration interface. This user-selectable number of containers makes the training process more intelligent.

[0016] In one possible implementation, when the user selects the second mode in the training configuration interface, the training configuration interface also allows the user to input or select the resource usage of the container running the training task; based on the user's selection in the training configuration interface, at least one training task is generated, including: generating at least one training task based on the training mode selected by the user in the training configuration interface and the resource usage of the container running the training task input or selected by the user.

[0017] The solution presented in this application allows the user to input or select the resource usage of the container running the training task when the user selects the second mode in the training configuration interface. The AI ​​platform can then generate at least one training task based on the training mode selected by the user and the resource usage. This user-selectable resource usage makes the training process more intelligent.

[0018] In one possible implementation, the resource usage of the container running the training task includes less GPU resource usage than a single graphics processing unit (GPU) and / or less video memory usage than a single piece of video memory. This allows for higher resource utilization because the resource usage of a single container is relatively small.

[0019] In one possible implementation, when the first mode is selected, at least one training task is executed to train an initial AI model, including: during the execution of at least one training task to train the initial AI model, when it is detected that the conditions for elastic scaling are met, obtaining the amount of idle computing resources in the computing resource pool; adjusting the number of at least one training task and the number of containers used to run the training tasks according to the amount of idle computing resources in the computing resource pool; and running the adjusted training task in the adjusted containers to train the initial AI model.

[0020] The solution presented in this application, when selecting the first mode, allows the AI ​​platform to detect whether at least one training task meets the conditions for elastic scaling during the execution of at least one training task to train an initial AI model. If the conditions are met, the AI ​​platform can obtain the available computing resources in the computing resource pool. Then, using these available resources, the AI ​​platform adjusts the number of the at least one training task and the number of containers running the training tasks. The AI ​​platform can then run the adjusted training tasks in the adjusted containers to train the initial AI model. This elastic scaling capability accelerates the training process.

[0021] In one possible implementation, adjusting the number of at least one training task and the number of containers used to run the training tasks, and running the adjusted training tasks in the adjusted containers to train the initial AI model, includes: adding a portion of the training tasks from at least one training task to a target container that has already run the training tasks from at least one training task; running multiple training tasks serially in the target container; and during training, using the average value of the model parameters obtained from running the multiple training tasks serially as the updated value of the model parameters.

[0022] The scheme presented in this application reduces the number of containers during scaling down. The scaled-down containers run a portion of at least one training task from the training tasks already running. This portion of the training tasks is then added to the target container, which already has at least one training task running. Since the target container already has training tasks running, adding a portion of the training tasks to the target container results in the target container running multiple training tasks. These multiple training tasks are run sequentially within the target container. The average of the model parameters obtained from running the multiple training tasks sequentially is used as the updated value for the model parameters. Thus, because the multiple training tasks are run sequentially within the target container, it is equivalent to multiple training tasks being executed in a distributed manner, the same as the execution method before scaling down, and this does not lead to a decrease in the training accuracy of the AI ​​model.

[0023] In one possible implementation, when the second mode is selected, the method includes: determining the remaining resources of the compute nodes corresponding to each container based on the resource usage of the containers running at least one training task in the second mode; and using the remaining resources of the compute nodes corresponding to each container to run one or more other training tasks.

[0024] The solution presented in this application, when selecting the second mode, allows the AI ​​platform to calculate the remaining resources of each container's computing nodes by subtracting the used resources from the total resources of the computing nodes corresponding to each container, based on the resource usage of the containers running at least one training task in the second mode. The AI ​​platform can then use these remaining resources to run one or more other training tasks. This maximizes the utilization of the remaining resources on each computing node, improving resource efficiency.

[0025] Secondly, this application provides an AI model training apparatus. The apparatus is applied to an AI platform associated with a computing resource pool, which includes computing nodes for model training. The apparatus includes: a training configuration module for providing a training configuration interface to a user, wherein the training configuration interface includes multiple training modes for the user to select, each training mode representing an allocation strategy for the computing nodes required to train the initial AI model; and a task management module for: generating at least one training task based on the user's selection in the training configuration interface; executing the at least one training task to train the initial AI model to obtain an AI model, which is then available for download or use by the user. In this way, the AI ​​platform provides the user with the function of selecting a training mode. The user can choose a suitable training mode to generate at least one training task, instead of using conventional distributed training, thus enabling flexible execution of distributed training and balancing the user's training needs with resource utilization.

[0026] In one possible implementation, the multiple training modes include a first mode and / or a second mode, wherein the first mode indicates that the number of training tasks is automatically adjusted during the training of the initial AI model, and the second mode indicates that different training tasks share the resources of the same computing node.

[0027] In one possible implementation, the at least one training task runs on a container, and the apparatus further includes:

[0028] The display module is used to provide the user with status information of the training process during the training of the initial AI model. The status information includes at least one of the following: the number of containers executing the training task, the resource usage of each container, the number of computing nodes executing the training task, and the resource usage of the computing nodes executing the training task.

[0029] In one possible implementation, the multiple training modes include a first mode and a second mode, and the task management module is used for:

[0030] At least one training task is generated based on the first mode and the second mode selected by the user in the training configuration interface.

[0031] In one possible implementation, when the user selects the first mode in the training configuration interface, the training configuration interface also allows the user to input or select the number of containers that can run training tasks.

[0032] The task management module is used for:

[0033] At least one training task is generated based on the training mode selected by the user in the training configuration interface and the number of containers for runnable training tasks input or selected by the user.

[0034] In one possible implementation, when the user selects the second mode in the training configuration interface, the training configuration interface also allows the user to input or select the resource usage of the container running the training task;

[0035] The task management module is used for:

[0036] At least one training task is generated based on the training mode selected by the user in the training configuration interface and the resource usage of the container for running the training task input or selected by the user.

[0037] In one possible implementation, the resource usage of the container running the training task includes GPU resource usage less than that of a single graphics processing unit (GPU) and / or memory usage less than that of a single memory unit.

[0038] In one possible implementation, when the first mode is selected, the task management module is configured to:

[0039] During the execution of the at least one training task to train the initial AI model, when it is detected that the conditions for elastic scaling are met, the amount of idle computing resources in the computing resource pool is obtained.

[0040] Based on the amount of idle computing resources in the computing resource pool, adjust the number of at least one training task and the number of containers used to run the training tasks.

[0041] The adjusted training task is run in the adjusted container to train the initial AI model.

[0042] In one possible implementation, the task management module is used for:

[0043] A portion of the training tasks from the at least one training task is added to a target container that has already run the at least one training task. Multiple training tasks are run serially in the target container. During training, the average value of the model parameters obtained from running the multiple training tasks serially is used as the updated value of the model parameters.

[0044] In one possible implementation, when the second mode is selected, the task management module is further configured to:

[0045] The remaining resources of the computing node corresponding to each container are determined based on the resource usage of the container running at least one training task in the second mode.

[0046] Use the remaining resources of the computing node corresponding to each container to run one or more other training tasks.

[0047] Thirdly, a computing device is provided, comprising a processor and a memory, wherein the memory stores computer instructions, and the processor executes the computer instructions to implement the method of the first aspect and its possible implementations.

[0048] Fourthly, a computer-readable storage medium is provided, which stores computer instructions that, when executed by a computing device, cause the computing device to perform the method of the first aspect and its possible implementations, or cause the computing device to perform the function of the apparatus of the second aspect and its possible implementations.

[0049] Fifthly, a computer program product containing instructions is provided, which, when run on a computing device, causes the computing device to perform the method of the first aspect and its possible implementations, or causes the computing device to perform the function of the apparatus of the second aspect and its possible implementations. Attached Figure Description

[0050] Figure 1 A schematic diagram of the structure of an AI platform 100 provided as an exemplary embodiment of this application;

[0051] Figure 2 A schematic diagram illustrating an application scenario of the AI ​​platform 100 provided in an exemplary embodiment of this application;

[0052] Figure 3 A deployment diagram of an AI platform 100 provided as an exemplary implementation of this application;

[0053] Figure 4 A schematic diagram of the structure of a computing device 400 for deploying an AI platform 100 provided as an exemplary implementation of this application;

[0054] Figure 5 A flowchart illustrating a training method for an AI model provided as an exemplary implementation of this application;

[0055] Figure 6 A schematic diagram illustrating the state information of a training process provided as an exemplary implementation of this application;

[0056] Figure 7 A flowchart illustrating a training method for an AI model provided as an exemplary implementation of this application;

[0057] Figure 8 A schematic diagram of a training configuration interface provided for an exemplary implementation of this application;

[0058] Figure 9 A schematic diagram illustrating the expansion provided as an exemplary implementation of this application;

[0059] Figure 10 A schematic diagram illustrating a scaled-down implementation of this application;

[0060] Figure 11 A flowchart illustrating a training method for an AI model provided as an exemplary implementation of this application;

[0061] Figure 12 A flowchart illustrating a training method for an AI model provided as an exemplary implementation of this application;

[0062] Figure 13 This is a schematic diagram of the structure of a computing device provided as an exemplary embodiment of this application. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0064] Currently, the artificial intelligence (AI) boom continues, and machine learning is a core means of realizing AI. Machine learning is permeating various industries such as medicine, transportation, education, and finance. Not only professionals, but also non-AI professionals in various industries are looking forward to using AI and machine learning to complete specific tasks.

[0065] To facilitate understanding of the technical solutions and embodiments provided in this application, the concepts of AI model, AI model training, distributed training, and AI platform are explained in detail below:

[0066] AI models AI models are a class of mathematical algorithm models that use machine learning concepts to solve practical problems. AI models include a large number of parameters and calculation formulas (or rules). The parameters in an AI model are numerical values ​​obtained by training the model on a training dataset. For example, AI model parameters are the weights of calculation formulas or factors within the model. AI models also contain hyperparameters, which are parameters that cannot be obtained through training on a training dataset. Hyperparameters are used to guide the construction or training of the AI ​​model. There are various types of hyperparameters, such as the number of iterations, learning rate, batch size, number of layers, and number of neurons per layer. In other words, the difference between hyperparameters and parameters in an AI model is that the values ​​of hyperparameters cannot be obtained by analyzing the training dataset, while the values ​​of parameters can be modified and determined by analyzing the training dataset during the training process.

[0067] AI models are diverse, with neural network models being one of the most widely used. Neural network models are mathematical algorithm models that mimic the structure and function of biological neural networks (the central nervous system of animals). A neural network model can include multiple neural network layers with different functions, each containing parameters and calculation formulas. Different layers in a neural network model have different names depending on their calculation formulas or functions. For example, the layer that performs convolution calculations is called a convolutional layer, which is often used for feature extraction from input signals (such as images). A neural network model can also be composed of a combination of multiple existing neural network models. Different neural network models can be used in different scenarios (such as classification and recognition) or provide different results when used in the same scenario. Differences in neural network model structure specifically include one or more of the following: different numbers of network layers, different order of network layers, and different weights, parameters, or calculation formulas in each network layer. The industry already has many different neural network models with high accuracy for applications such as recognition or classification. Some neural network models can be trained on specific training datasets and used alone to complete a task or combined with other neural network models (or other functional modules) to complete a task.

[0068] Generally, AI models need to be trained before they can be used to complete a task.

[0069] Training AI models,Supervised training refers to using existing data and certain methods to make an AI model fit the patterns in the existing data, thereby determining the parameters in the AI ​​model. Training an AI model requires a training dataset. Depending on whether the training data in the training dataset is labeled (i.e., whether the data corresponds to specific label information, such as type, name, and bounding boxes), AI model training can be divided into supervised training and unsupervised training. In supervised training, the training data in the training dataset is labeled. During training, the training data in the training dataset is used as input to the AI ​​model. The AI ​​model calculates the output value based on the input training data, using the corresponding labels as a reference. A loss function is used to calculate the loss between the AI ​​model's output value and the corresponding labels, and the parameters in the AI ​​model are adjusted based on the loss value. The AI ​​model is trained iteratively using each piece of training data in the training dataset, and the parameters are continuously adjusted until the AI ​​model can output values ​​that are the same as or similar to the corresponding labels of the input training data with high accuracy. Unsupervised training of AI models involves using unlabeled training data in a dataset. This data is sequentially fed into the AI ​​model, which gradually identifies relationships and underlying rules between the training data until it can determine or identify the type or features of the input data. For example, in clustering, an AI model receiving a large amount of training data can learn the characteristics of each data point and the relationships and differences between them, automatically categorizing the data into multiple types. Different task types can employ different AI models. Some models can only be trained using supervised learning, some only using unsupervised learning, and some using both methods. A trained AI model can then be used to complete a specific task. Generally, AI models in machine learning require supervised training. Supervised training allows the AI ​​model to more effectively learn the relationships between training data and their corresponding labels in a labeled dataset, resulting in higher accuracy when predicting other input inference data.

[0070] loss functionThe loss function is used to measure the degree to which an AI model has been trained (i.e., to calculate the difference between the AI ​​model's prediction and the true target). During AI model training, because we want the AI ​​model's output to be as close as possible to the desired predicted value, we compare the current AI model's predicted value based on the input data with the actual target value (i.e., the labeled input data), and update the parameters of the AI ​​model based on the difference between the two (of course, there is usually an initialization process before the first update, i.e., pre-configuring initial values ​​for the AI ​​model's parameters). Each training iteration uses the loss function to determine the difference between the current AI model's predicted value and the actual target value, updating the AI ​​model's parameters accordingly, until the AI ​​model can predict the actual target value or a value very close to the actual target value, at which point the AI ​​model is considered to have completed training.

[0071] Distributed training Distributed training is a commonly used method to accelerate AI model training. Distributed training involves splitting the training process across multiple independent computing nodes, then periodically aggregating and redistributing the results, thereby speeding up the AI ​​model training process. Current mainstream distributed computing topologies include ps-worker and all-reduce. Distributed training can include data-parallel distributed training and model-parallel distributed training.

[0072] Data-parallel distributed training The training data in the training dataset is distributed across multiple computing nodes for simultaneous computation. Training of the AI ​​model is performed on each computing node, and the gradients of the model parameters generated on each node are aggregated before updating the model parameters. Specifically, there are two options when splitting the training dataset across K computing nodes: 1. The batch size on each of the K computing nodes is the same as the batch size when using a single computing node for computation. The batch size refers to the number of training data points selected from the training dataset before each parameter adjustment. 2. The batch size on each computing node is the batch size when using a single computing node for computation divided by K, thus keeping the aggregated global batch size unchanged. In the following description of the embodiments of this application, the training method of the AI ​​model is described using data-parallel distributed training as an example.

[0073] Model-parallel distributed training This involves splitting the model across multiple computing nodes, while the data itself doesn't need to be split. For large-scale deep learning or machine learning models, memory or GPU memory consumption is very high, so model splitting is a viable option. There are various ways to split models in parallel distributed training. For example, for hierarchical models like neural networks, they can be split layer by layer, with each layer or more placed on a separate computing node.

[0074] AI Platform This is a platform that provides AI developers and users with a convenient AI development environment and convenient development tools. The AI ​​platform has built-in various pre-trained AI models or AI sub-models for solving different problems. The platform can search for and build suitable AI models based on user needs. Users only need to define their needs within the AI ​​platform and prepare and upload their training dataset as prompted. The platform can then train an AI model that can be used to achieve the user's requirements. Alternatively, users can prepare their own algorithm (also known as an initial AI model) and training dataset as prompted and upload them to the AI ​​platform. Based on the user's own algorithm and training dataset, the platform can train an AI model that can be used to achieve the user's requirements. Users can then use the trained AI model to complete their specific tasks. It should be understood that the AI ​​model before being trained by the AI ​​platform (e.g., the algorithm uploaded by the user, the algorithm pre-built by the AI ​​platform, or a pre-trained model) is referred to as the initial AI model.

[0075] To enable more flexible distributed training and to balance user training needs with resource utilization requirements, this application provides an AI platform that introduces multiple training modes, each representing a strategy for allocating computing nodes required for the initial AI model.

[0076] It should be noted that the AI ​​model mentioned above is a general term, and AI models include deep learning models, machine learning models, etc.

[0077] Figure 1 This is a schematic diagram of the structure of the AI ​​platform 100 in the embodiments of this application. It should be understood that... Figure 1 This is merely an illustrative structural diagram of the AI ​​platform 100, and this application does not limit the division of modules within the AI ​​platform 100. Figure 1 As shown, the AI ​​platform 100 includes an algorithm management module 101, a training configuration module 102, a task management module 103, and a data storage module 104. The AI ​​platform is associated with a computing resource pool, which includes multiple computing nodes for model training. The AI ​​platform can schedule computing nodes in the computing resource pool for model training.

[0078] The following is a brief description of the functions of each module in AI Platform 100:

[0079] Algorithm Management Module 101: Provides an initial AI model management interface for users to upload initial AI models created based on their own training objectives; alternatively, users can retrieve existing initial AI models from the initial AI model library. Alternatively, Algorithm Management Module 101 can also be used to retrieve pre-built initial AI models from the AI ​​platform based on the user-input task objective. Initial AI models created by users based on their own training objectives can be written using the framework provided by the AI ​​platform. Initial AI models can include untrained AI models and trained but not fully trained AI models. Untrained AI models refer to AI models that have not yet been trained using a training dataset, and the parameters in the constructed AI model are all preset values.

[0080] Training configuration module 102: Provides a training configuration interface for users. Users can select a training mode in the training configuration interface, which can include normal mode, mode one, and mode two. Mode one can also be called turbo mode or performance mode, and mode two can also be called economic mode or shared mode. In the following description, mode one will be referred to as performance mode, and mode two as shared mode. Normal mode is the existing distributed training mode.

[0081] Among them, performance mode refers to dynamically adjusting the resources used by the initial AI model during the training process.

[0082] Sharing mode: This refers to the situation where, during the training of an AI model, different AI models can share the resources of the same computing node, or different training tasks of the same AI model can share the resources of the same computing node.

[0083] Normal mode: This refers to the training process of AI models, where the training of each AI model occupies all the resources of one or more computing nodes and does not dynamically adjust.

[0084] Optionally, if the user selects the shared training mode, the user can also select the resource usage of the container running the training task in the training configuration interface.

[0085] Optionally, if the user selects performance mode as the training mode, the user can also select the number of containers that can run training tasks in the training configuration interface.

[0086] Optionally, users can also select the initial AI model and configure the input / output object storage service (OBS) path in the training configuration interface.

[0087] Optionally, users can also select the specifications of the computing nodes used to train the initial AI model in the training configuration interface, such as the required size of the graphics processing unit (GPU) and the amount of video memory for the computing nodes used to train the initial AI model.

[0088] Optionally, users can also enter a training dataset for training the initial AI model in the training configuration interface. This training dataset can contain labeled or unlabeled data. Specifically, users can enter the access address of the training dataset.

[0089] Optionally, users can also input their expected performance of the AI ​​model for the task objective and the expected training completion time in the training configuration interface. For example, they can input or select that the final AI model used for face recognition should have an accuracy rate higher than 99%, and expect to complete training within 24 hours.

[0090] The training configuration module 102 can communicate with the algorithm management module 101 to obtain the access address of the initial AI model from the algorithm management module 101. The training configuration module 102 is also used to package training jobs based on the access address of the initial AI model and some content entered or selected by the user in the training configuration interface.

[0091] The configuration module 102 can also communicate with the task management module 103 to submit training jobs to the task management module 103.

[0092] Task Management Module 103: The core module for managing the AI ​​model training process. Task Management Module 103 can communicate with Algorithm Management Module 101, Training Configuration Module 102, and Data Storage Module 104. Specific processing is as follows:

[0093] Based on the training mode, number of containers, container resource usage, and access address of the initial AI model provided by the training configuration module 102, the task management module 103 pulls the corresponding training image and initial AI model, and generates containers to run at least one training task. The containers for at least one training task are then distributed to the computing nodes of the computing resource pool for execution.

[0094] Optionally, the task management module 103 is also used to monitor whether at least one training task meets the scaling conditions, and if the scaling conditions are met, dynamically adjust at least one training task and the container of at least one training task.

[0095] Optionally, the task management module 103 is also used to configure the shared resources of each container. For example, scheduling container 1 and container 2 to a compute node in the compute resource pool.

[0096] Data storage module 104 (e.g., data storage resources corresponding to OBS provided by a cloud service provider): used to store user-uploaded training datasets, user-uploaded initial AI models, other user-uploaded initial AI models, and some configuration items for training modes, etc.

[0097] Optionally, the AI ​​platform also includes a display module 105. Figure 1 (Not shown in the image), the display module 105 communicates with the task management module 103 to obtain the status information of the training process, the AI ​​model that has been trained, etc., and provides the status information and the AI ​​model to the user.

[0098] It should be noted that the AI ​​platform in this application can be a system that can interact with users. This system can be a software system, a hardware system, or a combination of software and hardware. This application does not impose any restrictions.

[0099] Through the functions of the above modules, the AI ​​platform provided in this application embodiment can provide users with flexible distributed training services, enabling the AI ​​platform to balance users' training needs and resource utilization needs.

[0100] Figure 2 This is a schematic diagram illustrating an application scenario of an AI platform 100 provided in an embodiment of this application, such as... Figure 2 As shown, in one embodiment, the AI ​​platform 100 can be entirely deployed in a cloud environment. A cloud environment is an entity that provides cloud services to users using basic resources under a cloud computing model. A cloud environment includes a cloud data center and a cloud service platform. The cloud data center includes a large amount of basic resources (including computing resource pools, storage resources, and network resources) owned by the cloud service provider. The computing resource pool included in the cloud data center can be a large number of computing nodes (e.g., servers). The AI ​​platform 100 can be deployed independently on servers or virtual machines in the cloud data center, or it can be distributed and deployed on multiple servers in the cloud data center, or distributed and deployed on multiple virtual machines in the cloud data center, or distributed and deployed on servers and virtual machines in the cloud data center. Figure 2As shown, the AI ​​platform 100 is abstracted into an AI cloud service by a cloud service provider and offered to users. After purchasing this cloud service on the cloud service platform (pre-payment is possible, with settlement based on final resource usage), the cloud environment utilizes the AI ​​platform 100 deployed in the cloud data center to provide the AI ​​platform cloud service to the user. When using the AI ​​platform cloud service, users can specify the tasks to be performed by the AI ​​model and upload training datasets to the cloud environment through an application program interface (API) or a graphical user interface (GUI). The AI ​​platform 100 in the cloud environment receives the user's task information and training datasets, and performs data preprocessing and AI model training. The AI ​​platform returns status information of the AI ​​model's training process to the user through the API or GUI. The trained AI model can be downloaded by the user or used online to complete specific tasks.

[0101] In another embodiment of this application, when the AI ​​platform in the cloud environment is abstracted as an AI cloud service and provided to users, if the user chooses a shared mode, the user can purchase a fixed amount of container usage time. With a fixed amount of resource usage, the longer the usage time, the higher the cost, and vice versa. Within this usage time, the AI ​​platform trains the AI ​​model. Alternatively, if the user chooses a shared mode, the user can pre-charge, and then settle the bill based on the final number of GPUs used and the usage time after training is complete.

[0102] When a user selects the performance mode, the user can pre-charge and then settle the bill based on the number of GPUs used and the duration of use after training is complete.

[0103] In another embodiment of this application, when the AI ​​platform 100 in the cloud environment is abstracted into an AI cloud service and provided to users, it can be divided into two parts: basic AI cloud service and AI elastic training cloud service. Users can initially purchase only the basic AI cloud service on the cloud service platform, and then purchase the AI ​​elastic training cloud service when needed. After purchase, the cloud service provider provides the AI ​​elastic training cloud service API, and the AI ​​elastic training cloud service is charged additionally based on the number of API calls.

[0104] The AI ​​platform 100 provided in this application has relatively flexible deployment, such as Figure 3As shown, in another embodiment, the AI ​​platform 100 provided in this application can also be deployed in a distributed manner in different environments. The AI ​​platform 100 provided in this application can be logically divided into multiple parts, each with different functions. For example, in one embodiment, the AI ​​platform 100 includes an algorithm management module 101, a training configuration module 102, a task management module 103, and a data storage module 104. Each part of the AI ​​platform 100 can be deployed in any two or three environments: terminal computing devices, edge environments, and cloud environments. Terminal computing devices include: terminal servers, smartphones, laptops, tablets, personal desktop computers, smart cameras, etc. The edge environment is an environment that includes a set of edge computing devices located close to the terminal computing devices, including: edge servers, edge stations with computing capabilities, etc. The various parts of the AI ​​platform 100 deployed in different environments or devices work together to provide users with functions such as training AI models. For example, in one scenario, the algorithm management module 101, training configuration module 102, and data storage module 104 of the AI ​​platform 100 are deployed in a terminal computing device, while the task management module 103 of the AI ​​platform 100 is deployed in an edge computing device in an edge environment. The user sends an initial AI model to the algorithm management module 101 in the terminal computing device, which then stores the initial AI model in the data storage module 104. The user selects a training mode through the training configuration module 102. The task management module 103 in the edge computing device generates at least one training task and executes it. It should be understood that this application does not restrict the specific deployment environment of which parts of the AI ​​platform 100 are deployed; in practical applications, deployment can be adaptively made based on the computing power of the terminal computing device, the resource availability of the edge and cloud environments, or specific application requirements.

[0105] AI Platform 100 can also be deployed independently on a computing device in any environment (such as on an edge server in an edge environment). Figure 4 This is a schematic diagram of the hardware structure of a computing device 400 on which an AI platform 100 is deployed. Figure 4 The computing device 400 shown includes a memory 401, a processor 402, a communication interface 403, and a bus 404. The memory 401, processor 402, and communication interface 403 are interconnected via the bus 404.

[0106] The memory 401 can be a read-only memory (ROM), random access memory (RAM), hard disk, flash memory, or any combination thereof. The memory 401 can store programs. When the program stored in the memory 401 is executed by the processor 402, the processor 402 and the communication interface 403 are used to train an AI model for the user using the AI ​​platform 100. The memory can also store training datasets. For example, a portion of the storage resources in the memory 401 is allocated to a data storage module 104 for storing the data required by the AI ​​platform 100.

[0107] Processor 402 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), a GPU, or any combination thereof. Processor 402 may include one or more chips. Processor 402 may include an AI accelerator, such as a neural processing unit (NPU).

[0108] The communication interface 403 uses a transceiver module, such as a transceiver, to enable communication between the computing device 400 and other devices or communication networks. For example, data can be acquired through the communication interface 403.

[0109] Bus 404 may include a pathway for transmitting information between various components of computing device 400 (e.g., memory 401, processor 402, communication interface 403).

[0110] The following is combined Figure 5 The following describes the specific process of training an AI model in one embodiment, using the example of the method being executed by an AI platform:

[0111] Step 501: The AI ​​platform provides the user with a training configuration interface, which includes multiple training modes for the user to choose from. Each training mode represents an allocation strategy for the computing nodes required to train the initial AI model.

[0112] In this embodiment, if a user wants to train an AI model using the AI ​​platform, they can open the training configuration interface within the AI ​​platform. This training configuration interface can include multiple training modes for the user to choose from. Each training mode represents an allocation strategy for the computing nodes required to train the initial AI model. Specifically, the training configuration interface not only displays multiple training modes but also provides selection options for each mode, along with a description of each mode. Users can select the appropriate training mode for their AI model using the selection options and the descriptions provided.

[0113] Step 502: The AI ​​platform generates at least one training task based on the user's selection in the training configuration interface.

[0114] In this embodiment, the AI ​​platform can obtain the user's selections on the training configuration interface and generate at least one training task based on the user's selections and the initial AI model. This at least one training task is used to train the initial AI model. Executing the training of the initial AI model can be referred to as executing a training job; that is, a training job includes at least one training task.

[0115] Step 503: The AI ​​platform performs at least one training task to train the initial AI model, thereby obtaining an AI model. The obtained AI model is available for download or use by the user for a specific application.

[0116] In this embodiment, the AI ​​platform is associated with a computing resource pool, which includes computing nodes for model training. The computing nodes execute at least one training task to train an initial AI model, obtaining the AI ​​model. The computing nodes then feed the AI ​​model back to the AI ​​platform. The AI ​​platform can provide users with an interface to download AI models, allowing them to download the models and use them to perform corresponding tasks. Alternatively, users can upload inference datasets to the AI ​​platform, where the AI ​​model performs the inference process on the dataset.

[0117] Thus, through the embodiments of this application, users can select a suitable training mode to generate at least one training task, enabling distributed training to be executed flexibly, thereby balancing the user's training needs and resource utilization.

[0118] The following is for Figure 5 The process is further explained below:

[0119] In one possible implementation, multiple training modes may include a performance mode and a shared mode. The performance mode automatically adjusts the number of training tasks based on a certain strategy during the training of the initial AI model. The shared mode means that different training tasks share the resources of the same computing node. These resources may include GPU resources and / or video memory. Here, different training tasks can belong to the same training job or different training jobs. For example, in an AI platform, user A's training job A and user B's training job B are executed respectively. Training job A includes training tasks a and b, and training job B includes training tasks c and d. User A selects the shared mode in the training configuration interface. Then, the AI ​​platform can determine the remaining resources of the computing node corresponding to the container running training task a based on the resource usage of the container running training task a in training job A. If the AI ​​platform determines that the remaining resources are greater than the resource usage of the container running training task c, the AI ​​platform can schedule the container running training task c to the computing node corresponding to the container running training task a.

[0120] In one possible implementation, at least one training task runs on different containers, each container containing a complete runtime environment: a training task, all dependencies required to execute that training task, etc. Here, runtime refers to the dependencies of a program running. Specifically, in step 503, the AI ​​platform can distribute the containers for running at least one training task to computing nodes in the computing resource pool and start the distributed containers. The containers execute at least one training task to train the initial AI model and obtain the AI ​​model.

[0121] During the initial AI model training process, the AI ​​platform can also provide users with training status information, which may include the number of containers executing training tasks and the resource usage of each container. For example... Figure 6 The display shows the number of containers executing training tasks at various time points during the initial AI model training process, represented by a curve of time and container count. It also shows the resource usage of each container. This provides users with a real-time view of the number of containers executing training tasks, intuitively demonstrating the training performance.

[0122] Optionally, the status information may also include the number of computing nodes executing the training task, or the resource usage of the computing nodes executing the training task.

[0123] Optionally, the interface displaying status information may also include information such as the name of the initial AI model (e.g., AA), the specifications of the computing nodes used in the training mode (e.g., performance mode) (e.g., 8 cores), the training input, and the start time (e.g., 2020 / 9 / 27 / 10:38).

[0124] like Figure 7 The diagram shown illustrates the training process of an AI model when the user selects only the performance mode:

[0125] Step 701: The AI ​​platform provides the user with a training configuration interface, which includes multiple training modes for the user to choose from. When the user selects a performance mode in the training configuration interface, the interface also allows the user to input or select the number of containers that can run training tasks.

[0126] In this embodiment, the AI ​​platform can provide a training configuration interface to the user, which includes various training modes for the user to choose from. For example, such as... Figure 8 As shown, the training mode selection interface includes performance mode, shared mode, and normal mode, etc. The user selected performance mode.

[0127] When the user selects performance mode as the training mode, the training configuration interface also provides an option for the user to input or select the number of containers that can run training tasks. The number of containers that can run training tasks is to constrain the number of containers that can be used for each training job when scaling up or down.

[0128] Users can input or select the number of containers available for running training tasks in the training configuration interface. Specifically, the training configuration interface displays the number of containers available for selection, from which users can choose the number of containers that can run training tasks. For example, if the available container numbers are 1, 2, 4, and 8, the user can input or select 1, 2, and 4 as the number of containers available for running training tasks. Alternatively, the training configuration interface displays a range of container numbers, from which users can select the number of containers available for running training tasks. For example, if the container number range is [1, 8], the user can input or select 1, 2, and 4 as the number of containers available for running training tasks.

[0129] It's important to note here that the maximum number of containers that can run training tasks is the maximum number of containers used to run training tasks, and the minimum number of containers that can run training tasks is the minimum number of containers used to run training tasks. Limiting the number of containers is to constrain the range of elastic scaling when executing at least one training task in performance mode.

[0130] Optionally, to facilitate scaling, the number of containers running the training task during scaling can be 2. n And n is greater than or equal to 0 and less than or equal to the target value. For example, the target value can be 4.

[0131] Optional, such as Figure 8As shown, the training configuration interface also displays the dataset source, allowing users to select the training dataset and version.

[0132] Optional, such as Figure 8 As shown, the training configuration interface also displays the resource usage of the containers.

[0133] Optional, such as Figure 8 As shown, the training configuration interface also displays the billing method to prompt the user.

[0134] Optional, such as Figure 8 As shown, the training configuration interface also displays the initial AI model source, which is used to display the initial AI model that has been selected.

[0135] Optional, such as Figure 8 As shown, the training configuration interface also displays options for a public resource pool and a dedicated resource pool for the corresponding computing resource pool. When the user selects the public resource pool, the computing nodes in the public resource pool can be used by multiple training jobs. When the user selects the dedicated resource pool, the computing nodes in the dedicated resource pool are used only by the user's training job. Each computing node in the dedicated resource pool executes multiple training tasks, realizing resource sharing among multiple training tasks and improving resource utilization.

[0136] If a user chooses a public resource pool, billing will be conducted as described above. If a user chooses a dedicated resource pool, billing will be based on the number of compute nodes used and the duration of use.

[0137] Step 702: The AI ​​platform generates at least one training task based on the training mode selected by the user in the training configuration interface and the number of containers for runnable training tasks entered or selected by the user.

[0138] In this embodiment, the AI ​​platform obtains that the training mode selected by the user in the training configuration interface is performance mode. The AI ​​platform can obtain the resource usage of each container in performance mode. When the user selects only performance mode as the training mode, the resource usage of the container running the training task is a preset value, such as the resource usage of the container being all GPU resources and all video memory usage on a single computing node, or the resource usage of the container being two GPU resources and two video memory usages on a single computing node, etc. The AI ​​platform can generate at least one training task based on the idle computing nodes in the current computing resource pool, the number of containers that can run training tasks input or selected by the user, the resource usage of the containers, and the initial AI model. For example, if the resource usage of the container is all GPU resources and all video memory usage on a single computing node, and there are 16 idle computing nodes in the current resource pool, the maximum number of containers that can run training tasks is 8. The AI ​​platform can generate 8 training tasks, each training task running on one container, and each container occupying one computing node.

[0139] It's important to note that when the AI ​​platform generates a training task for the first time, it obtains the maximum number of containers that can run the training task, as well as the resource usage of each container. The AI ​​platform generates this maximum number of training tasks. If, based on the resource usage of each container, the AI ​​platform determines that the current idle resources in the computing resource pool can be used to run the maximum number of containers, then it creates that maximum number of containers. Each training task runs on one container, and different training tasks run on different containers. If, based on the resource usage of each container, the AI ​​platform determines that the current idle resources in the computing resource pool cannot be used to run the maximum number of containers, it determines the number of containers that can run the task and creates that number of training tasks. Since this number is less than the maximum, multiple training tasks run on a single container.

[0140] Step 703: The AI ​​platform performs at least one training task to train the initial AI model and obtain the AI ​​model.

[0141] In this embodiment, the AI ​​platform can distribute containers to computing nodes in the computing resource pool. The computing nodes then run the containers to execute at least one training task to train the initial AI model and obtain the AI ​​model. For example, in step 702, the AI ​​platform identifies eight training tasks and eight containers to run different training tasks. Each container runs on eight different computing nodes. The initial AI model is trained using these eight computing nodes.

[0142] When performance mode is selected, the AI ​​platform can dynamically adjust the number of containers during the training of the AI ​​model, and the processing can be as follows:

[0143] During the execution of at least one training task to train the initial AI model, when the condition for elastic scaling is detected, the amount of idle computing resources in the computing resource pool is obtained; based on the amount of idle computing resources in the computing resource pool, the number of at least one training task and the number of containers running the adjusted training task are adjusted; the adjusted training task is run in the adjusted container to train the initial AI model.

[0144] In this embodiment, during the execution of at least one training task to train the initial AI model, the AI ​​platform can periodically determine whether the proportion of idle computing resources in the computing resource pool exceeds a target value. If it does, it can further obtain the running information of each training task in the computing resource pool, including running time and other information. The running phase can include the training dataset loading phase and the training phase. The AI ​​platform can determine the ratio of the remaining running time to the running time of each training task in the computing resource pool, and determine the speedup ratio of each training task. For a training task, the speedup ratio can be represented by the ratio of the maximum number of containers to the number of containers currently in use. A training task with a speedup ratio of 1 indicates that it has already reached the maximum number of containers, and no adjustment of the number of containers is required.

[0145] The AI ​​platform can determine the ratio of the running time to the remaining running time of each training job, as well as the weighted value of the speedup ratio. The AI ​​platform sorts the training jobs according to their weighted values ​​from largest to smallest. Based on the available computing resources in the computing resource pool and the number of containers that each training job can run, the AI ​​platform identifies the training jobs whose available resources can be used for scaling up, and these are designated as container adjustment targets. As mentioned earlier, training jobs include training tasks. When the AI ​​platform selects at least one training task mentioned in step 701 as a container adjustment target, it determines that the at least one training task meets the scaling conditions. The AI ​​platform can then use the maximum number of containers for at least one training task as the adjusted number of containers. The AI ​​platform can adjust the number of training tasks to match the adjusted number of containers. Then, the AI ​​platform distributes the newly added containers to the computing nodes, where the adjusted training tasks from existing containers run.

[0146] For example, the number of containers capable of running at least one training task is 1, 2, or 4. Figure 9As shown, at least one training task is a training task A, which includes four training processes (training process 1, training process 2, training process 3, and training process 4) and uses one container. Training processes 1, 2, 3, and 4 run on one container, and this container occupies exactly one compute node's resources; currently, one compute node is occupied. If only this one training task A exists in the current compute resource pool, it can be split into four training tasks (training task i, training task j, training task k, and training task o). In this case, each training task includes training process 1, training process 2, training process 3, and training process 4, and the four training tasks run on four containers, with each container located on one compute node. After adjustment, it is equivalent to using four containers, and the four containers occupy four compute nodes.

[0147] The AI ​​platform can determine if a new training task exists. If so, it checks if the computing resources of the nodes in the computing resource pool are sufficient to execute the task. If so, it directly distributes the containers for the training task to the computing nodes. If the task cannot be executed, the AI ​​platform obtains the runtime information of each training task in the computing resource pool, including runtime and other information. The runtime phases can include the training dataset loading phase and the training phase. The AI ​​platform can determine the ratio of the running time to the remaining running time for each training task in the computing resource pool, and also determine the speedup ratio for each task. For a training task, the speedup ratio can be represented by the ratio of the number of containers currently used to the minimum number of containers. A speedup ratio of 1 indicates that the minimum number of containers is already used, and no container adjustment is required. The AI ​​platform can also determine the ratio of the remaining running time to the running time and the weighted value of the speedup ratio for each training task. The AI ​​platform sorts the training tasks in ascending order of weighted values. Based on the amount of idle computing resources in the computing resource pool and the number of containers that can run each training task, the AI ​​platform determines the training tasks that can be scaled down by these idle resources in the sequentially arranged training tasks, and uses these as the objects for container adjustment.

[0148] When the AI ​​platform uses at least one training task mentioned in step 701 as a container adjustment target, it determines that the at least one training task meets the scaling conditions. To ensure that the training tasks on the AI ​​platform can be executed quickly, the AI ​​platform can reduce the number of containers for the at least one training task by one level, resulting in the adjusted number of containers. The AI ​​platform can then adjust the number of training tasks to match the adjusted number of containers. Finally, the AI ​​platform deletes the container and moves the training tasks on that container to other containers for the at least one training task to run.

[0149] For example, the number of containers capable of running at least one training task is 1, 2, or 4. Figure 10 As shown, at least one training task consists of four training tasks (training task 1 includes training process 1, training task 2 includes training process 2, training task 3 includes training process 3, and training task 4 includes training process 4). The four training tasks use four containers, with each training task running on one container, and each container occupying exactly one computing node's resources. Currently, four nodes are occupied. By scaling down the four training tasks, every two training tasks (training process 1 and training process 3 belong to one training task a, and training process 2 and training process 4 belong to one training task b) can run on one container, and each container is located on one computing node. After adjustment, it is equivalent to using two containers, and the two containers occupy two computing nodes.

[0150] It should be noted here that, regardless of whether it is scaling up or down, the ultimate goal of the performance mode is to minimize the overall expected runtime of at least one training task.

[0151] In this way, when there are plenty of idle resources in the computing resource pool, the capacity of at least one training task that is still running is expanded to accelerate its operation, so that it can complete the training in the shortest possible time and avoid occupying the computing resources of the next busy period. Therefore, the training can be completed as quickly as possible.

[0152] In one possible implementation, to ensure that training accuracy does not decrease after scaling (i.e., both scaling and shrinking), the processing can be as follows:

[0153] Add a portion of the training tasks from at least one training task to the target container that has already run at least one training task. Run multiple training tasks sequentially in the target container. During training, use the average value of the model parameters obtained from running multiple training tasks sequentially as the updated value of the model parameters.

[0154] In this embodiment, the number of containers is reduced during scaling down. At least a portion of the training tasks from at least one training task is run on the reduced-size containers. This portion of the training tasks is then added to the target container, which already has at least one training task running. Since the target container already has training tasks running, adding more training tasks to it results in the target container running multiple training tasks. These multiple training tasks are run sequentially within the target container. The AI ​​platform uses the average of the model parameters obtained from running these multiple training tasks sequentially as the updated value for the model parameters. Thus, because the multiple training tasks are run sequentially in the target container, it is equivalent to multiple training tasks being executed in a distributed manner, the same as the execution method before scaling down, and this does not lead to a decrease in the training accuracy of the AI ​​model.

[0155] The above processing can be called a batch processing approximation, used to simulate the running of a task in a distributed manner with N containers. This is equivalent to mimicking distributed training with an integer multiple of containers during scaling down, ensuring that accuracy is not reduced. For example, in... Figure 10 In the example, training process 1 and training process 3 belong to one training task a after adjustment, and training process 2 and training process 4 belong to one training task b after adjustment. Training task a runs on container a, and training task b runs on container b. Container a runs training process 1 and training process 3 sequentially, and container b runs training process 2 and training process 4 sequentially.

[0156] For example, if an AI model is trained on 16 containers, with each container using 64 data points, the model parameters obtained from training in each of the 16 containers are averaged to obtain the final AI model. Similarly, when scaling down to a single container, the AI ​​model is trained sequentially using 16 sets of data (each set containing 64 data points), and the final model parameters are averaged to obtain the final AI model. Therefore, the training accuracy of the AI ​​model will not decrease.

[0157] By running each training task before adjustment sequentially, it can be ensured that the training accuracy does not decrease after scaling down.

[0158] In one possible implementation, to ensure that training accuracy does not decrease after scaling up or down, the AI ​​platform can perform adaptive parameter adjustments. The AI ​​platform can use historical training experience, offline test parameter sets, etc., to adaptively adjust the corresponding hyperparameters while scaling up or down, so that the training accuracy remains unchanged.

[0159] like Figure 11 The diagram shown illustrates the training process of an AI model when the user selects only the shared mode.

[0160] Step 1101: The AI ​​platform provides the user with a training configuration interface, which includes multiple training modes for the user to choose from. When the user selects the shared mode in the training configuration interface, the interface also allows the user to input or select the resource usage of the container running the training task.

[0161] In this embodiment, the AI ​​platform can provide a training configuration interface to the user, which includes multiple training modes for the user to choose from. The training configuration interface displays the resource usage options available to the user, allowing the user to select or input the resource usage of the container corresponding to the runnable training task. Alternatively, the training configuration interface displays a resource usage range, within which the user can select or input the resource usage of the container corresponding to the runnable training task. For example, if the resource usage range is from 0.1 GPUs to 1 GPU, the user can select 0.5 GPUs.

[0162] Step 1102: The AI ​​platform generates at least one training task based on the training mode selected by the user in the training configuration interface and the resource usage of the container for running the training task as input or selected by the user.

[0163] In this embodiment, the AI ​​platform can obtain the training mode selected by the user in the training configuration interface as shared mode. The AI ​​platform can also obtain the resource usage of each container in shared mode. When the user selects only shared mode as the training mode, the resource usage of the container running the training task is a preset value. The AI ​​platform can generate at least one training task based on the available computing nodes in the current computing resource pool, the preset number of containers, the resource usage of the containers, and the initial AI model. The preset number of containers here can be the number of usable containers specified by the AI ​​platform for the at least one training task, or the number of containers specified by the user for the at least one training task.

[0164] Optionally, the resource usage of the container running the training task includes GPU resource usage less than that of a single GPU and / or memory usage less than that of a single memory block. This allows for a more granular allocation of computing resources on compute nodes, resulting in higher resource utilization.

[0165] Step 1103: The AI ​​platform performs at least one training task to train the initial AI model and obtain the AI ​​model.

[0166] Optionally, to reduce resource fragmentation, step 1103 can be processed as follows:

[0167] Based on the resource usage of each container running at least one training task in the shared mode and the remaining resources of each computing node in the computing resource pool, determine the computing node on which the container for each training task runs; start at least one container for a training task on the determined computing node to train the initial AI model.

[0168] In this embodiment, the AI ​​platform can calculate the remaining resources of each computing node in the computing resource pool and obtain the resource usage of each container running at least one training task. If the remaining resources of a computing node that has partially occupied resources are greater than the resource usage of each container, the AI ​​platform can distribute a container to that computing node. If the remaining resources of all computing nodes that have partially occupied resources are less than the resource usage of each container, the AI ​​platform can distribute at least one container running a training task to a computing node with unoccupied resources. In this way, the AI ​​platform distributes containers to computing nodes. Then, the AI ​​platform starts the containers on the computing nodes, thereby enabling the training of the initial AI model.

[0169] Optionally, to reduce resource fragmentation, step 1103 can be processed in another way:

[0170] The remaining resources of the compute nodes corresponding to each container are determined based on the resource usage of the containers running at least one training task in the second mode; the remaining resources of the compute nodes corresponding to each container are used to run one or more other training tasks.

[0171] In this embodiment, the AI ​​platform can use the resource usage of the container running at least one training task in the second mode to determine the remaining resources of the computing node corresponding to that container. When the AI ​​platform is performing other training tasks, if the remaining resources of a computing node used by the at least one training task are sufficient to execute one or more other training tasks, then the remaining resources of that computing node can be used to run one or more other training tasks, thus achieving resource sharing within the same computing node.

[0172] In this way, under the shared mode, while ensuring the resource requirements of the container, we can make the most of the computing nodes that have already occupied some resources, which can reduce resource fragmentation and improve the overall utilization of resources.

[0173] It should be noted that, in Figure 11 In the process, since the shared mode involves multiple containers sharing the resources of a computing node, a memory limit function needs to be added to the initial AI model to prevent memory leaks in other containers caused by a single task overusing memory when multiple containers share a computing node.

[0174] like Figure 12 The diagram illustrates the training process of an AI model when the user selects performance mode and sharing mode.

[0175] Step 1201: The AI ​​platform provides the user with a training configuration interface, which includes multiple training modes for the user to choose from. When the user selects performance mode and shared mode in the training configuration interface, the interface also allows the user to input or select the number of containers that can run training tasks and the resource usage of the containers running the training tasks.

[0176] The processing procedure in step 1201 is a combination of steps 701 and 1101. Please refer to the descriptions of steps 701 and 1101 for details, which will not be repeated here.

[0177] Step 1202: The AI ​​platform generates at least one training task based on the training mode selected by the user in the training configuration interface, the number of containers that can run training tasks input or selected by the user, and the resource usage of the containers that run training tasks input or selected by the user.

[0178] In this embodiment, the AI ​​platform can obtain the training mode selected by the user in the training configuration interface as shared mode. The AI ​​platform can also obtain the resource usage of each container in shared mode. If the user selects only shared mode as the training mode, the resource usage of the container running the training task is a preset value. The AI ​​platform can generate at least one training task based on the available computing nodes in the current computing resource pool, the number of containers that can run training tasks as input or selected by the user, the resource usage of the containers, and the initial AI model. Here, the number of containers determined by the AI ​​platform belongs to the number of containers that can run training tasks as input or selected by the user.

[0179] Optionally, the resource usage of the container running the training task includes GPU resource usage less than that of a single GPU and / or memory usage less than that of a single memory block. This allows for a more granular allocation of computing resources on compute nodes, resulting in higher resource utilization.

[0180] Step 1203: The AI ​​platform performs at least one training task to train the initial AI model and obtain the AI ​​model.

[0181] In step 1203, it can be combined with Figure 7 The handling of dynamic scaling in the process, and Figure 11 Shared resources in the process. See details. Figure 7 and Figure 11 The description in the text will not be repeated here.

[0182] In this way, in shared mode, while ensuring the resource requirements of containers, the use of computing nodes with partially occupied resources can be maximized, reducing resource fragmentation and improving overall resource utilization, thus lowering the cost for individual users to train AI models. Furthermore, in performance mode, by dynamically adjusting the number of containers, training can be accelerated as much as possible, improving the efficiency of training AI models.

[0183] In addition, regarding Figure 7 , Figure 11 and Figure 12 The process involves downloading the entire training dataset from OBS to the compute node's storage space (which can be solid-state storage, SSS) when a single container loads its training data for the initial training job. Since the training dataset is stored in OBS, this process reduces the time required to download the training dataset from OBS for each container. This allows subsequent compute nodes executing the training job to directly read data from this storage space. For example, newly added containers added through expansion can directly read data from this storage space, reducing the time needed to re-download the training dataset from OBS.

[0184] It should be noted that in this embodiment, the container is generated by the AI ​​platform by pulling an image and an initial AI model. Furthermore, by running training tasks on the container, since the container provides good isolation, even if multiple containers are deployed on the same node, the training tasks running on each container will not interfere with each other.

[0185] Through the embodiments of this application, multiple training modes are provided in the AI ​​platform for users to choose from. By selecting the appropriate training mode, users can flexibly execute distributed training, thereby balancing the user's training needs and resource utilization.

[0186] Figure 1 This is a structural diagram of an AI model training device provided in an embodiment of this application. The device is applied to an AI platform, which is associated with a computing resource pool. The computing resource pool includes computing nodes for model training. This device can be implemented as part or all of the device through software, hardware, or a combination of both. The device provided in this embodiment of this application can implement the embodiments of this application. Figure 7 , Figure 11 , Figure 12 The process described herein, the device includes: a training configuration module 102, a task management module 103, and a display module 105, wherein:

[0187] The training configuration module 102 is used to provide a training configuration interface to the user. The training configuration interface includes multiple training modes for the user to select. Each training mode represents an allocation strategy for the computing nodes required to train the initial AI model. Specifically, it can be used to implement the training configuration function of step 701 and execute the implicit steps included in step 701.

[0188] Task management module 103 is used for:

[0189] Based on the user's selection in the training configuration interface, at least one training task is generated;

[0190] The at least one training task is performed to train the initial AI model to obtain an AI model. The obtained AI model is available for download or use by the user. Specifically, it can be used to implement the task management functions of steps 702 and 703 and to execute the implicit steps contained in steps 702 and 703.

[0191] In one possible implementation, the multiple training modes include a first mode and / or a second mode, wherein the first mode indicates that the number of training tasks is automatically adjusted during the training of the initial AI model, and the second mode indicates that different training tasks share the resources of the same computing node.

[0192] In one possible implementation, the at least one training task runs on a container, and the apparatus further includes:

[0193] The display module 105 is used to provide the user with status information of the training process during the training of the initial AI model. The status information includes at least one of the following: the number of containers executing the training task, the resource usage of each container, the number of computing nodes executing the training task, and the resource usage of the computing nodes executing the training task.

[0194] In one possible implementation, the multiple training modes include a first mode and a second mode, and the task management module 103 is used for:

[0195] At least one training task is generated based on the first mode and the second mode selected by the user in the training configuration interface.

[0196] In one possible implementation, when the user selects the first mode in the training configuration interface, the training configuration interface also allows the user to input or select the number of containers that can run training tasks.

[0197] The task management module 103 is used for:

[0198] At least one training task is generated based on the training mode selected by the user in the training configuration interface and the number of containers for runnable training tasks input or selected by the user.

[0199] In one possible implementation, when the user selects the second mode in the training configuration interface, the training configuration interface also allows the user to input or select the resource usage of the container running the training task;

[0200] The task management module 103 is used for:

[0201] At least one training task is generated based on the training mode selected by the user in the training configuration interface and the resource usage of the container for running the training task input or selected by the user.

[0202] In one possible implementation, the resource usage of the container running the training task includes GPU resource usage less than that of a single GPU and / or memory usage less than that of a single memory block.

[0203] In one possible implementation, when the first mode is selected, the task management module 103 is configured to:

[0204] During the execution of the at least one training task to train the initial AI model, when it is detected that the conditions for elastic scaling are met, the amount of idle computing resources in the computing resource pool is obtained.

[0205] Based on the amount of idle computing resources in the computing resource pool, adjust the number of at least one training task and the number of containers used to run the training tasks.

[0206] The adjusted training task is run in the adjusted container to train the initial AI model.

[0207] In one possible implementation, the task management module 103 is used for:

[0208] A portion of the training tasks from the at least one training task is added to a target container that has already run the at least one training task. Multiple training tasks are run serially in the target container. During training, the average value of the model parameters obtained from running the multiple training tasks serially is used as the updated value of the model parameters.

[0209] In one possible implementation, when the second mode is selected, the task management module 103 is further configured to:

[0210] The remaining resources of the computing node corresponding to each container are determined based on the resource usage of the container running at least one training task in the second mode.

[0211] Use the remaining resources of the computing node corresponding to each container to run one or more other training tasks.

[0212] The module division in this embodiment is illustrative and represents only one logical functional division. In actual implementation, other division methods are possible. Furthermore, the functional modules in each embodiment of this application can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0213] This application also provides a method such as Figure 4 The computing device 400 shown has a processor 402 that reads the program and data set stored in the memory 401 to execute the method executed by the aforementioned AI platform.

[0214] Since the various modules in the AI ​​platform 100 provided in this application can be distributed and deployed on multiple computers in the same or different environments, this application also provides a... Figure 13 The computing device shown includes multiple computers 1300, each computer 1300 including a memory 1301, a processor 1302, a communication interface 1303, and a bus 1304. The memory 1301, processor 1302, and communication interface 1303 are interconnected via the bus 1304.

[0215] The memory 1301 can be a read-only memory, a static storage device, a dynamic storage device, or a random access memory. The memory 1301 can store programs, and when the program stored in the memory 1301 is executed by the processor 502, the processor 1302 and the communication interface 1303 are used to execute parts of the methods for training AI models on the AI ​​platform. The memory can also store training datasets; for example, a portion of the storage resources in the memory 1301 may be allocated as a dataset storage module to store the training datasets required by the AI ​​platform.

[0216] The processor 1302 may be a general-purpose central processing unit, microprocessor, application-specific integrated circuit, graphics processor, or one or more integrated circuits.

[0217] The communication interface 1303 uses a transceiver module, such as, but not limited to, a transceiver, to enable communication between the computer 1300 and other devices or communication networks. For example, a training dataset can be obtained through the communication interface 1303.

[0218] Bus 504 may include a pathway for transmitting information between various components of computer 1300 (e.g., memory 1301, processor 1302, communication interface 1303).

[0219] Each of the aforementioned computers 1300 establishes a communication path through a communication network. Each computer 1300 runs any one or more of the following modules: algorithm management module 101, training configuration module 102, task management module 103, data storage module 104, and display module 105. Any computer 1300 can be a computer in a cloud data center (e.g., a server), a computer in an edge data center, or a terminal computing device.

[0220] The descriptions of the processes corresponding to the above-mentioned figures each have their own emphasis. For parts of a process that are not described in detail, please refer to the relevant descriptions of other processes.

[0221] In the above embodiments, implementation can be achieved, in whole or in part, by software, hardware, or a combination thereof. When implemented using software, it can be implemented, in whole or in part, as a computer program product. The computer program product providing the AI ​​platform includes one or more computer instructions for accessing the AI ​​platform. When these computer program instructions are loaded and executed on a computer, all or part of the implementation according to the embodiments of this application is generated. Figure 7 , Figure 11 or Figure 12 The process or function described.

[0222] The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic cable, twisted pair) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium stores computer program instructions that provide an AI platform. The computer-readable storage medium may be any medium accessible to a computer or a data storage device such as a server or data center that integrates one or more media. The medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., optical disc), or a semiconductor medium (e.g., solid-state drive).

Claims

1. A method for training an artificial intelligence (AI) model, characterized in that, The method is applied to an AI platform associated with a computing resource pool, which includes computing nodes for model training, including: A training configuration interface is provided to the user, wherein the training configuration interface includes multiple training modes for the user to select, each training mode represents an allocation strategy for the computing nodes required to train the initial AI model, and the multiple training modes include at least a first mode, which represents automatically adjusting the number of training tasks during the training of the initial AI model. Based on the user's selection in the training configuration interface, at least one training task is generated; Perform the at least one training task to train the initial AI model, obtain an AI model, and make the obtained AI model available for download or use by the user; Wherein, when the user selects the first mode in the training configuration interface, executing the at least one training task to train the initial AI model includes: During the execution of the at least one training task to train the initial AI model, when the proportion of idle computing resources in the computing resource pool to all computing resources in the computing resource pool is higher than a target value, the expansion situation in the condition of elastic scaling is determined, and the number of the at least one training task and the number of containers used to run the training task are adjusted according to the idle computing resources in the computing resource pool to speed up the training. The adjusted training task is run in the adjusted container to train the initial AI model.

2. The method according to claim 1, characterized in that, The multiple training modes also include a second mode, in which different training tasks share the resources of the same computing node.

3. The method according to claim 1, characterized in that, The at least one training task runs on a container, and the method further includes: During the training of the initial AI model, the user is provided with status information of the training process, wherein the status information includes at least one of the following: the number of containers executing the training task, the resource usage of each container, the number of computing nodes executing the training task, and the resource usage of the computing nodes executing the training task.

4. The method according to claim 2 or 3, characterized in that, The multiple training modes include a first mode and a second mode. Based on the user's selection on the training configuration interface, at least one training task is generated, including: At least one training task is generated based on the first mode and the second mode selected by the user in the training configuration interface.

5. The method according to claim 2 or 3, characterized in that, When the user selects the first mode in the training configuration interface, the training configuration interface also allows the user to input or select the number of containers that can run training tasks; The step of generating at least one training task based on the user's selection on the training configuration interface includes: At least one training task is generated based on the training mode selected by the user in the training configuration interface and the number of containers for runnable training tasks input or selected by the user.

6. The method according to claim 1 or 3, characterized in that, The multiple training modes also include a second mode, which indicates that different training tasks share the resources of the same computing node. When the user selects the second mode in the training configuration interface, the training configuration interface also allows the user to input or select the resource usage of the container running the training task. The step of generating at least one training task based on the user's selection on the training configuration interface includes: At least one training task is generated based on the training mode selected by the user in the training configuration interface and the resource usage of the container for running the training task input or selected by the user.

7. The method according to claim 5, characterized in that, The resource usage of the container running the training task includes GPU resource usage less than that of a single graphics processing unit (GPU) and / or memory usage less than that of a single memory unit.

8. The method according to claim 2 or 3, characterized in that, When the first mode is selected, performing the at least one training task to train the initial AI model further includes: During the execution of the at least one training task to train the initial AI model, when a scaling-down situation is detected that satisfies the conditions for elastic scaling-up and scaling-down, the amount of idle computing resources in the computing resource pool is obtained. Based on the amount of idle computing resources in the computing resource pool, adjust the number of at least one training task and the number of containers used to run the training tasks. The adjusted training task is run in the adjusted container to train the initial AI model.

9. The method according to claim 8, characterized in that, The step of adjusting the number of the at least one training task and adjusting the number of containers used to run the training tasks, and running the adjusted training tasks in the adjusted containers to train the initial AI model, includes: A portion of the training tasks from the at least one training task is added to a target container that has already run the at least one training task. Multiple training tasks are run serially in the target container. During training, the average value of the model parameters obtained from running the multiple training tasks serially is used as the updated value of the model parameters.

10. The method according to claim 1 or 3, characterized in that, The multiple training modes also include a second mode, which indicates that different training tasks share the resources of the same computing node. When the second mode is selected, the method includes: The remaining resources of the computing node corresponding to each container are determined based on the resource usage of the container running at least one training task in the second mode. Use the remaining resources of the computing node corresponding to each container to run one or more other training tasks.

11. A training device for an artificial intelligence (AI) model, characterized in that, The device is applied to an AI platform, which is associated with a computing resource pool, the computing resource pool including computing nodes for model training, including: The training configuration module is used to provide a training configuration interface to the user. The training configuration interface includes multiple training modes for the user to select. Each training mode represents an allocation strategy for the computing nodes required to train the initial AI model. The multiple training modes include at least a first mode, which represents automatically adjusting the number of training tasks during the training of the initial AI model. The task management module is used for: Based on the user's selection in the training configuration interface, at least one training task is generated; Perform the at least one training task to train the initial AI model, obtain an AI model, and make the obtained AI model available for download or use by the user; Wherein, when the user selects the first mode in the training configuration interface, the task management module is used to: During the execution of the at least one training task to train the initial AI model, when the proportion of idle computing resources in the computing resource pool to all computing resources in the computing resource pool is higher than a target value, the expansion situation in the condition of elastic scaling is determined, and the number of the at least one training task and the number of containers used to run the training task are adjusted according to the idle computing resources in the computing resource pool to speed up the training. The adjusted training task is run in the adjusted container to train the initial AI model.

12. The apparatus according to claim 11, characterized in that, The multiple training modes also include a second mode, in which different training tasks share the resources of the same computing node.

13. The apparatus according to claim 11, characterized in that, The at least one training task runs on a container, and the device further includes: The display module is used to provide the user with status information of the training process during the training of the initial AI model. The status information includes at least one of the following: the number of containers executing the training task, the resource usage of each container, the number of computing nodes executing the training task, and the resource usage of the computing nodes executing the training task.

14. The apparatus according to claim 12 or 13, characterized in that, The multiple training modes include a first mode and a second mode; the task management module is used for: At least one training task is generated based on the first mode and the second mode selected by the user in the training configuration interface.

15. The apparatus according to claim 12 or 13, characterized in that, When the user selects the first mode in the training configuration interface, the training configuration interface also allows the user to input or select the number of containers that can run training tasks; The task management module is used for: At least one training task is generated based on the training mode selected by the user in the training configuration interface and the number of containers for runnable training tasks input or selected by the user.

16. The apparatus according to claim 11 or 13, characterized in that, The multiple training modes also include a second mode, which indicates that different training tasks share the resources of the same computing node. When the user selects the second mode in the training configuration interface, the training configuration interface also allows the user to input or select the resource usage of the container running the training task. The task management module is used for: At least one training task is generated based on the training mode selected by the user in the training configuration interface and the resource usage of the container for running the training task input or selected by the user.

17. The apparatus according to claim 15, characterized in that, The resource usage of the container running the training task includes GPU resource usage less than that of a single graphics processing unit (GPU) and / or memory usage less than that of a single memory unit.

18. The apparatus according to claim 12 or 13, characterized in that, When the first mode is selected, the task management module is further configured to: During the execution of the at least one training task to train the initial AI model, when a scaling-down situation is detected that satisfies the conditions for elastic scaling-up and scaling-down, the amount of idle computing resources in the computing resource pool is obtained. Based on the amount of idle computing resources in the computing resource pool, adjust the number of at least one training task and the number of containers used to run the training tasks. The adjusted training task is run in the adjusted container to train the initial AI model.

19. The apparatus according to claim 18, characterized in that, The task management module is used for: A portion of the training tasks from the at least one training task is added to a target container that has already run the at least one training task. Multiple training tasks are run serially in the target container. During training, the average value of the model parameters obtained from running the multiple training tasks serially is used as the updated value of the model parameters.

20. The apparatus according to claim 11 or 13, characterized in that, The multiple training modes also include a second mode, which indicates that different training tasks share the resources of the same computing node. When the second mode is selected, the task management module is further configured to: The remaining resources of the computing node corresponding to each container are determined based on the resource usage of the container running at least one training task in the second mode. Use the remaining resources of the computing node corresponding to each container to run one or more other training tasks.

21. A computing device, characterized in that, The computing device includes a memory and a processor, the memory being used to store computer instructions; The processor executes the computer instructions stored in the memory to perform the method according to any one of claims 1-10.

22. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program code, which, when executed by a computing device, performs the method described in any one of claims 1-10.

Images