Management scheduling method for model training tasks and related device
By calculating the resource requirements of model training tasks and electing based on the resource provision of hardware accelerator and calling priority, the cost of high in model training task management scheduling is solved, and efficient resource utilization and priority processing of emergency tasks are achieved.
Patent Information
- Application Number
- CN202411134586.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-19
- Publication Date
- 2024-11-19
AI Technical Summary
The prior art has problems with excessive cost in the management scheduling of model training tasks, especially in terms of resource scheduling and computing resource utilization efficiency.
By obtaining task information of the target training task, compute resource requirements, and perform resource elections based on the resource provision of callable hardware accelerators and call priority, it is preferred to use a lower-priced hardware accelerator to perform model training tasks.
It reduces the management and scheduling cost of model training tasks, improves resource utilization efficiency, ensures priority execution of emergency tasks, and dynamically allocates computing resources to shorten the response time.
Smart Images

Figure CN118981367A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of model training technology, and in particular to a management and scheduling method for model training tasks and related devices. Background Art
[0002] With the development and application of AI (Artificial Intelligence) technology, AI models have become the focus of research and application due to their excellent performance in various application fields. For example, large language models can capture the complex structure and semantics of natural language and can handle a variety of natural language tasks. They are widely used in text classification, question answering, dialogue and other fields. Another example is that AI painting models perform well in the field of image generation and can generate high-quality painting images.
[0003] In the process of acquiring artificial intelligence models, a large amount of training data and computing resources are usually required for model training. Therefore, how to schedule hardware accelerators such as GPU (Graphics Processing Unit) and TPU (Tensor Processing Unit) has become one of the factors affecting model training. Existing technologies can achieve distributed training through tools such as Horovod and DeepSpeed to utilize cluster computing resources or cloud computing resources to perform model training tasks and conduct large-scale model training.
[0004] However, the inventors have discovered through research that the existing technology has the problem of excessively high costs when managing and scheduling model training tasks. Summary of the invention
[0005] The purpose of this application is to solve at least one of the above-mentioned technical deficiencies, especially the technical defect of excessively high cost in the prior art.
[0006] In a first aspect, an embodiment of the present application provides a method for managing and scheduling a model training task, including:
[0007] In response to an execution instruction of a target training task, task information of the target training task is acquired; the task information includes model training parameters, a data volume of a training data set, and a model size of a target model, where the target model is a basic model to be trained;
[0008] Calculate the resource requirement corresponding to the target training task according to the model training parameters, the data size and the model size;
[0009] Perform resource selection based on the resource demand, the resource supply corresponding to each callable hardware accelerator, and the call priority corresponding to each callable hardware accelerator to obtain a target hardware accelerator; wherein the lower the usage price corresponding to the callable hardware accelerator, the higher the call priority;
[0010] When a preset calling rule is met, the target hardware accelerator is called to execute the target training task, so that the target hardware accelerator performs model training on the target model according to the model training parameters and the training data set.
[0011] In a second aspect, an embodiment of the present application provides a management and scheduling device for a model training task, including:
[0012] A task information acquisition module, used to obtain task information of the target training task in response to an execution instruction of the target training task; the task information includes model training parameters, the data size of the training data set and the model size of the target model, and the target model is a basic model to be trained;
[0013] A resource requirement calculation module, used to calculate the resource requirement corresponding to the target training task according to the model training parameters, the data size and the model size;
[0014] A resource selection module, configured to select resources based on the resource demand, the resource supply corresponding to each callable hardware accelerator, and the call priority corresponding to each callable hardware accelerator, to obtain a target hardware accelerator; wherein the lower the usage price corresponding to the callable hardware accelerator, the higher the call priority;
[0015] A calling module is used to call the target hardware accelerator to execute the target training task when a preset calling rule is met, so that the target hardware accelerator performs model training on the target model according to the model training parameters and the training data set.
[0016] In a third aspect, an embodiment of the present application provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the management and scheduling method for model training tasks described in any of the above embodiments.
[0017] In a fourth aspect, an embodiment of the present application provides a computer device, the computer device comprising: one or more processors, and a memory;
[0018] The memory stores computer-readable instructions, and when the computer-readable instructions are executed by the one or more processors, the steps of the management and scheduling method of model training tasks described in any of the above embodiments are performed.
[0019] In the management and scheduling methods and related devices for model training tasks provided in some embodiments of the present application, the resource requirements required to perform the target training task can be estimated based on the model training parameters of the target training task, the data volume of the training data set, and the model size of the basic model to be trained, and resource elections are performed based on the resource requirements, the resource supply corresponding to each callable hardware accelerator, and the call priority to determine the target hardware accelerator used to perform the target training task. Since the call priority corresponding to the callable hardware accelerator is determined based on the usage price, the lower the usage price corresponding to the callable hardware accelerator, the higher the call priority, therefore, the present application can combine the resource requirements and the resource supply of each resource, and give priority to using the callable hardware accelerator with a lower price to perform the target training task. In this way, the management and scheduling costs of model training tasks can be reduced. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0021] Figure 1 A flowchart of a method for managing and scheduling model training tasks in one embodiment;
[0022] Figure 2 A flowchart of steps for calling a target hardware accelerator to execute a target training task in one embodiment;
[0023] Figure 3 A schematic diagram of a process of resource selection steps in one embodiment;
[0024] Figure 4 1 is a schematic diagram of the structure of a management and scheduling device for model training tasks in one embodiment;
[0025] Figure 5 FIG. 1 is a diagram of the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION
[0026] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0027] The following is an explanation of the terms involved in this application:
[0028] Large Language Model (LLM): A deep learning model for processing and generating natural language text.
[0029] AI painting model: an AI model used to automatically generate images, which can be but is not limited to the AI painting Lora model, which can generate high-quality paintings through fine-tuning training.
[0030] k8s cluster: Kubernetes cluster, an open source system for automating the deployment, scaling, and management of containerized applications.
[0031] GPU: Graphics processing unit, used to accelerate computationally intensive tasks and can be used to perform training tasks for deep learning models.
[0032] Ansible: An open-source automation engine for configuration management, application deployment, and task automation.
[0033] In one embodiment, the present application provides a method for managing and scheduling model training tasks. The following embodiments are described by taking the method applied to a computer device as an example. It can be understood that the computer device described herein can be a device with a resource scheduling function, which can be, but is not limited to, a server, a desktop computer, a laptop computer, etc.
[0034] like Figure 1 As shown, the management and scheduling method of the model training task of the present application may include the following steps:
[0035] S102: In response to an execution instruction of a target training task, obtaining task information of the target training task.
[0036] Among them, the target training task can be a model training task to be executed, such as a fine-tuning training task of a large language model or a training task of an AI painting model. It can be understood that the present application can achieve the determination of the target training task in any way, and this document does not impose specific restrictions on this. For example, when a computer device receives an execution instruction, the target training task can be determined based on the instruction information of the execution instruction.
[0037] The task information of the target training task may include model training parameters, the data size of the training data set and the model size of the target model. The model training parameters may be parameters that affect model training, and may be, but are not limited to, the number of trainings, the initial model parameters of the target model, etc. The target model may be a basic model to be trained, that is, the target training task is a model training task for the target model.
[0038] In this step, upon receiving the execution instruction of the target training task, the computer device may obtain the task information of the target training task to determine the model training parameters, data volume and model size involved in the target training task.
[0039] S104: Calculate the resource requirements corresponding to the target training task according to the model training parameters, data size and model size.
[0040] Among them, the resource demand may be the computing resources required to perform the target training task. It can be understood that the computing resources required to perform the target training task are associated with the model training parameters, the data volume of the training data set, and the model size of the model to be trained. For example, the resource demand may be positively correlated with the number of training times, the resource demand may be positively correlated with the data volume, and the resource demand may be positively correlated with the model size. Therefore, this step can estimate the resource demand required to perform the target training task based on the model training parameters of the target training task, the data volume of the training data set, and the model size corresponding to the target training task.
[0041] It should be noted that the present application may use any method to calculate the resource requirement, and this document does not impose any specific restrictions on this. In one example, the resource requirement may be estimated by using the Model Memory Calculator tool.
[0042] S106: Perform resource selection based on the resource demand, the resource provision corresponding to each callable hardware accelerator, and the call priority corresponding to each callable hardware accelerator to obtain a target hardware accelerator.
[0043] The resource provision refers to the computing resources that can be provided by the callable hardware accelerator. The target hardware accelerator can be a callable hardware accelerator used to perform the target training task. In this application, the callable hardware accelerator can be, but is not limited to, a GPU and a TPU.
[0044] The call priority corresponding to the callable hardware accelerator is associated with the usage price. The lower the usage price of the callable hardware accelerator, the higher its call priority. In this step, the computer device can perform resource election based on the resource demand, the resource supply corresponding to each callable hardware accelerator, and the call priority corresponding to each callable hardware accelerator, so as to select a callable hardware accelerator with a higher cost performance as the target hardware accelerator.
[0045] Exemplarily, the computer device may preliminarily determine a candidate hardware accelerator from among the callable hardware accelerators. The candidate hardware accelerator refers to a callable hardware accelerator whose resource provision can meet the resource demand. The computer device may select the candidate accelerator with the highest call priority as the target hardware accelerator according to the call priority corresponding to each candidate accelerator.
[0046] Furthermore, in one example, the computer device may estimate the training duration corresponding to the target training task based on the resource provision corresponding to the target hardware accelerator, return the training duration to the terminal device, and display it to the user who initiated the target training task.
[0047] S108: When a preset calling rule is met, the target hardware accelerator is called to execute the target training task, so that the target hardware accelerator performs model training on the target model according to the model training parameters and the training data set.
[0048] Among them, the preset calling rules can be rules used to determine whether to call the target hardware accelerator to perform the model training task, and the specific rule content can be determined based on actual conditions.
[0049] In this step, when the target hardware accelerator is determined and the preset calling rules are met, the computer device can call the target hardware accelerator to perform the target training task to achieve the training of the target model through the target hardware accelerator.
[0050] In this application, the resource demand required to perform the target training task can be estimated based on the model training parameters of the target training task, the data size of the training data set, and the model size of the basic model to be trained, and resource election can be performed based on the resource demand, the resource supply corresponding to each callable hardware accelerator, and the call priority to determine the target hardware accelerator used to perform the target training task. Since the call priority corresponding to the callable hardware accelerator is determined based on the usage price, the lower the usage price corresponding to the callable hardware accelerator, the higher the call priority, therefore, this application can combine the resource demand and each resource supply, and give priority to using the callable hardware accelerator with a lower price to perform the target training task. In this way, the management and scheduling costs of the model training task can be reduced.
[0051] In the case of relatively scarce computing resources, the simultaneous implementation of multiple training tasks will result in a large number of training tasks being unable to obtain computing resources. This application can schedule tasks according to the multi-task queue strategy, and can provide the function of resource preemption according to task priority, so that resource scheduling can be combined with task priority.
[0052] In one embodiment, the task information further includes a task priority of the target training task, and the task priority can be used to indicate the priority execution degree of the target training task. In one example, the task priority of the target training task can be preset according to the business urgency.
[0053] like Figure 2 As shown, when the preset calling rules are met, the target hardware accelerator is called to perform the target training task, including:
[0054] S202: If the task priority of the target training task is a preset emergency priority, and the target hardware accelerator is in a training state, then obtaining the task priority of the preempted task; wherein the preempted task is a currently executed task of the target hardware accelerator;
[0055] S204: If the task priority of the preempted task is not an emergency priority, sending a pause execution instruction to the target hardware accelerator; the pause execution instruction is used to instruct the target hardware accelerator to pause the execution of the preempted task and return the execution information of the preempted task;
[0056] S206: Save the task information of the preempted task, and call the target hardware accelerator to execute the target training task.
[0057] The emergency priority may include one or more preset task priorities, which may be determined according to actual conditions, and this document does not impose any specific restrictions on this. For ease of explanation, this application describes P0 as the emergency priority.
[0058] The target hardware accelerator can be in an idle state or a training state. When the target hardware accelerator is in an idle state, it indicates that the target hardware accelerator is not currently executing a model training task. When the target hardware accelerator is in a training state, it indicates that the target hardware accelerator is currently executing a model training task.
[0059] Specifically, if the task priority of the target training task is P0, and the target hardware accelerator is in the training state, it means that the target hardware accelerator is executing a model training task other than the target training task, and the computing resources of the target hardware accelerator are occupied by other model training tasks. In this case, the computer device can determine whether the target training task can preempt resources according to the task priority of the model training task currently executed by the target hardware accelerator (i.e., the task priority of the preempted task described in this article).
[0060] If the task priority of the preempted task is not P0, it indicates that the target hardware accelerator is not currently executing an urgent model training task, and the task priority of the preempted task is lower than the task priority of the target training task. The target training task can preempt the target hardware accelerator so that the target hardware accelerator has priority in executing the target training task. Therefore, when the task priority of the preempted task is not an urgent priority, the computer device can send a pause execution instruction to the target hardware accelerator to instruct the target hardware accelerator to pause the execution of the preempted task and return the execution information of the preempted task. Among them, the execution information can be the training context information of the preempted task, such as the current training weight corresponding to the preempted task.
[0061] The computer device can receive and save the preempted execution information so that when the preempted task is restarted, the model training can be continued based on the saved execution information, thereby improving the training efficiency of the preempted task. The computer device can also call the target hardware accelerator to execute the target training task to complete resource preemption.
[0062] In this embodiment, the training tasks with urgent priority can preempt the computing resources of the training tasks with low priority. In this way, the dynamic allocation of computing resources can be realized, ensuring that the training tasks with urgent priority can be executed first and shortening their response time, thereby improving the flexibility of resource scheduling.
[0063] In one embodiment, Figure 2 As shown, after the step of sending the pause execution instruction to the target hardware accelerator, the following steps may also be included:
[0064] S208: Add the preempted task to the head of the task queue corresponding to the target hardware accelerator.
[0065] In this embodiment, after the target training task preempts resources, the preempted task can be added to the head of the corresponding task queue to wait for the next execution. In this way, the impact of resource preemption on the execution of the preempted task can be reduced.
[0066] In one embodiment, when a preset calling rule is met, calling a target hardware accelerator to perform a target training task also includes:
[0067] If the task priority of the target training task is not an emergency priority, the target training task is added to the task queue corresponding to the target hardware accelerator according to the task priority of the target training task, and based on the queue order of the target training task in the task queue, the target hardware accelerator is called to execute the target training task;
[0068] If the task priority of the target training task is the emergency priority, and the task priority of the preempted task is the emergency priority, the target training task is added to the head of the task queue corresponding to the target hardware accelerator, and when the preempted task is completed, the target hardware accelerator is called to execute the target training task.
[0069] In this embodiment, if the target training task is a non-urgent task, and / or the currently executed task of the target hardware accelerator is an urgent task, scheduling can be performed based on the non-preemptive task queuing situation to avoid frequent suspension of task execution.
[0070] Specifically, when the task priority of the target training task is a non-urgent priority, and / or the task priority of the preempted task is an urgent priority, the target training task can be added to the task queue corresponding to the target hardware accelerator according to the task priority of the target training task. In the task queue, each task can be arranged in order from high to low according to the task priority. In other words, the higher the task priority, the earlier the queue order. Conversely, the lower the task priority, the later the queue order.
[0071] After adding the target training task to the corresponding task queue, the computer device can determine whether to call the target hardware accelerator to execute the target training task based on the task execution status of the target hardware accelerator and the queue order of the target training task in the task queue. If yes, the target hardware accelerator can be called to execute the target training task, otherwise, the target training task continues to wait.
[0072] For example, in the task queue corresponding to the target hardware accelerator, the first task, the second task, and the third task are arranged in order from the head to the tail, wherein the task priority of the first task is P1, the task priority of the second task is P3, and the task priority of the third task is P4. If the task priority of the target training task is P2, then after the target training task is added to the task queue, the order of the tasks from the head to the tail is: the first task, the target training task, the second task, and the third task.
[0073] In one embodiment, the computer device may also calculate the queuing time corresponding to the training task according to the above-mentioned scheduling strategy. Specifically, for each training task in the task queue, the computer device may calculate the number of batches corresponding to the training task according to the amount of data corresponding to the training task and the predetermined amount of data processed in a single batch. When calculating the queuing time corresponding to a training task, the computer device may count the number of batches corresponding to each task in the same task queue whose queuing order precedes the training task, and sum up the number of batches to obtain the total number of batches. And the product of the total number of batches and the preset single batch processing time is used as the queuing time corresponding to the training task.
[0074] In one embodiment, Figure 3 As shown, based on the resource demand, the resource supply corresponding to each callable hardware accelerator and the call priority corresponding to each callable hardware accelerator, resource selection is performed to obtain the target hardware accelerator, including:
[0075] S302: Determine the working state of each callable hardware accelerator respectively; wherein the working state is an idle state or a training state;
[0076] S304: judging whether there is at least one target idle accelerator according to the number of idle accelerators and / or the amount of resources provided by each idle accelerator; wherein an idle accelerator is a callable hardware accelerator in an idle state, and a target idle accelerator is an idle accelerator whose amount of resources provided can meet the amount of resource demand;
[0077] S306: If there is not at least one target idle accelerator, then when the task priority corresponding to the target training task is the emergency priority, according to the resource demand, the resource supply corresponding to each training accelerator, the calling priority corresponding to each training accelerator and the current task priority corresponding to each training accelerator, a resource election is performed in each training accelerator to obtain the target hardware accelerator; wherein the training accelerator is a callable hardware accelerator in a training state.
[0078] In this embodiment, when allocating a hardware accelerator for a target training task, the computer device may first determine whether there is an idle accelerator that can meet the resource demand. If it is determined that there is no idle accelerator that can meet the resource demand based on the number of idle accelerators and / or the resource provision amount corresponding to each idle accelerator, then a resource election may be performed in each training accelerator in combination with the task priority of the target training task and the current task priority corresponding to each training accelerator, so as to preferentially allocate the target training task with an urgent priority to the hardware accelerator that can preempt resources as much as possible, thereby improving the response efficiency of the urgent priority task.
[0079] Specifically, during the resource selection process, the computer device can determine the working status of each callable hardware accelerator respectively, and divide each hardware accelerator into an idle accelerator and a training accelerator according to the working status. The computer device can determine whether there is at least one target idle accelerator based on the number of idle accelerators and / or the amount of resources provided corresponding to each idle accelerator. For example, when the number of idle accelerators is zero, it indicates that each callable hardware accelerator is executing a task, so it can be determined that there is no target idle accelerator. For another example, when the idle accelerator is not zero, it can be determined whether there is a target idle accelerator based on the resource demand and the amount of resources provided corresponding to each idle accelerator.
[0080] If there is no target idle accelerator, it can be determined whether the task priority of the target training task is an emergency priority. If so, it indicates that the target training task needs to be responded to quickly. Therefore, resource election can be performed in combination with the call priority corresponding to each training accelerator and the current task priority corresponding to each training accelerator, so as to allocate the callable hardware accelerator that can preempt resources to the target training task as much as possible, so that the target training task can preempt the computing resources of the target hardware accelerator and achieve a quick response.
[0081] In one embodiment, Figure 3 As shown, resource selection is performed based on resource demand, resource provision corresponding to each callable hardware accelerator, and call priority corresponding to each callable hardware accelerator to obtain a target hardware accelerator, which may also include:
[0082] S308: If there is a target idle accelerator, determine the target idle accelerator as the target hardware accelerator;
[0083] S310: If there are multiple target idle accelerators, then according to the calling priority corresponding to each target idle accelerator, resource selection is performed in each target idle accelerator to obtain a target hardware accelerator.
[0084] In this embodiment, during the resource selection process, if there is at least one target idle accelerator, the target training task may be assigned to one of the target idle accelerators for execution, so as to avoid frequent suspension of task execution of the training accelerator.
[0085] Specifically, if there is one target idle accelerator, the target idle accelerator can be directly used as the target hardware accelerator, and the target training task can be assigned to the target idle accelerator for execution. If there are multiple target idle accelerators, the computer device can perform resource election based on the call priority corresponding to each target idle accelerator, so as to preferentially determine the target idle accelerator with a lower usage price as the target hardware accelerator.
[0086] In one embodiment, calling a target hardware accelerator to execute a target training task includes:
[0087] If the target hardware accelerator is deployed in a k8s cluster system, the target training task is initiated to the k8s cluster system to which the target hardware accelerator belongs through the k8s API;
[0088] If the target hardware accelerator is deployed in a cloud host, the target training task is initiated to the cloud host to which the target hardware accelerator belongs through the cloud resource operation and maintenance tool;
[0089] If the target hardware accelerator is deployed locally, the target hardware accelerator is directly called to execute the target training task.
[0090] In this embodiment, the computer device can support the computing resources of the k8s cluster, cloud host and local computer room, thereby improving the flexibility of resource scheduling and efficiently utilizing local cluster and cloud resources.
[0091] Specifically, in the process of initiating a target training task to the target hardware accelerator, the resource call can be made through the corresponding calling method according to the resource type corresponding to the target hardware accelerator. When the target hardware accelerator is a k8s resource, the computer device can initiate a job to the cluster where the target hardware accelerator is located through the k8s API (Application Programming Interface) to call the target hardware accelerator to perform the target training task. When the target hardware accelerator is a cloud host resource, a docker training task can be initiated to the cloud host through a cloud resource operation and maintenance tool such as Ansible. When the target hardware accelerator is a local resource, the target hardware accelerator can be called directly.
[0092] Furthermore, in one example, after calling the target hardware accelerator to execute the target training task, the running status of the target training task can be monitored in real time, and the training log and training report can be synchronized to the model training service at regular intervals.
[0093] In one embodiment, obtaining task information of a target training task includes:
[0094] Obtain training configuration information configured by the user, the training configuration information including the storage path of the training data set, the model identifier of the target model, and the model training parameters;
[0095] Determine the number of training data items in the training data set according to the storage path of the training data set, and calculate the data volume based on the number of training data items and the preset single data volume;
[0096] According to the model identification, the target model is determined in the preset basic model, and the model size of the target model is calculated.
[0097] Considering the existing model training task management and scheduling tools, such as Kubeflow, the task configuration and task management are complex, which increases the difficulty and cost of user use. In this embodiment, the user only needs to configure the storage path of the training data set and the model training parameters, and select the target model, and then the model training can be started with one click, without writing complex training scripts and training instructions, which can greatly reduce the difficulty and cost of user use.
[0098] Specifically, after the user inputs the training configuration information, the computer device can obtain the training configuration information, and determine the number of training data items (i.e., the number of tokens) of the training data set according to the storage path of the training data set in the training configuration information, and automatically calculate the data volume of the training data set accordingly. The computer device can also automatically determine the model size of the target model according to the model identifier of the target model in the training configuration information, and execute each step of S102 to S108 to call the hardware accelerator to perform the target training task.
[0099] In one embodiment, calling a target hardware accelerator to execute a target training task includes:
[0100] Step A1: Create a dataset directory and a basic model directory respectively;
[0101] Step A3: Import the training dataset into the dataset directory according to the storage path;
[0102] Step A5: According to the model identification, soft-link the target model to the basic model directory;
[0103] Step A7: Assemble the training command according to the dataset directory, basic model directory and model training parameters;
[0104] Step A9: Send a training command to the target hardware accelerator so that the target hardware accelerator performs the target training task according to the data set directory, the basic model directory and the model training parameters.
[0105] Specifically, the computer device may first initialize the training environment. In the process of initializing the training environment, the computer device may respectively create a data set directory and a basic model directory. Further, in one example, the computer device may also respectively create an output model directory, a log directory, and a training report directory.
[0106] The computer device can import the training data set into the data set directory according to the storage path of the training data set, and soft-connect the target model to the basic model directory, and automatically splice the parameters such as the data set directory, the basic model directory, the output model directory, the log directory and the training report directory, and assemble the training command in combination with the model training parameters. The computer device can send a training command to the target hardware accelerator to trigger the target hardware accelerator to perform the target training task based on the training command.
[0107] Exemplarily, the computer device may have built-in SDLora model training scripts, Sd basic models and optimal training parameters. Users only need to upload training data sets and create training tasks to start one-click training and obtain the corresponding SDLora model.
[0108] For another example, a computer device may have built-in large language model training scripts, model weight synthesis scripts, model quantization scripts, large language basic models, and optimal training parameters. Users only need to upload training data sets and create training tasks to start one-click training and obtain the corresponding large language model. When the target model is a large language model, the computer device may also assemble weight merging commands and model quantization commands, and send weight merging commands to the target hardware accelerator to instruct the target hardware accelerator to merge the trained weight model with the target model and obtain a model that can be directly applied to model reasoning. The computer device may also send a model quantization command to the target hardware accelerator to instruct the target hardware accelerator to perform model quantization to reduce the model size.
[0109] In one embodiment, in order to solve the shortcomings of the prior art in the insufficient logging and visual display of the training process, the present application can synchronize the training log, training report, and output model at regular intervals during the training process, and can display the log at the front end. The computer device can also display charts at the front end based on the report content of the training report, for example, the changes in the learning rate and loss value can be displayed through charts. In this way, users can monitor and adjust training tasks in real time.
[0110] The following is a description of the management and scheduling device for the model training tasks provided in an embodiment of the present application. The management and scheduling device for the model training tasks described below and the management and scheduling method for the model training tasks described above can be referenced to each other.
[0111] In one embodiment, Figure 4 As shown, the present application provides a management and scheduling device 400 for model training tasks, including:
[0112] The task information acquisition module 402 is used to obtain the task information of the target training task in response to the execution instruction of the target training task; the task information includes model training parameters, the data size of the training data set and the model size of the target model, and the target model is the basic model to be trained;
[0113] A resource requirement calculation module 404 is used to calculate the resource requirement corresponding to the target training task according to the model training parameters, the data size and the model size;
[0114] The resource selection module 406 is used to select resources based on the resource demand, the resource supply corresponding to each callable hardware accelerator and the call priority corresponding to each callable hardware accelerator to obtain a target hardware accelerator; wherein the lower the usage price corresponding to the callable hardware accelerator, the higher the call priority;
[0115] The calling module 408 is used to call the target hardware accelerator to execute the target training task when the preset calling rules are met, so that the target hardware accelerator performs model training on the target model according to the model training parameters and the training data set.
[0116] In one embodiment, the task information also includes the task priority. The calling module 408 of the present application includes:
[0117] A priority acquisition unit, configured to acquire the task priority of the preempted task if the task priority of the target training task is a preset emergency priority and the target hardware accelerator is in a training state; wherein the preempted task is a currently executed task of the target hardware accelerator;
[0118] A pausing unit, configured to send a suspend execution instruction to the target hardware accelerator if the task priority of the preempted task is not the emergency priority; the suspend execution instruction is used to instruct the target hardware accelerator to suspend the execution of the preempted task and return the execution information of the preempted task;
[0119] The preemption unit is used to save the task information of the preempted task and call the target hardware accelerator to execute the target training task.
[0120] In one embodiment, the calling module 408 of the present application further includes:
[0121] A first queuing unit is used for adding the target training task to the task queue corresponding to the target hardware accelerator according to the task priority of the target training task if the task priority of the target training task is not the emergency priority, and calling the target hardware accelerator to execute the target training task based on the queuing order of the target training task in the task queue;
[0122] The second queuing unit is used to add the target training task to the head of the task queue corresponding to the target hardware accelerator if the task priority of the target training task is the emergency priority and the task priority of the preempted task is the emergency priority, and to call the target hardware accelerator to execute the target training task when the preempted task is completed.
[0123] In one embodiment, the resource selection module 406 of the present application includes:
[0124] A working state determining unit, used to respectively determine the working state of each of the callable hardware accelerators; the working state is an idle state or a training state;
[0125] A first judgment unit is used to judge whether there is at least one target idle accelerator according to the number of idle accelerators and / or the resource provision amount corresponding to each idle accelerator; wherein the idle accelerator is a callable hardware accelerator in an idle state, and the target idle accelerator is an idle accelerator whose resource provision amount can meet the resource demand amount;
[0126] The first election unit is used for, if there is no at least one target idle accelerator, then when the task priority corresponding to the target training task is the emergency priority, performing resource election in each of the training accelerators according to the resource demand, the resource supply corresponding to each training accelerator, the calling priority corresponding to each of the training accelerators and the current task priority corresponding to each of the training accelerators to obtain the target hardware accelerator; wherein the training accelerator is a callable hardware accelerator in a training state.
[0127] In one embodiment, the resource selection module 406 of the present application further includes:
[0128] an accelerator allocation unit, configured to determine the target idle accelerator as the target hardware accelerator if there is one target idle accelerator;
[0129] The second election unit is configured to, if there are multiple target idle accelerators, perform resource election in each of the target idle accelerators according to the call priority corresponding to each of the target idle accelerators to obtain the target hardware accelerator.
[0130] In one embodiment, the calling module 408 of the present application includes:
[0131] A first calling unit is used to initiate the target training task to the k8s cluster system to which the target hardware accelerator belongs through the k8sAPI if the target hardware accelerator is deployed in the k8s cluster system;
[0132] A second calling unit is configured to initiate the target training task to the cloud host to which the target hardware accelerator belongs through a cloud resource operation and maintenance tool if the target hardware accelerator is deployed in a cloud host;
[0133] The third calling unit is used to directly call the target hardware accelerator to execute the target training task if the target hardware accelerator is deployed locally.
[0134] In one embodiment, the task information acquisition module 402 of the present application includes:
[0135] A configuration information acquisition unit, used to acquire training configuration information configured by a user, wherein the training configuration information includes a storage path of the training data set, a model identifier of the target model, and the model training parameters;
[0136] A data volume calculation unit, configured to determine the number of training data items of the training data set according to the storage path of the training data set, and calculate the data volume based on the number of training data items and a preset single data volume;
[0137] A model size calculation unit is used to determine the target model in a preset basic model according to the model identifier, and calculate the model size of the target model.
[0138] In one embodiment, the calling module 408 of the present application includes:
[0139] Path creation unit, used to create dataset directory and basic model directory respectively;
[0140] A training data importing unit, used to import the training data set into the data set directory according to the storage path;
[0141] A model importing unit, used for soft-linking the target model to the basic model directory according to the model identifier;
[0142] A command assembling unit, used to assemble a training command according to the data set directory, the basic model directory and the model training parameters;
[0143] A command sending unit is used to send the training command to the target hardware accelerator so that the target hardware accelerator performs the target training task according to the data set directory, the basic model directory and the model training parameters.
[0144] In one embodiment, the present application also provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the management and scheduling method of model training tasks in any embodiment.
[0145] In one embodiment, the present application also provides a computer device having computer-readable instructions stored therein. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the method for managing and scheduling model training tasks in any embodiment.
[0146] Indicatively, Figure 5 The internal structure diagram of a computer device provided in an embodiment of the present application is shown in FIG. 1 . In one example, the computer device may be a server. Figure 5, the computer device 900 includes a processing component 902, which further includes one or more processors, and a memory resource represented by a memory 901, for storing instructions executable by the processing component 902, such as an application. The application stored in the memory 901 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 902 is configured to execute instructions to perform the steps of the management and scheduling method of the model training task described in any of the above embodiments.
[0147] The computer device 900 may further include a power supply component 903 configured to perform power management of the computer device 900, a wired or wireless network interface 904 configured to connect the computer device 900 to a network, and an input / output (I / O) interface 905. The computer device 900 may operate based on an operating system stored in the memory 901, such as Windows Server TM, Mac OS X TM, Unix TM, Linux TM, Free BSD TM, or the like.
[0148] Those skilled in the art will understand that the internal structure of the computer device shown in the present application is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0149] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the term "include", "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements not only includes those elements, but also includes other elements not clearly listed, or also includes elements inherent to such process, method, article or equipment. In the absence of more restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or equipment including the elements. Herein, "one", "one", "said", "the" and "it" may also include plural forms, unless the context clearly indicates another way. A plurality refers to at least two cases, such as 2, 3, 5 or 8, etc. "And / or" includes any and all combinations of the relevant listed items.
[0150] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can refer to each other.
[0151] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method for managing and scheduling model training tasks, characterized in that: include: In response to an execution instruction of a target training task, acquiring task information of the target training task; The task information includes model training parameters, the data size of the training data set and the model size of the target model, and the target model is the basic model to be trained; Calculate the resource requirement corresponding to the target training task according to the model training parameters, the data size and the model size; Perform resource selection based on the resource demand, the resource supply corresponding to each callable hardware accelerator, and the call priority corresponding to each callable hardware accelerator to obtain a target hardware accelerator; wherein the lower the usage price corresponding to the callable hardware accelerator, the higher the call priority; When a preset calling rule is met, the target hardware accelerator is called to execute the target training task, so that the target hardware accelerator performs model training on the target model according to the model training parameters and the training data set.
2. The method according to claim 1, characterized in that The task information also includes task priority; The step of calling the target hardware accelerator to execute the target training task when a preset calling rule is met includes: If the task priority of the target training task is a preset emergency priority and the target hardware accelerator is in a training state, the task priority of the preempted task is obtained; wherein the preempted task is a currently executed task of the target hardware accelerator; If the task priority of the preempted task is not the emergency priority, sending a pause execution instruction to the target hardware accelerator; the pause execution instruction is used to instruct the target hardware accelerator to pause the execution of the preempted task and return the execution information of the preempted task; The task information of the preempted task is saved, and the target hardware accelerator is called to execute the target training task.
3. The method according to claim 2, characterized in that The step of calling the target hardware accelerator to execute the target training task when a preset calling rule is met also includes: If the task priority of the target training task is not the emergency priority, adding the target training task to the task queue corresponding to the target hardware accelerator according to the task priority of the target training task, and calling the target hardware accelerator to execute the target training task based on the queuing order of the target training task in the task queue; If the task priority of the target training task is the emergency priority, and the task priority of the preempted task is the emergency priority, the target training task is added to the head of the task queue corresponding to the target hardware accelerator, and when the preempted task is completed, the target hardware accelerator is called to execute the target training task.
4. The method according to claim 2, characterized in that: The step of performing resource selection based on the resource demand, the resource provision corresponding to each callable hardware accelerator, and the call priority corresponding to each callable hardware accelerator to obtain a target hardware accelerator includes: Determine the working state of each of the callable hardware accelerators respectively; the working state is an idle state or a training state; According to the number of idle accelerators and / or the amount of resources provided by each idle accelerator, it is determined whether there is at least one target idle accelerator; wherein the idle accelerator is a callable hardware accelerator in an idle state, and the target idle accelerator is an idle accelerator whose amount of resources provided can meet the resource demand; If there is no at least one target idle accelerator, then when the task priority corresponding to the target training task is the emergency priority, according to the resource demand, the resource supply corresponding to each training accelerator, the calling priority corresponding to each training accelerator and the current task priority corresponding to each training accelerator, resource election is performed in each training accelerator to obtain the target hardware accelerator; wherein the training accelerator is a callable hardware accelerator in a training state.
5. The method according to claim 4, characterized in that The step of performing resource selection based on the resource demand, the resource provision corresponding to each callable hardware accelerator, and the call priority corresponding to each callable hardware accelerator to obtain a target hardware accelerator further includes: If there is a target idle accelerator, determining the target idle accelerator as the target hardware accelerator; If there are multiple target idle accelerators, resource selection is performed in each target idle accelerator according to the calling priority corresponding to each target idle accelerator to obtain the target hardware accelerator.
6. The method according to claim 1, characterized in that The calling the target hardware accelerator to execute the target training task includes: If the target hardware accelerator is deployed in a k8s cluster system, the target training task is initiated to the k8s cluster system to which the target hardware accelerator belongs through the k8s API; If the target hardware accelerator is deployed in a cloud host, initiating the target training task to the cloud host to which the target hardware accelerator belongs through a cloud resource operation and maintenance tool; If the target hardware accelerator is deployed locally, the target hardware accelerator is directly called to execute the target training task.
7. The method according to any one of claims 1 to 6, characterized in that: The obtaining of task information of the target training task includes: Acquire training configuration information configured by a user, the training configuration information including a storage path of the training data set, a model identifier of the target model, and the model training parameters; Determine the number of training data items in the training data set according to the storage path of the training data set, and calculate the data volume based on the number of training data items and a preset single data volume; According to the model identifier, the target model is determined in a preset basic model, and a model size of the target model is calculated.
8. The method according to claim 7, characterized in that The calling the target hardware accelerator to execute the target training task includes: Create dataset directories and basic model directories respectively; According to the storage path, import the training data set into the data set directory; According to the model identifier, the target model is soft-linked to the basic model directory; Assembling a training command according to the data set directory, the basic model directory and the model training parameters; The training command is sent to the target hardware accelerator so that the target hardware accelerator performs the target training task according to the data set directory, the basic model directory and the model training parameters.
9. A management and scheduling device for model training tasks, characterized in that: include: A task information acquisition module, used to obtain task information of the target training task in response to an execution instruction of the target training task; The task information includes model training parameters, the data size of the training data set and the model size of the target model, and the target model is the basic model to be trained; A resource requirement calculation module, used to calculate the resource requirement corresponding to the target training task according to the model training parameters, the data size and the model size; A resource selection module, configured to select resources based on the resource demand, the resource supply corresponding to each callable hardware accelerator, and the call priority corresponding to each callable hardware accelerator, to obtain a target hardware accelerator; wherein the lower the usage price corresponding to the callable hardware accelerator, the higher the call priority; A calling module is used to call the target hardware accelerator to execute the target training task when a preset calling rule is met, so that the target hardware accelerator performs model training on the target model according to the model training parameters and the training data set.
10. A storage medium, characterized in that: The storage medium stores computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the steps of the method for managing and scheduling model training tasks as described in any one of claims 1 to 8.
11. A computer device, characterized in that: include: one or more processors, and memory; The memory stores computer-readable instructions, which, when executed by the one or more processors, perform the steps of the method for managing and scheduling model training tasks as described in any one of claims 1 to 8.
