A model training task scheduling method and device, equipment and storage medium
By acquiring training container logs and automatically generating task scheduling schemes using adaptive scheduling strategies, the problem of traditional model training task scheduling relying on manual intervention is solved, achieving efficient and flexible resource management and improving the stability and efficiency of model training.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHANGHAI YINGMIAO INTELLIGENT TECHNOLOGY CO LTD
- Filing Date
- 2026-03-04
- Publication Date
- 2026-06-05
AI Technical Summary
Traditional model training task scheduling methods rely on manual intervention, resulting in low resource utilization, low task scheduling flexibility and efficiency, and an inability to guarantee the efficiency and reliability of model training.
By acquiring the training container logs of the target model training task, a target task scheduling scheme is generated using the task adaptive scheduling strategy. Resource configuration is automatically adjusted to achieve adaptive scheduling, reducing the cost of manual intervention.
It improves resource utilization and the flexibility and efficiency of task scheduling, ensures the efficiency and reliability of model training, and reduces redundant calculations and progress loss caused by interruptions.
Smart Images
Figure CN122152464A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of model training technology, and in particular to a model training task scheduling method, apparatus, device, and storage medium. Background Technology
[0002] With the rapid development of artificial intelligence models, in model training scenarios, models are usually large, training cycles are long, and data processing chains are complex. Any training interruption may cause a large amount of repetitive calculations. Model training task scheduling plays a crucial role in ensuring the quality and efficiency of model training.
[0003] Currently, traditional model training task scheduling methods often require manual intervention, with operations teams manually troubleshooting, adjusting resource quotas, or restarting tasks. However, the manual intervention costs of traditional model training task scheduling are high, and resource utilization is low, resulting in low flexibility and efficiency in task scheduling, thus failing to guarantee the efficiency and reliability of model training. Summary of the Invention
[0004] This invention provides a model training task scheduling method, apparatus, device, and storage medium to achieve adaptive scheduling of model training tasks, reduce the cost of manual intervention, improve resource utilization, and enhance the flexibility and efficiency of task scheduling, thereby ensuring the efficiency and reliability of model training.
[0005] According to one aspect of the present invention, a model training task scheduling method is provided, the method comprising: In response to the abnormal state of the target model training task, the target training container log corresponding to the target model training task is obtained; Based on the task adaptive scheduling strategy and the target training container logs, the target task scheduling scheme corresponding to the target model training task is determined; Based on the target task scheduling scheme, the target model training task is scheduled to obtain the target task scheduling result.
[0006] According to another aspect of the present invention, a model training task scheduling apparatus is provided, the apparatus comprising: The log acquisition module is used to acquire the target training container logs corresponding to the target model training task when the task execution status of the target model training task is in an abnormal state. The scheme determination module is used to determine the target task scheduling scheme corresponding to the target model training task based on the task adaptive scheduling strategy and the target training container log; The task scheduling module is used to schedule the target model training task based on the target task scheduling scheme and obtain the target task scheduling result.
[0007] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to execute the model training task scheduling method according to any embodiment of the present invention.
[0008] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the model training task scheduling method according to any embodiment of the present invention.
[0009] The technical solution of this invention, in response to an abnormal task execution state of the target model training task, obtains the target training container log corresponding to the target model training task, providing a data foundation for subsequent analysis. Based on the task adaptive scheduling strategy and the target training container log, a target task scheduling scheme corresponding to the target model training task is determined, improving resource utilization. Based on the target task scheduling scheme, the target model training task is scheduled to obtain the target task scheduling result, improving scheduling efficiency. This invention automatically generates a target task scheduling scheme through the target training container log and the task adaptive scheduling strategy, and then achieves adaptive scheduling of the training task according to the target task scheduling scheme, greatly reducing the cost of manual intervention, improving resource utilization, and enhancing the flexibility and efficiency of task scheduling, thereby ensuring the efficiency and reliability of model training.
[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description
[0011] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0012] Figure 1 This is a flowchart of a model training task scheduling method provided in Embodiment 1 of the present invention; Figure 2This is a flowchart of a model training task scheduling method provided in Embodiment 2 of the present invention; Figure 3 This is a schematic diagram of the structure of a model training task scheduling device according to Embodiment 3 of the present invention; Figure 4 This is a schematic diagram of the structure of an electronic device that implements the model training task scheduling method of this invention. Detailed Implementation
[0013] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.
[0014] It should be noted that the terms "first," "second," "target," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0015] Example 1 Figure 1 This is a flowchart illustrating a model training task scheduling method provided in Embodiment 1 of the present invention. This embodiment is applicable to situations where model training tasks are scheduled when training tasks are abnormally interrupted. This method can be executed by a model training task scheduling device, which can be implemented in hardware and / or software. This model training task scheduling device can be configured in an electronic device. Figure 1 As shown, the method includes: S110. In response to the abnormal state of the target model training task, obtain the target training container log corresponding to the target model training task.
[0016] Here, the target model training task can refer to the ongoing training of the target model. The task execution status can refer to the real-time running status of the training task in the distributed system. Abnormal states can refer to unexpected states that occur during task execution, such as execution interruption. The target training container logs can refer to the structured logs generated during container runtime.
[0017] Specifically, the task execution status of the ongoing target model training task can be detected in real time. If the task execution status of the target model training task is detected to be abnormal, the target training container log corresponding to the target model training task can be automatically obtained, which can provide original evidence for subsequent analysis.
[0018] For example, S110 may include: obtaining the task execution status corresponding to the target model training task; in response to the target model training task's task execution status being an abnormal state, extracting logs from the training container corresponding to the target model training task based on a preset number of log lines, and obtaining the target training container logs corresponding to the target model training task.
[0019] Here, the preset log line count can refer to a pre-defined range of log lines that can be extracted. The training container can refer to a containerized unit that hosts the model's runtime environment in distributed training.
[0020] Specifically, the task execution status of the target model training task can be collected in real time by integrating the monitoring component of the container orchestration platform (such as Kubernetes). If the task execution status of the target model training task is abnormal, the logs of the training container corresponding to the target model training task can be extracted according to the preset number of log lines. Limiting the number of log lines can avoid network congestion and storage pressure caused by excessive log transmission, improve the log extraction speed, and quickly obtain the logs of the target training container corresponding to the target model training task, providing reliable data input for subsequent adaptive scheduling decisions.
[0021] For example, the target model training task can be a medical image training task. During the task training process, the state changes of PyTorchJob can be monitored in real time. When the medical image training task is detected to enter the Failed abnormal state, the last N lines of logs of the training container can be obtained (e.g., N=100).
[0022] S120. Based on the task adaptive scheduling strategy and the target training container log, determine the target task scheduling scheme corresponding to the target model training task.
[0023] The task adaptive scheduling strategy can refer to a pre-set intelligent algorithm that formulates a scheduling plan for the training task when it is abnormally interrupted. The target task scheduling scheme can refer to a scheduling execution plan designed for a specific task.
[0024] Specifically, by parsing the logs of the target training container, the error types corresponding to the abnormal states of the target model training task can be identified. Combined with the task adaptive scheduling strategy, a target task scheduling scheme corresponding to the target model training task can be generated, realizing differentiated processing of different error types and improving resource utilization.
[0025] S130. Based on the target task scheduling scheme, perform task scheduling on the target model training task to obtain the target task scheduling result.
[0026] The target task scheduling result can refer to the result used to indicate the successful scheduling of the training task.
[0027] Specifically, the target model training task can be scheduled according to the target task scheduling scheme, and the target task scheduling result can be generated after the scheduling operation is completed. This can significantly improve the scheduling efficiency of the training task and enhance the stability of the model training task.
[0028] In this embodiment, by responding to an abnormal task execution state of the target model training task, the target training container log corresponding to the target model training task is obtained, providing a data foundation for subsequent analysis. Based on the task adaptive scheduling strategy and the target training container log, the target task scheduling scheme corresponding to the target model training task is determined, which can improve resource utilization. The target model training task is scheduled based on the target task scheduling scheme to obtain the target task scheduling result, improving scheduling efficiency. This invention automatically generates a target task scheduling scheme through the target training container log and the task adaptive scheduling strategy, and then achieves adaptive scheduling of training tasks according to the target task scheduling scheme, greatly reducing the cost of manual intervention, improving resource utilization, and increasing the flexibility and efficiency of task scheduling, thereby ensuring the efficiency and reliability of model training.
[0029] Example 2 Figure 2 This is a flowchart of a model training task scheduling method provided in Embodiment 2 of the present invention. Based on the above embodiments, this embodiment optimizes the step "determining the target task scheduling scheme corresponding to the target model training task based on the task adaptive scheduling strategy and the target training container log". Explanations of terms that are the same as or corresponding to those in the above embodiments are not repeated here.
[0030] See Figure 2 The alternative model training task scheduling method provided in this embodiment specifically includes the following steps: S210. In response to the abnormal state of the target model training task, obtain the target training container log corresponding to the target model training task.
[0031] S220. Based on the target training container logs, determine the task fault type corresponding to the target model training task.
[0032] Among them, the task failure type can refer to the classification of abnormal states that occur during the execution of the target model training task.
[0033] Specifically, word segmentation, entity recognition, and relation extraction can be performed on the target training container logs. Combined with preset classification criteria, the task fault type corresponding to the target model training task can be determined, which can improve the efficiency and accuracy of fault diagnosis and provide a basis for determining subsequent scheduling schemes.
[0034] For example, S220 may include: determining the training exit code corresponding to the target model training task based on the target training container log; and determining the task fault type corresponding to the target model training task based on the graphics processing unit fault value range and the training exit code.
[0035] Among them, the training exit code can refer to the integer status code returned when the training container terminates, reflecting the program execution result or the cause of the exception. The graphics processing unit (GPU) fault value range can refer to the range of training exit codes when a graphics processing unit (GPU) malfunctions.
[0036] Specifically, the exit code information in the target training container log can be parsed to obtain the training exit code corresponding to the target model training task. The fault value range of the graphics processing unit is matched with the training exit code. By determining whether the training exit code is within the fault value range of the graphics processing unit, the task fault type corresponding to the target model training task can be determined. This transforms the fuzzy log information into structured fault type labels, improving the accuracy of fault location.
[0037] For example, the numerical range of GPU faults is matched with the training exit code. If the exit code matches the numerical range of GPU faults (i.e., the training exit code is within the numerical range of GPU faults), it is determined to be a "GPU fault"; otherwise, it is determined to be a "non-GPU fault" (e.g., model code error, data anomaly, etc.).
[0038] S230. Based on the task adaptive scheduling strategy and task fault type, determine the target task scheduling scheme corresponding to the target model training task.
[0039] Specifically, an adaptive scheduling strategy library can be pre-built. By mapping the fault type to the strategy, the optimal strategy can be automatically matched to generate the target task scheduling scheme corresponding to the target model training task, thereby avoiding manual intervention and improving scheduling efficiency.
[0040] For example, S230 may include: in response to a task fault type of graphics processing unit fault, triggering a task migration operation, updating the original graphics processing unit corresponding to the target model training task based on a task adaptive scheduling strategy, determining the target graphics processing unit, and determining the target graphics processing unit as the target task scheduling scheme corresponding to the target model training task.
[0041] The original graphics processing unit (GPU) can refer to the GPU in which the target model training task terminated abnormally. The target GPU can refer to the GPU resource selected for task migration that meets the resource requirements and compatibility requirements, serving as the execution carrier for the training task.
[0042] Specifically, if the fault is determined to be non-GPU fault, the migration process will not be triggered, the failure status will be retained, and manual processing will be performed. When a task fault type of GPU fault is detected, the task migration operation process will be automatically triggered. The task resource requirements corresponding to the target model training task will be analyzed according to the task adaptive scheduling strategy. Based on the analysis results, the GPU cluster where the original GPU is located will be screened, the original GPU will be updated, and the GPU that meets the screening conditions will be determined as the target GPU. The target GPU will then be determined as the target task scheduling scheme corresponding to the target model training task, which will greatly improve the efficiency and accuracy of task scheduling.
[0043] S240. Based on the target task scheduling scheme, perform task scheduling on the target model training task to obtain the target task scheduling result.
[0044] For example, S240 may include: migrating the target model training task to the target graphics processing unit based on the target task scheduling scheme; in response to the task start operation of the target graphics processing unit, obtaining the model intermediate state file corresponding to the target model training task, and performing a state recovery operation on the target graphics processing unit based on the model intermediate state file to obtain the target task scheduling result.
[0045] Among them, the intermediate state file of the model can refer to the binary / structured file (checkpoint file) that is saved periodically during the model training process. It is used to record the key state information of the training task and supports breakpoint resume training and fault recovery.
[0046] Specifically, the graphics processing unit (GPU) corresponding to the target model training task can be configured according to the target task scheduling scheme, and the target model training task of the original GPU can be migrated to the target GPU by calling the Kubernetes API. In response to the task start operation of the target GPU, the intermediate state file of the model corresponding to the target model training task can be obtained through object storage, and the intermediate state file of the model can be loaded into the target GPU. The state recovery operation of the target GPU can be performed to obtain the target task scheduling result. This can avoid progress loss or model performance degradation caused by task interruption, ensure training progress, thereby minimizing redundant calculations and improving resource utilization and system self-healing capabilities.
[0047] For example, based on the intermediate state file of the model, a state recovery operation is performed on the target graphics processing unit to obtain the target task scheduling result, including: determining the target recovery data corresponding to the target model training task based on the intermediate state file of the model, wherein the target recovery data includes the target model parameters, the target optimizer state and the target iteration progress; and performing a state recovery operation on the target graphics processing unit based on the target recovery data to obtain the target task scheduling result.
[0048] Specifically, the target recovery data can refer to the specific dataset parsed from the model's intermediate state file, used to recover the training task. The target model parameters can refer to the core weights and biases of the neural network, determining the model's predictive ability and training dynamics. The target optimizer state can refer to the dynamic state of the optimizer during training, including hyperparameters and gradient caching, affecting gradient update rules and convergence speed. The target iteration progress can refer to key indicators recording the current execution progress of the training task, which can be used to accurately locate the training position before interruption.
[0049] Specifically, by parsing the structured data in the intermediate state file of the model, the target model parameters, target optimizer state, and target iteration progress can be obtained, which are the target recovery data corresponding to the target model training task. Based on the target recovery data, the target model parameters and optimizer state are loaded into the video memory of the target GPU, and the target iteration progress is injected into the training loop to ensure that the training counter and learning rate scheduler are completely consistent with those before the interruption. This completes the state recovery operation of the target graphics processing unit and obtains the target task scheduling result. This ensures that the training task can be seamlessly resumed after migration, avoids progress loss or model performance fluctuations caused by interruption, and improves the stability of model training.
[0050] The technical solution of this embodiment determines the task fault type corresponding to the target model training task based on the target training container logs, ensuring the accuracy and efficiency of fault identification. Based on the task adaptive scheduling strategy and the task fault type, it determines the target task scheduling scheme corresponding to the target model training task, reducing manual intervention costs and improving task scheduling efficiency. This invention, by determining the task fault type and then determining the corresponding target task scheduling scheme based on the fault type, achieves accurate fault type diagnosis and intelligent scheduling scheme matching, which can greatly improve task scheduling efficiency, thereby enhancing the stability and efficiency of the model training task.
[0051] Example 3 Figure 3 This is a schematic diagram of a model training task scheduling device provided in Embodiment 3 of the present invention. Figure 3 As shown, the device includes: a log acquisition module 310, a scheme determination module 320, and a task scheduling module 330; The log acquisition module 310 is used to acquire the target training container log corresponding to the target model training task when the task execution status of the target model training task is abnormal. The scheme determination module 320 is used to determine the target task scheduling scheme corresponding to the target model training task based on the task adaptive scheduling strategy and the target training container log; The task scheduling module 330 is used to schedule the target model training task based on the target task scheduling scheme and obtain the target task scheduling result.
[0052] In this embodiment, by responding to an abnormal task execution state of the target model training task, the target training container log corresponding to the target model training task is obtained, providing a data foundation for subsequent analysis. Based on the task adaptive scheduling strategy and the target training container log, a target task scheduling scheme corresponding to the target model training task is determined, which can improve resource utilization. Based on the target task scheduling scheme, the target model training task is scheduled to obtain the target task scheduling result, improving scheduling efficiency. This invention automatically generates a target task scheduling scheme through the target training container log and the task adaptive scheduling strategy, and then realizes adaptive scheduling of training tasks according to the target task scheduling scheme, greatly reducing the cost of manual intervention, improving resource utilization, and increasing the flexibility and efficiency of task scheduling, thereby ensuring the efficiency and reliability of model training.
[0053] Optionally, the log acquisition module 310 is specifically used to: acquire the task execution status corresponding to the target model training task; in response to the task execution status of the target model training task being an abnormal state, extract logs from the training container corresponding to the target model training task based on a preset number of log lines, and acquire the target training container logs corresponding to the target model training task.
[0054] Optionally, the scheme determination module 320 includes: The fault type determination unit is used to determine the task fault type corresponding to the target model training task based on the target training container logs. The scheduling scheme determination unit is used to determine the target task scheduling scheme corresponding to the target model training task based on the task adaptive scheduling strategy and the task fault type.
[0055] Optionally, the fault type determination unit is specifically used to: determine the training exit code corresponding to the target model training task based on the target training container log; and determine the task fault type corresponding to the target model training task based on the fault value range of the graphics processing unit and the training exit code.
[0056] Optionally, the scheduling scheme determination unit is specifically used to: in response to the task fault type being a graphics processing unit fault, trigger a task migration operation, update the original graphics processing unit corresponding to the target model training task based on the task adaptive scheduling strategy, determine the target graphics processing unit, and determine the target graphics processing unit as the target task scheduling scheme corresponding to the target model training task.
[0057] Optionally, the task scheduling module 330 includes: The target unit determination unit is used to transfer the target model training task to the target graphics processing unit based on the target task scheduling scheme. The task scheduling unit is used to respond to the task start operation of the target graphics processing unit, obtain the model intermediate state file corresponding to the target model training task, and perform a state recovery operation on the target graphics processing unit based on the model intermediate state file to obtain the target task scheduling result.
[0058] Optionally, the task scheduling unit is specifically used to: determine the target recovery data corresponding to the target model training task based on the model intermediate state file, wherein the target recovery data includes target model parameters, target optimizer state, and target iteration progress; and perform a state recovery operation on the target graphics processing unit based on the target recovery data to obtain the target task scheduling result.
[0059] The above-described apparatus can execute the model training task scheduling method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of executing the model training task scheduling method.
[0060] Example 4 Figure 4 This is a schematic diagram of the structure of an electronic device implementing the model training task scheduling method of this invention. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.
[0061] like Figure 4 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded into the RAM 13 from storage unit 18. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.
[0062] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.
[0063] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as model training task scheduling methods.
[0064] In some embodiments, the model training task scheduling method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the model training task scheduling method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to execute the model training task scheduling method by any other suitable means (e.g., by means of firmware).
[0065] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication unit 19, or installed from storage unit 18, or installed from ROM 12. When the computer program is executed by processor 11, it performs the functions defined in the methods of the embodiments of the present invention.
[0066] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.
[0067] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.
[0068] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.
[0069] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).
[0070] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.
[0071] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.
[0072] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.
[0073] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.
Claims
1. A method for scheduling model training tasks, characterized in that, include: In response to the abnormal state of the target model training task, the target training container log corresponding to the target model training task is obtained; Based on the task adaptive scheduling strategy and the target training container logs, the target task scheduling scheme corresponding to the target model training task is determined; Based on the target task scheduling scheme, the target model training task is scheduled to obtain the target task scheduling result.
2. The method according to claim 1, characterized in that, In response to an abnormal task execution state, the target training container log corresponding to the target model training task is obtained, including: Obtain the task execution status corresponding to the target model training task; In response to the abnormal state of the target model training task, the logs of the training container corresponding to the target model training task are extracted based on a preset number of log lines to obtain the target training container logs corresponding to the target model training task.
3. The method according to claim 1, characterized in that, The step of determining the target task scheduling scheme corresponding to the target model training task based on the task adaptive scheduling strategy and the target training container logs includes: Based on the target training container logs, determine the task fault type corresponding to the target model training task; Based on the task adaptive scheduling strategy and the task fault type, the target task scheduling scheme corresponding to the target model training task is determined.
4. The method according to claim 3, characterized in that, The step of determining the task fault type corresponding to the target model training task based on the target training container logs includes: Based on the target training container logs, determine the training exit code corresponding to the target model training task; Based on the fault value range of the graphics processing unit and the training exit code, the task fault type corresponding to the target model training task is determined.
5. The method according to claim 3, characterized in that, The step of determining the target task scheduling scheme corresponding to the target model training task based on the task adaptive scheduling strategy and the task fault type includes: In response to the task failure type being a graphics processing unit failure, a task migration operation is triggered. Based on the task adaptive scheduling strategy, the original graphics processing unit corresponding to the target model training task is updated to determine the target graphics processing unit, and the target graphics processing unit is determined as the target task scheduling scheme corresponding to the target model training task.
6. The method according to claim 5, characterized in that, The step of scheduling the target model training task based on the target task scheduling scheme to obtain the target task scheduling result includes: Based on the target task scheduling scheme, the target model training task is transferred to the target graphics processing unit; In response to the task initiation operation of the target graphics processing unit, the intermediate state file of the model corresponding to the target model training task is obtained, and based on the intermediate state file of the model, the state recovery operation of the target graphics processing unit is performed to obtain the target task scheduling result.
7. The method according to claim 6, characterized in that, The step of performing a state recovery operation on the target graphics processing unit based on the intermediate state file of the model to obtain the target task scheduling result includes: Based on the intermediate state file of the model, the target recovery data corresponding to the target model training task is determined, wherein the target recovery data includes the target model parameters, the target optimizer state, and the target iteration progress; Based on the target recovery data, a state recovery operation is performed on the target graphics processing unit to obtain the target task scheduling result.
8. A model training task scheduling device, characterized in that, include: The log acquisition module is used to acquire the target training container logs corresponding to the target model training task when the task execution status of the target model training task is in an abnormal state. The scheme determination module is used to determine the target task scheduling scheme corresponding to the target model training task based on the task adaptive scheduling strategy and the target training container log; The task scheduling module is used to schedule the target model training task based on the target task scheduling scheme and obtain the target task scheduling result.
9. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the model training task scheduling method according to any one of claims 1-7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the model training task scheduling method according to any one of claims 1-7.