A task scheduling method and device

Abstract

The application discloses a task scheduling method and device, and relates to the field of deep learning underlying resource scheduling.When a reasoning task causes a resource conflict, the method safely freezes a node to intercept execution right after the end of back propagation and before the update of an optimizer parameter through a hook mechanism embedded in a deep learning framework execution engine, performs atomic state collection, generates a state package containing at least a model weight tensor, an optimizer momentum state and a mixed precision training factor, and releases GPU memory through asynchronous direct memory access (DMA); then, the reasoning task is executed in a pre-allocated GPU reserved area, and after the reasoning is completed, a redundant verification link is removed, and the training task is directly recovered based on the state package.The application crosses the limitation of traditional container-level macro scheduling, realizes accurate preservation and extremely fast switching of the micro state of the framework bottom layer, guarantees the second-level response of reasoning, completely avoids loss function jitter caused by training interruption, and realizes real lossless continuation.

Description

Technical Field

[0001] This invention relates to the field of resource allocation technology, and in particular to a task scheduling method and apparatus. Background Technology

[0002] In resource-constrained environments, GPUs need to simultaneously handle high-priority, low-latency inference tasks and long-cycle, high-occupancy training tasks in deep learning. Ensuring real-time inference response while achieving lossless interruption and recovery of training tasks, and improving GPU resource turnover efficiency, has become a key technical requirement.

[0003] Currently, GPU heterogeneous task scheduling mainly adopts two schemes: static priority queue and time slice round-robin. The former assigns a fixed priority to tasks, and inference tasks must wait for the training task to finish or terminate before execution. The latter divides the GPU computing time into fixed time slices, and the two types of tasks alternately occupy the GPU. Both only schedule at the process level and do not consider the internal state characteristics of deep learning tasks.

[0004] In existing solutions, static scheduling lacks interruption capability, resulting in uncontrollable inference latency. The fixed time slice of round-robin scheduling cannot adapt to the low latency requirements of inference. Furthermore, existing technologies can only forcibly terminate or simply pause training tasks, and cannot perceive the internal state of the framework, which can easily lead to the loss of critical data and make it impossible to achieve lossless interruption recovery of training.

[0005] Furthermore, while existing solutions (such as container-level scheduling based on cloud-native environments) can pause training tasks and save checkpoints when resources are scarce, they have significant drawbacks: First, the timing of the pause is uncontrollable; container-level pauses often occur midway through computation (such as during parameter updates), leading to a mismatch between weights and gradient states after resumption, causing training curve jitter or even preventing the loss function from converging. Second, the state saving granularity is coarse, only saving model weights and logs, losing internal framework states such as optimizer momentum and mixture precision factors, making true lossless continuation impossible. Third, they do not perform memory isolation and acceleration for inference tasks, failing to meet the stringent requirements of visual AI applications for millisecond / second end-to-end latency in inference tasks. Summary of the Invention

[0006] This invention provides a task scheduling method and apparatus that solves the technical problems of resource conflicts when training and inference tasks run in parallel under GPU computing power, the inability to recover without loss after training tasks are interrupted, low utilization of video memory resources, and insufficient hardware operation stability.

[0007] According to one aspect of the present invention, a task scheduling method is provided, the method comprising: When a new inference task arrives, determine the resource conflict situation of the inference task; When a resource conflict occurs, a preemption instruction is generated. Based on the preemption instruction, the training task is paused at a safe freeze point, which is the time window after the backpropagation ends and before the optimizer parameters are updated. At the safe freeze point, execution rights are intercepted through the hook mechanism of the deep learning framework execution engine, and atomic state collection is performed to generate a state package. The state package includes at least the model weight tensor, the optimizer momentum state, and the factors of mixed precision training. The state packet is transferred to the host memory via asynchronous direct memory access (DMA), and the GPU memory occupied by the training task is released. The inference task is executed in the pre-allocated memory reserved area for the inference task, and after the inference task is completed, the training task is directly restored to the safe freeze point based on the state packet to continue execution.

[0008] Optionally, the method further includes: obtaining a task request; determining the task type based on the calling interface of the task request, wherein the task type includes inference tasks and training tasks; defining attributes for tasks of different task types and recording the task attribute information of each task to form a task attribute table, wherein the task attribute information includes priority, interruptibility, peak video memory, expected computing power ratio and model family to which it belongs, with inference tasks being high priority and uninterruptible, and training tasks being low priority and safe to suspend.

[0009] Optionally, determining the resource conflict status of the inference task includes: obtaining the peak video memory and expected computing power ratio of the inference task based on the task attribute table to determine the video memory requirement of the inference task; obtaining the available video memory of the current graphics processing unit (GPU); determining whether the available video memory meets the video memory requirement; if so, determining the resource conflict status as non-conflict; otherwise, determining the resource conflict status as conflict.

[0010] Optionally, the state package may further include: gradient accumulation buffer, current training phase, dynamic learning rate curve state, and random number generator state.

[0011] Optionally, at the safe freeze point, execution rights are intercepted through a hook mechanism embedded in the deep learning framework's execution engine, and atomic state acquisition is performed to generate a state package. This includes: at the safe freeze point, execution rights are intercepted through a hook mechanism embedded in the deep learning framework's execution engine, and atomic state acquisition is performed to obtain a state snapshot of the training task. The state snapshot includes the model weight tensor, optimizer momentum state, gradient accumulation buffer, current training stage, dynamic learning rate curve state, mixed precision training factor, and random number generator state; and a state package is generated based on the state snapshot.

[0012] Optionally, generating a state packet based on a state snapshot includes: determining the generation type of the state snapshot, wherein the generation type includes periodic strong refresh and regular interruption; when the generation type is periodic strong refresh, directly serializing and encapsulating the state snapshot to generate the state packet for the training task; when the generation type is regular interruption, obtaining the preceding snapshot, extracting incremental data based on the preceding snapshot and the state snapshot, and serializing and encapsulating the incremental data into the state packet for the training task.

[0013] Optionally, restoring the training task to a safe freeze point based on the state packet includes: deserializing and parsing the state packet to obtain the restored state of the training task, wherein the restored state is a full state or an incremental state; loading the restored state into the graphics processor, restoring the running state of the training task to the safe freeze point, and controlling the training task to continue execution from the safe freeze point.

[0014] Optionally, the method also includes: real-time collection of operational monitoring metrics for the training task, wherein the monitoring metrics include GPU utilization, GPU core temperature, memory allocator fragmentation rate, and computing power contention latency; preset multi-level response thresholds and corresponding graded response levels; comparing the real-time collected monitoring metrics with the preset multi-level response thresholds to determine the current graded response level; and executing the corresponding graded response action according to the graded response level to achieve graded response monitoring of the training task.

[0015] Optionally, generating preemption instructions includes: querying the peak video memory and priority of each training task in the task attribute table; selecting the training task with the lowest priority whose peak video memory exceeds the video memory requirement as the preemption target; and generating the corresponding preemption instructions based on the preemption target.

[0016] Optionally, the method also includes: loading the weight data of the corresponding inference model into the reserved memory area according to the model family to which each inference task belongs in the task attribute table; when a new inference task of the same model family arrives, directly reusing the weight data already loaded in the reserved memory area to perform inference calculation.

[0017] According to another aspect of the present invention, a task scheduling apparatus is provided, the apparatus comprising: The resource conflict determination module is used to determine the resource conflict status of a new inference task when it arrives. The memory release and state packet generation module is used to generate a preemption instruction when resource conflicts occur. Based on the preemption instruction, the training task is paused at a safe freeze point, which is the time window after backpropagation ends and before the optimizer parameters are updated. At the safe freeze point, execution rights are intercepted through a hook mechanism embedded in the deep learning framework's execution engine. Atomic state acquisition is performed to generate a state packet, which includes at least the model weight tensor, the optimizer momentum state, and the mixed-precision training factor. The state packet is transferred to the host memory via asynchronous direct memory access (DMA), and the memory occupied by the training task is released. The training task recovery module is used to execute inference tasks in the pre-allocated video memory reserved area for inference tasks, and after the inference task is completed, directly restore the training task to the safe freeze point based on the state packet to continue execution.

[0018] The technical solution of this invention, by determining the resource conflict situation of the inference task, can judge the resource allocation status and provide an accurate decision basis for whether to perform preemption operations. By intercepting the execution right and performing atomic state acquisition at a specific computational semantic node (safe freeze point) after the backpropagation ends and before the optimizer parameters are updated, the hook mechanism embedded in the underlying framework is used to fundamentally ensure that the acquired model weights and gradient data are strictly matched numerically and logically, avoiding the problem of weight and gradient state mismatch caused by container-level random pauses. At the same time, the state packet is forced to include the internal state of the micro-framework, such as the optimizer momentum state and the factors of mixed precision training. Combined with the physical isolation of asynchronous DMA transfer and the reserved memory area, a closed loop for fast switching of underlying memory is constructed. After inference, by eliminating redundant verification steps such as hash verification and tensor shape checking, the logic from the end of inference to the restart of training is significantly shortened, and the resource switching latency is controlled at the second level. This solution achieves a micro-level qualitative change from "avoiding training from scratch" to "zero jitter and strict alignment of training loss curves," ensuring high real-time system response and continuity of training trajectory. It completely solves the technical defects of existing container-level scheduling solutions, such as coarse-grained state preservation and inability to achieve truly lossless retransmission.

[0019] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart of a task scheduling method provided in Embodiment 1 of the present invention; Figure 2 This is a flowchart of another task scheduling method provided according to Embodiment 2 of the present invention; Figure 3 This is a schematic diagram of the hierarchical architecture of a task scheduling system according to Embodiment 2 of the present invention; Figure 4 This is a schematic diagram of the structure of a task scheduling device according to Embodiment 3 of the present invention; Figure 5 This is a schematic diagram of the structure of an electronic device that implements a task scheduling method according to an embodiment of the present invention. Detailed Implementation

[0022] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0023] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0024] Example 1 Figure 1 The flowchart illustrates a task scheduling method provided in Embodiment 1 of the present invention. This embodiment is applicable to heterogeneous task scheduling scenarios in visual AI. The method can be executed by a task scheduling device, which can be implemented in hardware and / or software and can be configured in a computer controller. Figure 1 As shown, the method includes: S110. When a new reasoning task arrives, determine the resource conflict situation of the reasoning task.

[0025] In this context, inference tasks refer to real-time inference computation tasks in visual AI scenarios. These are short-term tasks that call upon a trained AI model, input visual data, and quickly output inference results. Resource conflict refers to the determination of whether the GPU's currently available video memory meets the memory requirements of a new inference task when it arrives.

[0026] Optionally, the method further includes: obtaining a task request; determining the task type based on the calling interface of the task request, wherein the task type includes inference tasks and training tasks; defining attributes for tasks of different task types and recording the task attribute information of each task to form a task attribute table, wherein the task attribute information includes priority, interruptibility, peak video memory, expected computing power ratio and model family to which it belongs, with inference tasks being high priority and uninterruptible, and training tasks being low priority and safe to suspend.

[0027] Specifically, the system first receives various task requests submitted externally. During processing, it does not interfere with the underlying memory read / write instructions. Instead, it distinguishes the task type by identifying the type of application programming interface called by the task request, clarifying whether the task request belongs to an inference task or a training task. After determining the task type, the system defines task attributes according to the different business characteristics and execution requirements of inference and training tasks.

[0028] As we know, inference tasks are short-term tasks in visual AI scenarios, with strict real-time requirements for execution latency. Their memory usage can be accurately predicted during initialization and remains constant throughout. Therefore, the system defines inference tasks as high-priority and uninterruptible. Training tasks are long-term model iteration tasks that can be interrupted at specific safe points and can be recovered without loss. Therefore, the system defines training tasks as low-priority and safely suspendable. Furthermore, the system comprehensively records and integrates the task attribute information for each task to form a task attribute table. The task attribute information recorded in the table includes priority, interruptibility, peak memory usage, expected computing power percentage, and the model family to which the task belongs. Peak memory usage is the core basis for subsequent resource conflict determination, while the expected computing power percentage and the model family to which the task belongs provide auxiliary references for subsequent scheduling decisions and resource allocation.

[0029] Optionally, determining the resource conflict status of the inference task includes: obtaining the peak video memory and expected computing power ratio of the inference task based on the task attribute table to determine the video memory requirement of the inference task; obtaining the available video memory of the current graphics processing unit (GPU); determining whether the available video memory meets the video memory requirement; if so, determining the resource conflict status as non-conflict; otherwise, determining the resource conflict status as conflict.

[0030] Specifically, the system retrieves the previously constructed task attribute table and extracts the peak video memory data and expected computing power ratio corresponding to the inference task. Peak video memory serves as the key basis for determining the video memory requirements of the inference task, while the expected computing power ratio is only used as auxiliary reference information for scheduling. Combining these two data points completes the definition of the actual video memory requirements of the inference task, providing a data foundation for subsequent resource conflict determination. Then, the system collects the device-side video memory status of the current graphics processing unit (GPU) in real time to obtain the available video memory of the current GPU. Available video memory is the amount of video memory resources on the GPU that are not occupied by training tasks or other tasks and can be allocated and used immediately. By directly comparing the video memory requirements of the inference task with the available video memory value of the current GPU, it can be determined whether the available video memory can fully meet the video memory requirements of the inference task. If the available video memory value of the current GPU is greater than or equal to the peak video memory requirements of the inference task, it means that the existing GPU video memory resources are sufficient to support the execution of the inference task, and no resource preemption or scheduling is required. The system directly determines that the resource conflict situation of the inference task is non-conflictual. If the available GPU memory is less than the peak memory requirement of the inference task, it means that the existing memory resources cannot support the execution of the inference task, which will trigger the GPU resource preemption scheduling logic. At this time, the resource conflict is determined to be a conflict.

[0031] S120. When the resource conflict is a conflict, a preemption instruction is generated. Based on the preemption instruction, the training task is paused at a safe freeze point, where the safe freeze point is the time window after the backpropagation ends and before the optimizer parameters are updated.

[0032] The preemption instruction contains the target training task ID and the expected amount of GPU memory to be released. It is the control instruction that triggers the training task to enter the safe freeze point and execute state sealing. The training task refers to the deep learning model training task in a visual AI scenario, which is a long-term task of training an untrained / to-iterate AI model with data and updating model parameters. The safe freeze point is a specific time window between the end of backpropagation and the update of optimizer parameters during the training task iteration process; it is a safe interruption point for the training task.

[0033] Optionally, generating preemption instructions includes: querying the peak video memory and priority of each training task in the task attribute table; selecting the training task with the lowest priority whose peak video memory exceeds the video memory requirement as the preemption target; and generating the corresponding preemption instructions based on the preemption target.

[0034] Among them, the video memory requirement refers to the total video memory required to run the new inference task that arrives at the moment. It is determined by the peak video memory field corresponding to the inference task in the task attribute table. The preemption instruction is a control instruction issued by the core scheduling module to the collaborative security sealing module. It is used to specify the training task object that needs to be interrupted and trigger the state sealing process.

[0035] Specifically, in multi-training task scenarios, priority levels can be further subdivided based on task importance. When the system determines that a resource conflict has occurred and multiple running training tasks exist, the core scheduling module first retrieves the task attribute table, iterating through and reading the peak GPU memory value and priority marker corresponding to all running training tasks. Then, it compares the peak GPU memory of each training task with the GPU memory requirement of the current inference task, selecting training tasks with peak GPU memory values ​​greater than or equal to the inference task's GPU memory requirement as candidate preemption targets, ensuring that the GPU memory space released by the target task can fully cover the running needs of the inference task. Then, the selected candidate task set is sorted according to priority from low to high, and the lowest priority training task is selected as the final preemption target, thus ensuring that high-priority core training tasks are not interrupted as much as possible, reducing the impact of scheduling operations on the core training process. Finally, the core scheduling module generates a corresponding preemption instruction based on the determined preemption target. The preemption instruction carries the identifier of the target training task and the expected GPU memory release baseline information, and sends the preemption instruction to the collaborative security sealing module to initiate subsequent security freeze and state sealing operations.

[0036] Optionally, the method also includes: loading the weight data of the corresponding inference model into the reserved memory area according to the model family to which each inference task belongs in the task attribute table; when a new inference task of the same model family arrives, directly reusing the weight data already loaded in the reserved memory area to perform inference calculation.

[0037] Here, "model family" refers to the classification and identification of inference models with similar functions and common network structures. Inference tasks within the same model family share the same set of basic model weights, differing only in output branches or post-processing logic. The inference model weight data is the set of parameter tensors obtained by training the deep learning inference model.

[0038] Specifically, after defining the attributes of all tasks and generating a task attribute table, the system first iterates through the model family identifiers corresponding to all inference task entries in the task attribute table, deduplicating duplicate model families to obtain a list of all inference model families to be deployed. Then, for each model family in the list, the system retrieves the corresponding inference model weight data and pre-loads it into the GPU's reserved memory area for permanent storage via a high-speed host-to-device memory copy operation, completing the pre-deployment of the inference runtime environment. When a new inference task arrives at the system, it first identifies the model family to which the task belongs, then queries the model family identifier corresponding to the already loaded weights in the reserved memory area. If the newly arrived inference task and the already loaded weights in the reserved memory area belong to the same model family, no additional weight loading and memory allocation operations are required; the system directly calls the already resident weight data in the reserved memory area to start forward inference computation. This eliminates the time overhead of dynamically loading weights for inference tasks, shortens the end-to-end startup latency of inference tasks, and avoids the waste of memory resources and additional computing power caused by repeatedly loading the same weights.

[0039] S130. At the safe freeze point, the execution right is intercepted by the hook mechanism of the deep learning framework execution engine, and the atomic state collection is performed to generate a state package. The state package includes at least the model weight tensor, the optimizer momentum state, and the factors of mixed precision training.

[0040] Intercepting training task execution and collecting state data at a safe freeze point prevents model parameter corruption and training state loss. The hook mechanism is a callback interception technique embedded in the framework's execution flow, allowing custom logic to be inserted at a specified execution node and take over program control without modifying the framework's core source code. Atomicity state collection means the entire collection process cannot be interrupted by any scheduling or interruption signal, ensuring all state data belongs to a consistent snapshot of the same training iteration, eliminating data mixing issues between iterations.

[0041] S140. Transfer the state packet to the host memory via asynchronous direct memory access (DMA) and release the GPU memory occupied by the training task.

[0042] The state packet refers to a self-contained data packet formed by serializing and encapsulating the collected training task state snapshots. It serves as the carrier connecting the interruption and resumption of the training task. This state packet is transferred asynchronously via Direct Memory Access (DMA) to a pre-allocated buffer in the host memory for storage. Video memory (VRAM) refers to the device-side memory of the graphics processor. It is the core hardware resource for storing model parameters and computational data during GPU inference and training tasks. It is divided into a dynamic VRAM area occupied by the training task and a fixed VRAM area pre-allocated for the inference task (VRAM reservation area). Released VRAM refers to the VRAM space originally occupied by the training task that is now freed up. When preemptive scheduling is triggered, the system archives and moves the training task, freeing up this portion of VRAM to work with the VRAM reservation area to support the complete operation of the inference task.

[0043] S150. Execute the inference task in the pre-allocated video memory reserved area for the inference task, and after the inference task is completed, directly restore the training task to the safe freeze point based on the state packet to continue execution.

[0044] The reserved memory area refers to a fixed memory area in the GPU that is pre-allocated specifically for inference tasks. It will not be occupied by training tasks, nor will it be dynamically allocated to other tasks.

[0045] Furthermore, this application eliminates redundant verification steps such as hash verification and tensor shape checking. Redundant verification steps refer to hash value comparison and tensor shape verification steps performed during routine cross-process or cross-node task recovery to prevent data corruption, version mismatch, or tensor dimension misalignment. Since the status packet in this application is generated atomically within the same process, data consistency and dimension matching are naturally guaranteed. Therefore, these verifications can be skipped to significantly reduce the overhead and time consumption of task recovery.

[0046] Optionally, restoring the training task to a safe freeze point based on the state packet includes: deserializing and parsing the state packet to obtain the restored state of the training task, wherein the restored state is a full state or an incremental state; loading the restored state into the graphics processor, restoring the running state of the training task to the safe freeze point, and controlling the training task to continue execution from the safe freeze point.

[0047] Specifically, the system reads the state packets stored in the host memory buffer and performs deserialization parsing on them. It's important to note that the parsing operation determines the snapshot type based on the state packet's markings using a chained state snapshot management mechanism. If the state packet is marked as a full snapshot, deserialization directly yields the full state of the training task. This state includes all training context information, such as the complete model weight tensor, optimizer momentum state, gradient accumulation buffer, current training stage, dynamic learning rate curve state, mixed-precision training factors, and random number generator state. If the state packet is marked as an incremental snapshot, the chained state snapshot management mechanism first retrieves the corresponding previous full snapshot data, merges the incremental snapshot with the full snapshot, and then performs deserialization to obtain the complete incremental state of the training task. This state includes parameter changes based on the full baseline state, current iteration step information, and updated context information such as the latest learning rate scheduler and stage metadata, ensuring that the parsed restored state fully reproduces all operational characteristics before the training task was interrupted.

[0048] Furthermore, after completing the deserialization and parsing of the state packet and obtaining the corresponding restored state, the collaborative secure sealing module is responsible for loading the restored state onto the graphics processor. This module will write various types of data in the restored state into the corresponding GPU memory area in sequence according to the storage layout requirements of the GPU memory. At the same time, it restores the execution stack configuration of the deep learning framework and the context state of the Compute Unified Device Architecture (CUDA), reconstructing the complete running state of the training task at the secure freeze point, ensuring that this state is completely consistent with the hardware, framework, and data state when the training task was interrupted. After recovery, the core scheduling and seamless recovery process sends a continuation instruction to the deep learning framework's execution engine, releasing the training task from the waiting state at the safe freeze point. The framework's execution engine will resume the subsequent training process from the interruption point between the end of backpropagation and the update of optimizer parameters. It will adjust the learning rate according to the dynamic learning rate curve saved in the restored state, perform parameter updates based on the optimizer momentum state, and conduct subsequent sample sampling and calculations in conjunction with the random number generator state. Throughout the process, the training task's loss function curve and parameter update trajectory are strictly aligned with those before the interruption, without any curve jitter or numerical offset issues. This achieves lossless continuation of the training task from the safe freeze point, and the entire recovery and continuation process is completely transparent to the user. The user does not need to perform any manual intervention, and the training task can continue to run normally.

[0049] Optionally, the method also includes: real-time collection of operational monitoring metrics for the training task, wherein the monitoring metrics include GPU utilization, GPU core temperature, memory allocator fragmentation rate, and computing power contention latency; preset multi-level response thresholds and corresponding graded response levels; comparing the real-time collected monitoring metrics with the preset multi-level response thresholds to determine the current graded response level; and executing the corresponding graded response action according to the graded response level to achieve graded response monitoring of the training task.

[0050] Specifically, during the execution of training tasks, the real-time monitoring module continuously collects operational monitoring metrics. These metrics include GPU utilization, GPU core temperature, memory allocator fragmentation rate, and computational contention latency. GPU utilization reflects the workload of the GPU cores in model computation per unit time, serving as a basis for determining whether computing resources are saturated. GPU core temperature monitors the real-time heat dissipation status of the GPU hardware, directly affecting hardware stability and lifespan. Memory allocator fragmentation rate calculates the percentage of unused and non-contiguous free space in GPU memory, reflecting the degree of memory fragmentation and allocation efficiency. Computational contention latency records the time spent queuing when training tasks request GPU computing resources, reflecting resource competition in multi-task parallel scenarios. All collected metrics are transmitted to the metric analysis module in real time to ensure data timeliness and accuracy. After the real-time collection of monitoring indicators is completed, the system will call the preset multi-level response threshold configuration module. The module is pre-set with multi-level response thresholds corresponding to the above monitoring indicators, and at the same time, it is set with corresponding three-level response levels. The three-level response levels are: Level 1 lossless adjustment, Level 2 selective suspension, and Level 3 emergency circuit breaker. The multi-level response thresholds of each monitoring indicator are set around the GPU hardware safe operation boundary and the resource requirements for normal execution of training tasks.

[0051] Optionally, corresponding graded response actions are executed according to the graded response level, including: when the graded response level is Grade 1, a Grade 1 lossless adjustment action is executed; when the graded response level is Grade 2, a Grade 2 preferential suspension action is executed; when the graded response level is Grade 3, a Grade 3 emergency circuit breaker action is executed.

[0052] Furthermore, the training-side tiered circuit breaker monitoring module compares real-time collected GPU utilization, GPU core temperature, memory allocator fragmentation rate, and computational contention latency with preset multi-level response thresholds. The comparison process is based on the actual operating values ​​of each indicator; if any indicator reaches the threshold standard corresponding to a certain response level, the current tiered response level is determined according to the rules. Finally, the module executes the corresponding tiered response action based on the determined current tiered response level, realizing tiered response monitoring of the training task. For example, if the system determines it to be a level 1 response, a level 1 lossless adjustment action is executed, reducing GPU load by dynamically reducing the training batch size, alleviating resource pressure without interrupting the training task. If it is determined to be a level 2 response, a level 2 selective suspension action is executed, freezing non-core tasks in the training task, and simultaneously performing a state sealing operation on the frozen tasks using the collaborative security sealing module to save the task's running state. If the response level is determined to be Level 3, a Level 3 emergency circuit breaker action will be executed. If an error-correcting code (ECC) hardware error or a deterioration of the computing power environment is detected, a full checkpoint operation will be forcibly executed. At the same time, all computing power resources used by the training task will be cleared, and only the inference resident service will be retained to avoid system crashes caused by hardware overheating, memory overflow, or other issues.

[0053] The technical solution of this invention, by determining the resource conflict situation of the inference task, can judge the resource allocation status and provide an accurate decision basis for whether to perform preemption operations. By intercepting the execution right and performing atomic state acquisition at a specific computational semantic node (safe freeze point) after the backpropagation ends and before the optimizer parameters are updated, the hook mechanism embedded in the underlying framework is used to fundamentally ensure that the acquired model weights and gradient data are strictly matched numerically and logically, avoiding the problem of weight and gradient state mismatch caused by container-level random pauses. At the same time, the state packet is forced to include the internal state of the micro-framework, such as the optimizer momentum state and the factors of mixed precision training. Combined with the physical isolation of asynchronous DMA transfer and the reserved memory area, a closed loop for fast switching of underlying memory is constructed. After inference, by eliminating redundant verification steps such as hash verification and tensor shape checking, the logic from the end of inference to the restart of training is significantly shortened, and the resource switching latency is controlled at the second level. This solution achieves a micro-level qualitative change from "avoiding training from scratch" to "zero jitter and strict alignment of training loss curves," ensuring high real-time system response and continuity of training trajectory. It completely solves the technical defects of existing container-level scheduling solutions, such as coarse-grained state preservation and inability to achieve truly lossless retransmission.

[0054] Example 2 Figure 2This is a flowchart of a task scheduling method provided in Embodiment 2 of the present invention. This embodiment adds a specific process to Embodiment 1, whereby a preemptive command is used to pause the training task at a safe freeze point and collect a state snapshot to generate a state packet. The specific content of steps S210 and S260-S270 is largely the same as steps S110 and S140-S150 in Embodiment 1, and therefore will not be repeated in this embodiment. Figure 2 As shown, the method includes: S210. When a new reasoning task arrives, determine the resource conflict situation of the reasoning task.

[0055] Optionally, the method further includes: obtaining a task request; determining the task type based on the calling interface of the task request, wherein the task type includes inference tasks and training tasks; defining attributes for tasks of different task types and recording the task attribute information of each task to form a task attribute table, wherein the task attribute information includes priority, interruptibility, peak video memory, expected computing power ratio and model family to which it belongs, with inference tasks being high priority and uninterruptible, and training tasks being low priority and safe to suspend.

[0056] Optionally, determining the resource conflict status of the inference task includes: obtaining the peak video memory and expected computing power ratio of the inference task based on the task attribute table to determine the video memory requirement of the inference task; obtaining the available video memory of the current graphics processing unit (GPU); determining whether the available video memory meets the video memory requirement; if so, determining the resource conflict status as non-conflict; otherwise, determining the resource conflict status as conflict.

[0057] S220. When the resource conflict is a conflict, generate a preemption command.

[0058] Optionally, generating preemption instructions includes: querying the peak video memory and priority of each training task in the task attribute table; selecting the training task with the lowest priority whose peak video memory exceeds the video memory requirement as the preemption target; and generating the corresponding preemption instructions based on the preemption target.

[0059] Optionally, the method also includes: loading the weight data of the corresponding inference model into the reserved memory area according to the model family to which each inference task belongs in the task attribute table; when a new inference task of the same model family arrives, directly reusing the weight data already loaded in the reserved memory area to perform inference calculation.

[0060] S230. The training task is paused at a safe freeze point based on the preemption instruction. The safe freeze point is the time window after the backpropagation ends and before the optimizer parameters are updated.

[0061] Backpropagation is a computational step in deep learning training, used to calculate the gradients of model parameters based on the forward propagation results; it is a prerequisite for parameter updates. Optimizer parameter updates are the operations in deep learning training that adjust model weights based on the gradients.

[0062] Specifically, after the core scheduling and seamless recovery process generates and sends the preemption command, the collaborative safety sealing module receives the command and then monitors the execution flow of the target training task in real time. It no longer intervenes in the normal computational logic of the training task, but waits only until the training task reaches a specific time window before the backpropagation ends and the optimizer parameters are updated—that is, the safety freeze point. The safety freeze point is a critical node in the training iteration process where the state is stable and there is no intermediate computational data. Pausing training at this point will not cause parameter corruption or data loss, making it the optimal time to perform state acquisition. The collaborative safety sealing module will continue to wait until the training task enters this safety freeze point, ensuring the security of state acquisition.

[0063] S240. At the safe freeze point, the execution right is intercepted by the hook mechanism of the deep learning framework execution engine, and atomic state acquisition is performed to obtain a state snapshot of the training task. The state snapshot includes the model weight tensor, optimizer momentum state, gradient accumulation buffer, current training stage, dynamic learning rate curve state, mixed precision training factor and random number generator state.

[0064] The hook mechanism, a callback interception mechanism provided by the deep learning framework, allows the insertion of custom logic at a specified execution node. This logic intercepts task execution at a safe freeze point, triggering state acquisition. Execution control represents the training task's control over GPU computing resources and the framework's operational flow. Intercepting execution control pauses the original task flow and switches to state acquisition. Atomic state acquisition refers to an indivisible, one-time acquisition of all training task state information at a safe freeze point. If any step during acquisition fails, the original training state is preserved to prevent model weight corruption and ensure the acquired state snapshot is complete, valid, and free from data loss or errors. A state snapshot is the complete training context information obtained by atomic state acquisition at a safe freeze point. It is the original data used to generate the state packet, including model weight tensors, optimizer momentum state, gradient accumulation buffer, current training stage, dynamic learning rate curve state, mixed-precision training factors, and random number generator state.

[0065] The state snapshot includes the model weight tensor, optimizer momentum state, gradient accumulation buffer, current training stage, dynamic learning rate curve state, mixed-precision training factor, and random number generator state. The model weight tensor contains the parameter data of the deep learning model, stored in tensor form, and determines the model's computational output and predictive ability. The optimizer momentum state contains the momentum parameters accumulated by the optimizer during training, used to improve the stability of parameter updates and convergence speed. The gradient accumulation buffer is a cache area storing gradient data from multiple iterations, used for gradient calculation and accumulation during large-batch training. The current training stage indicates the progress of the training task, including iteration steps, rounds, and the current training phase. The dynamic learning rate curve state records the current parameters and trends of the learning rate scheduler, ensuring logical consistency in learning rate adjustments after training resumes. The mixed-precision training factor is a core parameter used in mixed-precision training to control numerical precision conversion and gradient scaling, ensuring numerical stability during training. The random number generator state is the current seed and running state of the random number generator, ensuring that the randomness of data sampling, random discarding, and other operations after training is resumed remains consistent with that before the interruption.

[0066] Specifically, when the training task successfully enters the safe freeze point, the collaborative safe sealing module intercepts the training task's execution rights over GPU computing power and framework execution flow through a hook mechanism pre-embedded in the deep learning framework's execution engine. At this time, the normal computation flow of the training task is temporarily suspended, and the system transfers execution rights to the state acquisition logic. This interception operation is matched with the execution node of the training task, preventing acquisition anomalies caused by node offset. After the execution rights are intercepted, the collaborative safe sealing module immediately performs an atomic state acquisition operation. This acquisition operation is an indivisible and complete transaction, and its execution will not be interrupted by any other scheduling instructions or computation tasks, ensuring that the acquired state data is complete, consistent, and free of intermediate state deviations. The acquired content is a snapshot of the training task's state at the safe freeze point.

[0067] S250, Generate state packages based on state snapshots.

[0068] Optionally, generating a state packet based on a state snapshot includes: determining the generation type of the state snapshot, wherein the generation type includes periodic strong refresh and regular interruption; when the generation type is periodic strong refresh, directly serializing and encapsulating the state snapshot to generate the state packet for the training task; when the generation type is regular interruption, obtaining the preceding snapshot, extracting incremental data based on the preceding snapshot and the state snapshot, and serializing and encapsulating the incremental data into the state packet for the training task.

[0069] Specifically, after the collaborative security sealing module completes the atomic state acquisition to obtain a state snapshot of the training task, it synchronizes the snapshot data to the chained state snapshot management mechanism. This mechanism first performs the operation to determine the state snapshot generation type, which includes periodic strong refresh and regular interruption. When determining the generation type, it combines the system's scheduling trigger scenario. If the acquisition of this state snapshot is a routine state saving triggered by a system-preset periodic time node or iteration step, it is determined to be a periodic strong refresh type. If the acquisition of this state snapshot is triggered by a regular interruption of the training task caused by a preemption instruction triggered by a high-priority inference task, it is determined to be a regular interruption type.

[0070] For example, when the chain-based state snapshot management mechanism determines that the generation type is periodic strong refresh, it will directly instruct the collaborative security sealing module to perform a serialization and encapsulation operation on the complete state snapshot. This encapsulation operation will serialize all the full state information contained in the state snapshot, such as the model weight tensor, optimizer momentum state, gradient accumulation buffer, and current training stage, according to the system's preset unified data format, and integrate various scattered state data into a self-contained full data structure, finally generating a full state package for the corresponding training task. This state package serves as the baseline snapshot for subsequent training recovery, containing all the on-site information required for training recovery, and can be used independently for lossless continuation of training tasks. When the chained state snapshot management mechanism determines that the generation type is a regular interruption type, it first retrieves the latest previous snapshot corresponding to the training task from its own maintained chained snapshot structure. This previous snapshot is either the previously generated full state packet or the latest incremental state packet, which is the complete state baseline of the previous node of the training task. Subsequently, the mechanism compares the previous snapshot with the state snapshot obtained in this acquisition. Through parameter-by-parameter and tensor-by-tensor difference analysis, it extracts incremental data such as parameter changes and current iteration step update information between the two snapshots. Only the content that has changed in the state snapshot compared to the previous snapshot is retained. Then, the mechanism instructs the collaborative security sealing module to serialize and encapsulate the extracted incremental data to generate the incremental state packet corresponding to the training task. This state packet only contains the change information of the training task state. Relying on the chained snapshot structure, it can work with the previous snapshot to realize the complete restoration of the training scene, effectively reducing the storage overhead of the state packet and adapting to high-frequency task switching scenarios.

[0071] It's important to note that when the generation type is periodic strong refresh, the system generates a complete full snapshot, including all training information such as model weights, optimizer state, gradient buffer, learning rate, and random number seed. This is encapsulated into a full state packet, which can be used directly during restoration. When the generation type is regular interruption, the system does not save all information, only incremental data such as parameters and iteration steps that have changed compared to the previous full snapshot. This incremental data is encapsulated into an incremental state packet. During restoration, the incremental state needs to be merged with the previous full state to reconstruct the complete training scenario.

[0072] S260: Transfer the state packet to the host memory via asynchronous direct memory access (DMA) and release the GPU memory occupied by the training task.

[0073] S270. Execute the inference task in the pre-allocated video memory reserved area for the inference task, and after the inference task is completed, directly restore the training task to the safe freeze point based on the state packet to continue execution.

[0074] Optionally, restoring the training task to a safe freeze point based on the state packet includes: deserializing and parsing the state packet to obtain the restored state of the training task, wherein the restored state is a full state or an incremental state; loading the restored state into the graphics processor, restoring the running state of the training task to the safe freeze point, and controlling the training task to continue execution from the safe freeze point.

[0075] Optionally, the method also includes: real-time collection of operational monitoring metrics for the training task, wherein the monitoring metrics include GPU utilization, GPU core temperature, memory allocator fragmentation rate, and computing power contention latency; preset multi-level response thresholds and corresponding graded response levels; comparing the real-time collected monitoring metrics with the preset multi-level response thresholds to determine the current graded response level; and executing the corresponding graded response action according to the graded response level to achieve graded response monitoring of the training task.

[0076] Optionally, corresponding graded response actions are executed according to the graded response level, including: when the graded response level is Grade 1, a Grade 1 lossless adjustment action is executed; when the graded response level is Grade 2, a Grade 2 preferential suspension action is executed; when the graded response level is Grade 3, a Grade 3 emergency circuit breaker action is executed.

[0077] Specific application scenarios: Figure 3 This is a schematic diagram of the hierarchical architecture of a task scheduling system provided in Embodiment 2 of the present invention. Figure 3The system presents a complete system design for collaborative scheduling of training and inference tasks based on GPU computing resources. Through a four-layer progressive structure, it achieves closed-loop operation of task awareness, resource management, dynamic scheduling and state protection, ensuring efficient utilization of GPU resources and lossless switching of training tasks in multi-task scenarios.

[0078] Specifically, the top layer is the task awareness and definition layer, which receives inference tasks to be executed. Through the task attribute awareness module, it extracts full-dimensional features of the tasks, ultimately generating a standardized task attribute table. This table labels attributes such as peak memory requirements, estimated computing power ratio, and model family ID, providing a task profile for subsequent resource allocation and scheduling decisions. This ensures that the scheduling logic accurately matches the resource demands of different tasks. The second layer is the scheduling core and resource management layer, which is the resource control center of the system. This layer consists of two modules: first, a tiered circuit breaker monitoring module, which continuously monitors operating indicators such as GPU temperature, utilization, memory fragmentation rate, and computing power contention latency to achieve full-cycle monitoring of the training task's running status, providing a basis for tiered response and task preemption decisions; second, a memory management module, responsible for the partitioning management of GPU memory, dividing the memory into a dynamically scalable space for training tasks and a statically anchored area reserved for inference tasks. This ensures the stable operation of inference tasks while providing elastic resources for training tasks, achieving resource isolation and dynamic adaptation between the two types of tasks. The third layer is the dynamic scheduling logic axis (horizontal time / event flow), which is the system's task scheduling execution link, connecting the complete scheduling process along the time and event flow. First, conflict detection identifies resource conflicts between training and inference tasks, triggering preemption logic. Then, a state sealing operation is performed to completely save the running state of the training task at the safe freeze point. Next, resource scheduling reallocates GPU computing power and memory resources, releasing the dynamic scaling space resources of the training task to the inference task. Then, a task switching operation is performed to start the high-priority inference task. Finally, after the inference task completes, a state recovery operation is performed to seamlessly restore the training task from its sealed state, achieving seamless task continuation. The bottom layer is the storage protection layer, which is the security guarantee system for the training task state. This layer adopts a chained storage structure of full snapshots and incremental snapshots. Based on the full snapshot generated by periodic strong refresh, only incremental snapshots are generated in subsequent normal interruption scenarios. The chained structure of the first incremental snapshot, the second incremental snapshot, ... up to the Nth incremental snapshot through the full snapshot ensures the complete traceability of the training state, reduces the space overhead of state storage, provides reliable data support for lossless recovery of training tasks, and realizes efficient storage and fast reading of state data.

[0079] The technical solution of this invention, by determining the type of state snapshot generation, can adapt to the storage requirements of different interruption scenarios; during periodic strong refresh, it directly serializes and encapsulates the full state snapshot to generate a state packet, which can retain complete training context information as a recovery benchmark; during regular interruptions, it extracts incremental data and serializes and encapsulates it to generate a state packet, which can reduce the storage overhead of the state packet and improve data transmission and storage efficiency. By loading the restored state onto the graphics processor and restoring it to the safe freeze point running context, the running state before the training interruption can be completely reproduced, ensuring the accuracy of recovery.

[0080] Example 3 Figure 4 This is a schematic diagram of a task scheduling device provided in Embodiment 3 of the present invention. Figure 4 As shown, the device includes: a resource conflict determination module 310, used to determine the resource conflict status of a reasoning task when a new reasoning task arrives; The video memory release and state packet generation module 320 is used to generate a preemption instruction when the resource conflict is a conflict, and to pause the training task at a safe freeze point based on the preemption instruction. The safe freeze point is a time window after the end of backpropagation and before the optimizer parameters are updated. At the safe freeze point, the execution right is intercepted through a hook mechanism embedded in the deep learning framework execution engine, and atomic state acquisition is performed to generate a state packet. The state packet includes at least the model weight tensor, the optimizer momentum state, and the mixed precision training factor. The state packet is transferred to the host memory through asynchronous direct memory access (DMA), and the video memory occupied by the training task is released. The training task recovery module 330 is used to execute the inference task in the pre-allocated video memory reserved area for the inference task, and after the inference task is completed, directly restore the training task to the safe freeze point based on the state packet to continue execution.

[0081] Optionally, the device further includes a task attribute table generation module, used to: obtain task requests; determine the task type based on the calling interface of the task request, wherein the task type includes inference tasks and training tasks; define attributes for tasks of different task types and record the task attribute information of each task to form a task attribute table, wherein the task attribute information includes priority, interruptibility, peak video memory, expected computing power ratio and model family to which it belongs, with inference tasks being high priority and uninterruptible, and training tasks being low priority and safe to suspend.

[0082] Optionally, the resource conflict determination module 310 is specifically used to: obtain the peak video memory and expected computing power ratio of the inference task based on the task attribute table to determine the video memory requirement of the inference task; obtain the available video memory of the current graphics processor GPU; determine whether the available video memory meets the video memory requirement; if so, determine that the resource conflict is not a conflict; otherwise, determine that the resource conflict is a conflict.

[0083] Optionally, the memory release and state packet generation module 320 is specifically used to: intercept execution rights at the safe freeze point through the hook mechanism of the deep learning framework execution engine, perform atomic state acquisition to obtain a state snapshot of the training task, wherein the state snapshot includes the model weight tensor, optimizer momentum state, gradient accumulation buffer, current training stage, dynamic learning rate curve state, mixed precision training factor and random number generator state; and generate a state packet based on the state snapshot.

[0084] Optionally, the state packet generation unit is specifically used to: determine the generation type of the state snapshot, wherein the generation type includes periodic strong refresh and regular interruption; when the generation type is periodic strong refresh, directly serialize and encapsulate the state snapshot to generate the state packet of the training task; when the generation type is regular interruption, obtain the previous snapshot, extract incremental data based on the previous snapshot and the state snapshot, and serialize and encapsulate the incremental data into the state packet of the training task.

[0085] Optionally, the training task recovery module 330 is specifically used for: deserializing and parsing the state packet to obtain the restored state of the training task, wherein the restored state is a full state or an incremental state; loading the restored state into the graphics processor, restoring the training task to the running state of the safe freeze point, and controlling the training task to continue execution from the safe freeze point.

[0086] Optionally, the device also includes: a training task monitoring module, used to: collect real-time monitoring indicators of training task operation, including GPU utilization, GPU core temperature, memory allocator fragmentation rate, and computing power contention latency; preset multi-level response thresholds and corresponding graded response levels; compare the real-time collected monitoring indicators with the preset multi-level response thresholds to determine the current graded response level; and execute corresponding graded response actions according to the graded response level to realize graded response monitoring of training tasks.

[0087] Optionally, the memory release and status packet generation module 320 specifically includes: a preemption instruction generation unit, used to: query the peak memory and priority of each training task in the task attribute table; select the training task with peak memory exceeding memory requirements and the lowest priority as the preemption target; and generate the corresponding preemption instruction based on the preemption target.

[0088] Optionally, the device further includes a weight data reuse module, used to: load the weight data of the corresponding inference model into the reserved area of ​​video memory according to the model family to which each inference task belongs in the task attribute table; when a new inference task of the same model family arrives, directly reuse the weight data already loaded in the reserved area of ​​video memory to perform inference calculation.

[0089] The technical solution of this invention, by determining the resource conflict situation of the inference task, can judge the resource allocation status and provide an accurate decision basis for whether to perform preemption operations. By intercepting the execution right and performing atomic state acquisition at a specific computational semantic node (safe freeze point) after the backpropagation ends and before the optimizer parameters are updated, the hook mechanism embedded in the underlying framework is used to fundamentally ensure that the acquired model weights and gradient data are strictly matched numerically and logically, avoiding the problem of weight and gradient state mismatch caused by container-level random pauses. At the same time, the state packet is forced to include the internal state of the micro-framework, such as the optimizer momentum state and the factors of mixed precision training. Combined with the physical isolation of asynchronous DMA transfer and the reserved memory area, a closed loop for fast switching of underlying memory is constructed. After inference, by eliminating redundant verification steps such as hash verification and tensor shape checking, the logic from the end of inference to the restart of training is significantly shortened, and the resource switching latency is controlled at the second level. This solution achieves a micro-level qualitative change from "avoiding training from scratch" to "zero jitter and strict alignment of training loss curves," ensuring high real-time system response and continuity of training trajectory. It completely solves the technical defects of existing container-level scheduling solutions, such as coarse-grained state preservation and inability to achieve truly lossless retransmission.

[0090] The task scheduling device provided in this embodiment of the invention can execute a task scheduling method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0091] Example 4 Figure 5 A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0092] like Figure 5As shown, the electronic device 10 includes at least one processor 11 and a memory, such as ROM 12, RAM 13, etc., communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded into the RAM 13 from the storage unit 18. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An I / O interface 15 is also connected to the bus 14. The ROM 12 is a read-only memory, the RAM 13 is a random access memory, and the I / O interface 15 is an input / output interface.

[0093] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0094] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as a task scheduling method.

[0095] In some embodiments, a task scheduling method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the task scheduling method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform a task scheduling method by any other suitable means (e.g., by means of firmware).

[0096] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A task scheduling method, characterized in that, include: When a new inference task arrives, determine the resource conflict status of the inference task; When the resource conflict is a conflict, a preemption instruction is generated, and the training task is paused at a safe freeze point based on the preemption instruction. The safe freeze point is a time window after the backpropagation ends and before the optimizer parameters are updated. At the safe freeze point, execution rights are intercepted through a hook mechanism embedded in the deep learning framework execution engine, and atomic state acquisition is performed to generate a state package, which includes at least the model weight tensor, optimizer momentum state, and mixed precision training factors. The state packet is transferred to the host memory via asynchronous direct memory access (DMA), and the GPU memory occupied by the training task is released. The inference task is executed in the pre-allocated memory reserved area for the inference task, and after the inference task is completed, the training task is directly restored to the safe freeze point based on the state packet to continue execution.

2. The method according to claim 1, characterized in that, The method further includes: Obtain a task request, and determine the task type based on the calling interface of the task request, wherein the task type includes inference tasks and training tasks; Attributes are defined for different task types and task attribute information is recorded to form a task attribute table. The task attribute information includes priority, interruptibility, peak video memory, expected computing power ratio and model family. Inference tasks are high priority and uninterruptible, while training tasks are low priority and safe to suspend.

3. The method according to claim 2, characterized in that, Determining the resource conflict situation of the inference task includes: Based on the task attribute table, the peak video memory and expected computing power ratio of the inference task are obtained to determine the video memory requirements of the inference task. Get the available video memory of the current graphics processing unit (GPU); Determine whether the available video memory meets the video memory requirements; if so, determine that the resource conflict is not a conflict. Otherwise, the resource conflict situation is determined to be a conflict.

4. The method according to claim 1, characterized in that, The process of intercepting execution rights at the secure freeze point through a hook mechanism embedded in the deep learning framework's execution engine, and performing atomic state acquisition to generate a state packet, includes: At the safe freeze point, execution rights are intercepted through a hook mechanism embedded in the deep learning framework execution engine to perform atomic state acquisition to obtain a state snapshot of the training task. The state snapshot includes the model weight tensor, optimizer momentum state, gradient accumulation buffer, current training stage, dynamic learning rate curve state, mixed precision training factor and random number generator state. A state package is generated based on the state snapshot.

5. The method according to claim 4, characterized in that, The generation of the state packet based on the state snapshot includes: Determine the generation type of the state snapshot, wherein the generation type includes periodic strong refresh and regular interruption; When the generation type is a periodic strong refresh type, the state snapshot is directly serialized and encapsulated to generate the state package of the training task; When the generation type is a regular interruption type, a preceding snapshot is obtained, incremental data is extracted based on the preceding snapshot and the state snapshot, and the incremental data is serialized and encapsulated into a state packet for the training task.

6. The method according to any one of claims 1, 4-5, characterized in that, The step of restoring the training task to a safe freeze point based on the state packet and continuing execution includes: The state packet is deserialized and parsed to obtain the restored state of the training task, wherein the restored state is either the full state or the incremental state. The restored state is loaded into the graphics processor, the training task is restored to the running state of the safe freeze point, and the training task is controlled to continue execution from the safe freeze point.

7. The method according to claim 1, characterized in that, The method further includes: Real-time collection of operational monitoring metrics for training tasks, including GPU utilization, GPU core temperature, memory allocator fragmentation rate, and computing power contention latency. Preset multi-level response thresholds and corresponding graded response levels; The real-time collected monitoring indicators are compared with preset multi-level response thresholds to determine the current graded response level; Execute the corresponding graded response action according to the graded response level to realize graded response monitoring of the training task.

8. The method according to claim 3, characterized in that, The generation of the preemption instruction includes: Query the peak memory and priority of each training task in the task attribute table; Select the training task whose peak video memory exceeds the video memory requirement and has the lowest priority as the preemption target; A corresponding preemption command is generated based on the preemption target.

9. The method according to claim 2, characterized in that, The method further includes: Based on the model family to which each inference task belongs in the task attribute table, the weight data of the corresponding inference model is loaded into the reserved area of ​​video memory. When a new inference task of the same model family arrives, the weight data already loaded in the reserved memory area is directly reused to perform inference calculation.

10. A task scheduling device, characterized in that, include: The resource conflict determination module is used to determine the resource conflict status of a new inference task when the task arrives. The memory release and state packet generation module is used to generate a preemption instruction when the resource conflict is detected, and to pause the training task at a safe freeze point based on the preemption instruction. The safe freeze point is a time window after backpropagation ends and before the optimizer parameters are updated. At the safe freeze point, execution rights are intercepted through a hook mechanism embedded in the deep learning framework execution engine, and atomic state acquisition is performed to generate a state packet. The state packet includes at least the model weight tensor, the optimizer momentum state, and the mixed precision training factor. The state packet is transferred to the host memory via asynchronous direct memory access (DMA), and the memory occupied by the training task is released. The training task recovery module is used to execute the inference task in the pre-allocated video memory reserved area for the inference task, and after the inference task is completed, directly restore the training task to the safe freeze point based on the state packet to continue execution.

Images