A large model training and inference hybrid scheduling method, device, equipment and medium

Abstract

The application provides a large model training and inference hybrid scheduling method and device, equipment and medium, relating to the technical field of cloud computing, which comprises the following steps: in response to a night switching on signal, selecting a candidate stage node from a main inference cluster, stopping the candidate stage node from receiving new requests and performing local emptying; starting a guaranteed inference service; in response to the guaranteed inference service passing the readiness detection, switching the new requests to the guaranteed inference service and performing global emptying; releasing the candidate stage node as a training node to the training resource pool to perform a training task; triggering a checkpoint barrier when the training node reaches a safe training stop time, outputting and verifying the training breakpoint; in response to the verification passing, releasing the training node to restore the main inference cluster, and after the main inference cluster passes the health check, switching the new requests back to the main inference cluster, and integrating the guaranteed inference service into the main inference cluster after the guaranteed inference service is emptied. The application efficiently reuses computing power and realizes double guarantee of inference and training.

Description

Technical Field

[0001] This invention relates to the field of cloud computing technology, and more specifically, to a method, apparatus, device, and medium for large-scale training and push hybrid scheduling. Background Technology

[0002] In AI computing clusters, the hybrid deployment of online inference services and training tasks has become a common method to improve computing power utilization. During the day, inference requests are concentrated, requiring multiple nodes to process them in parallel to ensure service response speed; at night, inference requests decrease, and some nodes can be released for training, fine-tuning, or replay tasks, thereby achieving time-division multiplexing of computing resources.

[0003] Existing scheduling methods mainly include fixed partitioning of inference and training resource pools, static hard switching at fixed intervals, migration based on Kubernetes service rolling updates, preemption and recovery of training tasks relying on the checkpoint mechanism of the training framework itself, and global switching of traffic from one inference backend to another through gateway global priority switching. These solutions achieve resource reuse or switching scheduling between inference and training tasks to a certain extent, but their limitations are becoming increasingly apparent with the widespread deployment of large-model inference services. Large-model inference services are characterized by long-term GPU memory occupation, continuous streaming response, and decoupling of multi-stage computations, which are fundamentally different from ordinary web services. When existing scheduling methods release inference nodes to training tasks at night, established streaming connections may be truncated, existing requests on candidate stage nodes may be preempted by training tasks, and incomplete exit of old inference instances may cause GPU memory resource conflicts. There is a risk that the main inference cluster may not recover in a timely manner when training tasks end. Furthermore, the use of unified static switching parameters in heterogeneous clusters needs improvement in its adaptability to the actual hardware performance, such as different chip types, GPU memory capacities, model loading times, and checkpoint write bandwidth. Summary of the Invention

[0004] The present invention aims to solve at least one of the above-mentioned technical problems.

[0005] To address the aforementioned problems, this invention provides a large-scale training and push hybrid scheduling method, apparatus, equipment, and medium.

[0006] In a first aspect, the present invention provides a large-scale model training and push hybrid scheduling method, comprising: In response to the night-time switching start signal, candidate stage nodes are selected from the main inference cluster, the candidate stage nodes are stopped from receiving new requests, and the existing streaming requests are partially drained. After the candidate stage node is partially emptied and the old inference instances that have been processed are released from memory, the guaranteed inference service is started. In response to the guaranteed inference service passing the readiness detection, the new request is switched to the guaranteed inference service through the application layer traffic scheduling gateway, and the existing streaming requests of all stage nodes are globally drained. After all the aforementioned stage nodes are globally cleared, the candidate stage nodes are released as training nodes to the training resource pool to perform training tasks. The safe training stop time is calculated by working backward from the recovery deadline of the main inference cluster. When the training node reaches the safe training stop time, a checkpoint barrier is triggered, and the training breakpoint is output and verified. In response to the successful verification of the training breakpoint, the training node is released and the main inference cluster is restored. After the main inference cluster passes the health check, the new request is switched back to the main inference cluster, and the backup inference service is drained and then merged into the main inference cluster.

[0007] Optionally, after the old inference instances that have been partially cleared and processed at the candidate stage node are released from memory, the step of starting the guaranteed inference service includes: Monitor the memory usage status of the candidate stage nodes, and start the guaranteed inference service after the old inference instance exits and the memory usage is lower than the preset release threshold; And / or, the response to the guaranteed inference service passing a readiness probe, switching new requests to the guaranteed inference service via an application-layer traffic scheduling gateway, includes: The guaranteed inference service is tested for container status, model loading, and initial metaword detection. After the continuous detection passes, the guaranteed inference service is confirmed to be ready. Update the routing rules and backend availability flags of the application layer traffic scheduling gateway so that the new requests enter the backup inference service, and ensure that the global emptying process of existing streaming requests at all stage nodes is not interrupted.

[0008] Optionally, triggering a checkpoint barrier when the training node reaches the safe training stop time, and outputting and verifying the training breakpoint, includes: When the training node reaches the safe training stop time, a training stop signal is sent to the training task. When a checkpoint barrier is triggered, the training process enters the barrier procedure, saving model parameters, optimizer state, training steps, data cursor, and scheduling metadata as a temporary state file. After the temporary state file is written, an atomic renaming operation is performed to obtain the checkpoint file; An integrity check is performed on the renamed temporary state file. In response to the successful check, the global pointer of the renamed temporary state file is updated, and the training breakpoint is confirmed to have been saved successfully. The global pointer is used as an index to point to the renamed temporary state file.

[0009] Optionally, the global draining of existing streaming requests across all stage nodes also includes drain timeout degradation handling: When the emptying time reaches the preset soft emptying limit, freeze new training admissions and send an emptying timeout alarm; When the draining time reaches the preset absolute draining limit, an active intervention degradation strategy is triggered to terminate the abnormal long connection. Wherein, the upper limit of soft evacuation is less than the upper limit of absolute evacuation.

[0010] Optionally, the triggering of the proactive intervention degradation strategy to terminate the abnormal long connection includes: Save the state of abnormal long-lived connections that have not yet ended; The application layer traffic scheduling gateway returns the standard interrupt status code or retry indicator for this abnormal state to the client. Terminate the abnormal long connection and reconfirm the exit status and memory release status of the old inference instance.

[0011] Optionally, the large model training and push hybrid scheduling method further includes heterogeneous cluster differential control: Collect heterogeneous hardware characteristics of different nodes, including AI acceleration chip type, model loading path, node interconnection network and checkpoint storage path; Based on the heterogeneous hardware characteristics, the timeout threshold of the local emptying, the upper bound of the ready waiting for the guaranteed inference service, and the conservative estimate in the backward calculation are dynamically adjusted. Specifically, based on the memory capacity and driver version of the AI ​​acceleration chip type, the memory release threshold and the upper bound of the guaranteed inference service startup wait are adjusted. Based on the cache hit rate and read bandwidth of the model loading path, adjust the conservative estimates of the upper bound of the ready wait for the backup inference service and the recovery time of the main inference. Adjust the checkpoint write safety time and the safety buffer time based on the write throughput and historical failure rate of the checkpoint storage path.

[0012] Optionally, the large model training-pull hybrid scheduling method further includes exception rollback handling: If the guaranteed inference service fails the readiness test, the candidate stage node resumes receiving the new request; If the main inference cluster fails the health check or the recovery times out, the backup inference service will continue to accept the new request, and new training admissions will be frozen.

[0013] Secondly, the present invention provides a large-scale model training and push hybrid scheduling device, comprising: The initial module is used to select candidate stage nodes from the main inference cluster in response to the night switching start signal, stop the candidate stage nodes from receiving new requests, and partially drain the existing streaming requests. The startup module is used to start the backup inference service after the old inference instances that have been partially cleared and processed in the candidate stage node have released their video memory. The switching module is used to switch the new request to the guaranteed inference service through the application layer traffic scheduling gateway in response to the guaranteed inference service passing the readiness detection, and to globally drain the existing streaming requests of all stage nodes. The release module is used to release the candidate stage node as a training node to the training resource pool to perform training tasks after all the stage nodes are globally cleared. The training module is used to calculate the safe training stop time by working backward from the recovery deadline of the main inference cluster. When the training node reaches the safe training stop time, a checkpoint barrier is triggered, and the training breakpoint is output and verified. The rollback module is used to release the training node and restore the main inference cluster in response to the passing of the training breakpoint verification. After the main inference cluster passes the health check, the new request is rolled back to the main inference cluster, and the backup inference service is drained and then merged into the main inference cluster.

[0014] Thirdly, the present invention provides an electronic device, including a memory and a processor; The memory is used to store computer programs; The processor is configured to implement the large model training and push hybrid scheduling method as described in the first aspect when executing the computer program.

[0015] Fourthly, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the large-model training-pushing hybrid scheduling method as described in the first aspect.

[0016] The beneficial effects of the large-model training-inference hybrid scheduling method of this invention are as follows: It leverages the low-traffic window during nighttime inference to initiate the resource switching process, locking candidate stage nodes and blocking new request inflows, thus avoiding the continuous accumulation of node resource occupation caused by the continuous generation of new business loads during the emptying process. It only performs partial emptying on existing streaming requests for candidate stage nodes, allowing other main inference nodes to continue carrying business, achieving layered business termination, and orderly completing the natural processing of existing inference sessions on nodes. It does not forcibly truncate long-text streaming inference connections for users, ensuring the integrity of online inference services. It simultaneously and gradually releases the GPU memory resources occupied by inference instances, reserving complete hardware resource space for subsequent node role switching and avoiding GPU memory contention conflicts.

[0017] Setting the complete release of GPU memory as a prerequisite for starting the backup inference service ensures that all GPU memory resources, such as model weights and KV cache, occupied by the old inference instance are reclaimed before loading the backup inference service, completely eliminating the issues of GPU memory contention and memory overflow between old and new inference instances. The backup inference service independently handles off-peak traffic at night, continuously providing backup inference computing power during the complete window of switching from candidate nodes to training nodes, forming uninterrupted business disaster recovery capabilities, eliminating service gaps during resource switching, and ensuring the continuous and stable achievement of standards for large-scale online inference.

[0018] Using the readiness detection result of the backup inference service as a hard prerequisite for traffic switching, new requests are only scheduled after confirming that the backup service model is loaded and the business inference capability is fully normal, achieving lossless traffic migration and avoiding failures or timeouts of new requests due to the backup service not being ready. Traffic switching only applies to new requests; existing streaming connections are retained and processed continuously in the original inference backend, differentiating between new and old requests for differentiated scheduling to adapt to large-model, long-session inference scenarios. After traffic splitting is completed, all existing requests in the entire cluster are globally drained, releasing all inference instances and GPU memory resources of the main inference nodes at once. This removes resource occupation obstacles for batch recycling of nodes and their transfer to the training resource pool. The draining process is layered and orderly, with no concurrent conflicts between traffic scheduling and resource recycling operations.

[0019] With the entire cluster's inference operations completely idle, no GPU memory usage, and no online request backlog, candidate nodes are released in batches for training tasks. The GPU memory and computing power resources of these nodes can then be fully utilized for offline tasks such as large model fine-tuning and continuous training. This fully leverages idle computing power during nighttime inference to achieve tidal reuse of training and inference resources, eliminating the need for separate deployment or reservation of dedicated training computing power clusters, significantly reducing the hardware procurement and maintenance costs of AI computing power clusters. Inference and training operations utilize hardware resources in a staggered, time-sharing manner, improving the all-weather resource utilization of GPUs and other acceleration devices, and eliminating the waste of idle computing power in the nighttime inference cluster.

[0020] By using the hard constraint time for cluster recovery during peak daytime inference periods to deduce safe training stop nodes, and comprehensively reserving time for cluster expansion and recovery, checkpoint writing, and anomaly buffering, the runtime window for training tasks is precisely controlled. This prevents continuous training from occupying nodes, which could cause the main inference cluster to be unable to resume handling peak traffic on time the following day. Upon reaching the safe training stop time, a checkpoint barrier is automatically triggered, uniformly solidifying complete training metadata such as model parameters, optimizer status, training iteration steps, and data cursors, forming a consistent breakpoint file. Simultaneously, integrity checks are performed on the breakpoint file to ensure the complete retention of training progress, preventing node reclamation from interrupting training and rendering all previous iteration computing power useless, and reducing the computing power and time losses caused by repeated training.

[0021] Training nodes are only released after the breakpoints are verified to be complete and usable, ensuring that subsequent training restarts can seamlessly continue the entire iteration progress and avoiding training task failures caused by damaged or incomplete breakpoints. After node reclamation, the complete main inference cluster is rebuilt and a full-link health check is performed. Once the cluster's inference throughput and long session capacity are confirmed to have recovered to peak standards, traffic is switched back to the main cluster, ensuring stable support for high-concurrency inference services during the day. Finally, the remaining requests for the backup inference service are cleared and its resources are merged into the main inference cluster, completing the complete and reversible restoration of training and inference resource roles. During peak inference periods during the day, the original full inference computing power scale can be fully restored, with no idle or wasted hardware resources for the backup service. The entire switching process forms a closed-loop reversible scheduling logic.

[0022] This invention significantly improves the 24 / 7 utilization of expensive AI acceleration hardware such as GPUs, reducing the deployment and maintenance costs of independent training clusters. The entire resource switching logic distinguishes between new requests and existing long-running inference connections, adapting to the characteristics of large-scale model long-session inference services. There is no forced session interruption or service downtime, ensuring stable continuity of online inference services. By using a recovery deadline to determine the safe training stoppage window, coupled with checkpoint barriers and breakpoint consistency verification mechanisms, it precisely balances the nighttime training computing power supply with the business constraint of timely recovery of the inference cluster during the day. Nodes smoothly transition between inference and training roles, avoiding issues such as memory contention, training progress loss, and traffic scheduling conflicts throughout the process. It balances high availability of online inference services with the offline training computing power supply for large models, adapting to the day-night tidal load scheduling scenarios of enterprise large-scale model integrated training and inference computing power clusters. Attached Figure Description

[0023] Figure 1 This is a flowchart illustrating the large-model training-promotion hybrid scheduling method according to an embodiment of the present invention. Figure 1 ; Figure 2 This is a schematic diagram of the structure of the large-scale model training and push hybrid scheduling device according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention; Figure 4 This is a flowchart illustrating the large-model training-promotion hybrid scheduling method according to an embodiment of the present invention. Figure 2 . Detailed Implementation

[0024] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Although some embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present invention. It should be understood that the accompanying drawings and embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.

[0025] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0026] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to"; the term "based on" means "at least partially based on"; the term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments"; and the term "optionally" means "optional embodiments". Definitions of other terms will be given in the following description. It should be noted that the concepts of "first," "second," etc., mentioned in this invention are used only to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.

[0027] It should be noted that the terms "a" and "a plurality of" used in this invention are illustrative rather than restrictive. Those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0028] The names of the messages or information exchanged between the multiple devices in the embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of these messages or information.

[0029] To address the existing problems, this embodiment provides a large-scale model training and push hybrid scheduling method, apparatus, equipment, and medium.

[0030] like Figure 1 and Figure 4 As shown in the figure, an embodiment of the present invention provides a large-scale model training and push hybrid scheduling method, including: Step S1: In response to the night switch-on signal, select candidate stage nodes from the main inference cluster, stop the candidate stage nodes from receiving new requests, and partially drain the existing streaming requests.

[0031] This embodiment is based on the tidal load of large model training and inference, i.e., more inference requests during the day and fewer at night. Training, fine-tuning, or replay tasks typically require more computing power for scheduling at night. A multi-stage computation decoupling topology is adopted, which breaks down large model inference into multiple independent computation stages and deploys them on separate nodes. Alternatively, an input preprocessing / incremental generation separation architecture can be used, with each stage node independently managed. Candidate stage nodes are selected from the decoupling stage nodes of the main inference cluster during off-peak hours at night to switch computing power roles, prioritizing the clearing of existing requests and ensuring a smooth inference startup process. New requests are user inference tasks arriving after the nighttime switch and not yet having established a streaming long connection; these are routed and managed by the gateway. Existing streaming requests are inference sessions that have established long connections and are continuously outputting terms / text fragments before the switch is initiated; they are not forcibly interrupted and are allowed to complete naturally. Partial clearing restricts only candidate stage nodes from receiving new requests, waiting for all existing streaming requests within that node to be processed, while other inference stage nodes continue to handle traffic. It should be noted that in this embodiment, daytime and nighttime are not limited to specific times, but are used to represent the number of inference requests.

[0032] Specifically, a nighttime switching signal is generated when the preset nighttime training window is detected to be open, or when the real-time inference concurrency falls below a low threshold. Candidate stage nodes are selected based on the memory usage, number of active connections, hardware heterogeneity, and historical flushing time of each stage node in the cluster. A scheduling prohibition command is issued to cut off all new request scheduling paths to the candidate stage nodes. The active streaming request count of the candidate stage nodes is continuously polled, and the process continues until the selected stage node completes frame-by-frame output of existing sessions and the number of active connections is cleared, thus completing the partial flushing of the candidate stage nodes. All other stage nodes, except for the candidate stage nodes, maintain normal traffic handling capacity, without affecting current user inference access.

[0033] Step S2: After the old inference instances that have been partially cleared and processed in the candidate stage node are released from memory, start the guaranteed inference service.

[0034] In this embodiment, the old inference instance refers to the original inference service process running on the candidate stage node, which occupies GPU memory, KV cache, and hardware device handles. Memory release refers to the reclamation of GPU / NPU memory and runtime cache resources after the old inference instance exits, reducing memory usage to a preset safety threshold and maintaining it stably. The backup inference service is a lightweight inference service started on the candidate stage node after it has been cleared during the low-load period at night, taking over all new requests at night and freeing up computing power for training tasks in the main cluster.

[0035] Specifically, continuously monitor the candidate stage node process list, hardware device handles, and GPU memory usage metrics. Confirm that the old inference instance process has completely exited, the service port is closed, and memory usage remains below the minimum safe startup threshold for a specified buffer duration, thus determining that the memory has been completely released. Deploy a lightweight minimum safe inference service in the current candidate stage node container environment, load the same version of the large model weights, and complete the basic inference environment initialization.

[0036] Step S3: In response to the guaranteed inference service passing the readiness detection, the new request is switched to the guaranteed inference service through the application layer traffic scheduling gateway, and the existing streaming requests of all stage nodes are globally drained.

[0037] In this embodiment, readiness detection involves the controller performing a health check on the backup inference service, including multi-dimensional checks on container status, model loading integrity, first-word generation latency, and business request status codes. Readiness is determined when all checks are met. The application-layer traffic scheduling gateway is a Layer 7 gateway / service mesh component, capable of adjusting backend weights, service routing, and service discovery flags, and distinguishing between existing long-lived connections and new requests for independent routing. Global drain ensures that all existing streaming requests within all stages of the main inference cluster have been processed and that there are no active inference sessions.

[0038] Specifically, a multi-dimensional readiness probe is performed on the backup inference service, verifying metrics such as container running status, model loading completion markers, first metaword generation time, and standard business inference return codes. Service readiness is determined once all metrics meet the criteria. A routing update command is issued to the Layer 7 application layer traffic scheduling gateway, adjusting backend weights and service availability markers. All newly arriving requests are uniformly routed to the backup inference service. The gateway maintains the original backend binding relationships of existing streaming connections; long-lived connections established before the switch remain processed at each stage of the main inference process. A global traversal of all stage nodes in the main cluster is performed, continuously calculating the total number of active existing connections across the entire cluster, and initiating a global drain monitoring process.

[0039] Step S4: After all the stage nodes are globally empty, the candidate stage nodes are released as training nodes to the training resource pool to perform training tasks.

[0040] In this embodiment, the training resource pool uniformly collects and empties all inference nodes, and only assigns training tasks with breakpoint recovery capabilities. The recovery deadline is a hard business time constraint that the main inference cluster must complete expansion and be healthy and online before the peak inference period arrives the following morning.

[0041] Specifically, the system aggregates active connection data for all stages of the main inference process in real time. When all existing streaming requests across the cluster have been processed and there are no active inference sessions, the system determines that global resource allocation is complete. The node's inference service resource occupancy flag is removed, and the hardware resources of that candidate stage node are included in the training resource pool for unified scheduling. Only tasks supporting checkpoint breakpoint saving and state recovery logic—such as model fine-tuning, continuous training, and data replay—are allocated to this node. Offline tasks without breakpoint mechanisms are prohibited from occupying the reusable node.

[0042] Step S5: Calculate the safe training stop time by working backward from the recovery deadline of the main inference cluster. When the training node reaches the safe training stop time, trigger the checkpoint barrier, output and verify the training breakpoint.

[0043] In this embodiment, the safe training stop time is calculated backward from the recovery deadline to obtain the training termination node, reserving time for cluster recovery, breakpoint writing, and safety buffering. The checkpoint barrier is a consistent saving process that uniformly saves model weights, optimizer parameters, training steps, data cursors, and scheduling metadata after the training process receives a stop instruction. The training breakpoint is a temporary state file output by the barrier process, which, after atomic renaming and metadata updates, forms a recoverable complete training snapshot. Breakpoint verification involves reading the breakpoint file, shard data, and metadata pointers to verify file integrity and write consistency; only after successful verification can the computing nodes be released. The main inference cluster health check is performed after cluster expansion, conducting full-link probing of containers, models, and inference services on nodes at each stage; if all probing passes, the cluster is considered ready.

[0044] Specifically, the system reads the main inference recovery deadline from the business configuration. Based on historical observation data, it takes three parameters: the conservative upper bound of the main cluster expansion recovery time, the safe time for writing training breakpoints, and the safe buffer duration for scheduling. The safe training stop time is then calculated by working backward from the recovery deadline. When the system clock reaches the safe training stop time during training, a training stop signal is sent to all nodes running training tasks, triggering a global checkpoint barrier. The training process synchronously writes a temporary breakpoint file containing model parameters, optimizer status, training steps, data cursors, and scheduling metadata. After the file is sharded and written, an atomic renaming operation is performed to update the global breakpoint metadata pointer. Finally, the complete breakpoint file, shard checksum, and metadata version are read to complete consistency verification.

[0045] Step S6: In response to the successful verification of the training breakpoint, release the training node and restore the main inference cluster. After the main inference cluster passes the health check, switch the new request back to the main inference cluster and empty the backup inference service before merging it into the main inference cluster.

[0046] Specifically, if the training breakpoint integrity and consistency checks pass, the controller releases the computing resources of all training nodes, initiates the main inference cluster expansion and reconstruction process, and simultaneously deploys the pre-filled + decoded decoupled inference topology. A full-link health check is performed on the expanded main inference cluster; if all node model loading and inference business probing at each stage pass, the cluster is deemed ready. A gateway routing back-switch command is issued, rerouting all new inference requests back to the main inference cluster. The system continuously monitors the existing streaming requests within the backup inference service, waiting for all sessions to naturally end and be cleared. After clearing, the backup inference service is shut down, and the node is reintegrated into the main inference cluster's standard stage node pool, restoring the full inference computing power scale during the daytime peak.

[0047] This invention leverages the low-traffic window during nighttime to initiate the resource switching process, locking candidate stage nodes and blocking new requests from entering, thus preventing the continuous accumulation of node resource consumption caused by the continuous generation of new business load during the emptying process. Only existing streaming requests to candidate stage nodes are partially emptyed, allowing other main inference nodes to continue carrying business, achieving layered business termination and orderly completion of existing inference sessions on nodes. It does not forcibly truncate long-text streaming inference connections for users, ensuring the integrity of online inference services. Simultaneously, the GPU memory resources occupied by inference instances are gradually released, reserving complete hardware resource space for subsequent node role switching and avoiding GPU memory contention conflicts.

[0048] Setting the complete release of GPU memory as a prerequisite for starting the backup inference service ensures that all GPU memory resources, such as model weights and KV cache, occupied by the old inference instance are reclaimed before loading the backup inference service, completely eliminating the issues of GPU memory contention and memory overflow between old and new inference instances. The backup inference service independently handles off-peak traffic at night, continuously providing backup inference computing power during the complete window of switching from candidate nodes to training nodes, forming uninterrupted business disaster recovery capabilities, eliminating service gaps during resource switching, and ensuring the continuous and stable achievement of standards for large-scale online inference.

[0049] Using the readiness detection result of the backup inference service as a hard prerequisite for traffic switching, new requests are only scheduled after confirming that the backup service model is loaded and the business inference capability is fully normal, achieving lossless traffic migration and avoiding failures or timeouts of new requests due to the backup service not being ready. Traffic switching only applies to new requests; existing streaming connections are retained and processed continuously in the original inference backend, differentiating between new and old requests for differentiated scheduling to adapt to large-model, long-session inference scenarios. After traffic splitting is completed, all existing requests in the entire cluster are globally drained, releasing all inference instances and GPU memory resources of the main inference nodes at once. This removes resource occupation obstacles for batch recycling of nodes and their transfer to the training resource pool. The draining process is layered and orderly, with no concurrent conflicts between traffic scheduling and resource recycling operations.

[0050] With the entire cluster's inference operations completely idle, no GPU memory usage, and no online request backlog, candidate nodes are released in batches for training tasks. The GPU memory and computing power resources of these nodes can then be fully utilized for offline tasks such as large model fine-tuning and continuous training. This fully leverages idle computing power during nighttime inference to achieve tidal reuse of training and inference resources, eliminating the need for separate deployment or reservation of dedicated training computing power clusters, significantly reducing the hardware procurement and maintenance costs of AI computing power clusters. Inference and training operations utilize hardware resources in a staggered, time-sharing manner, improving the all-weather resource utilization of GPUs and other acceleration devices, and eliminating the waste of idle computing power in the nighttime inference cluster.

[0051] By using the hard constraint time for cluster recovery during peak daytime inference periods to deduce safe training stop nodes, and comprehensively reserving time for cluster expansion and recovery, checkpoint writing, and anomaly buffering, the runtime window for training tasks is precisely controlled. This prevents continuous training from occupying nodes, which could cause the main inference cluster to be unable to resume handling peak traffic on time the following day. Upon reaching the safe training stop time, a checkpoint barrier is automatically triggered, uniformly solidifying complete training metadata such as model parameters, optimizer status, training iteration steps, and data cursors, forming a consistent breakpoint file. Simultaneously, integrity checks are performed on the breakpoint file to ensure the complete retention of training progress, preventing node reclamation from interrupting training and rendering all previous iteration computing power useless, and reducing the computing power and time losses caused by repeated training.

[0052] Training nodes are only released after the breakpoints are verified to be complete and usable, ensuring that subsequent training restarts can seamlessly continue the entire iteration progress and avoiding training task failures caused by damaged or incomplete breakpoints. After node reclamation, the complete main inference cluster is rebuilt and a full-link health check is performed. Once the cluster's inference throughput and long session capacity are confirmed to have recovered to peak standards, traffic is switched back to the main cluster, ensuring stable support for high-concurrency inference services during the day. Finally, the remaining requests for the backup inference service are cleared and its resources are merged into the main inference cluster, completing the complete and reversible restoration of training and inference resource roles. During peak inference periods during the day, the original full inference computing power scale can be fully restored, with no idle or wasted hardware resources for the backup service. The entire switching process forms a closed-loop reversible scheduling logic.

[0053] This invention significantly improves the 24 / 7 utilization of expensive AI acceleration hardware such as GPUs, reducing the deployment and maintenance costs of independent training clusters. The entire resource switching logic distinguishes between new requests and existing long-running inference connections, adapting to the characteristics of large-scale model long-session inference services. There is no forced session interruption or service downtime, ensuring stable continuity of online inference services. By using a recovery deadline to determine the safe training stoppage window, coupled with checkpoint barriers and breakpoint consistency verification mechanisms, it precisely balances the nighttime training computing power supply with the business constraint of timely recovery of the inference cluster during the day. Nodes smoothly transition between inference and training roles, avoiding issues such as memory contention, training progress loss, and traffic scheduling conflicts throughout the process. It balances high availability of online inference services with the offline training computing power supply for large models, adapting to the day-night tidal load scheduling scenarios of enterprise large-scale model integrated training and inference computing power clusters.

[0054] Optionally, after the old inference instances that have been partially cleared and processed at the candidate stage node are released from memory, the step of starting the guaranteed inference service includes: Monitor the memory usage status of the candidate stage nodes, and start the guaranteed inference service after the old inference instance exits and the memory usage is lower than the preset release threshold.

[0055] Specifically, the hardware monitoring interface of the candidate stage nodes is continuously polled to collect four types of monitoring data in real time: memory usage status, process list, inference service port, and hardware device handles. Two judgment logics are executed simultaneously: First, it verifies whether the old inference instance has completely exited, and the judgment criteria are that the inference process is destroyed, the service port is inaccessible, and all GPU / NPU device handles are released; second, it compares the real-time memory usage value with the preset release threshold corresponding to the hardware, requiring that the memory usage remain below the threshold and maintain the configured buffer duration to filter out instantaneous memory fluctuations. The preset release threshold is a memory safety threshold configured based on the characteristics of heterogeneous hardware, and is set separately according to the memory capacity of different chips and model loading overhead; the judgment requires that the memory usage be below the threshold and maintain the preset buffer duration to exclude the interference of instantaneous memory fluctuations. Only when both conditions are met simultaneously—the old inference instance has completely exited and the memory usage is stably below the preset release threshold—is the backup inference service deployed and started in the container environment of the candidate stage node. If any condition is not met, the hardware and process metrics will be continuously monitored in a loop, and the startup of the backup inference service will be delayed to avoid memory contention and GPU OOM crashes caused by the coexistence of new and old inference processes.

[0056] Optionally, the step of switching new requests to the guaranteed inference service via the application layer traffic scheduling gateway in response to the guaranteed inference service through readiness detection includes: The guaranteed inference service is tested for container status, model loading, and initial metaword detection. After the continuous detection passes, the guaranteed inference service is confirmed to be ready. Update the routing rules and backend availability flags of the application layer traffic scheduling gateway so that the new requests enter the backup inference service, and ensure that the global emptying process of existing streaming requests at all stage nodes is not interrupted.

[0057] Specifically, after the guaranteed inference service starts, the controller automatically executes three rounds of continuous joint probing, with probing dimensions including container status, model loading, and first-term meta-test. Container status probing verifies the container's running status, confirming no crashes, restarts, or resource allocation failures. Model loading probing verifies the complete loading of model weights, quantization configurations, and inference runtime dependency files, ensuring no read interruptions or file corruption. First-term meta-test probing issues standardized test inference requests, collecting first-term meta-test generation latency and business response status codes to verify inference service availability. Only when all rounds of continuous probing pass can the guaranteed inference service be considered officially ready; if any round of probing fails, the service is deemed unavailable, traffic switching is not performed, and the training node release process is simultaneously postponed. After the guaranteed service is ready, the controller issues a configuration update command to the application-layer traffic scheduling gateway, simultaneously completing two configuration modifications: first, updating the gateway routing rules to switch the distribution path of all newly arriving requests to the guaranteed inference service; second, updating the backend availability flag, marking the guaranteed inference service as online and the original main inference backend as compatible only with existing connections. Throughout the gateway traffic switching process, the controller does not pause or interrupt the global drain monitoring logic for all existing streaming requests across the entire cluster. It continuously traverses all stages of the main inference process, summarizing the number of active existing connections on each node in real time, and driving the global drain process throughout. Furthermore, traffic differentiation constraints are set up to switch only newly added requests to the backup service. Existing streaming requests established before the switch maintain their binding relationship with the original main inference backend, without forced migration or interruption of inference output, preventing truncation of user text responses.

[0058] This invention monitors node memory usage in real time, only activating the backup service when an old instance exits or memory usage falls below a threshold. This avoids memory contention between the old and new inference services, ensuring the stable operation of the backup service. Service status is determined through multi-layered continuous probing using containers, models, and initial metawords, ensuring the backup inference service has complete business processing capabilities and preventing errors from occurring with new requests after traffic switching. Only new requests are switched to the backup service; existing streaming requests continue to be processed, and the global drain process remains undisturbed, balancing smooth traffic migration with orderly node resource reclamation.

[0059] Optionally, triggering a checkpoint barrier when the training node reaches the safe training stop time, and outputting and verifying the training breakpoint, includes: When the training node reaches the safe training stop time, a training stop signal is sent to the training task. When a checkpoint barrier is triggered, the training process enters the barrier procedure, saving model parameters, optimizer state, training steps, data cursor, and scheduling metadata as a temporary state file. After the temporary state file is written, an atomic renaming operation is performed to obtain the checkpoint file; An integrity check is performed on the renamed temporary state file. In response to the successful check, the global pointer of the renamed temporary state file is updated, and the training breakpoint is confirmed to have been saved successfully. The global pointer is used as an index to point to the renamed temporary state file.

[0060] Specifically, the system clock is compared in real time with the pre-calculated safe training stop time. When the training node reaches the safe training stop time, the controller synchronously sends a training stop signal to all nodes running training tasks. Upon receiving the training stop signal, the training node immediately triggers the checkpoint barrier process, temporarily pausing training iteration operations such as gradient backpropagation, parameter updates, and data loading, freezing the current training state. The model parameters, optimizer internal state, current global training steps, dataset read cursor, and cluster scheduling metadata are completely serialized and written to a temporary state file in the storage path. Write locks are used throughout the writing process to prevent concurrent modification of the file content by multiple processes. After all fragmented data and metadata have been completely written to the temporary state file and the file write stream has been closed normally, an atomic renaming operation is performed by the operating system to replace the cache file with the temporary suffix with a standard format checkpoint file. Atomic operations are uninterruptible; if a system crashes during the writing process, only an invalid temporary file remains, and no incomplete or corrupted breakpoints are generated. Read the renamed checkpoint file and perform multi-level integrity checks to verify the completeness of file fragment count, file hash digest, model parameter dimensions, optimizer fields, training steps, data cursor, and scheduling metadata fields. If the integrity check passes, update the global pointer index file in the storage directory, pointing the global pointer to the newly generated checkpoint file, marking this snapshot as the current valid training breakpoint, and confirming that the breakpoint for this barrier process has been successfully saved. If the integrity check fails, do not update the global pointer, retain the previous valid breakpoint, report a breakpoint write exception alarm to the scheduling monitoring module, temporarily suspend the release of training nodes, and postpone the main inference cluster recovery process.

[0061] It's important to note that in distributed multi-training node scenarios, all nodes synchronously trigger checkpoint barriers, uniformly completing temporary file writing and atomic renaming before unified verification to ensure global consistency of training snapshots across multiple machines. Write buffer durations are adjusted differently for different storage media (e.g., local SSDs, distributed shared storage) to ensure that atomic renaming is performed only after temporary files are fully persisted. Global pointers are persistently stored, allowing direct reading of the pointers to locate the latest valid breakpoint after system restarts or cluster failures, without needing to scan all historical snapshot files.

[0062] This invention's embodiments trigger barriers on time to uniformly solidify the entire set of training state data, preventing the loss of training progress while strictly adhering to cluster recovery time constraints. A temporary state file containing complete training metadata is written, retaining all information such as the model, optimizer, iterations, and data cursors at once, ensuring breakpoints are fully recoverable. After the file is written, an atomic renaming is performed to generate a checkpoint file, using atomic operations to ensure the integrity of the file write transaction and preventing incomplete or corrupted breakpoint files caused by read / write interruptions. Invalid breakpoints are verified and filtered, and a global pointer provides a unified index, ensuring the system only reads reliable and usable training snapshots. Through barrier solidification, atomic writing, integrity verification, and global pointer indexing, a closed-loop breakpoint protection is formed, accurately matching the cluster recovery sequence while ensuring the integrity and reliability of training snapshots, avoiding redundant training and computational waste, and improving the overall fault tolerance and stability of training-push scheduling.

[0063] Optionally, the global draining of existing streaming requests across all stage nodes also includes drain timeout degradation handling: When the emptying time reaches the preset soft emptying limit, freeze new training admissions and send an emptying timeout alarm; When the draining time reaches the preset absolute draining limit, an active intervention degradation strategy is triggered to terminate the abnormal long connection. Wherein, the upper limit of soft evacuation is less than the upper limit of absolute evacuation.

[0064] Optionally, the triggering of the proactive intervention degradation strategy to terminate the abnormal long connection includes: Save the state of abnormal long-lived connections that have not yet ended; The application layer traffic scheduling gateway returns the standard interrupt status code or retry indicator for this abnormal state to the client. Terminate the abnormal long connection and reconfirm the exit status and memory release status of the old inference instance.

[0065] In this embodiment, the drain timeout degradation handling is a tiered control mechanism for abnormal scenarios where existing connections cannot be cleared for extended periods. It consists of two levels of time thresholds: a soft drain upper limit and an absolute drain upper limit, with rate limiting, alarms, and session termination operations executed at each level. The preset soft drain upper limit is the first-level buffer timeout threshold, used to warn of delayed drain progress. It does not forcibly terminate existing sessions, only restricting the access of new training tasks. The preset absolute drain upper limit is the second-level hard timeout threshold, serving as a fallback upper limit for the scheduling process to prevent excessively long abnormal connections from permanently occupying inference nodes and blocking nighttime training windows. Abnormal long connections are defined as connections where the client has not read data for a long time, the inference process is stuck, there is malicious keep-alive behavior, or there are excessively long, unterminated text generation-like streaming sessions that cannot be naturally terminated by the normal drain process. The standard interruption status code is a standardized business exception response code uniformly returned by the gateway, used to inform the client that the current inference session has been abnormally terminated. The retry flag is a business tag carried in the response body. After the client identifies this tag, it can continue inference with the already generated content, without having to start a complete request from scratch.

[0066] Specifically, a global drain timer is initiated, continuously summarizing the number of active existing connections for nodes at each stage across the entire cluster, and comparing the current drain duration with the two-level preset timeout thresholds in real time. The first level of control, i.e., reaching the soft drain limit: When the cumulative global drain duration reaches the pre-configured soft drain limit, two operations are performed: freezing the new task access channel for the training resource pool, no longer allocating currently pending candidate stage nodes to new training tasks, and delaying the training start time; and pushing a drain timeout structured alarm to the operation and maintenance monitoring system, marking abnormal nodes, the number of active connections, the consumed drain duration, and hardware heterogeneous parameters, while simultaneously recording the scheduling state machine log for post-event tracing. During this stage, existing existing streaming requests are not interrupted, and the session continues to wait for natural completion and output. The second level of control, namely reaching the absolute empty limit: If the emptying time after continuous waiting further reaches the preset absolute emptying limit, an active intervention and degradation strategy is triggered to uniformly handle abnormal long connections. All incomplete existing sessions are traversed, and the session ID, request context, output term range, node identifier, timestamp, and audit flag of each abnormal connection are persistently saved; if the inference engine supports KV cache snapshots, the runtime summary is cached synchronously; a standard interruption status code and a resume / retry prompt are returned to the corresponding client through the application layer traffic scheduling gateway; and an instruction is issued to terminate all timeout abnormal long connections, releasing the ports, video memory, and hardware device handles occupied by the connections.

[0067] In this embodiment of the invention, during the timeout warning phase, new training tasks are first blocked and anomalies are reported. Sufficient time is allowed for the abnormal long-lived connections to terminate naturally, prioritizing the complete output of user streaming inference. Simultaneously, operations and maintenance can promptly detect any issues with data rollout obstruction. Upon reaching a hard threshold, long-lived sessions are forcibly cleared, releasing long-occupied GPU memory and connection resources. This prevents the data rollout process from being indefinitely blocked, ensuring nodes are released on time for training tasks and preventing significant shortening of the training window at night. A two-tiered timeout degradation strategy is employed, balancing the complete termination of normal sessions with controllable scheduling efficiency. This minimizes the probability of user inference interruptions while promptly releasing computing resources to ensure timely progress of training and push scheduling, thus improving the cluster's fault tolerance.

[0068] Optionally, the large model training and push hybrid scheduling method further includes heterogeneous cluster differential control: Collect heterogeneous hardware characteristics of different nodes, including AI acceleration chip type, model loading path, node interconnection network, and checkpoint storage path.

[0069] Based on the heterogeneous hardware characteristics, the timeout threshold of the local emptying, the upper bound of the ready wait for the guaranteed inference service, and the conservative estimate in the backward calculation are dynamically adjusted.

[0070] In this embodiment, the heterogeneous cluster differentiation control targets a hybrid computing cluster composed of multiple AI acceleration chips, storage devices, and network nodes. Instead of using a globally unified scheduling threshold, it configures emptying, startup, and training stop parameters individually based on the actual hardware characteristics of each node, eliminating scheduling timing mismatches caused by hardware differences. The local emptying timeout threshold has two levels: a soft emptying upper limit and an absolute emptying upper limit, used to manage the waiting window of existing long connections. The upper bound of the guaranteed inference service readiness wait is the longest allowed waiting time for containers, models, and initial metaword detection after starting the guaranteed inference service; a timeout indicates startup failure. The conservative estimates in the backward calculation are three types of conservative parameters used when calculating the safe training stop time: the main inference recovery time reserve, the checkpoint write safety time, and the safety buffer duration.

[0071] Specifically, the scheduling controller automatically collects four types of hardware characteristics for each node during cluster initialization and node launch, and persistently stores the hardware profile. AI acceleration chip feature extraction includes: identifying GPU / NPU model, total video memory capacity, driver stack version, and video memory read / write bandwidth; model loading path feature extraction includes: collecting local weight cache hit rate, remote storage read bandwidth, and average time for sharded model loading; node interconnection network feature extraction includes: collecting cross-node communication latency, link bandwidth, and packet loss rate; checkpoint storage path feature extraction includes: distinguishing between local SSD / distributed shared storage, collecting write throughput, historical write failure rate, and shard synchronization time.

[0072] Optionally, the step of dynamically adjusting the timeout threshold of the local emptying, the upper bound of the ready wait for the guaranteed inference service, and the conservative estimate in the backward calculation based on the heterogeneous hardware characteristics includes: Adjust the memory release threshold and the upper bound of the guaranteed inference service startup wait based on the memory capacity and driver version of the AI ​​acceleration chip type.

[0073] Specifically, two sets of parameters are adjusted according to the type of AI acceleration chip, memory capacity, and driver version: (1) Memory release threshold: the larger the memory capacity, the higher the memory release safety threshold is set; the old driver has low memory recycling efficiency, so the threshold is lowered and the memory stable buffer time is extended; (2) Guaranteed inference service startup waiting upper limit: small memory chip model loading speed is slow, so the ready detection waiting upper limit is extended; high-end high bandwidth chip shortens the waiting time.

[0074] Based on the cache hit rate and read bandwidth of the model loading path, adjust the conservative estimates of the upper bound of the ready wait for the guaranteed inference service and the recovery time of the main inference.

[0075] Specifically, based on the model loading path cache hit rate and read bandwidth, two sets of parameters are adjusted: (1) Upper bound of guaranteed inference service readiness wait: In scenarios with low cache hit rate in remote storage, model loading takes longer, so the upper bound of probe wait is increased; the wait time for local high cache nodes is shortened; (2) Conservative estimate of main inference recovery time: In scenarios with slow remote storage, the conservative value of recovery time reservation is increased, and the conservative reservation time for local cache nodes is decreased. Based on the write throughput and historical failure rate of the checkpoint storage path, the checkpoint write safety time and the safety buffer time are adjusted.

[0076] Adjust the checkpoint write safety time and the safety buffer time based on the write throughput and historical failure rate of the checkpoint storage path.

[0077] Specifically, two sets of parameters are adjusted based on the checkpoint storage path write throughput and historical failure rate: (1) Checkpoint write security time: For distributed shared storage and nodes with low write throughput, the snapshot write reservation time is extended; local high-speed SSDs shorten the security time; (2) Security buffer time: For nodes with high storage write failure rate, the security redundancy buffer during reverse calculation is increased to ensure that the cluster recovery is completed before the morning peak.

[0078] The embodiments of this invention abandon globally unified scheduling parameters and adjust the thresholds related to emptying, starting, and stopping training according to the performance differences of heterogeneous hardware of each node. This adapts to multi-model GPU / NPU hybrid clusters, which not only reduces hardware resource idleness, memory conflicts, and service startup failures, but also accurately controls the training stoppage boundary, greatly improving the scheduling stability and hardware resource utilization of heterogeneous training and push clusters.

[0079] Optionally, the large model training-pull hybrid scheduling method further includes exception rollback handling: If the guaranteed inference service fails the readiness test, the candidate stage node resumes receiving the new request; If the main inference cluster fails the health check or the recovery times out, the backup inference service will continue to accept the new request, and new training admissions will be frozen.

[0080] Specifically, the guaranteed inference service, once started, undergoes multiple rounds of continuous, repeated probing of containers, models, and initial terms. If any round of probing fails, the guaranteed inference service is deemed abnormal and unavailable. A rollback operation is immediately executed, which involves revoking the routing rules and backend availability flags previously updated by the application-layer traffic scheduling gateway, and resuming the candidate stage nodes from receiving all new requests. Simultaneously, the subsequent process of emptying candidate stage nodes is paused, the transfer of computing power after memory release is not performed, and the current nighttime training window is postponed or canceled to prevent premature node release from causing inference interruptions.

[0081] After the training breakpoint verification passes, the main inference cluster is expanded and rebuilt, and a full-link health check is performed. At the same time, the total recovery time of the cluster is monitored. The fallback strategy is triggered in two failure scenarios: after the main inference cluster is expanded, the health check fails multiple times; the total time for cluster expansion and model loading exceeds the preset recovery upper limit, and the recovery is judged to have timed out. After the failure is triggered, two fallback actions are performed: (1) the current gateway routing configuration is maintained, and all new requests are continuously handled by the backup inference service without performing traffic back-switching to the main cluster; (2) the new task admission channel of the training resource pool is frozen, and no new fine-tuning and pre-training tasks are issued to avoid the training computing power occupation affecting the resource supply of the backup inference service. The main inference cluster health check is continuously retried periodically until all verifications pass, and then traffic back-switching and the training admission freeze are lifted.

[0082] This invention provides a two-layer fault rollback logic to offer fallback solutions for two types of faults: guaranteed startup and main cluster recovery. Traffic can be rolled back quickly, and training resource access is synchronously controlled, which greatly improves the fault tolerance capability of the training-inference hybrid scheduling process and stably ensures the continuity of online inference services.

[0083] Optionally, the large model training and push hybrid scheduling method further includes a state machine and security constraints, as shown in Table 1.

[0084] Table 1

[0085] like Figure 2 As shown in the figure, an embodiment of the present invention provides a large model training and push hybrid scheduling device, comprising: The initial module is used to select candidate stage nodes from the main inference cluster in response to the night switching start signal, stop the candidate stage nodes from receiving new requests, and partially drain the existing streaming requests. The startup module is used to start the backup inference service after the old inference instances that have been partially cleared and processed in the candidate stage node have released their video memory. The switching module is used to switch the new request to the guaranteed inference service through the application layer traffic scheduling gateway in response to the guaranteed inference service passing the readiness detection, and to globally drain the existing streaming requests of all stage nodes. The release module is used to release the candidate stage node as a training node to the training resource pool to perform training tasks after all the stage nodes are globally cleared. The training module is used to calculate the safe training stop time by working backward from the recovery deadline of the main inference cluster. When the training node reaches the safe training stop time, a checkpoint barrier is triggered, and the training breakpoint is output and verified. The rollback module is used to release the training node and restore the main inference cluster in response to the passing of the training breakpoint verification. After the main inference cluster passes the health check, the new request is rolled back to the main inference cluster, and the backup inference service is drained and then merged into the main inference cluster.

[0086] like Figure 3 As shown in the figure, an electronic device provided by an embodiment of the present invention includes a memory and a processor; the memory is used to store a computer program; the processor is used to implement the large model training and push hybrid scheduling method as described above when executing the computer program.

[0087] Alternatively, an electronic device includes a memory and a processor coupled to the memory; the memory is configured to store a computer program; the processor is configured to perform the following operations when the computer program is executed: In response to the night-time switching start signal, candidate stage nodes are selected from the main inference cluster, the candidate stage nodes are stopped from receiving new requests, and the existing streaming requests are partially drained. After the candidate stage node is partially emptied and the old inference instances that have been processed are released from memory, the guaranteed inference service is started. In response to the guaranteed inference service passing the readiness detection, the new request is switched to the guaranteed inference service through the application layer traffic scheduling gateway, and the existing streaming requests of all stage nodes are globally drained. After all the aforementioned stage nodes are globally cleared, the candidate stage nodes are released as training nodes to the training resource pool to perform training tasks. The safe training stop time is calculated by working backward from the recovery deadline of the main inference cluster. When the training node reaches the safe training stop time, a checkpoint barrier is triggered, and the training breakpoint is output and verified. In response to the successful verification of the training breakpoint, the training node is released and the main inference cluster is restored. After the main inference cluster passes the health check, the new request is switched back to the main inference cluster, and the backup inference service is drained and then merged into the main inference cluster.

[0088] This invention provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the large-model training and push hybrid scheduling method as described above.

[0089] Alternatively, a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform the following operations: In response to the night-time switching start signal, candidate stage nodes are selected from the main inference cluster, the candidate stage nodes are stopped from receiving new requests, and the existing streaming requests are partially drained. After the candidate stage node is partially emptied and the old inference instances that have been processed are released from memory, the guaranteed inference service is started. In response to the guaranteed inference service passing the readiness detection, the new request is switched to the guaranteed inference service through the application layer traffic scheduling gateway, and the existing streaming requests of all stage nodes are globally drained. After all the aforementioned stage nodes are globally cleared, the candidate stage nodes are released as training nodes to the training resource pool to perform training tasks. The safe training stop time is calculated by working backward from the recovery deadline of the main inference cluster. When the training node reaches the safe training stop time, a checkpoint barrier is triggered, and the training breakpoint is output and verified. In response to the successful verification of the training breakpoint, the training node is released and the main inference cluster is restored. After the main inference cluster passes the health check, the new request is switched back to the main inference cluster, and the backup inference service is drained and then merged into the main inference cluster.

[0090] The present invention will now describe electronic devices that can serve as servers or clients of the present invention, which are examples of hardware devices that can be applied to various aspects of the present invention. Electronic devices are intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0091] Electronic devices include a computing unit that can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) or loaded from a storage unit into random access memory (RAM). The RAM can also store various programs and data required for device operation. The computing unit, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.

[0092] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc. In this application, the units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of the present invention according to actual needs. Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units can be implemented in hardware or as software functional units.

[0093] While the present invention has been disclosed above, its scope of protection is not limited thereto. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention, and all such changes and modifications will fall within the scope of protection of the present invention.

Claims

1. A large-scale model training and push hybrid scheduling method, characterized in that, include: In response to the night-time switching start signal, candidate stage nodes are selected from the main inference cluster, the candidate stage nodes are stopped from receiving new requests, and the existing streaming requests are partially drained. After the candidate stage node is partially emptied and the old inference instances that have been processed are released from memory, the guaranteed inference service is started. In response to the guaranteed inference service passing the readiness detection, the new request is switched to the guaranteed inference service through the application layer traffic scheduling gateway, and the existing streaming requests of all stage nodes are globally drained. After all the aforementioned stage nodes are globally cleared, the candidate stage nodes are released as training nodes to the training resource pool to perform training tasks. The safe training stop time is calculated by working backward from the recovery deadline of the main inference cluster. When the training node reaches the safe training stop time, a checkpoint barrier is triggered, and the training breakpoint is output and verified. In response to the successful verification of the training breakpoint, the training node is released and the main inference cluster is restored. After the main inference cluster passes the health check, the new request is switched back to the main inference cluster, and the backup inference service is drained and then merged into the main inference cluster.

2. The large-scale model training and push hybrid scheduling method according to claim 1, characterized in that, After the old inference instances that have been partially cleared and processed at the candidate stage node are released from memory, the backup inference service is started, including: Monitor the memory usage status of the candidate stage nodes, and start the guaranteed inference service after the old inference instance exits and the memory usage is lower than the preset release threshold; And / or, the response to the guaranteed inference service passing a readiness probe, switching new requests to the guaranteed inference service via an application-layer traffic scheduling gateway, includes: The guaranteed inference service is tested for container status, model loading, and initial metaword detection. After the continuous detection passes, the guaranteed inference service is confirmed to be ready. Update the routing rules and backend availability flags of the application layer traffic scheduling gateway so that the new requests enter the backup inference service, and ensure that the global emptying process of existing streaming requests at all stage nodes is not interrupted.

3. The large-scale model training and push hybrid scheduling method according to claim 1, characterized in that, The step of triggering a checkpoint barrier when the training node reaches the safe training stop time, outputting and verifying the training breakpoint, includes: When the training node reaches the safe training stop time, a training stop signal is sent to the training task. When a checkpoint barrier is triggered, the training process enters the barrier procedure, saving model parameters, optimizer state, training steps, data cursor, and scheduling metadata as a temporary state file. After the temporary state file is written, an atomic renaming operation is performed to obtain the checkpoint file; An integrity check is performed on the renamed temporary state file. In response to the successful check, the global pointer of the renamed temporary state file is updated, and the training breakpoint is confirmed to have been saved successfully. The global pointer is used as an index to point to the renamed temporary state file.

4. The large-scale model training and push hybrid scheduling method according to claim 1, characterized in that, The global draining of existing streaming requests across all stage nodes also includes draining timeout degradation handling: When the emptying time reaches the preset soft emptying limit, freeze new training admissions and send an emptying timeout alarm; When the draining time reaches the preset absolute draining limit, an active intervention degradation strategy is triggered to terminate the abnormal long connection. Wherein, the upper limit of soft evacuation is less than the upper limit of absolute evacuation.

5. The large-scale model training and push hybrid scheduling method according to claim 4, characterized in that, The triggering of the proactive intervention degradation strategy to terminate abnormal long connections includes: Save the state of abnormal long-lived connections that have not yet ended; The application layer traffic scheduling gateway returns the standard interrupt status code or retry indicator for this abnormal state to the client. Terminate the abnormal long connection and reconfirm the exit status and memory release status of the old inference instance.

6. The large-scale model training and push hybrid scheduling method according to claim 1, characterized in that, It also includes differentiated control for heterogeneous clusters: Collect heterogeneous hardware characteristics of different nodes, including AI acceleration chip type, model loading path, node interconnection network and checkpoint storage path; Based on the heterogeneous hardware characteristics, the timeout threshold of the local emptying, the upper bound of the ready waiting for the guaranteed inference service, and the conservative estimate in the backward calculation are dynamically adjusted. Specifically, based on the memory capacity and driver version of the AI ​​acceleration chip type, the memory release threshold and the upper bound of the guaranteed inference service startup wait are adjusted. Based on the cache hit rate and read bandwidth of the model loading path, adjust the conservative estimates of the upper bound of the ready wait for the backup inference service and the recovery time of the main inference. Adjust the checkpoint write safety time and the safety buffer time based on the write throughput and historical failure rate of the checkpoint storage path.

7. The large-scale model training and push hybrid scheduling method according to claim 1, characterized in that, It also includes handling of abnormal rollbacks: If the guaranteed inference service fails the readiness test, the candidate stage node resumes receiving the new request; If the main inference cluster fails the health check or the recovery times out, the backup inference service will continue to accept the new request, and new training admissions will be frozen.

8. A large-scale model training-pushing hybrid scheduling device, characterized in that, include: The initial module is used to select candidate stage nodes from the main inference cluster in response to the night switching start signal, stop the candidate stage nodes from receiving new requests, and partially drain the existing streaming requests. The startup module is used to start the backup inference service after the old inference instances that have been partially cleared and processed in the candidate stage node have released their video memory. The switching module is used to switch the new request to the guaranteed inference service through the application layer traffic scheduling gateway in response to the guaranteed inference service passing the readiness detection, and to globally drain the existing streaming requests of all stage nodes. The release module is used to release the candidate stage node as a training node to the training resource pool to perform training tasks after all the stage nodes are globally cleared. The training module is used to calculate the safe training stop time by working backward from the recovery deadline of the main inference cluster. When the training node reaches the safe training stop time, a checkpoint barrier is triggered, and the training breakpoint is output and verified. The rollback module is used to release the training node and restore the main inference cluster in response to the passing of the training breakpoint verification. After the main inference cluster passes the health check, the new request is rolled back to the main inference cluster, and the backup inference service is drained and then merged into the main inference cluster.

9. An electronic device, characterized in that, Including memory and processor; The memory is used to store computer programs; The processor is configured to implement the large model training and push hybrid scheduling method as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the large model training and push hybrid scheduling method as described in any one of claims 1 to 7.

Images