Real-time fault-tolerant and task migration system and method for process-oriented GPUs

By combining incremental differential and lightweight compression techniques with an RDMA network, health indicators are collected in real time and failure risks are predicted, which solves the problems of low reliability and low resource utilization efficiency of long-term training tasks in GPU clusters and enables rapid task migration and recovery.

CN121560461BActive Publication Date: 2026-04-17HANHOU (BEIJING) TECH CO LTD
3 Cites 0 Cited by

Patent Information

Application Number
CN202610092073.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2026-01-23
Publication Date
2026-04-17
Estimated Expiration
2046-01-23

AI Technical Summary

Technical Problem

Existing technologies suffer from poor reliability and low resource utilization efficiency in the continuous operation of long-term training tasks in GPU clusters, mainly due to the large storage and transmission overhead, long recovery time, and idle spare resources caused by the full checkpoint scheme.

Method used

The incremental differential algorithm is used to capture key data blocks of the training task state. Combined with lightweight compression and multi-level storage, multi-dimensional health indicators are collected in real time and failure risks are predicted through a time series prediction model. The RDMA network is used to realize task migration and quickly restore the training task state.

Benefits of technology

It reduces storage overhead, shortens data processing time, improves the accuracy of fault identification and resource utilization efficiency, and ensures rapid recovery and continuity of long-term training tasks.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader

Abstract

The application provides a real-time fault-tolerant and task migration system and method for a process GPU, and relates to the technical field of distributed computing, wherein the method comprises the following steps: capturing a training task state in GPU memory and generating difference data by using an incremental differential algorithm, and storing the difference data in a multi-level storage system after light compression; simultaneously, real-time collection of GPU node health indicators is performed, node states are analyzed by using a time series prediction model, and a migration process is triggered when a fault risk is predicted or a preemption warning is received; the task is paused during migration and a final incremental checkpoint is generated, the checkpoint data is quickly transmitted to a target healthy GPU node through an RDMA network, a complete context state is reconstructed, and the task execution is resumed, so that real-time fault-tolerance and seamless migration of the GPU task are realized. The application improves the continuous operation reliability and resource utilization efficiency of the GPU cluster in a long-time training task.
Need to check novelty before this filing date? Find Prior Art