Real-time fault-tolerant and task migration system and method for process-oriented GPUs

Abstract

The application provides a real-time fault-tolerant and task migration system and method for a process GPU, and relates to the technical field of distributed computing, wherein the method comprises the following steps: capturing a training task state in GPU memory and generating difference data by using an incremental differential algorithm, and storing the difference data in a multi-level storage system after light compression; simultaneously, real-time collection of GPU node health indicators is performed, node states are analyzed by using a time series prediction model, and a migration process is triggered when a fault risk is predicted or a preemption warning is received; the task is paused during migration and a final incremental checkpoint is generated, the checkpoint data is quickly transmitted to a target healthy GPU node through an RDMA network, a complete context state is reconstructed, and the task execution is resumed, so that real-time fault-tolerance and seamless migration of the GPU task are realized. The application improves the continuous operation reliability and resource utilization efficiency of the GPU cluster in a long-time training task.

Description

Technical Field

[0001] This application relates to the field of distributed computing technology, and in particular to a real-time fault-tolerant and task migration system and method for procedural GPUs. Background Technology

[0002] As the scale of current deep learning models continues to increase, a single training task often needs to run continuously for several days or even weeks, which places extremely high demands on the stability of computing nodes. However, in a shared cluster environment, hardware failures and preemption of high-priority tasks occur frequently. Therefore, there is an urgent need for a fault-tolerant and rapid recovery mechanism that can ensure the continuity of long-running tasks, thereby minimizing the waste of computing resources and the loss of training progress caused by unplanned interruptions.

[0003] Current mainstream solutions generally adopt periodic full checkpointing technology. Specifically, by setting a fixed time interval during training, the complete running state of the task is saved to the shared storage system. When a computing node abnormality is detected or a scheduling instruction such as task preemption is received, the solution reloads the data from the most recently saved checkpoint to resume task execution. The entire process relies on the cluster management component to complete the reallocation of resources and the restart of tasks.

[0004] However, the full checkpointing scheme incurs significant storage and transmission overhead during execution, thus consuming valuable computing bandwidth. Furthermore, because the checkpoint saving interval is constrained by operation time, a failure could lead to a prolonged loss of computational progress. During the recovery phase, the need to fully read checkpoint data and reinitialize the entire runtime environment results in a lengthy task interruption. Additionally, the backup computing resources reserved for failures are typically idle during non-failure periods, leaving considerable room for optimization in overall resource utilization. Summary of the Invention

[0005] This application provides a real-time fault-tolerant and task migration system and method for procedural GPUs, which solves the problems of poor reliability and low resource utilization efficiency of GPU clusters in long-term training tasks in the prior art.

[0006] To address the aforementioned technical problems, in a first aspect, this application provides a real-time fault-tolerance and task migration method for procedural GPUs, comprising:

[0007] The system captures the training task state in the graphics processing unit (GPU) memory, uses an incremental difference algorithm to determine the key data blocks between the training task state at the current checkpoint and the training task state at the previous checkpoint, and generates difference data based on the key data blocks.

[0008] The differential data is lightly compressed to form compressed incremental data, and the compressed incremental data is stored in a multi-level storage system.

[0009] While storing compressed incremental data, multi-dimensional health indicators of the current GPU node are collected in real time, and the multi-dimensional health indicators are input into the time series prediction model for joint analysis. When the time series prediction model predicts that the failure risk of the current GPU node is greater than the preset risk threshold, or when it receives a preemption warning signal sent by the cluster scheduler, the task migration process is triggered.

[0010] In the task migration process, the training task on the current GPU node is paused, the final operation is triggered, and the final incremental data is generated.

[0011] The final incremental data and the latest incremental data in the hot backup of the multi-level storage system are transferred to the pre-assigned target healthy GPU node via a Remote Direct Memory Access (RDMA) network.

[0012] On the target healthy GPU node, the complete context state of the training task is reconstructed using the final incremental data and the latest incremental data in the hot backup, so as to restore the execution state of the training task and realize real-time fault tolerance and task migration for procedural GPUs.

[0013] Secondly, this application provides a real-time fault-tolerant and task migration system for procedural GPUs, comprising:

[0014] The determination module is used to capture the training task state in the GPU memory, use the incremental difference algorithm to determine the key data block between the training task state at the current checkpoint and the training task state at the previous checkpoint, and generate difference data based on the key data block.

[0015] A compression module is used to perform lightweight compression on the differential data to form compressed incremental data, and to store the compressed incremental data in a multi-level storage system.

[0016] The acquisition module is used to collect multi-dimensional health indicators of the current GPU node in real time while storing compressed incremental data, and input the multi-dimensional health indicators into the time series prediction model for joint analysis. When the time series prediction model predicts that the failure risk of the current GPU node is greater than the preset risk threshold, or receives a preemption warning signal sent by the cluster scheduler, the task migration process is triggered.

[0017] The generation module is used to pause the training task on the current GPU node, trigger the final operation, and generate the final incremental data during the task migration process.

[0018] The allocation module is used to transfer the final incremental data and the latest incremental data in the hot backup of the multi-level storage system to the pre-allocated target healthy GPU node via the RDMA network.

[0019] The reconstruction module is used to reconstruct the complete context state of the training task on the target healthy GPU node using the final incremental data and the latest incremental data in the hot backup, so as to restore the execution state of the training task and realize real-time fault tolerance and task migration for procedural GPUs.

[0020] Thirdly, this application provides an electronic device, comprising:

[0021] Memory, used to store computer programs;

[0022] A processor, used to implement the steps of the real-time fault-tolerant and task migration method for procedural GPUs as described in the first aspect above when executing the computer program.

[0023] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the steps of the real-time fault-tolerant and task migration method for procedural GPUs as described in the first aspect above.

[0024] The technical solution provided in this application has the following beneficial effects:

[0025] This application achieves accurate capture of key data blocks and generation of differential data for training task states through high-frequency state preservation and incremental differential algorithms, effectively reducing storage overhead and unnecessary data processing. Lightweight compression and a multi-level storage system optimize the storage and transmission efficiency of incremental data, balancing the dual requirements of rapid recovery and long-term archiving. By collecting multi-dimensional health indicators in real time and combining them with time-series prediction models, it can comprehensively grasp the node's operating status, promptly identify potential failure risks, and quickly trigger the migration process when a failure is predicted or a preemption warning is received. High-speed, low-latency data migration is achieved by transmitting the final incremental data and hot backup data to the target node via RDMA network. Based on complete context state reconstruction, the accuracy and continuity of task recovery are ensured, thereby improving the operational reliability and resource utilization efficiency of the GPU cluster in long-term training tasks.

[0026] Furthermore, this application achieves accurate early warning and early identification of node failures by combining the prediction model of Transformer and graph neural network, thereby improving prediction accuracy. By listening to the cluster scheduler instructions, the resource reservation and migration process is immediately initiated upon receiving resource preemption signals or failure warnings, establishing a complete migration preparation mechanism to ensure that resource allocation is completed before failure or preemption occurs, creating favorable conditions for smooth task migration.

[0027] These or other aspects of this application will become more apparent in the following description of the embodiments. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 A flowchart illustrating a real-time fault tolerance and task migration method for procedural GPUs provided in this application embodiment;

[0030] Figure 2 A schematic diagram illustrating a specific implementation of a real-time fault-tolerance and task migration method for procedural GPUs provided in this application embodiment;

[0031] Figure 3 This is a schematic diagram of the structure of a real-time fault-tolerant and task migration system for procedural GPUs provided in an embodiment of this application. Detailed Implementation

[0032] To address the problems of existing technologies, this application proposes a real-time fault tolerance and task migration method for procedural GPUs, achieving a breakthrough improvement through a collaborative mechanism of lightweight incremental checkpointing and intelligent predictive migration. The core of this scheme lies in using incremental differential technology to save only changed data blocks, combined with lightweight compression and multi-level storage strategies, thereby reducing checkpoint overhead by an order of magnitude and supporting high-frequency saving at the minute level. Simultaneously, a time-series prediction model is introduced to proactively assess node health and is linked with the cluster scheduler to actively trigger migration before a failure occurs or during resource preemption. High-speed transmission of checkpoint data is achieved through an RDMA network, and the complete task context is quickly reconstructed on the target node. This method not only compresses task interruption time from hours to within minutes but also reduces progress loss from hours to minutes. Furthermore, through a backup node resource reuse mechanism, it improves cluster resource utilization efficiency while ensuring high availability, thus systematically resolving the inherent contradiction between performance overhead, recovery speed, and resource economy in traditional solutions.

[0033] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0034] The core of this application is to provide a real-time fault-tolerant and task migration method for procedural GPUs, and a flowchart of one specific implementation is shown below. Figure 1 As shown, the method includes:

[0035] Step 101: Capture the training task state in the GPU memory, use the incremental difference algorithm to determine the key data blocks between the training task state at the current checkpoint and the training task state at the previous checkpoint, and generate difference data based on the key data blocks.

[0036] In step 101, the training task state in GPU memory refers to all the information stored in the GPU memory during model training that is used to maintain the continuity of training. This information mainly includes model parameters, optimizer state, etc.

[0037] Key data blocks refer to those data units whose content has changed, identified through incremental difference algorithms; difference data refers to a complete difference description dataset containing the content of all changed data blocks and their original storage location information, identified through incremental difference algorithms.

[0038] In the embodiments of this application, the specific implementation of step 101 can refer to steps 501 to 505 below, which will not be repeated here.

[0039] Step 102: Perform lightweight compression on the differential data to form compressed incremental data, and store the compressed incremental data in a multi-level storage system.

[0040] In step 102, the compressed incremental data refers to the differential data after compression, which has a more compact format and smaller size; a multi-level storage system is a hierarchical data storage architecture composed of storage devices with different performance, capacity and cost, which may include local high-speed access media and distributed file systems.

[0041] Among them, local high-speed access media refers to high-speed storage devices that are directly connected to computing nodes, such as solid-state drives; backup data, such as the latest incremental data, is stored on high-speed storage devices and can be read and used immediately.

[0042] A distributed file system is a file system that distributes data across multiple servers. It is used to store backup data for long-term archiving, such as consolidated full checkpoints, on low-cost, slow-access storage devices.

[0043] Furthermore, the integrated full checkpoint refers to the complete state snapshot generated by merging and reconstructing multiple consecutive incremental checkpoint data with a basic full checkpoint. The generation process is as follows: during the periodic execution of incremental checkpoints, when the preset integration conditions are met, the latest basic full checkpoint is fused with all subsequent incremental checkpoint data to regenerate a new full checkpoint containing the complete task state up to the current moment.

[0044] In the embodiments of this application, the specific implementation process of step 102 can be referred to steps 601 to 606, which will not be repeated here.

[0045] Step 103: While storing the compressed incremental data, collect multi-dimensional health indicators of the current GPU node in real time, and input the multi-dimensional health indicators into the time series prediction model for joint analysis. When the time series prediction model predicts that the failure risk of the current GPU node is greater than the preset risk threshold, or receives a preemption warning signal sent by the cluster scheduler, the task migration process is triggered.

[0046] In step 103, GPU memory is a core component of the GPU node. GPU memory is specifically used to store runtime data such as model parameters and optimizer status during the training process.

[0047] Multidimensional health metrics reflect the operational status of GPU nodes from different perspectives, including hardware metrics, software metrics, and system metrics; failure risk is a quantified value that indicates the likelihood of a GPU node failing in the near future.

[0048] The preset risk threshold is a manually set critical value used to determine whether the risk has reached a level that requires action. The embodiments of this application do not specifically limit the size of this threshold.

[0049] The cluster scheduler is a central control software used to manage and allocate computing cluster resources. When resources need to be reallocated for high-priority tasks, it sends a preemption warning signal to this method, thereby triggering the migration process.

[0050] The preemption warning signal is a message issued by the cluster scheduler to notify a low-priority task that its resources will soon be reclaimed to run a high-priority task.

[0051] The task migration process includes a complete sequence of operations from triggering the migration to the final recovery. State hot migration specifically refers to the key technical steps in this process that achieve seamless state migration through final incremental checkpoints, RDMA transfer, and context reconstruction. The two are related as the overall process and the core techniques.

[0052] In the embodiments of this application, the specific implementation process of step 103 can be referred to steps 201 to 205, which will not be repeated here.

[0053] Step 104: In the task migration process, pause the training task on the current GPU node, trigger the final operation, and generate the final incremental data.

[0054] In step 104, the current GPU node is a node that is executing a training task and whose health status is being monitored. When the node is predicted to fail or be preempted, it becomes the source GPU node that needs to be migrated out of the task.

[0055] After the final operation is triggered, all state changes since the last periodic checkpoint can be captured. Periodicity refers to the checkpoint saving process that is triggered regularly according to a preset time interval. The last periodic checkpoint refers to the most recent regular checkpoint executed according to this fixed period before the current migration trigger time.

[0056] The final incremental data is the last state difference data that is urgently generated when the migration process is triggered, and it is used to capture the state changes from the latest periodic checkpoint to the migration trigger time.

[0057] In this embodiment, when the task migration process is triggered, the first step is to issue a pause command to the task currently running on the GPU node to ensure that its task state remains temporarily static. Then, immediately after the task is paused, a final data capture operation is performed. The basic logic of this operation is similar to the periodic checkpoint creation process, but the key difference is that its comparison benchmark is set to the previously successfully saved periodic checkpoint state. The purpose of this design is to accurately capture all task state changes that occur from the last successful checkpoint moment until the current moment when the task is paused. Finally, through comparison and calculation, a final incremental data containing all state change data of the above final stage is generated.

[0058] Step 105: Transfer the final incremental data and the latest incremental data from the hot backup of the multi-level storage system to the pre-assigned target healthy GPU node via the RDMA network.

[0059] In step 105, the RDMA network is a network infrastructure that supports remote direct memory access technology, and the target healthy GPU node refers to an idle and normally functioning GPU node that is prepared in advance to receive and continue running the migration task.

[0060] In this embodiment of the application, an RDMA channel is first established between the source GPU node and the target healthy GPU node. The RDMA channel bypasses the operating system kernel and directly accesses the video memory space of the target healthy GPU node. With the help of this channel, the following steps 401 to 405 are executed, which will not be described in detail here.

[0061] RDMA channel is a high-speed data transfer path established between the memory of two computing nodes. This path allows data to be transferred directly from the memory of the source node to the memory of the target node without the need for extensive processing by the central processing units of both nodes.

[0062] Bypassing refers to kernel bypass technology implemented through RDMA network cards, which allows data to be transferred directly between user space and target video memory without going through data copying and processing by the operating system kernel. The operating system kernel is the basic software layer in a computer system that manages hardware resources and provides core services.

[0063] Step 106: On the target healthy GPU node, using the final incremental data and the latest incremental data in the hot backup, reconstruct the complete context state of the training task to restore the execution state of the training task, thereby achieving real-time fault tolerance and task migration for procedural GPUs.

[0064] In step 106, reconstructing the complete context state of the training task means, on the target node, using the received checkpoint data, reassembling all runtime states such as the model, optimizer, and number of iterations that are completely consistent with the time of the source node's interruption.

[0065] In this embodiment, the specific implementation process of step 106 can be referred to steps 701-703, and will not be repeated here. This method achieves real-time fault tolerance and rapid migration of GPU training tasks through multiple collaborative steps.

[0066] To address the limitations of existing technologies in achieving accurate prediction and efficient migration decisions, some embodiments, in step 103, input the multi-dimensional health indicators into a time-series prediction model for joint analysis. When the time-series prediction model predicts that the failure risk of the current GPU node exceeds a preset risk threshold, or receives a preemption warning signal from the cluster scheduler, the task migration process is triggered. Figure 2 As shown, it includes:

[0067] Step 201: Organize the multi-dimensional health indicators into an ordered data sequence according to the time series. The multi-dimensional health indicators include hardware indicators, software indicators, and system indicators.

[0068] In step 201, the ordered data sequence is a structured data set formed by arranging multi-dimensional health indicators in the order of collection time points.

[0069] In this embodiment, various operating status data of GPU nodes are continuously collected in the background, and these data are timestamped at fixed time intervals. The collected data are classified into different indicators in three dimensions: hardware, software and system, according to their attributes. Then, these indicator values ​​are arranged and combined according to the order of the timestamps to form a standardized data sequence. This standardized sequence provides a standardized input format for the subsequent time series analysis process.

[0070] Step 202: Input the ordered data sequence into a time-series prediction model based on the Transformer architecture, analyze the ordered data sequence through the time-series prediction model, and output an evaluation result representing the current health status of the GPU node.

[0071] In step 202, the evaluation result is a comprehensive score of the GPU node's health status by the model, representing the probability of the node failing.

[0072] In this embodiment, the specific implementation process of step 202 can refer to steps 301 to 306, and will not be repeated here. It should be noted that this embodiment does not specifically limit the specific structure of the time series prediction model.

[0073] Step 203: When the number of times the evaluation result exceeds the preset risk threshold reaches a set number, the correlation between the current GPU node and other GPU nodes is analyzed through a graph neural network to generate a node failure warning.

[0074] In step 203, the continuous judgment is achieved by performing sliding window analysis on the evaluation results of multiple consecutive time points output by the time series prediction model. For example, the system maintains a time window containing the 10 most recent evaluation results. When more than 80% of the evaluation results in the time window are higher than the preset risk threshold, and this state is maintained for at least 3 complete data collection cycles, the "continuously exceeding" condition is met and a node fault warning is generated.

[0075] Other GPU nodes refer to related nodes in the cluster that have resource competition, task dependency, or physical connection with the current GPU node. Node fault warning is a fault risk warning signal generated based on correlation analysis.

[0076] In this embodiment, when the failure risk probability value in the evaluation result exceeds the preset risk threshold for multiple consecutive monitoring periods, the early warning analysis process is triggered. At this time, a graph structure is constructed with all relevant GPU nodes as vertices and resource dependencies and communication links between nodes as edges. This graph structure is input into the graph neural network, which aggregates the state information from adjacent nodes in the graph through its message passing mechanism, thereby analyzing the degree of mutual influence between the current failure risk node and its surrounding related nodes. Finally, the local time-series prediction evaluation results are combined with the correlation influence results obtained from the graph structure analysis to generate a more comprehensive and accurate node failure early warning.

[0077] Step 204: When a resource preemption instruction is received through the cluster scheduler interface, the resource preemption instruction is parsed and a preemption warning signal is generated.

[0078] In step 204, the resource preemption instruction is a resource reclamation notification issued by the cluster scheduler, and the preemption warning signal is a migration trigger signal generated after parsing the instruction.

[0079] In this embodiment of the application, resource scheduling messages from the cluster scheduler are continuously monitored through a pre-defined interface. When the monitored message contains content related to resource reclamation, the instruction is parsed to extract the task identifier and time parameters. Subsequently, a preemption warning signal with a uniform and clear format is generated based on the specific parameters obtained from the parsing.

[0080] Step 205: After generating a node failure warning or preemption warning signal, start the task migration preparation process. In the task migration preparation process, send a resource reservation request to the target healthy GPU node, and trigger the task migration process after receiving the resource confirmation response from the target healthy GPU node.

[0081] In step 205, the task migration preparation process is the resource coordination process before migration. The resource reservation request is a message that requests computing resources from the target node, and the resource confirmation response is a resource readiness notification returned by the target node.

[0082] In this embodiment, after any warning signal is generated, the task migration preparation process is immediately initiated. First, according to the established resource scheduling strategy, a suitable target healthy GPU node is selected from the cluster, and a resource reservation request containing specific resource requirement information is sent to that node. After receiving this request, the target node checks its local available resources and reserves the necessary computing resources for the task to be migrated. After successfully reserving resources, the target node returns a confirmation response to the source node. When the source node receives this confirmation response, it formally triggers the subsequent complete task migration process.

[0083] In this embodiment of the application, the process achieves accurate assessment of fault risk through time series analysis of multi-dimensional indicators, and improves the accuracy of early warning by combining graph neural networks to consider the correlation between nodes.

[0084] It should be understood that the temporal prediction model includes: an encoder, a temporal convolution module, a feature fusion module, a prediction head module, a trend analysis module, and an output module. Furthermore, the specific design of these structures in this application embodiment is not specifically limited and can be configured according to actual conditions. To further improve the accuracy of fault prediction and the timeliness of early warning, in some embodiments, step 202: inputting the ordered data sequence into the temporal prediction model based on the Transformer architecture, analyzing the ordered data sequence through the temporal prediction model, and outputting an evaluation result characterizing the current GPU node health status, including:

[0085] Step 301: Input the ordered data sequence into a time series prediction model based on the Transformer architecture. Calculate the correlation weights between data at different time steps in the ordered data sequence through the encoder of the time series prediction model. Based on the correlation weights, and combined with the forward propagation algorithm in the self-attention mechanism, generate a global feature representation.

[0086] In step 301, different time steps refer to the various data collection time points arranged in chronological order in the ordered data sequence. Each time step corresponds to a set of multi-dimensional health indicator data collected at a specific time. The global feature representation is a comprehensive feature formed by the model after capturing the overall long-term dependency relationship of the input sequence.

[0087] In this embodiment, the ordered data sequence is first input into the encoder for processing. The encoder uses its internal self-attention mechanism to dynamically calculate the degree of correlation between the data at each time step in the sequence and the data at all other time steps. The calculated correlation weights directly determine the importance of information at different time points when synthesizing new sequence features. Subsequently, through the forward propagation calculation process, all weighted information is integrated and feature transformed. Finally, the encoder outputs a comprehensive feature representation that can effectively represent the long-term change pattern and global dependency of the entire input data sequence.

[0088] Step 302: The ordered data sequence is subjected to dilated convolution processing through the temporal convolution module of the temporal prediction model to obtain local context features.

[0089] In step 302, local context features refer to features that reflect the sudden changes in the health status of GPU nodes within adjacent time segments. These features include real-time status information that affects migration decisions, such as hardware indicator change patterns and short-term fluctuations in resource utilization.

[0090] In this embodiment, the temporal convolution module uses a convolution kernel with an expanded structure to scan the input ordered data sequence layer by layer. This structure allows the convolution kernel to skip some data points at intervals when covering the input sequence, thereby significantly expanding its effective sensing range while maintaining good computational efficiency. Through this localization operation of the sequence, the temporal convolution module can effectively capture key local context features such as short-term fluctuation patterns and abrupt change trends between adjacent time steps from the data.

[0091] Step 303: The global feature representation and the local context features are weighted and fused through the feature fusion module of the time series prediction model to form a health status representation vector.

[0092] In step 303, the health status representation vector is a comprehensive feature vector that can fully describe the health status of a node, formed by effectively combining global feature representation with local contextual features.

[0093] In this embodiment, the feature fusion module simultaneously receives global feature representations from the encoder and local contextual features from the temporal convolution module. The module first assigns appropriate fusion weights to these two complementary features and then performs a weighted summation operation on them. This fusion process enables the final generated health status representation vector to simultaneously contain the long-term evolution law of the data sequence and its short-term fluctuation details, thereby forming a more comprehensive and three-dimensional integrated description of the health status of the GPU node.

[0094] Step 304: Using the prediction head module of the time series prediction model, the health status representation vector is mapped into a fault risk probability value through a multilayer perceptron.

[0095] In step 304, the failure risk probability value is a specific numerical value used to quantify the likelihood of the current GPU node failing in the future.

[0096] In this embodiment, the prediction head module receives a health status representation vector from the feature fusion module and inputs the vector into a multilayer perceptron. The multilayer perceptron is composed of multiple fully connected layers and nonlinear activation functions, and through a series of nonlinear transformations, it gradually maps and condenses the high-dimensional and complex health status representation vector into a specific probability value representing the current node's failure risk.

[0097] Step 305: Using the trend analysis module of the time series prediction model, perform trend analysis on the fault risk probability value based on a sliding time window. When the fault risk probability value is detected to be rising continuously and the fluctuation range exceeds the preset range, generate a health warning result.

[0098] In step 305, the fluctuation amplitude is the relative volatility obtained by calculating the ratio of the standard deviation of the probability value to the moving average within the same time window. It represents the degree of instability and the intensity of abnormal fluctuations in the failure risk probability during a trend of increase. The health warning result is a signal indicating that the health status of the node is continuously deteriorating and the fluctuations are intensifying.

[0099] In this embodiment, the trend analysis module maintains a sliding time window of fixed length to continuously collect the failure risk probability values ​​output by the prediction head module. The module performs trend analysis on the probability value sequence arranged by time within the window. The trend analysis specifically includes calculating the overall slope of the sequence and evaluating its fluctuation. When the analysis results clearly show that the probability value exhibits a continuous upward trend, and its fluctuation amplitude exceeds a preset reasonable range threshold, the module triggers and generates a specific health warning result.

[0100] Step 306: The fault risk probability value is combined with the health warning result through the output module of the time series prediction model to output an evaluation result that represents the current health status of the GPU node.

[0101] In this embodiment, the output module integrates and encapsulates the failure risk probability value calculated by the prediction head module and the health warning result generated by the trend analysis module to form a structured comprehensive evaluation result. This result not only includes specific values ​​representing the current risk level, but also integrates clear warning information about whether there is a continuous deterioration trend, thereby providing a complete and reliable basis for subsequent warning decisions and task migration processes.

[0102] In this embodiment of the application, this step process achieves refined modeling of the node health status by integrating long-term and short-term time-series features.

[0103] To further improve the efficiency and reliability of data transmission during task migration, in some embodiments, step 105: transferring the final incremental data and the latest incremental data from the hot backup of the multi-level storage system to the pre-allocated target healthy GPU node via an RDMA network, includes:

[0104] Step 401: In the memory space of the source GPU node, the final incremental data is divided into multiple first data transmission units, and corresponding address mapping information is attached to each first data transmission unit. The address mapping information points to a predetermined storage area in the memory space of the corresponding target healthy GPU node.

[0105] In step 401, the first data transmission unit is an independent data block formed by dividing the final incremental data into fixed sizes. The address mapping information is information used to indicate the exact location where the data block should be stored in the target node's video memory. The predetermined storage area is a dedicated memory address range in the target GPU video memory that is pre-divided for receiving specific types of data.

[0106] In this embodiment, the final incremental data is first read from the memory space of the source GPU node and divided into multiple consecutive first data transmission units according to the preset block size. Then, an address mapping information is generated and attached to each transmission unit. This information accurately points to the corresponding storage area that has been pre-planned in the memory of the target healthy GPU node, thereby making full preparations for subsequent direct memory write operations through the RDMA channel.

[0107] Step 402: Through the RDMA write operation of the RDMA network, the data transmission unit with address mapping information is directly written to the corresponding storage location in the video memory space of the target healthy GPU node.

[0108] Step 403: Simultaneously, read the latest incremental data from the local high-speed access medium, divide the latest incremental data into multiple second data transmission units, attach corresponding address mapping information to each second data transmission unit, and transmit the second data transmission unit to the backup storage area of ​​the target healthy GPU node's memory space through RDMA one-sided operation of the RDMA network.

[0109] In this embodiment, steps 402 and 403 can be executed concurrently. Specifically, during the transmission process, two data transmission tasks are executed concurrently. One task is to write the first data transmission units, which already have address mapping information attached, directly from the source node's video memory to a predetermined storage area in the target node's video memory through an RDMA write operation. At the same time, the other task is to read the latest incremental data of the hot backup storage from the source node's local high-speed access medium, divide this data into multiple second data transmission units, and attach address mapping information pointing to the target node's backup storage area to each unit. Finally, these second data transmission units are also transmitted to the target node through a one-sided RDMA operation.

[0110] As a specific embodiment, steps 404 and 405 can be executed after step 403. Step 404: In any data transmission process, the integrity of each data transmission unit is verified by the verification mechanism of the RDMA network, and a retransmission mechanism is initiated for data transmission units that fail to transmit.

[0111] In step 404, the verification mechanism is a technical means to verify whether the data remains intact and error-free during transmission; the retransmission mechanism is a guarantee process that automatically retransmits the data unit when a data transmission failure or error is detected.

[0112] In this embodiment of the application, whenever a first data transmission unit or a second data transmission unit completes transmission through the RDMA network, the receiver will use a preset verification mechanism to verify whether the data content it receives is completely consistent with the original data sent. If the verification fails, the corresponding retransmission mechanism will be automatically triggered to retransmit the specific data unit that failed until it is successfully received by the target node and passes the integrity verification.

[0113] Step 405: After all data transmission units have been successfully written to the designated location in the memory space of the target healthy GPU node, a transmission completion confirmation signal is generated and the transmission completion confirmation signal is returned to the source GPU node.

[0114] In this embodiment of the application, when the target healthy GPU node confirms that all first data transmission units and second data transmission units have passed the integrity check and have been successfully written to the specified location of its video memory, it will generate a transmission completion confirmation signal and then return the signal to the source GPU node through the network, which marks the successful completion of this batch data transmission phase.

[0115] In this embodiment of the application, the process improves data migration efficiency by establishing a high-speed direct connection channel and a concurrent transmission mechanism, and ensures the reliability of data transmission by combining integrity verification and automatic retransmission, thereby laying a solid data foundation for achieving rapid task recovery.

[0116] To further improve the efficiency and accuracy of incremental checkpoint generation, in some embodiments, in step 101, an incremental difference algorithm is used to determine key data blocks between the training task state of the current checkpoint and the training task state of the previous checkpoint. Based on the key data blocks, difference data is generated, including:

[0117] Step 501: Divide the training task status of the current checkpoint into multiple data blocks according to storage units of fixed size.

[0118] In step 501, the fixed-size storage unit is a predefined basic unit used to divide the training task state data.

[0119] In this embodiment, the training task status of the current checkpoint is read. This status includes complete information such as model parameters and optimizer status. Then, this continuous status data is cut into multiple continuous data blocks according to a preset fixed size to prepare for subsequent fine-grained comparison.

[0120] Step 502: Calculate a unique identifier for each data block. Use the incremental difference algorithm to compare the unique identifier of each data block in the current checkpoint with the unique identifier of the corresponding data block in the previous checkpoint item by item to identify data blocks with inconsistent unique identifiers and mark the data blocks with inconsistent unique identifiers as candidate data blocks.

[0121] In step 502, the unique identifier is a short digital fingerprint that can uniquely represent the content of the data block by calculating the data block content. The candidate data block is a data block that is found after preliminary comparison, whose identifier is inconsistent but whose content has not yet been confirmed.

[0122] In this embodiment of the application, a unique identifier code that can sensitively reflect any changes in the content of the data block is calculated for each data block obtained in step 501. Then, the identifier codes of all data blocks at the current checkpoint are compared one by one with the identifier codes of the data blocks at the corresponding positions of the previous checkpoint, and all data blocks with inconsistent identifier codes are initially screened out and marked as candidate data blocks.

[0123] Step 503: Verify the candidate data blocks using a convolutional neural network. Based on the verification results, filter out data blocks with substantially the same content, mark the remaining data blocks as key data blocks, and record the storage location information of each key data block in the GPU memory.

[0124] In step 503, data blocks with substantially the same content refer to those data blocks whose unique identifiers may be inconsistent due to minor reasons such as changes in metadata, but whose actual effective data content is exactly the same. This is determined by extracting deep features of the data blocks through a convolutional neural network and calculating similarity.

[0125] In this embodiment of the application, the candidate data blocks obtained in step 502 are paired with the original data blocks corresponding to them in the previous checkpoint and then input together into a pre-trained convolutional neural network for content analysis. The network compares the actual content of the two data blocks and outputs a quantitative score representing the similarity between the two contents. Based on this similarity score, candidate data blocks that are only different in identifier but have the same content can be filtered out, and those data blocks that have been confirmed to have undergone real content changes are marked as key data blocks. At the same time as marking, the specific location information of each key data block in the GPU memory is recorded.

[0126] Step 504: Extract the original content data of the key data blocks from the GPU memory, and sort and combine the original content data of the key data blocks according to the storage location information.

[0127] In step 504, the raw content data is the unprocessed raw byte information of the key data block stored in the GPU memory.

[0128] In this embodiment of the application, based on the storage location information recorded in step 503, the original content data of all key data blocks are directly read from the GPU memory, and then arranged and spliced ​​according to the inherent logical order of these key data blocks in the original training task state data, and finally a continuous data stream containing all real change data content is formed.

[0129] Step 505: Combine the sorting and combination results with the corresponding storage location information to generate difference data.

[0130] In this embodiment, the ordered original content data obtained in step 504 is packaged and combined with the corresponding storage location information recorded in step 503 to generate a final difference data. This data not only includes which specific data content has changed, but also clearly records the precise location of these changed data in the GPU memory, thus providing a complete basis for subsequent accurate reconstruction and restoration of the task state. The reason for combining the sorted and combined content data with the storage location information is that sorting and combining the data content alone can only reflect "what has changed in the data", while it is necessary to combine it with its storage location information in order to accurately restore the data to the original memory layout during state reconstruction. The combination of the two ultimately forms a complete and self-consistent difference dataset.

[0131] In this embodiment of the application, the process involves a two-stage process combining coarse screening and fine judgment. This ensures accurate capture of all real state changes while effectively filtering out spurious changes caused by irrelevant fluctuations, thereby generating high-fidelity, small-volume difference data. This lays a solid foundation for achieving efficient and low-overhead incremental checkpoints.

[0132] To further improve data compression efficiency and storage reliability, in some embodiments, step 102: performing lightweight compression on the differential data to form compressed incremental data, and storing the compressed incremental data in a multi-level storage system, including:

[0133] Step 601: Compress the difference data to identify duplicate byte sequences in the difference data, and replace the duplicate byte sequences with a preset duplicate marker.

[0134] In this embodiment, the differential data is input into an attention-based neural network for processing. The network first analyzes the importance of different components in the data and simultaneously identifies the recurring byte sequences. Then, it replaces these recurring sequences with a set of preset unique tags that are much shorter than the original recurring sequences, thereby achieving initial compression of the original differential data.

[0135] Step 602: Perform entropy encoding on the replaced data to form compressed incremental data.

[0136] In this embodiment of the application, the frequency of occurrence of each different symbol in the data after the repeated sequence replacement process is counted, and then an appropriate length of code is assigned to each symbol according to the frequency. Specifically, the symbols with higher frequency will be assigned shorter codes, and finally all the original data are converted into this frequency-based variable-length code form, thereby generating compressed incremental data with a significantly reduced volume.

[0137] Step 603: Simultaneously write the newly generated compressed incremental data to two storage locations in the multi-level storage system: the local high-speed access medium as a hot backup and the temporary storage area as a standby node.

[0138] In this embodiment of the application, the compressed incremental data generated in step 602 is simultaneously written to two separate operations: first, it is immediately written to a local high-speed solid-state drive as a hot backup that can be read quickly; second, it is sent to another pre-configured backup node via the network and stored in a temporary storage area specified by that node, thereby achieving real-time off-site backup of the data.

[0139] Step 604: In the temporary storage area, when the cumulative number of compressed incremental data reaches a preset threshold, a full checkpoint integration process is triggered. The full checkpoint integration process includes: using a generative adversarial network to verify data integrity; reading basic full checkpoint data based on the integrity verification result; and merging and reconstructing multiple compressed incremental data with the basic full checkpoint data to generate a new complete state snapshot.

[0140] In this embodiment, the amount of compressed incremental data in the temporary storage area of ​​the backup node is continuously monitored. When the amount reaches a preset threshold, the integration process is automatically triggered. The process first uses a generative adversarial network to verify the integrity of all incremental data to be integrated, ensuring that the data has not been corrupted or tampered with. After the verification is passed, an earlier saved basic full checkpoint data is read, and then all accumulated incremental data are applied in sequence. The most accurate complete task state at the current moment is reconstructed by calculation as a new complete state snapshot.

[0141] Step 605: Convert the new full state snapshot into an integrated full checkpoint and write the integrated full checkpoint to the distributed file system.

[0142] In this embodiment, the newly generated complete state snapshot is converted into a standardized storage format to form a full checkpoint after integration. This complete data packet is then written to a distributed file system for permanent storage as a reliable cold backup. At the same time, the integrated incremental data is cleaned up as needed to free up storage space.

[0143] In this embodiment of the application, the process reduces storage overhead through intelligent compression technology, balances recovery speed and data security through multi-level backup strategy, and ensures the accuracy of data reconstruction through integrity verification, thereby constructing an efficient, reliable, and hierarchical checkpoint data storage and management system.

[0144] To further improve the accuracy and completeness of task recovery, in some embodiments, step 106: on the target healthy GPU node, using the final incremental data and the latest incremental data in the hot backup, reconstructs the complete context state of the training task to restore the execution state of the training task, including:

[0145] Step 701: In the target healthy GPU node, create a state recovery workspace, and load the final incremental data and the latest incremental data in the hot backup into the state recovery workspace after sorting them according to the address mapping information.

[0146] In this embodiment, the target healthy GPU node first allocates a dedicated state recovery work area in its video memory, and then sorts these data blocks according to their logical order in the original video memory based on the address mapping information attached to the final incremental data and the latest incremental data in the hot backup, and loads them into the work area in sequence.

[0147] Step 702: Reorganize the storage location information in the sorted checkpoint data to restore the execution environment of the training task, re-establish the CUDA kernel execution context, and set the corresponding training iteration counter and learning rate scheduler state to obtain the reconstructed complete context state.

[0148] The CUDA kernel is a parallel computing function that runs on the GPU, while the operating system kernel is the core software that manages computer hardware.

[0149] In a specific implementation, step 702 includes the following steps:

[0150] Step a1: Reorganize the storage location information in the sorted checkpoint data to obtain a complete set of model parameters and a complete set of optimizer states.

[0151] In this embodiment, the storage location information of all data blocks in the state recovery work area is read, and the data blocks scattered in the video memory are reassembled into a continuous and complete data structure based on these address information, thereby recovering the complete set of model parameters and the complete set of optimizer states required for the training task.

[0152] Step a2: Based on the complete set of model parameters, reconstruct the GPU computation graph structure to establish a computation pipeline that includes forward computation and backward propagation.

[0153] In this embodiment of the application, based on the recovered complete set of model parameters, the computational graph structure of the model is reconstructed to clarify the connection relationship and specific computation order between the layers in the model; thereby establishing a complete computational pipeline that includes a forward computation path from input to output and a backward propagation path from loss backpropagation to parameter update.

[0154] Step a3: Based on the complete set of optimizer states, reinitialize the optimizer's internal states.

[0155] In this embodiment, the complete set of optimizer states is read, and internal state variables such as momentum buffer and gradient squared accumulator are reloaded into the optimizer, so that the internal state of the optimizer is accurately restored to the precise state before the transition.

[0156] Step a4: Based on the computational pipeline and the internal state of the optimizer, restore the execution environment of the training task, re-establish the CUDA kernel execution context, and set the corresponding training iteration counter and learning rate scheduler state to obtain the reconstructed complete context state.

[0157] In this embodiment, based on the establishment of the computational pipeline and the restoration of the optimizer's internal state, the CUDA execution environment is further reinitialized and the GPU's computational context is restored. At the same time, the states of the training iteration counter and the learning rate scheduler are precisely set to the values ​​before the migration. Through the above steps, the entire training task execution environment is finally reconstructed, resulting in a reconstructed complete context state that is completely consistent with that before the migration.

[0158] Step 706: Verify the consistency between the reconstructed complete context state and the source GPU node through a preset verification mechanism. After confirming the consistency, continue training computation from the iteration point where the training task was interrupted.

[0159] In step 706, the preset verification mechanism refers to the pre-defined state verification method. The specific content of the verification mechanism includes verifying and comparing key data such as model parameters, optimizer state, and training progress. The iteration point refers to a specific iteration step in the training process. The iteration point contains complete state information such as model parameters, optimizer state, and training progress at that time.

[0160] In this embodiment of the application, the process of this step ensures the integrity and accuracy of the task state after migration through a systematic state reorganization and environment reconstruction process, thereby providing a reliable guarantee for the high availability of GPU computing tasks.

[0161] Figure 3 This application provides a schematic diagram of the structure of a real-time fault-tolerant and task migration system for procedural GPUs, as shown in the embodiments of this application. Figure 3As shown, the detailed implementation section describes:

[0162] The determination module 31 is used to capture the training task state in the GPU memory, use the incremental difference algorithm to determine the key data block between the training task state at the current checkpoint and the training task state at the previous checkpoint, and generate difference data based on the key data block.

[0163] Compression module 32 is used to perform lightweight compression on the differential data to form compressed incremental data, and store the compressed incremental data in a multi-level storage system.

[0164] The acquisition module 33 is used to collect multi-dimensional health indicators of the current GPU node in real time while storing compressed incremental data, and input the multi-dimensional health indicators into the time series prediction model for joint analysis. When the time series prediction model predicts that the failure risk of the current GPU node is greater than the preset risk threshold, or receives a preemption warning signal sent by the cluster scheduler, the task migration process is triggered.

[0165] The generation module 34 is used to pause the training task on the current GPU node and trigger the final operation to generate the final incremental data during the task migration process.

[0166] The allocation module 35 is used to transfer the final incremental data and the latest incremental data in the hot backup of the multi-level storage system to the pre-allocated target healthy GPU node via the RDMA network.

[0167] The reconstruction module 36 is used to reconstruct the complete context state of the training task on the target healthy GPU node using the final incremental data and the latest incremental data in the hot backup, so as to restore the execution state of the training task and realize real-time fault tolerance and task migration for procedural GPUs.

[0168] The real-time fault-tolerant and task migration system for procedural GPUs in this application is used to implement the aforementioned real-time fault-tolerant and task migration method for procedural GPUs. Therefore, the specific implementation of the real-time fault-tolerant and task migration system for procedural GPUs can be found in the embodiment section of the real-time fault-tolerant and task migration method for procedural GPUs above. The specific implementation can be referred to the description of the corresponding embodiments, and will not be repeated here.

[0169] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of any of the above-described procedural GPU real-time fault tolerance and task migration methods.

[0170] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the above-described methods for real-time fault tolerance and task migration for procedural GPUs.

[0171] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as USB flash drives, read-only memory, random access memory, portable hard drives, magnetic disks, or optical disks.

[0172] The embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above embodiments of the real-time fault-tolerant and task migration method for procedural GPUs.

[0173] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0174] The above provides a detailed description of a real-time fault-tolerant and task migration system and method for procedural GPUs provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A real-time fault-tolerant and task migration method for a process-oriented GPU, characterized in that, include: The training task state in the GPU memory is captured, and the key data block between the training task state at the current checkpoint and the training task state at the previous checkpoint is determined using the incremental difference algorithm. Based on the key data block, difference data is generated. The differential data is lightly compressed to form compressed incremental data, and the compressed incremental data is stored in a multi-level storage system. While storing compressed incremental data, multi-dimensional health indicators of the current GPU node are collected in real time, and the multi-dimensional health indicators are input into the time series prediction model for joint analysis. When the time series prediction model predicts that the failure risk of the current GPU node is greater than the preset risk threshold, or when it receives a preemption warning signal sent by the cluster scheduler, the task migration process is triggered. In the task migration process, the training task on the current GPU node is paused, the final operation is triggered, and the final incremental data is generated. The final incremental data and the latest incremental data in the hot backup of the multi-level storage system are transferred to the pre-assigned target healthy GPU node via the RDMA network. On the target healthy GPU node, the complete context state of the training task is reconstructed using the final incremental data and the latest incremental data in the hot backup, so as to restore the execution state of the training task and realize real-time fault tolerance and task migration for procedural GPUs. The multi-dimensional health indicators are input into the time-series prediction model for joint analysis. When the time-series prediction model predicts that the failure risk of the current GPU node is greater than a preset risk threshold, or when it receives a preemption warning signal sent by the cluster scheduler, the task migration process is triggered, including: The multi-dimensional health indicators are organized into an ordered data sequence according to the time series, and the multi-dimensional health indicators include hardware indicators, software indicators and system indicators; The ordered data sequence is input into a time-series prediction model based on the Transformer architecture. The time-series prediction model analyzes the ordered data sequence and outputs an evaluation result representing the current health status of the GPU node. When the evaluation result exceeds the preset risk threshold a set number of times, the correlation between the current GPU node and other GPU nodes is analyzed through a graph neural network to generate a node failure warning. When a resource preemption command is received through the cluster scheduler interface, the resource preemption command is parsed and a preemption warning signal is generated. After generating a node failure warning or preemption warning signal, the task migration preparation process is initiated. In the task migration preparation process, a resource reservation request is sent to the target healthy GPU node, and the task migration process is triggered after receiving the resource confirmation response from the target healthy GPU node.

2. The method of claim 1, wherein, The process of inputting the ordered data sequence into a Transformer-based temporal prediction model, analyzing the ordered data sequence through the temporal prediction model, and outputting an evaluation result characterizing the current health status of the GPU node includes: The ordered data sequence is input into a time series prediction model based on the Transformer architecture. The encoder of the time series prediction model calculates the correlation weights between data at different time steps in the ordered data sequence. Based on the correlation weights, combined with the forward propagation algorithm in the self-attention mechanism, a global feature representation is generated. The ordered data sequence is subjected to dilated convolution processing by the temporal convolution module of the temporal prediction model to obtain local context features. The feature fusion module of the time-series prediction model performs weighted fusion of the global feature representation and the local context features to form a health status representation vector. The health status representation vector is mapped to a fault risk probability value through a multilayer perceptron via the prediction head module of the time series prediction model. The trend analysis module of the time series prediction model performs trend analysis on the fault risk probability value based on a sliding time window. When the fault risk probability value is detected to be rising continuously and the fluctuation range exceeds the preset range, a health warning result is generated. The output module of the time-series prediction model combines the fault risk probability value with the health warning result to output an evaluation result representing the current health status of the GPU node.

3. The method of claim 1, wherein, The step of transferring the final incremental data and the latest incremental data from the hot backup of the multi-level storage system to the pre-allocated target healthy GPU node via the RDMA network includes: In the memory space of the source GPU node, the final incremental data is divided into multiple first data transmission units, and corresponding address mapping information is attached to each first data transmission unit. Through the RDMA write operation of the RDMA network, the data transmission unit with address mapping information is directly written to the corresponding storage location in the video memory space of the target healthy GPU node. Simultaneously, the latest incremental data is read from the local high-speed access medium, the latest incremental data is divided into multiple second data transmission units, and corresponding address mapping information is attached to each second data transmission unit. The second data transmission unit is then transmitted to the backup storage area of ​​the target healthy GPU node's memory space through the RDMA one-sided operation of the RDMA network.

4. The method of claim 1, wherein, The step involves using an incremental difference algorithm to determine key data blocks between the training task state at the current checkpoint and the training task state at the previous checkpoint, and generating difference data based on these key data blocks, including: The training task state at the current checkpoint is divided into multiple data blocks according to a fixed-size storage unit; A unique identifier is calculated for each data block. The incremental difference algorithm is used to compare the unique identifier of each data block in the current checkpoint with the unique identifier of the corresponding data block in the previous checkpoint item by item to identify data blocks with inconsistent unique identifiers and mark the data blocks with inconsistent unique identifiers as candidate data blocks. The candidate data blocks are verified by a convolutional neural network. Based on the verification results, data blocks with essentially the same content are filtered out, the remaining data blocks are marked as key data blocks, and the storage location information of each key data block in the GPU memory is recorded. Extract the original content data of key data blocks from the GPU memory, and sort and combine the original content data of the key data blocks according to the storage location information; The sorting and combination results are combined with the corresponding storage location information to generate differential data.

5. The method of claim 1, wherein, The step of performing lightweight compression on the differential data to form compressed incremental data, and storing the compressed incremental data in a multi-level storage system, includes: The difference data is compressed to identify duplicate byte sequences in the difference data, and the duplicate byte sequences are replaced using a preset duplicate marker; The replaced data is entropy encoded to form compressed incremental data. The newly generated compressed incremental data is written simultaneously to two storage locations in the multi-level storage system: a local high-speed access medium for hot backup and a temporary storage area for standby nodes. In the temporary storage area, when the cumulative number of compressed incremental data reaches a preset threshold, a full checkpoint integration process is triggered. The full checkpoint integration process includes: using a generative adversarial network to verify data integrity; reading basic full checkpoint data based on the integrity verification result; and merging and reconstructing multiple compressed incremental data with the basic full checkpoint data to generate a new complete state snapshot. The new full state snapshot is converted into an integrated full checkpoint, and the integrated full checkpoint is written to the distributed file system.

6. The method of claim 1, wherein, The step of reconstructing the complete context state of the training task on the target healthy GPU node using the final incremental data and the latest incremental data in the hot backup to restore the execution state of the training task includes: In the target healthy GPU node, a state recovery workspace is created, and the final incremental data and the latest incremental data in the hot backup are sorted according to the address mapping information and loaded into the state recovery workspace. The storage location information in the sorted checkpoint data is reorganized to restore the execution environment of the training task, re-establish the CUDA kernel execution context, and set the corresponding training iteration counter and learning rate scheduler state to obtain the reconstructed complete context state. The consistency between the reconstructed complete context state and the source GPU node is verified by a preset verification mechanism. Once the consistency is confirmed, the training computation continues from the iteration point where the training task was interrupted.

7. A real-time fault-tolerant and task migration system for a process-oriented GPU, characterized in that, include: The determination module is used to capture the training task state in the GPU memory, use the incremental difference algorithm to determine the key data block between the training task state at the current checkpoint and the training task state at the previous checkpoint, and generate difference data based on the key data block. A compression module is used to perform lightweight compression on the differential data to form compressed incremental data, and to store the compressed incremental data in a multi-level storage system. The acquisition module is used to collect multi-dimensional health indicators of the current GPU node in real time while storing compressed incremental data, and input the multi-dimensional health indicators into the time series prediction model for joint analysis. When the time series prediction model predicts that the failure risk of the current GPU node is greater than the preset risk threshold, or receives a preemption warning signal sent by the cluster scheduler, the task migration process is triggered. The generation module is used to pause the training task on the current GPU node, trigger the final operation, and generate the final incremental data during the task migration process. The allocation module is used to transfer the final incremental data and the latest incremental data in the hot backup of the multi-level storage system to the pre-allocated target healthy GPU node via the RDMA network. The reconstruction module is used to reconstruct the complete context state of the training task on the target healthy GPU node using the final incremental data and the latest incremental data in the hot backup, so as to restore the execution state of the training task and realize real-time fault tolerance and task migration for procedural GPUs. The multi-dimensional health indicators are input into the time-series prediction model for joint analysis. When the time-series prediction model predicts that the failure risk of the current GPU node is greater than a preset risk threshold, or when it receives a preemption warning signal sent by the cluster scheduler, the task migration process is triggered, including: The multi-dimensional health indicators are organized into an ordered data sequence according to the time series, and the multi-dimensional health indicators include hardware indicators, software indicators and system indicators; The ordered data sequence is input into a time-series prediction model based on the Transformer architecture. The time-series prediction model analyzes the ordered data sequence and outputs an evaluation result representing the current health status of the GPU node. When the evaluation result exceeds the preset risk threshold a set number of times, the correlation between the current GPU node and other GPU nodes is analyzed through a graph neural network to generate a node failure warning. When a resource preemption command is received through the cluster scheduler interface, the resource preemption command is parsed and a preemption warning signal is generated. After generating a node failure warning or preemption warning signal, the task migration preparation process is initiated. In the task migration preparation process, a resource reservation request is sent to the target healthy GPU node, and the task migration process is triggered after receiving the resource confirmation response from the target healthy GPU node.

8. An electronic device, comprising: include: Memory, used to store computer programs; A processor, configured to implement the steps of the real-time fault-tolerant and task migration method for procedural GPUs as described in any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, enables the implementation of the real-time fault-tolerant and task migration method for procedural GPUs as described in any one of claims 1 to 6.

Images