Fault Tolerance Recovery Method and System for Ultra-Mega-Card-Level Training and Inference Tasks

Abstract

The present invention relates to the field of gradient reconstruction technology, and in particular to a fault-tolerant recovery method and system for super-10,000-card-level training and inference tasks. The present invention proposes the following scheme: training data is managed through a hierarchical storage structure, and global gradient backtracking is used to perform gradient recovery when a computing node fails. During the recovery process, a dynamic time window is set based on the stored gradient snapshot, and the gradient recovery accuracy is optimized by adjusting the window length. When the backtracking fails, a micro-checkpoint rollback is used to ensure the stable recovery of the training process. Based on the causal dependency chain and the gradient interpolation regression method, version control gradient replay is performed on the critical path, and interpolation regression recovery is used for non-critical paths. Finally, the gradient recovery accuracy is improved through weighted correction. It can reduce the gradient loss caused by computing node failure and improve the fault tolerance and computing efficiency of ultra-large-scale training tasks.

Description

Technical Field

[0001] The present invention relates to the field of gradient reconstruction technology, and in particular to a fault-tolerant recovery method and system for multi-million-card-level training and inference tasks. Background Art

[0002] The training of ultra-large-scale deep learning models demands enormous computing resources, typically using distributed parallel computing. However, as the computational scale increases, the failure rate of compute nodes also increases, potentially leading to loss of training gradients, parameter desynchronization, and even training interruption. Existing training recovery methods primarily rely on periodically stored checkpoints for rollback or use interpolation methods to estimate lost gradients. However, these methods have significant shortcomings in ultra-large-scale training tasks. For example, checkpoint rollbacks result in extensive repeated computations, reducing training efficiency, while gradient interpolation methods struggle to ensure the accuracy of recovered gradients, potentially leading to unstable model convergence. Furthermore, existing methods lack effective tracking of gradient dependencies during training, making it impossible to accurately recover critical gradients, resulting in degraded training performance. Therefore, there is an urgent need for an efficient fault-tolerant recovery method that can accurately recover lost gradients in the event of a compute node failure and dynamically adjust the recovery strategy to minimize training interruptions and improve model training stability and computational efficiency.

[0003] The existing technologies all have the problems raised by this background technology: in ultra-large-scale model training, there are problems such as inaccurate gradient recovery, high rollback cost, missing key gradient dependencies and serious impact of training interruptions. It is difficult to effectively ensure efficient fault-tolerant recovery after computing node failure. To solve the above problems, this application designs a fault-tolerant recovery method and system for ultra-10,000-card-level training and inference tasks. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to address the deficiencies of the existing technology and provide a fault-tolerant recovery method and system for super-10,000-card-level training and inference tasks. The method manages training data through a hierarchical storage structure and uses global gradient backtracking to perform gradient recovery when a computing node fails. During the recovery process, a dynamic time window is set based on the stored gradient snapshot, and the gradient recovery accuracy is optimized by adjusting the window length. When backtracking fails, a micro-checkpoint rollback is used to ensure the stable recovery of the training process. Based on the causal dependency chain and gradient interpolation regression method, version control gradient replay is performed on the critical path, and interpolation regression recovery is used for non-critical paths. Finally, the gradient recovery accuracy is improved through weighted correction. It can reduce the gradient loss caused by computing node failure and improve the fault tolerance and computing efficiency of ultra-large-scale training tasks.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] A fault-tolerant recovery method for multi-million-card-scale training and inference tasks, applied to the training phase of large cross-modal models, includes:

[0007] Establish a hierarchical storage structure for computing nodes at the beginning of training;

[0008] When the computing node fails, the training gradient is restored by global gradient backtracking according to the hierarchical storage structure;

[0009] If the rollback fails, roll back to the micro-checkpoint for recovery.

[0010] The hierarchical storage structure is implemented through incremental gradient graphs and dynamic causal checkpoints. The hierarchical storage structure includes a high-speed layer and a low-speed layer, including:

[0011] Preset continuous time windows according to training time, and generate a corresponding gradient map for each window, wherein the gradient map includes map nodes, map edges and map metadata;

[0012] Identifying critical paths and non-critical paths based on the causal relationship of the gradient map, wherein the identification of the paths is determined based on the correction amplitude of the gradient paths;

[0013] Checkpoints are arranged according to the path attributes, and the gradient data collected at the checkpoints are stored in the corresponding high-speed layer and low-speed layer.

[0014] According to the causal relationship of the gradient map, the critical path and the non-critical path are identified, including:

[0015] Calculating a correction amplitude for each path in the gradient map within its continuous time window, wherein the correction amplitude represents the cumulative energy of the gradient changes of all computational nodes on the path;

[0016] Assigning a value to the correction amplitude according to the in-degree and out-degree of the path in the gradient amplitude;

[0017] The paths are sorted according to the assignment results, and the critical paths and non-critical paths are identified based on dynamic thresholds.

[0018] Arrange checkpoints according to path attributes, and store the gradient data collected at the checkpoints in the corresponding high-speed layer and low-speed layer, including:

[0019] Arranging high-frequency micro-checkpoints on the critical path, wherein the high-frequency micro-checkpoints are set according to the gradient change rate of the critical path;

[0020] Low-frequency micro-checkpoints are arranged on the non-critical path, wherein the low-frequency micro-checkpoints are set according to the gradient change rate of the non-critical path.

[0021] The recovery of the training gradient by global gradient backtracking includes:

[0022] Reading a first gradient snapshot corresponding to a first time window from the hierarchical storage structure, wherein the first time window represents a time window corresponding to a computing node failure, and the first gradient snapshot represents storage data of a previous micro-checkpoint in the first time window;

[0023] Setting a second time window between the previous micro-checkpoint and the first time window based on the gradient change rate of the first gradient snapshot, and restoring the second time window based on the gradient flow of the first gradient snapshot, wherein the left interval of the second time window is set as the previous micro-checkpoint;

[0024] Shifting the second time window toward the first time window by a set length, and restoring a third time window within the set length, wherein the set length is obtained based on the gradient change rate of the second time window;

[0025] Determine the second-order derivatives of the gradient at both ends of the right interval of the current second time window. If the termination condition is not met, continue to advance the set length until the termination condition is met.

[0026] The second time window is extended to be flush with the left interval of the first time window to restore the training gradient within the first time window.

[0027] Restoring the second time window according to the gradient flow of the first gradient snapshot includes:

[0028] calculating a gradient flow direction and a rate of the first gradient snapshot;

[0029] calculating a gradient update trend through flow dynamics according to the flow direction and rate;

[0030] According to the gradient update trend, incremental correction is performed starting from the left interval of the second time window until the correction of the right interval of the second time window is completed.

[0031] Recover the third time window within the set length, including:

[0032] Loading a gradient map based on the second time window before the shift, processing the gradient map by low-rank approximation, and generating a reconstructed gradient of the third time window;

[0033] Calculate the local consistent hash of the reconstructed gradient and the global parameters. If the hashes match, restore the third time window based on the reconstructed gradient. If they do not match, recalculate the set length of the second time window based on the local consistent hash, where the global parameters represent model parameters that are globally synchronized during the training process.

[0034] Restoring the third time window according to the reconstruction gradient includes:

[0035] Load the causal dependency chain of the critical path in the second time window from the high-speed layer, replay the gradient aggregation operation according to the version number sequence, and restore the gradient information of the critical path corresponding to the third time window;

[0036] Loading the gradient information of the non-critical path in the second time window from the low-speed layer, and restoring the gradient information of the critical path corresponding to the third time window based on gradient interpolation and linear regression;

[0037] The reconstructed gradient is weightedly corrected according to the restoration result.

[0038] The termination conditions include:

[0039] The second-order derivative of the left gradient is less than the set threshold and the first-order derivative tends to zero, while the second-order derivative of the right gradient is greater than the set threshold.

[0040] A fault-tolerant recovery system for multi-million-card training and inference tasks, comprising a preprocessing module and a gradient recovery module;

[0041] The preprocessing module is used to build a hierarchical storage structure of computing nodes based on training data and network topology before the training task begins, initialize relevant data of each time window, and generate a preliminary gradient map and checkpoint configuration;

[0042] The gradient recovery module is used to recover the lost gradients of the training task when a failure occurs.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] This method accurately recovers lost gradients in the event of a compute node failure by constructing a hierarchical storage structure, implementing global gradient rollback, and dynamic time window adjustment. It also combines incremental gradient graph analysis to analyze critical and non-critical paths, employing causal dependency chains for version-controlled replay of critical gradients. It also employs gradient interpolation and regression analysis to recover non-critical path gradients, ultimately optimizing gradient consistency through weighted correction. Compared to existing methods, this method reduces unnecessary global rollbacks, improves the accuracy of gradient recovery, reduces computational resource waste, and enhances the stability and computational efficiency of ultra-large-scale training tasks, thereby ensuring the continuity of the training process and efficient model convergence. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0046] Figure 1This is a flowchart of a fault-tolerant recovery method for super-10,000-card-level training and inference tasks according to Example 1 of the present invention;

[0047] Figure 2 A flowchart for constructing a hierarchical storage structure according to embodiment 1 of the present invention;

[0048] Figure 3 This is a schematic diagram of time window division in Example 1 of the present invention;

[0049] Figure 4 This is a schematic diagram of the time window shift in Example 1 of the present invention;

[0050] Figure 5 This is a schematic diagram of termination determination in Example 1 of the present invention;

[0051] Figure 6 This is a module diagram of the fault-tolerant recovery system for super-10,000-card-level training and inference tasks in Example 2 of the present invention. DETAILED DESCRIPTION

[0052] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0053] Example 1:

[0054] See also Figure 1 The present invention provides an embodiment: a fault-tolerant recovery method for multi-million-card-level training and inference tasks, which is applied to the training phase of a large cross-modal model. The specific steps are as follows:

[0055] S1: Establish a hierarchical storage structure of computing nodes at the beginning of training;

[0056] In this embodiment, a hierarchical storage structure is divided into high-speed and low-speed layers. Gradient information that requires frequent access and updates during training is stored in the high-speed layer, while data with a longer history is stored in the low-speed layer. This optimizes data storage and access, reduces hardware resource consumption, ensures efficient execution of training tasks, and effectively alleviates storage pressure on computing nodes, especially for large datasets.

[0057] The purpose is to ensure efficient data access during training, while avoiding the impact of storage bottlenecks and improving the speed and stability of overall training.

[0058] S2: When the computing node fails, the training gradient is restored through global gradient backtracking according to the hierarchical storage structure;

[0059] In this embodiment, if a compute node fails, the most recent gradient snapshot is read from the hierarchical storage structure and global gradient backtracking is performed based on this snapshot. This backtracking technology accurately restores the training state before the failure, ensuring the accuracy of the recovery process. The gradient backtracking process incorporates the causal relationship of the gradient graph and dynamically adjusts the length of the backtracking window to make the recovery operation more precise, thereby minimizing the loss of gradient information caused by compute node failures.

[0060] S3: If the backtracking fails, roll back to the micro-checkpoint.

[0061] In this embodiment, micro-checkpoints are snapshots of the state saved periodically during training. Rolling back to these micro-checkpoints effectively avoids excessive loss of training information and minimizes recovery time. The micro-checkpoint rollback design provides a reliable recovery point in the event of a rollback failure, ensuring that model training can be restored to a normal state within a reasonable timeframe.

[0062] This embodiment addresses the fault-tolerant recovery issues in training and inference tasks at the 10,000-card level, especially in large-scale cross-modal large-model training, and solves the problems of training data loss and difficulty in recovering training gradients due to computing node failures.

[0063] Specifically, traditional training recovery methods typically rely on simple rollback mechanisms (such as rolling back to the most recent checkpoint) or incremental gradient interpolation to recover lost data when facing compute node failures. However, these methods often fail to effectively recover high-quality training gradients in large-scale model training, especially when involving ultra-large-scale parameter updates. They may even lead to larger training errors, reducing the efficiency and accuracy of model convergence.

[0064] In this embodiment, by establishing a global gradient-based backtracking mechanism, when a computing node fails, the global gradient backtracking will first be performed based on the gradient snapshot and time window saved in the hierarchical storage structure. By calculating the gradient change rate, not only can the training state before the failure be restored, but the length of the recovery time window can also be automatically adjusted according to the gradient flow trend at different training stages. The innovation of this application is that traditional technologies often use a fixed preset window length, while this application can more accurately capture the gradient information before the training interruption by dynamically adjusting the window length, thereby reducing training errors and information loss.

[0065] For example, during the training of a large cross-modal model, when a computing node is interrupted due to hardware failure or communication problems, the traditional recovery method may backtrack part of the training data based on a fixed time window, which often cannot effectively restore to the optimal state. This application adopts a recovery technology based on global gradient backtracking. First, it determines the gradient change of the current time window, and then automatically adjusts the backtracking length according to the gradient change rate. It can even dynamically shift the window to accurately restore the training state. In this way, not only can the common problems of gradient explosion or gradient disappearance be avoided, but also the losses caused by training interruptions can be minimized, ensuring the efficient recovery of training tasks.

[0066] Furthermore, when global gradient backtracking fails, this application does not completely interrupt the training task. Instead, it rolls back the training process to the nearest micro-checkpoint, thereby quickly restoring the training state. Compared to the existing technology of simply rolling back to a fixed checkpoint, this application uses frequent micro-checkpoint collection to select the valid checkpoint closest to the current training state for rollback each time a failure occurs, thereby reducing the risk of training data loss and improving the accuracy of recovery.

[0067] In summary, the fault-tolerant recovery method for training and inference tasks at the scale of over 10,000 cards provided by this application not only theoretically fills the gaps in existing technologies for large-scale training tasks, but also significantly improves the fault tolerance and recovery accuracy of large-scale model training in practice. By combining dynamically adjusted time windows with micro-checkpoint rollbacks, this application provides an efficient and reliable fault-tolerant recovery solution that is particularly suitable for large-scale parameter updates and gradient recovery problems in cross-modal large-scale model training.

[0068] See also Figure 2 , a flowchart of the hierarchical storage structure construction of an embodiment of the present invention, in the training process of ultra-large-scale cross-modal large models, the fault-tolerant recovery of computing nodes faces huge technical challenges, especially in the storage and recovery of gradients. Traditional methods often rely on simple gradient snapshots and global backtracking, lack fine-grained storage management and recovery strategies, and easily lead to the inability to accurately restore gradient data when a computing node fails. In order to improve the accuracy and efficiency of fault-tolerant recovery, the present application realizes efficient gradient storage and recovery by establishing a hierarchical storage structure for computing nodes at the beginning of training. In this storage structure, the gradient information in the training process is divided into multiple levels, including a high-speed layer and a low-speed layer, and the gradient data is allocated to different levels for storage according to its importance and real-time nature. The high-speed layer is used to store the gradient information that is most critical to the current training state, such as the gradient data of the current time window, while the low-speed layer stores more redundant historical data or data with lower calculation frequency.

[0069] The specific steps of S1 are as follows:

[0070] S1.1: Preset continuous time windows according to the training time, and generate a corresponding gradient map for each window, where the gradient map includes map nodes, map edges, and map metadata;

[0071] Specifically, during large-scale training, as training progresses, the model parameters continue to change, and the gradient information of the training data is constantly updated. This requires a mechanism to accurately capture the changes and flow of gradients within each time period so that the training state can be accurately restored when a failure occurs. Therefore, this embodiment designs a time window-based gradient map generation strategy. By presetting continuous time windows according to the training time and generating a gradient map for each time window, the gradient change process within each time period can be effectively recorded.

[0072] Specifically, each gradient graph consists of graph nodes, graph edges, and graph metadata. Graph nodes represent the gradient values of each computing node during the training process, while graph edges represent the gradient dependencies between different computing nodes. Metadata includes information such as the dependencies between computing nodes, timestamps, and the number of training steps. By constructing a gradient graph, the changing trends of gradients within different time windows and the interdependencies between computing nodes can be clearly displayed. After a fault occurs, the gradient state before the fault can be accurately located based on the time and gradient information stored in the graph, thereby providing a refined time dimension index for fault-tolerant recovery. This allows data to be accurately traced back to a specific time window during data recovery, significantly improving recovery accuracy. In addition, by storing the data of each time window in a graph format, subsequent optimization and adjustment are facilitated, improving the flexibility and adaptability of fault-tolerant recovery.

[0073] S1.2: Identify critical paths and non-critical paths based on the causal relationship of the gradient map, wherein the identification of the paths is determined based on the correction amplitude of the gradient path;

[0074] Specifically, the gradient information during training is a multi-layered, multi-dimensional network, and different gradient paths have varying impacts on training results. Critical paths are those that significantly influence model parameter updates and training progress, while non-critical paths have less of an impact on training. Traditional recovery methods often fail to effectively distinguish between these paths, resulting in the recovery of a large number of irrelevant paths, impacting training efficiency and even introducing unnecessary errors.

[0075] In this embodiment, the importance of a path can be determined based on the magnitude of the gradient path correction—that is, the magnitude of the correction value on each gradient path. Paths with larger gradient correction magnitudes typically correspond to critical paths that significantly impact training results, while paths with smaller correction magnitudes can be considered non-critical paths. This approach effectively distinguishes which paths require priority recovery during training and which paths can be appropriately delayed, avoiding excessive recovery of irrelevant paths and improving recovery efficiency and training accuracy.

[0076] S1.3: Arrange checkpoints according to the path attributes, and store the gradient data collected at the checkpoints in the corresponding high-speed layer and low-speed layer.

[0077] Specifically, in ultra-large-scale training tasks, the arrangement of checkpoints is a crucial step in the recovery process. A checkpoint is an intermediate state in the training process that can record the gradient data of the current stage for future recovery. To improve the accuracy and efficiency of recovery, this embodiment arranges checkpoints based on the attributes of different paths and stores the collected gradient data in different layers. Gradient data on critical paths will be collected frequently and stored in the high-speed layer so that in the event of a failure, these gradient data can be quickly accessed for recovery. For gradient data on non-critical paths, less frequent checkpoints are used and stored in the low-speed layer.

[0078] In this embodiment, by combining the properties of the gradient path with the design of the storage hierarchy, resource utilization during the training process is made more efficient. When performing fault-tolerant recovery, the gradient data of the critical path can be first obtained from the high-speed layer, thereby quickly restoring important training data; and for the data of non-critical paths, it can be loaded on demand from the low-speed layer, avoiding unnecessary calculations and storage accesses. This on-demand access storage strategy can avoid frequent access to unimportant gradient data while ensuring the accuracy of fault-tolerant recovery, greatly improving computing efficiency. In large-scale model training, the storage and recovery of gradients are often the bottleneck of the system. The use of this strategy can significantly improve the execution efficiency of training tasks and the reliability of fault-tolerant recovery.

[0079] The specific steps of S1.2 are as follows:

[0080] S1.2.1: Calculate the correction amplitude of each path in the gradient map within its continuous time window, where the correction amplitude represents the cumulative energy of the gradient changes of all computational nodes on the path;

[0081] Specifically, during training, the gradient graph is composed of the gradient updates of multiple computing nodes. Each computing node generates different gradient values at different stages of training, and these gradient values change as training progresses. Therefore, the cumulative energy of the gradient change on each path can reflect the gradient update rate and impact of that path. To accurately calculate the correction amplitude, it is necessary to track the gradient changes within each time window. Using the gradient difference accumulation method, the gradient corrections for all computing nodes on each path are calculated and weighted accumulation is performed to obtain the correction amplitude for the path. The weighted accumulation method can be adjusted based on the activity of the computing node, the gradient update frequency, and the role of the path in the gradient propagation process. This ensures that the correction amplitude of the path can truly reflect the trend of the gradient change and effectively avoids misjudgments caused by gradient mutations at a single computing node.

[0082] As a preference, since the gradient update during the training process has a high degree of temporal correlation, the correction amplitude can also be calculated by means of a sliding time window to ensure the stability of the calculation results and prevent deviations in the overall path evaluation due to local abnormal gradients.

[0083] S1.2.2: Assign the correction amplitude according to the in-degree and out-degree of the path in the gradient amplitude;

[0084] In this embodiment, to further quantify the impact of each path on the global training process, the correction amplitude is assigned based on the in-degree and out-degree of the gradient amplitude. The in-degree and out-degree respectively represent the inflow and outflow of gradients involved in the calculation on the path during the gradient propagation process.

[0085] Specifically, the in-degree represents the number of gradient flows entering the path, while the out-degree represents the number of gradients flowing out of the path. Since training tasks are usually highly parallel, the gradient propagation relationship between different computing nodes is complex during the gradient calculation process. Therefore, it is necessary to analyze the in-degree and out-degree of the path to determine the contribution of the path to training. During the assignment process, the in-degree, out-degree, and correction amplitude of the path are comprehensively considered to ensure that the final assignment accurately reflects the importance of the path. Specifically, if a path has a high in-degree, it means that the path is a convergence path for the gradients of multiple computing nodes and has a strong gradient accumulation effect. Therefore, it is given a higher weight. For paths with a high out-degree, it means that the gradient on the path affects the calculation results of multiple computing nodes. Therefore, it also needs to be given a higher weight.

[0086] Preferably, a time decay factor can be introduced during the actual calculation process to reduce the weight of gradients far from the current time window, ensuring the rationality and timeliness of the assignment. This can accurately measure the impact of a path on global gradient propagation during training, avoid importance assessment bias caused by relying solely on gradient amplitude calculations, and improve the accuracy of path evaluation.

[0087] S1.2.3: Sort the paths based on the evaluation results and identify critical and non-critical paths based on dynamic thresholds.

[0088] In this embodiment, all paths are sorted based on the path assignment results calculated above, and critical paths and non-critical paths are identified based on dynamic thresholds. The purpose of sorting is to distinguish which paths have a greater impact on the training process, thereby ensuring that subsequent gradient recovery and computing resource allocation can be more accurate. In order to achieve effective sorting, a dynamic sorting algorithm based on path importance weights is adopted. This algorithm not only considers the correction amplitude, in-degree and out-degree of the path, but also globally sorts the paths based on the overall distribution of gradient propagation during the training process. During the sorting process, high-weight paths are given priority to ensure that these paths are given priority in the training task. Next, based on the sorting results, critical paths and non-critical paths are divided by dynamic thresholds, where the setting of the dynamic threshold is adjusted based on the real-time situation of the training task.

[0089] Specifically, in the early stages of training, the model's parameters are updated more drastically and the gradient fluctuates greatly. Therefore, a lower threshold is set to identify more critical paths and ensure that the critical gradient information is fully supported by computing resources. In the later stages of training, as the model converges, the gradient changes tend to stabilize. At this time, the threshold is gradually increased to make the screening of critical paths more stringent, thereby reducing unnecessary computing overhead and improving training efficiency.

[0090] See also Figure 3 , the schematic diagram of the time window division of an embodiment of the present invention illustrates the initial setting and hierarchical division of the time window during the gradient recovery process. After a computing node failure occurs, in order to accurately recover the lost gradient information, a time window system is first constructed based on the micro-checkpoint and divided into a first time window and a second time window. The first time window is used to mark the time interval when the failure occurs, and the second time window is set between the micro-checkpoint and the first time window to provide an initial recovery basis for gradient backtracking. This multi-level time window division can ensure that the recovery process has both accuracy and stability, avoid the accumulation of recovery errors due to excessive direct backtracking spans, and provide a structured data frame for subsequent gradient correction.

[0091] See also Figure 4The schematic diagram of the time window shift in the embodiment of the present invention further illustrates the dynamic adjustment process of the time window. The third time window is the key interval for further refining the gradient recovery. Its position follows the second time window. In the actual training process, since the gradient change has certain nonlinear characteristics, the fixed time window backtracking may not be able to accurately match the gradient state before the fault. In order to improve the recovery accuracy, this embodiment uses a dynamic time window shifting mechanism to gradually shift toward the first time window based on the initial second time window, and dynamically adjusts the window length according to the stability of the gradient in the current window. The setting of this shifting direction ensures that each backtracking can make full use of the existing gradient information, and enables the boundaries of the time window to be flexibly adjusted according to the actual situation of the training state. Compared with the traditional fixed time window backtracking method, this method can reduce redundant gradient calculations, improve recovery efficiency, and ensure that the recovered gradient information is more consistent with the actual trajectory during the training process.

[0092] See also Figure 5 Schematic diagram of termination determination in an embodiment of the present invention. During the time window advancement process, a second-order derivative determination mechanism is introduced to optimize gradient recovery accuracy. In ultra-large-scale training tasks, gradient changes are often affected by multiple factors. Especially in the early stages of training, gradients may exhibit significant volatility. Therefore, relying solely on static adjustments to the time window may not fully reflect the true evolution of the gradient. To more accurately identify key gradient change points, this embodiment introduces a second-order derivative determination mechanism. By calculating the second-order derivatives of the gradients within different time windows, the gradient trend within the current window is determined. A large absolute value of the second-order derivative indicates a significant gradient update within that time window. In this case, the recovery process should be refined to avoid information loss caused by coarse interpolation. A small second-order derivative indicates that the gradient is stabilizing, indicating good recovery results for the current window and can be used as a criterion for terminating the advancement process. The adaptive adjustment of the time window ensures that the recovery process accurately captures key gradient changes while reducing unnecessary computational overhead, thereby optimizing the recovery efficiency and stability of the training task.

[0093] The specific steps of S2 are as follows:

[0094] S2.1: Read a first gradient snapshot corresponding to a first time window from the hierarchical storage structure, where the first time window represents a time window corresponding to a computing node failure, and the first gradient snapshot represents stored data from a micro-checkpoint immediately preceding the first time window.

[0095] In this embodiment, first, after a computing node fails, in the process of restoring the training gradient, the first gradient snapshot corresponding to the first time window is extracted from the hierarchical storage structure. The hierarchical storage structure includes a high-speed layer and a low-speed layer, wherein the high-speed layer stores important gradient snapshots that are frequently accessed during recent training, while the low-speed layer stores earlier training state information. In order to ensure the accuracy of training recovery, the gradient snapshot of the most recent micro-checkpoint is first read from the high-speed layer as the first gradient snapshot, and combined with its version information and gradient change trend, it is ensured that the data closest to the failure point can be obtained during the backtracking process. During the reading process, the integrity of the gradient snapshot is automatically verified, and combined with the global parameter hash check to prevent data corruption caused by storage anomalies. The gradient information that can be used for recovery is obtained in the shortest time, while avoiding recovery failures caused by incomplete gradient snapshots.

[0096] S2.2: Based on the gradient change rate of the first gradient snapshot, a second time window is set between the previous micro-checkpoint and the first time window. The second time window is restored based on the gradient flow of the first gradient snapshot, with the left interval of the second time window being set to the previous micro-checkpoint.

[0097] In this embodiment, the range of the second time window is dynamically set according to the gradient change rate of the first gradient snapshot. Since the training gradient of the computing node does not change uniformly, there may be a situation where the gradient changes drastically in some training stages, while the gradient update in some stages is relatively stable. Therefore, by calculating the gradient change rate in the first gradient snapshot, the gradient stability in the time period is identified, and the length of the second time window is determined based on the change trend. If the gradient change rate of the first gradient snapshot is large, it means that the training gradient fluctuates greatly in the time period, and a shorter time window is required for fine recovery to reduce the error accumulation caused by gradient instability; if the gradient change rate is small, it means that the gradient update is relatively stable, and the time window can be appropriately lengthened to speed up the recovery progress. It can effectively reduce the error in the gradient recovery process, improve the recovery accuracy, and ensure that the training state after recovery is consistent with that before the failure.

[0098] The calculation formula for the gradient change rate is:

[0099] ,

[0100] Among them, R represents the gradient change rate, which is used to measure the intensity of the gradient update during training and reflects the fluctuation of the gradient between different time steps. N represents the total number of gradient data in the micro-checkpoint. and Respectively represent the start time and end time of the micro-checkpoint, and the gradient change rate is calculated within this time range. Represents the gradient increment (Gradient Update) at time t, that is, the gradient change value corresponding to this time step, Represents the gradient increment at time t+1, that is, the gradient change value of the next time step, Indicates the norm, represents a constant greater than zero;

[0101] In this embodiment, the length of the second time window is set by setting the change rate threshold, wherein the initial length W of the second time window can be set to the time window length of the micro-checkpoint, and then the gradient change rate R and the stability threshold are used to determine the length of the second time window. and instability threshold The difference between the initial length is weighted, if , indicating that the gradient changes smoothly, the length of the second time window can be set to ,if , indicating that the gradient changes dramatically, and the time window needs to be narrowed to accurately restore the lost gradient information. The length of the second time window can be set to If it is between the two, the time window size is adjusted by linear interpolation.

[0102] Furthermore, during the recovery process, corrections are made based on the gradient flow characteristics. By constructing a gradient flow direction map, the order of gradient updates during the recovery process is ensured to be correct, thereby avoiding the problem of inconsistent training states caused by incorrect gradient backtracking order.

[0103] S2.3: Shifting the second time window toward the first time window by a set length, and restoring a third time window within the set length, where the set length is determined based on the gradient change rate of the second time window;

[0104] In this embodiment, in order to further improve the accuracy of gradient recovery, based on the gradient change rate of the second time window, the time window is gradually shifted toward the first time window, and a third time window is defined within the set time range to restore the gradient data in stages. The calculation of the set length is similar to the steps in S2.2. Based on the gradient change trend of the second time window, if the gradient change rate within the second time window is relatively stable, a larger time window can be shifted to reduce the recovery time and improve the recovery efficiency; if the gradient change rate is large, a smaller time window is shifted to ensure the accuracy of the restored gradient. In this way, it is possible to gradually approach the first time window and perform gradient recovery in an incremental manner to ensure the stability and accuracy of the entire backtracking process. In addition, each time the time window is shifted, a hash consistency check is performed on the gradient of the current window to determine the consistency of the restored gradient with the global parameters. If the gradient deviation is found to be too large, the time window length is readjusted to ensure the stability of the restored gradient.

[0105] S2.4: Determine the second-order derivatives of the gradient at both ends of the right interval of the current second time window. If the termination condition is not met, continue to advance the set length until the termination condition is met;

[0106] In this embodiment, to ensure the accuracy of gradient recovery and the stability of the training process, the second-order derivative of the gradient at both ends of the right interval of the time window is calculated during the time window extension to determine whether the termination condition is met. The physical significance of the second-order derivative of the gradient is to measure the acceleration of the gradient change. If the second-order derivative is small, it indicates that the gradient update trend is stable, which means that the recovery process is close to a stable state. If the second-order derivative is large, it indicates that the gradient change still has large fluctuations and the time window needs to be further extended to ensure that the gradient is restored to the most stable state.

[0107] Specifically, the judgment condition is that the second-order derivative of the left-hand gradient is less than a set threshold and the first-order derivative approaches zero, while the second-order derivative of the right-hand gradient is greater than a set threshold. This effectively means that the left-hand gradient has stabilized, indicating that the recovered gradient is of high quality and has basically reached the normal state during training. However, the second-order derivative of the right-hand gradient is large, indicating that the gradient in this area is still fluctuating dramatically and may be significantly affected by the fault. If the time window is further advanced at this point, the recovery process may introduce more unstable factors, leading to further accumulation of gradient errors. Therefore, it can be considered that the current window has reached an optimal critical point. Further advancement to the right will not only fail to achieve better recovery results, but may also destabilize the existing recovery. Therefore, at this stage, the recovery process is deemed to have reached convergence conditions, and further advancement of the time window is stopped to ensure that the recovered gradient maintains a high quality while not affecting the overall stability of training due to the introduction of low-quality data.

[0108] S2.5: Extend the second time window until it is flush with the relatively far end of the first time window to restore the training gradient within the first time window;

[0109] In this embodiment, the second time window refers to a time window during the recovery process, and its position is based on the gradient snapshots before and after the failure point. This time window will start from a micro-checkpoint before the failure occurs and extend to a period of time before and after the failure point. The first time window refers to the time window when the failure occurs, which usually corresponds to the moment the failure occurs and the short time interval that follows. Between this time window and the micro-checkpoint, the computing node fails, resulting in the loss of some training gradients. The relatively far end is aligned, which means that the right end of the second time window (i.e., the end that is temporally closer to the failure point) will extend to be aligned with the relatively far end of the first time window (i.e., the time interval before the failure point), that is, the range of the shifted time window can span the portion of the training gradient lost within the first time window.

[0110] Specifically, the second time window is extended toward the starting point of the first time window to restore the lost gradient data in the first time window. This can also be understood as moving the right end of the second time window to the starting point of the first time window to cover and restore the gradient information in the first time window.

[0111] The specific steps of S2.2 are as follows:

[0112] S2.2.1: Calculate the gradient flow direction and rate of the first gradient snapshot;

[0113] In this embodiment, to ensure accurate gradient recovery, the gradient flow direction and rate of the first gradient snapshot must first be calculated. The gradient flow direction reflects the primary trend of the gradient update—that is, the overall direction of the gradient vector for parameter updates during training. This determines the path of the next gradient adjustment. When calculating the flow direction, the gradient changes over multiple historical time steps are analyzed based on the stored first gradient snapshot, combined with the calculation results of the compute nodes to extract key gradient change trends.

[0114] Specifically, by comparing the gradient snapshots of adjacent time windows, calculating the vector change of the gradient, identifying the direction in which the gradient mainly converges or diverges, the global gradient flow trend is determined.

[0115] In this example, during large-scale model training, the gradient update rate may vary nonlinearly. In some regions, the gradient changes slowly, while in others, the gradient may fluctuate dramatically due to the optimizer's dynamic adjustments. Therefore, when calculating the flow rate, it is necessary to measure the speed of the gradient change, that is, the magnitude of the gradient change per unit time.

[0116] Specifically, the gradient changes of continuous time windows are time-weighted, and the instantaneous noise is eliminated by the sliding window mean method, making the calculated gradient flow rate more stable.

[0117] S2.2.2: Calculate a gradient update trend using flow dynamics based on the flow direction and rate;

[0118] In this embodiment, calculating the gradient update trend is a crucial step in the gradient recovery process. Relying solely on static gradient information for recovery is likely to introduce significant historical errors, leading to inaccurate gradient recovery. Therefore, based on the principles of flow dynamics, the gradient flow is considered a process of varying forces, and an inertia term is introduced to correct the gradient's changing trend at different time steps.

[0119] Specifically, the gradient's rate of change within a historical time window is analyzed and combined with information about gradient flow from adjacent compute nodes to construct a gradient update trend model. The core concept of this model is that gradient changes depend not only on the gradient information at the current time step but also on the gradient flow from previous time steps. Therefore, a historical gradient contribution factor is introduced to calculate the impact of gradients from different time windows on the current time step. This factor, combined with the computational power of the current compute node, allows prediction of future gradient changes. This effectively addresses the issue of gradient loss caused by compute node failures, improves the accuracy of gradient backtracking, and ensures that the recovered gradient maintains the convergence of the global model.

[0120] To prevent excessive noise from being introduced during gradient recovery, the weight of historical gradients is adaptively adjusted based on the stability of the current training task when calculating the update trend. This ensures that the recovered gradients do not significantly impact the training process. This not only improves the stability of gradient recovery but also reduces parameter drift after recovery, making the training process more robust.

[0121] S2.2.3: Perform incremental correction starting from the left interval of the second time window according to the gradient update trend until the right interval of the second time window is corrected;

[0122] In this embodiment, the core idea of incremental correction is that gradient recovery should not be a one-time replacement process, but a dynamic adjustment process.

[0123] Specifically, this application will not directly use the stored gradient snapshot for recovery, but will combine the calculated gradient update trend to dynamically adjust the restored gradient value to make it closer to the actual gradient change during the training process. When performing incremental corrections, the information of different gradient points in the time window will be gradually introduced so that the restored gradient can be more in line with the overall gradient change trend rather than an isolated static value. It can effectively reduce the impact on the global training model after gradient recovery, so that the restored gradient can be naturally integrated into the current training state without causing model instability. In the specific execution process, starting from the left interval of the second time window, the correction is gradually expanded to the right. Each time step will be adjusted based on the correction result of the previous time step to ensure that the corrected gradient can maintain global consistency.

[0124] As an optimal approach, a dynamic adjustment strategy is introduced during the correction process. If the gradient at a certain time step changes abnormally, such as a large gradient mutation, the correction process is paused and the gradient flow direction at that time step is recalculated to prevent erroneous gradients from affecting the entire recovery process. Compared to traditional full recovery methods, this incremental correction method can minimize the training error caused by recovery, improve the accuracy of gradient recovery, and ensure model stability and training continuity.

[0125] The specific steps of S2.3 are as follows:

[0126] S2.3.1: Loading a gradient map based on the second time window before the shift, processing the gradient map by low-rank approximation, and generating a reconstructed gradient for the third time window;

[0127] In this embodiment, to improve the accuracy and computational efficiency of training gradient recovery, the corresponding gradient map is first loaded in the second time window before the shift. This gradient map is constructed based on the gradient transmission and computational dependencies between computing nodes during the training process, and contains local gradient information, parameter versions, historical gradient update trajectories, and data flow dependency information for each computing node. Due to the large-scale parallel computing involved in the training process, gradient information is usually high-dimensional and sparse data. If the complete gradient data is directly used for recovery, the computational overhead is huge and redundant information may be introduced, resulting in low recovery efficiency.

[0128] Specifically, this embodiment uses a low-rank approximation method to process the gradient map to reduce data complexity and extract core gradient information. The basic idea of low-rank approximation is to remove redundant information in the gradient data through dimensionality reduction methods, retaining only the main gradient components that affect the training state, thereby realizing the reconstruction of the gradient in the third time window. In practical applications, based on the gradient contribution, parameter update amplitude and gradient correlation of each computing node during the training process, a weight matrix is constructed and matrix decomposition is performed to extract a low-dimensional representation that mainly affects gradient recovery. It can retain the global structure of the gradient data, while effectively compressing the data scale and reducing the computational cost, so that the missing gradient can be calculated and reconstructed faster during the gradient backtracking process.

[0129] Furthermore, low-rank approximation can reduce errors caused by random noise or abnormal gradient changes, enhancing the stability of gradient recovery. After completing the low-rank approximation calculation, the extracted principal component gradients are used to construct a gradient reconstruction model for the third time window, ensuring that the gradient information in this window is as close as possible to the actual training state, avoiding gradient loss that could cause a decrease in computational accuracy and interruption of the training process.

[0130] S2.3.2: Calculate the local consistent hash of the reconstructed gradient and the global parameters. If the hashes match, restore the third time window based on the reconstructed gradient. If they do not match, recalculate the set length of the second time window based on the local consistent hash, where the global parameters represent the model parameters that are globally synchronized during the training process.

[0131] In this embodiment, to ensure the correctness and global consistency of the reconstructed gradient, a local consistent hash is first calculated between the reconstructed gradient and the global parameters before restoring the gradient in the third time window. Because gradient calculation in ultra-large-scale training tasks involves the interaction of multiple compute nodes, gradient calculation errors at any compute node can affect the convergence of the global model. Therefore, the gradient recovery process must ensure consistency with the global parameters. Local consistent hashing is a method that verifies the consistency of gradients with the global parameter state through hash calculation. Its core concept is to calculate a unique hash fingerprint based on the model parameter version, gradient update sequence, and historical training state, ensuring that the restored gradient data is consistent with the state during training. During the calculation process, a hierarchical hashing method is used to calculate hash values between the local compute nodes and the global parameter server. If the local consistent hash values match, it proves that the recovered gradients conform to the evolution trajectory of the global model and can be directly used for recovery in the third time window. If the hash values do not match, it indicates that the current reconstructed gradients may be inaccurate, possibly due to information loss caused by gradient approximation or incomplete recovery due to a too short lookback time window. In this case, the set length of the second time window needs to be readjusted.

[0132] Specifically, the new time window length is calculated based on the matching deviation of the local consistent hashing, so that the time window covers a more complete training gradient, ensuring that the recovered gradient information is as close as possible to the true evolution state of the global parameters. It can dynamically adjust the backtracking range to avoid recovery failures caused by unreasonable window settings, while reducing computational overhead, allowing training tasks to be quickly recovered after failures, ensuring model stability and training efficiency. In addition, local consistent hashing can detect gradient distortion problems caused by storage or communication errors, improving the robustness and reliability of gradient recovery.

[0133] The specific steps of S2.3.2 are as follows:

[0134] S2.3.2.1: Load the causal dependency chain of the critical path in the second time window from the high-speed layer, replay the gradient aggregation operation according to the version number sequence, and restore the gradient information of the critical path corresponding to the third time window;

[0135] In this embodiment, since the recovery of the second time window is based on the real data of the micro-checkpoint, and the recovery of the third time window does not have the same direct data support, it is necessary to use the causal dependency chain to accurately replay the gradient aggregation operation to maximize the recovery of the lost gradient information.

[0136] Specifically, the gradient information stored in the high-speed layer mainly contains the causal relationship of the critical path within the time window. By constructing a causal dependency chain, the order of gradient propagation of the computing nodes before the failure can be traced. Gradient updates are often serial or hierarchical. Especially in ultra-large-scale training, there is a strict time dependency between the gradient updates of different computing nodes. Therefore, based on the stored causal dependency information, starting from the forward propagation phase, the gradient aggregation operation is gradually replayed in the order of the version number to restore the gradient of the critical path in the third time window. The gradient aggregation operation here refers to the gradient information contributed by each computing node to the same batch of data during the asynchronous computing time in distributed training. A certain version control mechanism is required to ensure that the gradients of all computing nodes can be correctly synchronized and applied to the global model.

[0137] Specifically, the third time window is beyond the direct storage and impact range of the micro-checkpoint, so its gradient cannot be directly restored using traditional backtracking methods. Instead, it must rely on historical computation information. The causal dependency chain ensures that the order of gradient replay strictly follows the order in which the compute nodes were executed during training. This effectively avoids gradient confusion or gradient drift caused by incorrect replay order. It also minimizes gradient loss caused by communication delays or uneven compute node load, ensuring that the recovered gradient remains consistent with the original computation process, thereby improving recovery accuracy and training convergence.

[0138] S2.3.2.2: Load the gradient information of the non-critical path in the second time window from the low-speed layer, and restore the gradient information of the critical path corresponding to the third time window based on gradient interpolation and linear regression;

[0139] In this embodiment, since the gradients of non-critical paths generally do not significantly affect the stability of global training, restoring the gradients of the third time window relies on data stored in the low-speed layer and uses mathematical methods such as gradient interpolation and linear regression to infer the missing gradient information. Since the gradient information of non-critical paths is stored in the low-speed layer and its data integrity is lower than that of the critical path, it cannot be directly restored using the gradient backtracking method. Instead, it needs to be inferred based on the historical gradient trend.

[0140] Specifically, we first extract non-critical path gradient data from the low-speed layer for the second time window and analyze the changing trends of these gradients within that time window. By comparing the temporal variation patterns of these gradients, we can use an interpolation algorithm to perform a preliminary restoration to ensure that they conform to the overall gradient trend.

[0141] Furthermore, relying solely on interpolation can introduce certain errors. Therefore, a linear regression method is introduced to ensure the stability of the gradient recovery process. Linear regression is used to infer the optimal fit value of the missing gradient based on known gradient information, avoiding the gradient smoothness or mismatch with the global gradient update direction caused by simple interpolation. Especially in non-critical paths, due to the relatively low frequency of gradient updates, the time evolution of the gradient often exhibits a strong linear correlation, which enables the linear regression method to accurately fit the missing gradient points. During the recovery process, a gradient curve is first constructed using historical data. The fitted curve is optimized using the least squares method to obtain a regression equation that predicts the missing gradient. Finally, the interpolation and regression results are combined to infer the gradient of the third time window, ensuring that the recovered gradient is as close as possible to the gradient value during the actual calculation process. Instead of relying entirely on stored data, gradient recovery can be combined with historical trends, effectively avoiding data loss caused by storage hierarchies. At the same time, since the gradients of non-critical paths usually do not have a decisive impact on the gradient flow of the overall training, using interpolation and regression analysis for recovery not only has low computational overhead but also ensures the rationality of the recovered gradients, avoiding the impact of computational errors in the recovery process on the global training effect.

[0142] S2.3.2.3: Perform weighted correction on the reconstructed gradient according to the restoration result.

[0143] In this embodiment, since the recovery methods of the critical path and the non-critical path are different, there is a deviation in the value of the recovered gradient. Therefore, this application adopts a weighted correction method based on the gradient contribution to improve the reliability of the recovered gradient.

[0144] Specifically, it is necessary to assign higher weights to the gradients of critical paths and lower weights to the gradients of non-critical paths. The final corrected gradient value is then calculated by fusing the recovered gradient data. The weighted correction calculation process mainly includes the following steps: First, all recovered gradient data is normalized to eliminate inconsistencies in the gradient scale. Second, a weight distribution is calculated based on the gradient contributions of the critical and non-critical paths. The gradient weight of the critical path is determined by its historical calculation contribution rate, while the gradient weight of the non-critical path is determined by the fitting error of the linear regression. Finally, the final gradient update value is calculated using a weighted average method and applied to the parameter update of the training model. This ensures that the overall direction of the recovered gradient is consistent with the gradient update direction during the original training process, effectively reducing the problem of error accumulation during the gradient recovery process.

[0145] Furthermore, this weighted correction method improves the stability of recovered gradients, reducing the impact of noise generated during the recovery process on the final training results. This ensures training accuracy while improving the efficiency of fault-tolerant recovery. Even if a compute node fails during ultra-large-scale training tasks, gradients can be restored quickly and training can resume, avoiding model performance degradation caused by gradient loss.

[0146] Example 2:

[0147] See also Figure 6 ,The present invention provides an embodiment: a fault-tolerant recovery system for super-10,000-card-level training and inference tasks, the system comprising a preprocessing module and a gradient recovery module;

[0148] The preprocessing module is used to build a hierarchical storage structure of computing nodes based on training data and network topology before the training task begins, initialize relevant data of each time window, and generate a preliminary gradient map and checkpoint configuration;

[0149] The gradient recovery module is used to recover the lost gradients of the training task in the event of a failure;

[0150] The pre-processing module comprises:

[0151] The time window division unit automatically divides multiple time windows according to the specific requirements of the training task and sets the size and time step of each window;

[0152] The storage structure initialization unit initializes the hierarchical storage structure based on the training computing resource configuration (such as the number of computing nodes and memory capacity), and pre-allocates space for the high-speed layer and the low-speed layer;

[0153] The gradient map generation unit automatically generates the gradient map of each computing node during the training process through preprocessing of training data, and provides support for subsequent fault-tolerant recovery;

[0154] The gradient recovery module includes:

[0155] The global gradient backtracking unit quickly locates the failure time window when a computing node fails, and recovers the training gradient related to the failure through the global gradient backtracking mechanism;

[0156] The time window reconstruction unit, based on the gradient information of the current time window and the previous micro-checkpoint, moves to the new time window according to the gradient change rate and reconstructs the missing gradient through interpolation algorithms, linear regression and other methods;

[0157] The causal dependency chain recovery unit, based on the gradient recovery of the critical path and the reconstruction of the non-critical path, adopts the causal dependency chain strategy to replay the gradient aggregation operation, correct the errors in the recovery process, and ensure the accuracy of the recovery results.

[0158] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A fault-tolerant recovery method for training and reasoning tasks at the 10,000-card level, applied to the training phase of large cross-modal models, characterized by: The fault-tolerant recovery method includes: At the beginning of training, a hierarchical storage structure of computing nodes is established. The hierarchical storage structure is implemented by an incremental gradient graph and dynamic causal checkpoints. The hierarchical storage structure includes a high-speed layer and a low-speed layer. The steps of establishing the hierarchical storage structure include: Preset continuous time windows according to training time, and generate a corresponding gradient map for each window, wherein the gradient map includes map nodes, map edges and map metadata; Identifying critical paths and non-critical paths based on the causal relationship of the gradient map, wherein the identification of the paths is determined based on the correction amplitude of the gradient paths; Arrange checkpoints according to path attributes, and store gradient data collected at the checkpoints into corresponding high-speed layers and low-speed layers; When the computing node fails, the training gradient is restored by global gradient backtracking according to the hierarchical storage structure, wherein the restoring of the training gradient includes: Set the time window based on the stored gradient snapshot and gradually adjust the window length; Restore the gradient training state through incremental correction; If the backtracking fails, roll back to the micro-checkpoint for recovery. The micro-checkpoint is a snapshot of the state saved periodically during the training process.

2. The fault-tolerant recovery method for super-10,000-card-level training and inference tasks according to claim 1 is characterized in that: According to the causal relationship of the gradient map, the critical path and the non-critical path are identified, including: Calculating a correction amplitude for each path in the gradient map within its continuous time window, wherein the correction amplitude represents the cumulative energy of the gradient changes of all computational nodes on the path; Assigning a value to the correction amplitude according to the in-degree and out-degree of the path in the gradient amplitude; The paths are sorted according to the assignment results, and the critical paths and non-critical paths are identified based on dynamic thresholds.

3. The fault-tolerant recovery method for super-10,000-card-level training and inference tasks according to claim 2 is characterized in that: Arrange checkpoints according to path attributes, and store the gradient data collected at the checkpoints in the corresponding high-speed layer and low-speed layer, including: Arranging high-frequency micro-checkpoints on the critical path, wherein the high-frequency micro-checkpoints are set according to the gradient change rate of the critical path; Low-frequency micro-checkpoints are arranged on the non-critical path, wherein the low-frequency micro-checkpoints are set according to the gradient change rate of the non-critical path.

4. The fault-tolerant recovery method for super-10,000-card-level training and inference tasks according to claim 1 is characterized in that: The recovery of the training gradient by global gradient backtracking includes: Reading a first gradient snapshot corresponding to a first time window from the hierarchical storage structure, wherein the first time window represents a time window corresponding to a computing node failure, and the first gradient snapshot represents storage data of a previous micro-checkpoint in the first time window; Setting a second time window between the previous micro-checkpoint and the first time window based on the gradient change rate of the first gradient snapshot, and restoring the second time window based on the gradient flow of the first gradient snapshot, wherein the left interval of the second time window is set as the previous micro-checkpoint; Shifting the second time window toward the first time window by a set length, and restoring a third time window within the set length, wherein the set length is obtained based on the gradient change rate of the second time window; Determine the second-order derivatives of the gradient at both ends of the right interval of the current second time window. If the termination condition is not met, continue to advance the set length until the termination condition is met. The second time window is extended to the left interval of the first time window to restore the training gradient within the first time window.

5. The fault-tolerant recovery method for super-10,000-card-level training and inference tasks according to claim 4 is characterized in that: Restoring the second time window according to the gradient flow of the first gradient snapshot includes: Calculating the gradient flow direction and rate of the first gradient snapshot; calculating a gradient update trend through flow dynamics according to the flow direction and rate; According to the gradient update trend, incremental correction is performed starting from the left interval of the second time window until the correction of the right interval of the second time window is completed.

6. The fault-tolerant recovery method for super-10,000-card-level training and inference tasks according to claim 4 is characterized in that: Recover the third time window within the set length, including: Loading a gradient map based on the second time window before the shift, processing the gradient map by low-rank approximation, and generating a reconstructed gradient of the third time window; Calculate the local consistent hash of the reconstructed gradient and the global parameters. If the hashes match, restore the third time window based on the reconstructed gradient. If they do not match, recalculate the set length of the second time window based on the local consistent hash, where the global parameters represent model parameters that are globally synchronized during the training process.

7. The fault-tolerant recovery method for super-10,000-card-level training and inference tasks according to claim 6, characterized in that: Restoring the third time window according to the reconstruction gradient includes: Load the causal dependency chain of the critical path in the second time window from the high-speed layer, replay the gradient aggregation operation according to the version number sequence, and restore the gradient information of the critical path corresponding to the third time window; Loading the gradient information of the non-critical path in the second time window from the low-speed layer, and restoring the gradient information of the critical path corresponding to the third time window based on gradient interpolation and linear regression; The reconstructed gradient is weightedly corrected according to the restoration result.

8. The fault-tolerant recovery method for super-10,000-card-level training and inference tasks according to claim 4 is characterized in that: The termination conditions include: The second-order derivative of the left gradient is less than the set threshold and the first-order derivative tends to zero, while the second-order derivative of the right gradient is greater than the set threshold.

9. A fault-tolerant recovery system for super-10,000-card-level training and reasoning tasks, configured to implement the fault-tolerant recovery method for super-10,000-card-level training and reasoning tasks as described in any one of claims 1 to 8, characterized in that: The system includes a pre-processing module and a gradient recovery module; The preprocessing module is used to build a hierarchical storage structure of computing nodes based on training data and network topology before the training task begins, initialize relevant data of each time window, and generate a preliminary gradient map and checkpoint configuration; The gradient recovery module is used to recover the lost gradients of the training task when a failure occurs.

Images