Fault-tolerant recovery method and system for training and reasoning tasks exceeding ten thousand cards

Abstract

The invention relates to the technical field of gradient reconstruction, in particular to a fault-tolerant recovery method and a fault-tolerant recovery system for a training and reasoning task exceeding a ten thousand calorie level, and provides the following scheme: training data is managed through a hierarchical storage structure, and gradient recovery is carried out by adopting global gradient backtracking when a node fault is calculated. In the recovery process, a dynamic time window is set based on the stored gradient snapshot, and the gradient recovery precision is optimized by adjusting the length of the window. And when the backtracking fails, the stable recovery of the training process is ensured by adopting the micro check point rollback. And based on a causal dependency chain and a gradient interpolation regression method, version control gradient playback is carried out on the critical path, interpolation regression recovery is carried out on the non-critical path, and finally, gradient recovery precision is improved through weighted correction. Gradient loss caused by calculation node faults can be reduced, and the fault-tolerant capability and calculation efficiency of super-large-scale training tasks can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of gradient reconstruction, and particularly to a fault tolerance and recovery method and system for training and inference tasks with over ten thousand cards. Background Art

[0002] During the training process of ultra-large-scale deep learning models, huge computing resources are required, and distributed parallel computing is usually adopted for training. However, as the computing scale expands, the failure rate of computing nodes also increases, which may lead to the loss of training gradients, parameter asynchronization, and even the interruption of the training process. Existing training recovery methods mainly rely on periodically stored checkpoints for rollback, or use interpolation methods to estimate the lost gradients. However, these methods have significant deficiencies in ultra-large-scale training tasks. For example, checkpoint rollback will cause a large amount of repeated calculations, reducing the training efficiency, while gradient interpolation methods are difficult to ensure the accuracy of the recovered gradients, which may lead to unstable model convergence. In addition, existing methods lack effective tracking of the gradient dependency relationship during the training process, unable to accurately recover key gradients, resulting in a decline in training performance. Therefore, there is an urgent need for an efficient fault tolerance and recovery method that can accurately trace back the lost gradients when a computing node fails and dynamically adjust the recovery strategy to reduce the interruption of training tasks and improve the stability and computing efficiency of model training.

[0003] All existing technologies have the problems proposed in this background art: there are problems such as inaccurate gradient recovery, high rollback cost, lack of key gradient dependency relationship, and serious impact on training interruption in ultra-large-scale model training, making it difficult to effectively ensure efficient fault tolerance and recovery after a computing node fails. To solve the above problems, this application designs a fault tolerance and recovery method and system for training and inference tasks with over ten thousand cards. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a fault tolerance and recovery method and system for training and inference tasks with over ten thousand cards in view of the deficiencies of the prior art. The training data is managed through a hierarchical storage structure, and when a computing node fails, global gradient backtracking is used for gradient recovery. During the recovery process, a dynamic time window is set based on the stored gradient snapshots, and the gradient recovery accuracy is optimized by adjusting the window length. When the backtracking fails, micro-checkpoint rollback is used to ensure the stable recovery of the training process. And based on the causal dependency chain and gradient interpolation regression method, version control gradient replay is performed on the critical path, and interpolation regression recovery is used for the non-critical path. Finally, the gradient recovery accuracy is improved through weighted correction. It can reduce the gradient loss caused by computing node failures and improve the fault tolerance and computing efficiency of ultra-large-scale training tasks.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] Fault Tolerance Recovery Method for Ultra-Mega Card-Level Training and Inference Tasks, applied to the training phase of cross-modal large models, the fault tolerance recovery method includes:

[0007] Establish a hierarchical storage structure for computing nodes at the beginning of training;

[0008] When a computing node fails, according to the hierarchical storage structure, recover the training gradient through global gradient backtracking;

[0009] If the backtracking fails, roll back to the micro checkpoint for recovery.

[0010] The hierarchical storage structure is implemented through incremental gradient maps and dynamic causal checkpoints. The hierarchical storage structure includes a high-speed layer and a low-speed layer, including:

[0011] Preset consecutive time windows in advance according to training time, and generate corresponding gradient maps for each window, where the gradient maps include map nodes, map edges, and map metadata;

[0012] Identify critical paths and non-critical paths according to the causal relationship of the gradient maps, where the identification of paths is determined based on the correction amplitude of the gradient paths;

[0013] Arrange checkpoints according to path attributes, and store the gradient data collected by the checkpoints in the corresponding high-speed layer and low-speed layer.

[0014] Identify critical paths and non-critical paths according to the causal relationship of the gradient maps, including:

[0015] Calculate the correction amplitude of each path in the gradient map within its consecutive time window, where the correction amplitude represents the cumulative energy of the gradient changes of all computing nodes on the path;

[0016] Assign values to the correction amplitude according to the in-degree and out-degree of the path in the gradient amplitude;

[0017] Sort the paths according to the assignment results, and identify critical paths and non-critical paths according to dynamic thresholds.

[0018] Arrange checkpoints according to path attributes, and store the gradient data collected by the checkpoints in the corresponding high-speed layer and low-speed layer, including:

[0019] Arrange high-frequency micro checkpoints on the critical path, where the high-frequency micro checkpoints are set according to the gradient change rate of the critical path;

[0020] Arrange low-frequency micro checkpoints on the non-critical path, where the low-frequency micro checkpoints are set according to the gradient change rate of the non-critical path.

[0021] The recovery of the training gradient through global gradient backtracking includes:

[0022] Read a first gradient snapshot corresponding to a first time window from the hierarchical storage structure, where the first time window represents the time window corresponding to the computing node failure, and the first gradient snapshot represents the stored data of the previous micro-checkpoint of the first time window;

[0023] Set a second time window between the previous micro-checkpoint and the first time window according to the gradient change rate of the first gradient snapshot, and recover the second time window according to the gradient flow of the first gradient snapshot, where the left interval of the second time window is set as the previous micro-checkpoint;

[0024] Shift the second time window towards the first time window by a set length, and recover the third time window within the set length range, where the set length is obtained based on the gradient change rate of the second time window;

[0025] Judge the second-order derivative of the gradient at both ends of the right interval of the current second time window. If the termination condition is not met, continue to shift the set length until the termination condition is met;

[0026] Lengthen the second time window to be flush with the left interval of the first time window to recover the training gradient within the first time window.

[0027] The recovery of the second time window according to the gradient flow of the first gradient snapshot includes:

[0028] Calculate the gradient flow direction and rate of the first gradient snapshot;

[0029] According to the flow direction and rate, calculate the gradient update trend through flow dynamics;

[0030] According to the gradient update trend, perform incremental correction starting from the left interval of the second time window until the correction of the right interval of the second time window is completed.

[0031] The recovery of the third time window within the set length range includes:

[0032] Load a gradient map based on the second time window before shifting, process the gradient map through low-rank approximation to generate a reconstructed gradient of the third time window;

[0033] Calculate the local consistency hash of the reconstructed gradient and the global parameters. If the hash matches, recover the third time window according to the reconstructed gradient. If it does not match, recalculate the set length of the second time window according to the local consistency hash, where the global parameters represent the model parameters globally synchronized during training.

[0034] The restoration of the third time window according to the reconstructed gradient includes:

[0035] Loading the causal dependency chain of the critical path within the second time window from the high-speed layer, replaying the gradient aggregation operation according to the version number order, and restoring the gradient information of the critical path corresponding to the third time window;

[0036] Loading the gradient information of the non-critical path within the second time window from the low-speed layer, and restoring the gradient information of the critical path corresponding to the third time window according to gradient interpolation and linear regression;

[0037] Performing weighted correction on the reconstructed gradient according to the restoration result.

[0038] The termination conditions include:

[0039] The second derivative of the left gradient is less than the set threshold and the first derivative tends to zero, while the second derivative of the right gradient is greater than the set threshold.

[0040] A fault-tolerant recovery system for super multi-thousand-card-level training and inference tasks, the system includes a preprocessing module and a gradient recovery module;

[0041] The preprocessing module is used to construct a hierarchical storage structure of computing nodes based on training data and network topology before the training task starts, initialize the relevant data of each time window, and generate a preliminary gradient map and checkpoint configuration;

[0042] The gradient recovery module is used to recover the lost gradients of the training task when a failure occurs.

[0043] Compared with the prior art, the beneficial effects of the present invention are:

[0044] By constructing a hierarchical storage structure, global gradient backtracking, and dynamic time window adjustment, the present invention accurately recovers the lost gradients when a computing node fails, combines incremental gradient map analysis to analyze critical paths and non-critical paths, uses the causal dependency chain to perform version control replay on critical gradients, and at the same time uses gradient interpolation and regression analysis to recover non-critical path gradients, and finally optimizes gradient consistency through weighted correction. Compared with existing methods, the present invention can reduce unnecessary global rollbacks, improve the accuracy of gradient recovery, reduce waste of computing resources, improve the stability and computing efficiency of ultra-large-scale training tasks, and thus ensure the continuity of the training process and the efficient convergence of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] By reading the detailed description of the non-limiting embodiments with reference to the following drawings, other features, objects, and advantages of the present invention will become more apparent:

[0046] Figure 1Schematic flowchart of the fault tolerance and recovery method for the ultra - ten - thousand - card - level training and inference tasks in Embodiment 1 of the present invention;

[0047] Figure 2 Flowchart for constructing the hierarchical storage structure in Embodiment 1 of the present invention;

[0048] Figure 3 Schematic diagram of time window division in Embodiment 1 of the present invention;

[0049] Figure 4 Schematic diagram of time window progression in Embodiment 1 of the present invention;

[0050] Figure 5 Schematic diagram of termination determination in Embodiment 1 of the present invention;

[0051] Figure 6 Module diagram of the fault tolerance and recovery system for the ultra - ten - thousand - card - level training and inference tasks in Embodiment 2 of the present invention. Detailed implementation manners

[0052] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments.

[0053] Embodiment 1:

[0054] Please refer to Figure 1 , an embodiment provided by the present invention: A fault tolerance and recovery method for ultra - ten - thousand - card - level training and inference tasks, which is applied to the training stage of a cross - modal large model, and the specific steps are as follows:

[0055] S1: Establish a hierarchical storage structure for computing nodes at the beginning of training;

[0056] In this embodiment, by dividing the hierarchical storage structure into a high - speed layer and a low - speed layer, the gradient information that needs to be frequently accessed and updated during training is stored in the high - speed layer, while the data with a longer history is stored in the low - speed layer. It can optimize data storage and access, reduce the consumption of hardware resources, ensure the efficient execution of training tasks, and especially effectively relieve the storage pressure of computing nodes under large - scale data sets.

[0057] The purpose is to ensure efficient data access during training, avoid the influence of storage bottlenecks, and improve the overall training speed and stability.

[0058] S2: When a failure occurs in the computing node, recover the training gradient through global gradient backtracking according to the hierarchical storage structure;

[0059] In this embodiment, once a computing node fails, the most recent gradient snapshot is read from the hierarchical storage structure, and global gradient backtracking is performed based on it. Through the backtracking technology, the training state before the failure can be accurately restored to ensure the accuracy of the recovery process. In the process of gradient backtracking, the causal relationship of the gradient graph is combined, and the length of the backtracking window is dynamically adjusted to make the recovery operation more refined, thereby avoiding the loss of gradient information caused by computing node failure as much as possible.

[0060] S3: If the backtracking fails, roll back to the micro-checkpoint.

[0061] In this embodiment, micro-checkpoints are state snapshots saved periodically during the training process. By rolling back to these micro-checkpoints, it is possible to effectively avoid losing too much training information and minimize the recovery time. The design of micro-checkpoint rollback can provide a reliable recovery point when the backtracking fails, ensuring that the model training can be restored to a normal state within a reasonable time frame.

[0062] This embodiment aims at the fault-tolerant recovery problem in super-10,000-card-level training and reasoning tasks, especially in the large-scale cross-modal large model training process, and solves the problems of training data loss and difficulty in training gradient recovery caused by computing node failure.

[0063] Specifically, traditional training recovery methods usually rely on simple rollback mechanisms (such as rolling back to the most recent checkpoint) or incremental gradient interpolation to recover lost data when facing computing node failures. However, these methods often fail to effectively recover high-quality training gradients in large-scale model training, especially when ultra-large-scale parameter updates are involved. They may even lead to larger training errors and reduce the efficiency and accuracy of model convergence.

[0064] In this embodiment, by establishing a global gradient-based backtracking mechanism, when a computing node fails, the global gradient backtracking is first performed based on the gradient snapshot and time window saved in the hierarchical storage structure. By calculating the gradient change rate, not only can the training state before the failure be restored, but also the length of the restored time window can be automatically adjusted according to the gradient flow trend at different training stages. The innovation of this application is that traditional technologies often use a fixed preset window length, while this application can more accurately capture the gradient information before the training interruption by dynamically adjusting the window length, thereby reducing training errors and information loss.

[0065] Exemplarily, during the training process of a cross-modal large model, when a computing node is interrupted due to hardware failure or communication problems, traditional recovery methods may backtrack part of the training data based on a fixed time window, which often fails to effectively recover to the optimal state. By adopting a recovery technique based on global gradient backtracking, this application first determines the gradient change situation of the current time window, and then automatically adjusts the backtracking length according to the gradient change rate, and can even dynamically shift the window to accurately recover the training state. In this way, not only can common problems such as gradient explosion or gradient disappearance be avoided, but also the loss caused by training interruption can be minimized to ensure the efficient recovery of the training task.

[0066] Furthermore, when the global gradient backtracking fails, this application does not completely interrupt the training task, but rolls back the training process to the closest micro checkpoint, so as to quickly recover the training state. Compared with simply rolling back to a fixed checkpoint in the prior art, this application enables rolling back to the most effective checkpoint closest to the current training state at each failure by frequent micro checkpoint collection, thus reducing the risk of training data loss and improving the accuracy of recovery.

[0067] In summary, the fault tolerance and recovery method for ultra-thousands-of-cards training and inference tasks provided by this application not only fills the gap in the prior art for large-scale training tasks theoretically, but also significantly improves the fault tolerance and recovery accuracy of large model training in practice. By combining dynamic adjustment of the time window and micro checkpoint rollback, this application provides an efficient and reliable fault tolerance and recovery solution, which is particularly suitable for large-scale parameter updates and gradient recovery problems in cross-modal large model training.

[0068] Please refer to Figure 2 , the flowchart of constructing the hierarchical storage structure in the embodiment of the present invention. During the training process of an ultra-large-scale cross-modal large model, the fault tolerance and recovery of computing nodes face huge technical challenges, especially in the storage and recovery of gradients. Traditional methods often rely on simple gradient snapshots and global backtracking, lacking fine-grained storage management and recovery strategies, and are prone to failure to accurately recover gradient data when computing nodes fail. To improve the accuracy and efficiency of fault tolerance and recovery, this application realizes efficient gradient storage and recovery by establishing a hierarchical storage structure for computing nodes at the beginning of training. In this storage structure, the gradient information during the training process is divided into multiple levels, including a high-speed level and a low-speed level, and gradient data is stored in different levels according to its importance and real-time nature. The high-speed level is used to store the gradient information most critical to the current training state, such as the gradient data of the current time window, while the low-speed level stores relatively redundant historical data or data with a lower calculation frequency.

[0069] The specific steps of S1 are as follows:

[0070] S1.1: Preset consecutive time windows according to the training time, and generate corresponding gradient maps for each window, where the gradient maps include map nodes, map edges, and map metadata;

[0071] Specifically, during large-scale training, as the training time progresses, the parameters of the model continuously change, and the gradient information of the training data is also constantly updated. This requires a mechanism to accurately capture the changes and flows of gradients within each time period so that the training state can be precisely restored when a fault occurs. Therefore, this embodiment designs a gradient map generation strategy based on time windows. By presetting consecutive time windows according to the training time and generating a gradient map for each time window, the change process of gradients within each time period can be effectively recorded.

[0072] Specifically, each gradient map consists of map nodes, map edges, and map metadata. Map nodes represent the gradient values of each computing node during the training process, and map edges represent the gradient dependency relationships between different computing nodes. Metadata includes information such as the dependency relationships between computing nodes, timestamps, and training steps. By constructing the gradient map, the change trends of gradients within different time windows and the mutual dependency relationships between computing nodes can be clearly displayed. After a fault occurs, based on the time and gradient information stored in the map, the gradient state before the fault can be accurately located, thereby providing a refined time dimension index for fault-tolerant recovery, enabling precise backtracking to a specific time window during data recovery, and thus greatly improving the recovery accuracy. In addition, by storing the data of each time window in a mapped manner, it is also convenient for subsequent optimization and adjustment, improving the flexibility and adaptability of fault-tolerant recovery.

[0073] S1.2: Identify critical paths and non-critical paths according to the causal relationships of the gradient maps, where the identification of paths is determined based on the correction amplitude of the gradient paths;

[0074] Specifically, the gradient information during the training process is a multi-level and multi-dimensional network, and different gradient paths have different impacts on the training results. Critical paths refer to those gradient paths that have an important impact on model parameter updates and training progress, while non-critical paths have a smaller impact on training. Traditional recovery methods usually lack effective discrimination of these paths, resulting in the recovery of a large number of irrelevant paths during recovery, which affects the training efficiency and even introduces unnecessary errors.

[0075] In this embodiment, based on the correction amplitude of the gradient path, that is, the magnitude of the correction value on each gradient path, the importance of the path can be determined. Paths with a larger gradient correction amplitude usually correspond to key paths that have an important impact on the training result, while paths with a smaller correction amplitude can be regarded as non-critical paths. In this way, it is possible to effectively distinguish which paths need to be preferentially restored during training and which paths can be restored appropriately later, avoiding excessive restoration of irrelevant paths and improving the restoration efficiency and training accuracy.

[0076] S1.3: Arrange the checkpoints according to the path attributes, and store the gradient data collected by the checkpoints into the corresponding high-speed layer and low-speed layer.

[0077] Specifically, in ultra-large-scale training tasks, the arrangement of checkpoints is a crucial step in the restoration process. Checkpoints are intermediate states during training, which can record the gradient data at the current stage for future restoration. To improve the accuracy and efficiency of restoration, in this embodiment, the checkpoints are arranged according to the attributes of different paths, and the collected gradient data is stored in different levels. The gradient data on the key paths is collected frequently and stored in the high-speed layer, so that in case of a failure, these gradient data can be quickly accessed for restoration. For the gradient data of non-critical paths, checkpoint collection is performed at a lower frequency and stored in the low-speed layer.

[0078] In this embodiment, by combining the attributes of the gradient path with the design of the storage layer, the resource utilization during the training process is made more efficient. During fault-tolerant restoration, the gradient data of the key paths can be first obtained from the high-speed layer, so as to quickly restore the important training data; for the data of non-critical paths, it can be loaded on demand from the low-speed layer, avoiding unnecessary calculations and storage accesses. This on-demand access storage strategy can avoid frequent access to unimportant gradient data while ensuring the accuracy of fault-tolerant restoration, greatly improving the computing efficiency. In large-scale model training, the storage and restoration of gradients are often the bottlenecks of the system. Adopting this strategy can significantly improve the execution efficiency of training tasks and the reliability of fault-tolerant restoration.

[0079] The specific steps of S1.2 are as follows:

[0080] S1.2.1: Calculate the correction amplitude of each path in the gradient map within its continuous time window, where the correction amplitude represents the cumulative energy of the gradient changes of all computing nodes on this path;

[0081] Specifically, during the training process, the gradient map is composed of the gradient updates of multiple computing nodes. Each computing node generates different gradient values at different stages of training, and these gradient values change as the training progresses. Therefore, the cumulative energy of the gradient changes along each path can reflect the gradient update rate and influence degree of that path. To accurately calculate this correction amplitude, it is necessary to track the gradient changes within each time window, use the method of cumulative gradient difference to calculate the gradient correction amount for all computing nodes on each path, and perform weighted accumulation on the correction amounts to obtain the correction amplitude of that path. Among them, the way of weighted accumulation can be adjusted based on the activity of the computing node, the gradient update frequency, and the role of this path in the gradient propagation process. This can ensure that the correction amplitude of the path can truly reflect the trend of gradient changes and effectively avoid misjudgment caused by the gradient mutation of a single computing node.

[0082] Preferably, due to the high temporal correlation of gradient updates during the training process, the correction amplitude can also be calculated by means of a sliding time window to ensure the stability of the calculation result and prevent deviation in the overall path evaluation caused by local abnormal gradients.

[0083] S1.2.2: Assign a value to the correction amplitude according to the in-degree and out-degree of the path in the gradient amplitude;

[0084] In this embodiment, to further quantify the influence of each path on the global training process, a value is assigned to the correction amplitude according to the in-degree and out-degree in the gradient amplitude. The in-degree and out-degree respectively represent the situation of the gradient inflow and outflow participating in the calculation on this path during the gradient propagation process.

[0085] Specifically, the in-degree represents the number of gradient flows entering this path, and the out-degree represents the number of gradients flowing out of this path. Since the training tasks are usually highly parallel and the gradient propagation relationship between different computing nodes is complex during the gradient calculation process, it is necessary to analyze the in-degree and out-degree of the path to determine the contribution degree of this path in the training. During the value assignment process, the in-degree, out-degree, and correction amplitude of the path are comprehensively considered to ensure that the final value assignment can accurately reflect the importance of the path. Specifically, if a certain path has a high in-degree, it means that this path is a convergence path for the gradients of multiple computing nodes and has a strong gradient accumulation effect, so a higher weight is assigned to it; for a path with a high out-degree, it indicates that the gradients on this path affect the calculation results of multiple computing nodes, so a higher weight also needs to be assigned.

[0086] Preferably, in the actual calculation process, a time decay factor can also be introduced to reduce the influence weight of the gradient far from the current time window, ensuring the rationality and timeliness of the assignment. It can accurately measure the influence of the path on the global gradient propagation during the training process, avoid the deviation of importance evaluation caused by simply relying on the gradient magnitude calculation, and improve the accuracy of path evaluation.

[0087] S1.2.3: Sort the paths according to the assignment results, and identify critical paths and non-critical paths according to the dynamic threshold.

[0088] In this embodiment, based on the path assignment results calculated above, all paths are sorted, and critical paths and non-critical paths are identified according to the dynamic threshold. The purpose of sorting is to distinguish which paths have a greater impact on the training process, so as to ensure that the subsequent gradient recovery and computing resource allocation can be more accurate. To achieve effective sorting, a dynamic sorting algorithm based on path importance weights is adopted. This algorithm not only considers the correction amplitude, in-degree, and out-degree of the path, but also combines the overall distribution of gradient propagation during the training process to perform a global sort on the paths. During the sorting process, high-weight paths are given priority to ensure that these paths are processed first in the training task. Next, based on the sorting results, critical paths and non-critical paths are divided by a dynamic threshold, where the setting of the dynamic threshold is adjusted based on the real-time situation of the training task.

[0089] Specifically, in the initial stage of training, the model parameters are updated more violently and the gradient fluctuates greatly. Therefore, a lower threshold is set to be able to identify more critical paths and ensure that critical gradient information is supported by sufficient computing resources. In the later stage of training, as the model converges and the gradient changes tend to be stable, the threshold is gradually increased at this time to make the screening of critical paths more strict, thereby reducing unnecessary computing overhead and improving training efficiency.

[0090] Please refer to Figure 3 , the schematic diagram of time window division in the embodiment of the present invention, which illustrates the initial setting and hierarchical division of the time window during the gradient recovery process. After the computing node failure occurs, in order to accurately recover the lost gradient information, a time window system is first constructed based on the micro-checkpoint and divided into the first time window and the second time window. Among them, the first time window is used to mark the time interval when the failure occurs, while the second time window is set between the micro-checkpoint and the first time window to provide the initial recovery basis for gradient backtracking. This multi-level time window division can ensure that the recovery process has both accuracy and stability, avoid the accumulation of recovery errors caused by too large a direct backtracking span, and provide a structured data frame for subsequent gradient correction.

[0091] Please refer to Figure 4, the schematic diagram of the time window shift of the embodiment of the present invention further demonstrates the dynamic adjustment process of the time window. The third time window is the key interval for further refining the gradient recovery. Its position follows the second time window. In the actual training process, since the gradient change has certain nonlinear characteristics, the fixed time window backtracking may not accurately match the gradient state before the failure. In order to improve the recovery accuracy, this embodiment uses a dynamic time window shifting mechanism to gradually shift in the direction of the first time window on the basis of the initial second time window, and dynamically adjusts the length of the window according to the stability of the gradient in the current window. The setting of this shifting direction ensures that each backtracking can make full use of the existing gradient information, and enables the boundary of the time window to be flexibly adjusted according to the actual situation of the training state. Compared with the traditional fixed time window backtracking method, this method can reduce redundant gradient calculations, improve recovery efficiency, and ensure that the recovered gradient information is more consistent with the actual trajectory during the training process.

[0092] See also Figure 5 , a schematic diagram of termination determination in an embodiment of the present invention, in the process of time window shifting, a second-order derivative determination mechanism is introduced to optimize the accuracy of gradient recovery. In ultra-large-scale training tasks, the change of gradient is usually affected by multiple factors, especially in the early stage of training, the gradient may show greater volatility. Therefore, simply relying on the static adjustment of the time window may not fully reflect the true evolution of the gradient. In order to more accurately identify the key gradient change points, this embodiment introduces a second-order derivative determination mechanism, which determines the gradient change trend of the current window by calculating the second-order derivative of the gradient in different time windows. When the absolute value of the second-order derivative is large, it means that the gradient update of the time window is more intense. At this time, the recovery process should continue to be refined to avoid information loss caused by rough interpolation; when the second-order derivative is small, it means that the gradient change tends to be stable, indicating that the recovery effect of the current window is good, which can be used as a determination condition for terminating the shift. The time window can be adaptively adjusted to ensure that the recovery process can accurately capture the key gradient changes and reduce unnecessary computational overhead, thereby optimizing the recovery efficiency and stability of the training task.

[0093] The specific steps of S2 are as follows:

[0094] S2.1: Reading a first gradient snapshot corresponding to a first time window from the hierarchical storage structure, wherein the first time window represents a time window corresponding to a computing node failure, and the first gradient snapshot represents storage data of a micro-checkpoint before the first time window;

[0095] In this embodiment, first, after a computing node fails and during the process of restoring the training gradient, a first gradient snapshot corresponding to the first time window is extracted from the hierarchical storage structure. The hierarchical storage structure includes a high-speed layer and a low-speed layer. The high-speed layer stores important gradient snapshots that are frequently accessed during recent training processes, while the low-speed layer stores earlier training status information. To ensure the accuracy of training restoration, first, the gradient snapshot of the most recent micro-checkpoint is read from the high-speed layer as the first gradient snapshot, and combined with its version information and gradient change trend, it is ensured that the data closest to the failure point can be obtained during the backtracking process. During the reading process, the integrity of the gradient snapshot is automatically verified, and combined with the global parameter hash verification, to prevent data corruption problems caused by storage anomalies. Obtain the gradient information available for restoration within the shortest time, and at the same time avoid the restoration failure phenomenon caused by incomplete gradient snapshots.

[0096] S2.2: Set a second time window between the previous micro-checkpoint and the first time window according to the gradient change rate of the first gradient snapshot, and restore the second time window according to the gradient flow of the first gradient snapshot, where the left interval of the second time window is set as the previous micro-checkpoint;

[0097] In this embodiment, the range of the second time window is dynamically set according to the gradient change rate of the first gradient snapshot. Since the training gradient of the computing node does not change uniformly, there may be situations where the gradient changes violently in some training stages, while the gradient updates are relatively stable in some stages. Therefore, by calculating the gradient change rate in the first gradient snapshot, the gradient stability during this time period is identified, and the length of the second time window is determined according to this change trend. If the gradient change rate of the first gradient snapshot is large, it means that the training gradient fluctuates greatly during this time period, and a shorter time window needs to be used for fine restoration to reduce the error accumulation caused by gradient instability; if the gradient change rate is small, it means that the gradient update is relatively stable, and the time window can be appropriately lengthened to speed up the restoration progress. It can effectively reduce the error during the gradient restoration process, improve the restoration accuracy, and ensure that the restored training state is consistent with that before the failure.

[0098] The calculation formula for the gradient change rate is:

[0099] ,

[0100] where R represents the gradient change rate, which is used to measure the intensity of gradient update during the training process and reflects the fluctuation of the gradient between different time steps. N represents the total number of gradient data within the micro-checkpoint, and respectively represent the start time and end time of the micro-checkpoint, and the gradient change rate is calculated within this time range. denotes the gradient increment at time t, i.e., the gradient change value corresponding to this time step, denotes the gradient increment at time t + 1, i.e., the gradient change value at the next time step, denotes taking the norm, denotes a constant greater than zero;

[0101] In this embodiment, the length of the second time window is set by setting a change rate threshold. The initial length W of the second time window can be set to the time window length of the micro checkpoint, and then according to the gradient change rate R and the stable threshold and the unstable threshold the difference between them is used to weight the initial length. If , it indicates that the gradient change is stable, and the length of the second time window can be set to , if , it indicates that the gradient changes violently, and the time window needs to be reduced to finely recover the lost gradient information. The length of the second time window can be set to , if it is between the two, the time window size is adjusted by linear interpolation.

[0102] Furthermore, during the recovery process, corrections are made based on the gradient flow characteristics. By constructing a gradient flow direction map, the order of gradient updates during the recovery process is ensured to be correct, thus avoiding the problem of inconsistent training states caused by incorrect gradient backtracking order.

[0103] S2.3: Push the second time window towards the first time window by a set length, and recover the third time window within the set length range, where the set length is obtained based on the gradient change rate of the second time window;

[0104] In this embodiment, in order to further improve the accuracy of gradient recovery, based on the gradient change rate of the second time window, the time window is gradually pushed towards the first time window, and the third time window is defined within the set time range to recover the gradient data in stages. The calculation of the set length is similar to the steps in S2.2. Based on the gradient change trend of the second time window, if the gradient change rate within the second time window is relatively stable, a larger time window can be pushed to reduce the recovery time and improve the recovery efficiency; if the gradient change rate is large, a smaller time window is pushed to ensure the accuracy of the recovered gradient. In this way, it is possible to gradually approach the first time window and perform gradient recovery in an incremental manner to ensure the stability and accuracy of the entire backtracking process. In addition, when pushing the time window each time, a hash consistency check is performed on the gradient of the current window to judge the consistency between the recovered gradient and the global parameters. If a large gradient deviation is found, the time window length is readjusted to ensure the stability of the recovered gradient.

[0105] S2.4: Determine the second-order derivatives of the gradient at both ends of the right interval of the current second time window. If the termination condition is not met, continue to move the set length until the termination condition is met;

[0106] In this embodiment, in order to ensure the accuracy of gradient recovery and the stability of the training process, during the time window shift, the second-order derivatives of the gradient at both ends of the right interval of the time window are calculated to determine whether the termination condition is met. The physical meaning of the second-order derivative of the gradient is to measure the acceleration of the gradient change. If the second-order derivative is small, it means that the gradient update trend tends to be stable, which means that the recovery process is close to a stable state; if the second-order derivative is large, it means that the gradient change still has large fluctuations, and the time window needs to be continued to shift to ensure that the most stable gradient state is restored.

[0107] Specifically, the judgment condition is that the second-order derivative of the left gradient is less than the set threshold and the first-order derivative tends to zero, while the second-order derivative of the right gradient is greater than the set threshold. This actually means that the left gradient has tended to be stable, indicating that the quality of the restored gradient is high and is basically close to the normal state during the training process, while the second-order derivative of the right gradient is large, indicating that the gradient in this area still has drastic changes and may be greatly affected by the fault. If the time window continues to be pushed at this time, the recovery process may introduce more unstable factors, resulting in further accumulation of gradient errors. Therefore, it can be considered that the current window has reached an optimal critical point at this time. Moving it to the right will not only fail to bring better recovery effects, but may also destroy the stability of the existing recovery. Therefore, at this stage, it is determined that the recovery process has reached the convergence condition and the time window is stopped from being pushed to ensure that the restored gradient can maintain a high quality and will not affect the overall stability of the training due to the introduction of low-quality data.

[0108] S2.5: extending the second time window to be flush with the relatively far end of the first time window to restore the training gradient in the first time window;

[0109] In this embodiment, the second time window refers to a time window in the recovery process, and its position is based on the gradient snapshots before and after the failure point. This time window will start from a micro-checkpoint before the failure occurs and move to a period of time before and after the failure point. The first time window refers to the time window when the failure occurs, which usually corresponds to the moment when the failure occurs and the short time interval that follows. Between this time window and the micro-checkpoint, the computing node fails, resulting in the loss of part of the training gradient. The relatively far end is flush, which means that the right end of the second time window (i.e., the end close to the failure point in time) will extend to the left end of the first time window (i.e., the time interval before the failure point) and the relatively far end is flush, that is, the range of the shifted time window can span the lost part of the training gradient in the first time window.

[0110] Specifically, extend the second time window to the starting point of the first time window to recover the lost gradient data within the first time window. It can also be understood that move the right end of the second time window to the starting point of the first time window to cover and recover the gradient information within the first time window.

[0111] The specific steps of S2.2 are as follows:

[0112] S2.2.1: Calculate the gradient flow direction and rate of the first gradient snapshot;

[0113] In this embodiment, in order to ensure the accuracy of gradient recovery, it is necessary to first calculate the gradient flow direction and rate of the first gradient snapshot. The gradient flow direction reflects the main trend of gradient update, that is, during the training process, the gradient vector of parameter update generally tends to which direction, which determines the path of the next gradient adjustment. When calculating the flow direction, based on the stored first gradient snapshot, analyze the gradient changes in multiple historical time steps, and combine the calculation results of the computing nodes to extract the key gradient change trends.

[0114] Specifically, by comparing the gradient snapshots of adjacent time windows, calculate the vector change of the gradient, identify the direction where the gradient mainly converges or diverges, so as to determine the global gradient flow trend.

[0115] In this embodiment, because during the training of large-scale models, the update rate of the gradient may have non-linear changes, the gradient changes slowly in some areas, while in some areas, the gradient may fluctuate violently due to the dynamic adjustment of the optimizer. Therefore, when calculating the flow rate, it is necessary to measure the speed of gradient change, that is, the amplitude of gradient change per unit time.

[0116] Specifically, perform time weighting on the gradient changes of consecutive time windows, and eliminate instantaneous noise by the method of moving window mean, so that the calculated gradient flow rate is more stable.

[0117] S2.2.2: According to the flow direction and rate, calculate the gradient update trend through flow dynamics;

[0118] In this embodiment, the calculation of the gradient update trend is a crucial step in the gradient recovery process, because if relying solely on static gradient information for recovery, it is very likely to introduce excessive historical errors, resulting in inaccurate gradient recovery. Therefore, through the principle of flow dynamics, regard the gradient flow as a process of force change, and introduce an inertia term to correct the change trend of the gradient at different time steps.

[0119] Specifically, analyze the change rate of the gradient within the historical time window, and combine the gradient flow information of adjacent computing nodes to construct a gradient update trend model. The core idea of this model is that the change of the gradient not only depends on the gradient information at the current time step, but also is affected by the gradient flow in the previous few time steps. Therefore, introduce a historical gradient contribution factor, calculate the influence of gradients in different time windows on the current time step, and combine the computing power of the current computing node to predict the future gradient change. It can effectively solve the problem of gradient loss caused by computing node failures, improve the accuracy of gradient backtracking, and at the same time ensure that the restored gradient can maintain the convergence of the global model.

[0120] Preferably, in order to prevent excessive noise from being introduced during the gradient recovery process, when calculating the update trend, the weight of the historical gradient is adaptively adjusted according to the stability of the current training task to ensure that the restored gradient will not have too much impact on the training process. It can not only improve the stability of gradient recovery, but also reduce the parameter offset after recovery, making the training process more robust.

[0121] S2.2.3: According to the gradient update trend, start incremental correction from the left interval of the second time window until the correction of the right interval of the second time window is completed;

[0122] In this embodiment, the core idea of incremental correction is that gradient recovery should not be a one-time replacement process, but a dynamic adjustment process.

[0123] Specifically, this application does not directly use the stored gradient snapshot for recovery, but combines the calculated gradient update trend to dynamically adjust the restored gradient value to make it closer to the real gradient change during the training process. When performing incremental correction, the information of different gradient points within the time window is gradually introduced, so that the restored gradient can better conform to the overall gradient change trend, rather than an isolated static value. It can effectively reduce the impact on the global training model after gradient recovery, so that the restored gradient can naturally integrate into the current training state without causing model instability. During the specific execution process, start from the left interval of the second time window and gradually expand the correction to the right. Each time step will be adjusted based on the correction result of the previous time step to ensure that the corrected gradient can maintain global consistency.

[0124] Preferably, during the correction process, a dynamic adjustment strategy is also introduced. That is, if the gradient change at a certain time step is abnormal, for example, there is a large gradient mutation, the correction process will be paused, and the gradient flow direction at this time step will be recalculated to prevent incorrect gradients from affecting the entire recovery process. Compared with the traditional full-scale recovery method, this incremental correction method can minimize the training error caused by recovery to the greatest extent, improve the accuracy of gradient recovery, and ensure the stability of the model and the continuity of training.

[0125] The specific steps of S2.3 are as follows:

[0126] S2.3.1: Load the gradient map based on the second time window before the shift, and process the gradient map through low-rank approximation to generate the reconstructed gradient of the third time window;

[0127] In this embodiment, in order to improve the recovery accuracy and calculation efficiency of the training gradient, within the second time window before the shift, the corresponding gradient map is first loaded. This gradient map is constructed based on the gradient transmission and calculation dependency relationships between computing nodes during the training process, and includes local gradient information, parameter versions, historical gradient update trajectories, and data flow dependency information of each computing node. Due to large-scale parallel computing during the training process, gradient information is usually high-dimensional sparse data. If the complete gradient data is directly used for recovery, the computational cost is huge, and redundant information may be introduced, resulting in low recovery efficiency.

[0128] Specifically, this embodiment uses the low-rank approximation method to process the gradient map to reduce the data complexity and extract the core gradient information. The basic idea of low-rank approximation is to remove redundant information in the gradient data through dimensionality reduction methods, and only retain the main gradient components that affect the training state, so as to realize the reconstruction of the gradient within the third time window. In practical applications, based on the gradient contribution degree, parameter update amplitude, and gradient correlation of each computing node during the training process, a weight matrix is constructed and matrix decomposition is performed to extract the low-dimensional representation that mainly affects gradient recovery. It can retain the global structure of the gradient data, effectively compress the data scale, reduce the computational cost, and enable faster calculation and reconstruction of missing gradients during the gradient backtracking process.

[0129] Furthermore, low-rank approximation can also reduce errors caused by random noise or abnormal gradient changes, and enhance the stability of gradient recovery. After completing the low-rank approximation calculation, use the extracted principal component gradients to construct a gradient reconstruction model for the third time window to ensure that the gradient information in this window is as close as possible to the real training state, and avoid the decline in computational accuracy and the interruption of the training process caused by gradient loss.

[0130] S2.3.2: Calculate the local consistent hash of the reconstructed gradient and the global parameters. If the hash matches, restore the third time window according to the reconstructed gradient. If it does not match, recalculate the set length of the second time window according to the local consistent hash, where the global parameters represent the model parameters globally synchronized during the training process.

[0131] In this embodiment, to ensure the correctness and global consistency of the reconstructed gradient, before restoring the gradient in the third time window, first calculate the local consistent hash of the reconstructed gradient and the global parameters. Since in ultra-large-scale training tasks, gradient calculation involves the interaction of multiple computing nodes, the gradient calculation error of any computing node may affect the convergence of the global model. Therefore, the gradient restoration process must ensure its consistency with the global parameters. The local consistent hash is a method to verify whether the gradient is consistent with the global parameter state through hash calculation. Its core idea is to calculate a unique hash fingerprint based on the model parameter version, gradient update sequence, and historical training state to ensure that the data after gradient restoration is consistent with the state during the training process. During the calculation process, a hierarchical hash method is adopted to calculate the hash values between the local computing node and the global parameter server respectively. If the local consistent hash matches, it proves that the restored gradient conforms to the evolution trajectory of the global model and can be directly used for the restoration of the third time window. If the hash value does not match, it indicates that there may be an error in the current reconstructed gradient, which may be due to information loss caused by gradient approximation processing or incomplete restoration due to too short a backtracking time window. At this time, it is necessary to readjust the set length of the second time window.

[0132] Specifically, the new time window length will be calculated according to the matching deviation of the local consistent hash, so that the time window covers a more complete training gradient to ensure that the restored gradient information is as close as possible to the true evolution state of the global parameters. It can dynamically adjust the backtracking range, avoid recovery failure caused by unreasonable window setting, and reduce the computational overhead at the same time, enabling the training task to quickly recover after a failure and ensuring the stability and training efficiency of the model. In addition, the local consistent hash can also detect gradient distortion problems caused by storage or communication errors, improving the robustness and reliability of gradient restoration.

[0133] The specific steps of S2.3.2 are as follows:

[0134] S2.3.2.1: Load the causal dependency chain of the critical path within the second time window from the high-speed layer, replay the gradient aggregation operation according to the version number order, and restore the gradient information of the critical path corresponding to the third time window;

[0135] In this embodiment, since the recovery of the second time window is based on the real data of the micro-checkpoint, while the recovery of the third time window does not have the same direct data support, it is necessary to use the causal dependency chain to accurately replay the gradient aggregation operation to maximize the recovery of the lost gradient information.

[0136] Specifically, the gradient information stored in the high-speed layer mainly includes the causal relationships of the critical paths within the time window. By constructing a causal dependency chain, the gradient propagation order of the computing nodes before the failure can be traced. The update of gradients is often serial or hierarchical. Especially in ultra-large-scale training, there are strict time dependencies between the gradient updates of different computing nodes. Therefore, according to the stored causal dependency information, starting from the forward propagation stage, the gradient aggregation operation is gradually replayed in the order of the version numbers to recover the gradients of the critical paths within the third time window. Here, the gradient aggregation operation refers to the gradient information contributed by each computing node to the same batch of data during asynchronous computing times in distributed training. A certain version control mechanism is required to ensure that the gradients of all computing nodes can be correctly synchronized and applied to the global model.

[0137] Specifically, the third time window has exceeded the direct storage range and direct influence range of the micro-checkpoint. Therefore, it is impossible to directly recover its gradients using traditional backtracking methods, but it must rely on historical calculation information for recovery. Through the causal dependency chain, it can be ensured that the order of gradient replay is strictly in accordance with the execution order of the computing nodes during the training process. This can effectively avoid gradient confusion or gradient drift caused by incorrect replay order. It can also minimize the gradient loss caused by communication delays or uneven computing node loads, ensuring that the recovered gradients are consistent with the original calculation process, thereby improving the accuracy of recovery and the convergence of training.

[0138] S2.3.2.2: Load the gradient information of the non-critical paths within the second time window from the low-speed layer, and recover the gradient information of the corresponding critical paths in the third time window according to gradient interpolation and linear regression;

[0139] In this embodiment, since the gradients of the non-critical paths usually do not significantly affect the stability of global training, when recovering the gradients of the third time window, it relies on the data stored in the low-speed layer and uses mathematical methods such as gradient interpolation and linear regression to infer the missing gradient information. The gradient information of the non-critical paths is stored in the low-speed layer, and its data integrity is lower than that of the critical paths. Therefore, it is impossible to directly use the gradient backtracking method for recovery, but it is necessary to infer in combination with the changing trend of historical gradients.

[0140] Specifically, we first extract the non-critical path gradient data of the second time window from the low-speed layer, and analyze the change trend of these gradients in the time window. By comparing the time change patterns of these gradients, we can use the interpolation algorithm to perform preliminary recovery to make it consistent with the overall gradient change trend.

[0141] Furthermore, relying solely on interpolation may bring certain errors. Therefore, on this basis, a linear regression method is further introduced to ensure the stability of the gradient recovery process. The role of linear regression is to infer the optimal fitting value of the lost gradient based on the known gradient information, so as to avoid the gradient being not smooth or not matching the global gradient update direction due to simple interpolation calculation. Especially in non-critical paths, due to the relatively low frequency of gradient updates, the time evolution of the gradient often shows a strong linear correlation, which enables the linear regression method to fit the lost gradient points more accurately. During the recovery process, the gradient change curve is first constructed using historical data, and the fitting curve is optimized by the least squares method to obtain a regression equation that can predict the missing gradient. Finally, the gradient of the third time window is inferred by combining the interpolation result and the regression calculation result to ensure that the restored gradient is as close as possible to the gradient value in the actual calculation process. It does not need to rely entirely on stored data, but can be combined with historical trends for gradient recovery, thereby effectively avoiding the data loss problem caused by the storage level. At the same time, since the gradients of non-critical paths usually do not have a decisive impact on the gradient flow of the overall training, the use of interpolation and regression analysis for recovery not only has low computational overhead, but also ensures the rationality of the restored gradients, avoiding the impact of computational errors in the recovery process on the global training effect.

[0142] S2.3.2.3: Perform weighted correction on the reconstructed gradient according to the restoration result.

[0143] In this embodiment, since the recovery methods of the critical path and the non-critical path are different, there is a deviation in the value of the restored gradient. Therefore, this application adopts a weighted correction method based on the gradient contribution to improve the reliability of the restored gradient.

[0144] Specifically, higher weights need to be assigned to the gradients of the critical path, while lower weights are assigned to the gradients of the non-critical path. By fusing the recovered gradient data, the final corrected gradient value is calculated. The calculation process of weighted correction mainly includes the following steps: First, normalize all the recovered gradient data to eliminate the inconsistency in the gradient scale; Second, calculate the weight distribution according to the gradient contribution degrees of the critical path and the non-critical path, where the gradient weight of the critical path is determined by its historical calculation contribution rate, and the gradient weight of the non-critical path is determined by the fitting error of linear regression; Finally, use the weighted average method to calculate the final gradient update value and apply it to the parameter update of the training model. This can ensure that the overall direction of the recovered gradient is consistent with the gradient update direction in the original training process, thereby effectively reducing the problem of error accumulation in the gradient recovery process.

[0145] Furthermore, this weighted correction method can also improve the stability of the recovered gradient and reduce the impact of noise generated during the recovery process on the final training result. It can improve the efficiency of fault-tolerant recovery while ensuring the training accuracy, so that even if a computing node fails in an ultra-large-scale training task, the gradient can be recovered in a short time and training can continue, avoiding the problem of model performance degradation caused by gradient loss.

[0146] Embodiment 2:

[0147] Please refer to Figure 6 , the present invention provides an embodiment: a fault-tolerant recovery system for ultra-ten-thousand-card-level training and inference tasks, and the system includes a preprocessing module and a gradient recovery module;

[0148] The preprocessing module is used to construct a hierarchical storage structure of computing nodes based on training data and network topology before the training task starts, initialize the relevant data of each time window, and generate a preliminary gradient map and checkpoint configuration;

[0149] The gradient recovery module is used to recover the lost gradients of the training task when a failure occurs;

[0150] The preprocessing module includes:

[0151] The time window division unit automatically divides multiple time windows according to the specific requirements of the training task and sets the size and time step of each window;

[0152] The storage structure initialization unit initializes the hierarchical storage structure according to the computing resource configuration of the training (such as the number of computing nodes, memory capacity, etc.), and pre-allocates the space of the high-speed layer and the low-speed layer;

[0153] Gradient map generation unit, through preprocessing of training data, automatically generates gradient maps of each computing node during the training process, and provides support for subsequent fault-tolerant recovery;

[0154] The gradient recovery module includes:

[0155] Global gradient backtracking unit, when a computing node fails, quickly locates the fault time window, and recovers the training gradient related to the fault through the global gradient backtracking mechanism;

[0156] Time window reconstruction unit, based on the current time window and the gradient information of the previous micro-checkpoint, moves to a new time window according to the gradient change rate, and reconstructs the missing gradient through methods such as interpolation algorithm and linear regression;

[0157] Causal dependency chain recovery unit, based on the gradient recovery of the critical path and the reconstruction of the non-critical path, adopts the causal dependency chain strategy to replay the gradient aggregation operation, corrects the errors in the recovery process, and ensures the accuracy of the recovery result.

[0158] Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

Claims

1. A fault-tolerant recovery method for super-10,000-card-level training and reasoning tasks, applied to the training phase of a large cross-modal model, characterized by: The fault-tolerant recovery method comprises: Establish a hierarchical storage structure of computing nodes at the beginning of training; When the computing node fails, the training gradient is restored by global gradient backtracking according to the hierarchical storage structure, wherein the restoring of the training gradient includes: Set the time window according to the stored gradient snapshots and gradually adjust the window length; Restore the gradient training state through incremental correction; If the rollback fails, roll back to the micro-checkpoint for recovery.

2. The fault-tolerant recovery method for super-10,000-card-level training and reasoning tasks according to claim 1 is characterized in that: The hierarchical storage structure is implemented through incremental gradient graphs and dynamic causal checkpoints, including a high-speed layer and a low-speed layer, including: Preset continuous time windows according to the training time, and generate a corresponding gradient map for each window, wherein the gradient map includes map nodes, map edges and map metadata; According to the cause-effect relationship of the gradient map, identifying the critical path and the non-critical path, wherein the identification of the path is determined based on the correction amplitude of the gradient path; Checkpoints are arranged according to the path attributes, and the gradient data collected at the checkpoints are stored in the corresponding high-speed layer and low-speed layer.

3. The fault-tolerant recovery method for super-10,000-card-level training and reasoning tasks according to claim 2 is characterized in that: According to the causal relationship of the gradient map, the critical path and the non-critical path are identified, including: Calculating the correction amplitude of each path in the gradient map within its continuous time window, wherein the correction amplitude represents the cumulative energy of the gradient changes of all computing nodes on the path; Assigning a value to the correction amplitude according to the in-degree and out-degree of the path in the gradient amplitude; The paths are sorted according to the assignment results, and the critical paths and non-critical paths are identified based on dynamic thresholds.

4. The fault-tolerant recovery method for super-10,000-card-level training and reasoning tasks according to claim 3 is characterized in that: Arranging checkpoints according to path attributes and storing the gradient data collected at the checkpoints to the corresponding high-speed layer and low-speed layer includes: Arranging high-frequency micro-checkpoints on the critical path, wherein the high-frequency micro-checkpoints are set according to the gradient change rate of the critical path; A low-frequency micro-checkpoint is arranged on the non-critical path, wherein the low-frequency micro-checkpoint is set according to the gradient change rate of the non-critical path.

5. The fault-tolerant recovery method for super-10,000-card-level training and reasoning tasks according to claim 2 is characterized in that: The recovery of the training gradient by global gradient backtracking includes: Reading a first gradient snapshot corresponding to a first time window from the hierarchical storage structure, wherein the first time window represents a time window corresponding to a computing node failure, and the first gradient snapshot represents storage data of a previous micro-checkpoint of the first time window; According to the gradient change rate of the first gradient snapshot, a second time window is set between the previous micro-checkpoint and the first time window, and the second time window is restored according to the gradient flow of the first gradient snapshot, wherein the left interval of the second time window is set as the previous micro-checkpoint; Shifting the second time window toward the first time window by a set length, and restoring a third time window within the set length, wherein the set length is obtained based on the gradient change rate of the second time window; Determine the second-order derivatives of the gradient at both ends of the right interval of the current second time window. If the termination condition is not met, continue to move the set length until the termination condition is met. The second time window is extended to the left interval of the first time window to restore the training gradient in the first time window.

6. The fault-tolerant recovery method for super-10,000-card-level training and reasoning tasks according to claim 5 is characterized in that: The restoring the second time window according to the gradient flow of the first gradient snapshot includes: Calculating the gradient flow direction and rate of the first gradient snapshot; calculating a gradient update trend through flow dynamics according to the flow direction and rate; According to the gradient update trend, incremental correction is performed starting from the left interval of the second time window until the right interval of the second time window is corrected.

7. The fault-tolerant recovery method for super-10,000-card-level training and reasoning tasks according to claim 5 is characterized in that: Restoring the third time window within the set length range includes: Loading a gradient map based on the second time window before the shift, processing the gradient map by low-rank approximation, and generating a reconstructed gradient of the third time window; Calculate the local consistent hash of the reconstructed gradient and the global parameters. If the hashes match, restore the third time window according to the reconstructed gradient. If they do not match, recalculate the set length of the second time window according to the local consistent hash, where the global parameters represent the model parameters that are globally synchronized during the training process.

8. The fault-tolerant recovery method for super-10,000-card-level training and reasoning tasks according to claim 7 is characterized in that: The restoring the third time window according to the reconstruction gradient includes: Load the causal dependency chain of the critical path in the second time window from the high-speed layer, replay the gradient aggregation operation according to the version number sequence, and restore the gradient information of the critical path corresponding to the third time window; Loading the gradient information of the non-critical path in the second time window from the low-speed layer, and restoring the gradient information of the critical path corresponding to the third time window according to gradient interpolation and linear regression; The reconstructed gradient is weightedly corrected according to the restoration result.

9. The fault-tolerant recovery method for super-10,000-card-level training and reasoning tasks according to claim 5, characterized in that: The termination conditions include: The second-order derivative of the left gradient is less than the set threshold and the first-order derivative tends to zero, while the second-order derivative of the right gradient is greater than the set threshold.

10. A fault-tolerant recovery system for super-10,000-card-level training and reasoning tasks, used to implement the fault-tolerant recovery method for super-10,000-card-level training and reasoning tasks as described in any one of claims 1 to 9, characterized in that: The system includes a preprocessing module and a gradient recovery module; The preprocessing module is used to construct a hierarchical storage structure of computing nodes based on training data and network topology before the training task starts, initialize relevant data of each time window, and generate a preliminary gradient map and checkpoint configuration; The gradient recovery module is used to recover the lost gradient of the training task when a failure occurs.

Images