Checkpoint remote backup method based on large model parallel policy awareness

Abstract

A check point remote backup method based on large model parallel strategy perception comprises the following steps: S1, constructing a Checkpoint backup scheme: S11, constructing different cross-node parameter backup mapping relationships according to different parallel modes; s12, constructing a model parameter fragment distribution scheme according to the parameter load capacity of each card and the redundant copy number; s13, grouping the training nodes according to the number of backup copies set by a user, and generating a plurality of full mutual backup groups with equal sizes; the training nodes in the full mutual backup group are mutually backed up; s2, managing memory allocation in a Checkpoint backup process on the basis of a memory multiplexing mechanism; scheduling of CPU resources in the Checkpoint data transmission process is managed according to the asynchronous transmission process; s3, realizing punctual writing of the Checkpoint through a round and time threshold combined triggering method; and S4, controlling the startup and scheduling of the Checkpoint backup. According to the method, when the node breaks down, the system can quickly recover the model state from other nodes according to a set backup strategy, and the continuity of a training task and the fault-tolerant capability of the system are ensured.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of large model distributed training, and in particular to a checkpoint remote backup method based on large model parallel strategy perception. BACKGROUND

[0002] With the rapid development of deep learning technology, especially the continuous evolution of large-scale models such as large language models and generative models, the demand for computing resources, storage space, and network bandwidth required for model training continues to grow. To meet the high-intensity training needs, current mainstream training tasks usually rely on distributed clusters composed of thousands or even tens of thousands of GPUs or NPUs, and the training period may last for several weeks or even months. As the cluster size expands, the probability of any node failure in the system significantly increases, and the training process is more susceptible to hardware failure, system crash, or network interruption, which can cause training interruption, resource waste, and even model convergence failure. To improve the fault tolerance and stability of training tasks, Checkpoint technology is widely used in distributed deep learning systems. This technology periodically saves intermediate states such as model parameters, optimizer states, and training steps during the training process, allowing the system to continue training from the last Checkpoint state after a system failure, avoiding starting from scratch, and significantly reducing resource consumption and time loss.

[0003] However, as the scale of distributed training continues to expand, the traditional Checkpoint mechanism exposes many shortcomings in data backup and recovery efficiency. Once a node fails, its local Checkpoint data may be lost, and the system needs to pull data from remote storage for recovery. Limited by network bandwidth, storage access speed, and the stability of the recovery path, the recovery process often takes a long time, affecting the overall efficiency of training. In addition, existing solutions do not fully consider factors such as node failure probability, network structure, and training task characteristics, making it difficult to achieve an efficient and reliable recovery strategy. Therefore, improving the high availability of Checkpoint backup and the rapid response capability of recovery has become a key technical problem that needs to be solved in distributed large-scale deep learning training systems. SUMMARY

[0004] The application aims at the problems of poor checkpoint backup reliability and long recovery time in large-scale distributed deep learning training, and proposes a checkpoint remote backup method based on large model parallel strategy perception. According to the type of parallel strategy adopted by each node and whether the checkpoint uses a redundant storage mechanism, the backup strategy is dynamically adjusted to complete the cross-node checkpoint data backup without interference in the training process. When a node fails, the system can quickly recover the model state from other nodes according to the established backup strategy, ensuring the continuity of the training task and the fault tolerance of the system.

[0005] In order to achieve the above purpose, the technical solutions adopted by the application are as follows:

[0006] The checkpoint remote backup method based on large model parallel strategy perception comprises the following steps:

[0007] S1, constructing a checkpoint backup scheme, comprising:

[0008] S11, constructing different cross-node parameter backup mapping relationships according to different parallel modes;

[0009] S12, constructing a model parameter sharding allocation scheme according to the parameter load of each card and the number of redundant copies;

[0010] S13, grouping the training nodes according to the number of backup copies set by the user to generate multiple full mutual backup groups with equal size; the training nodes in the full mutual backup group backup each other;

[0011] S2, managing memory allocation in the checkpoint backup process based on the memory reuse mechanism; managing the scheduling of CPU resources in the checkpoint data transmission process according to the asynchronous transmission process;

[0012] S3, realizing the on-time writing of checkpoint through the round and time threshold joint triggering method;

[0013] S4, controlling the starting and scheduling of checkpoint backup according to the checkpoint backup scheme and triggering conditions.

[0014] As a preferred, in the step S11, if the parallel mode is data parallel, the cross-node parameter backup mapping relationship is that a complete model parameter copy is saved on each training node;

[0015] If the parallel mode is tensor parallel, pipeline parallel or hybrid parallel, the cross-node parameter backup mapping relationship is that the model parameter copy is divided into multiple non-overlapping subsets, and each training node holds part of the model parameter data.

[0016] As preferred, in step S13, for the remaining training nodes which cannot be evenly divided, a one-way dependent cycle backup chain is constructed through a ring topology, so that each training node is the backup target of its predecessor node.

[0017] As preferred, in step S4, the backup node sends a wait_flag signal through a semaphore mechanism to detect whether the previous backup task is completed; if the previous task has not been completed, the backup sub-process is blocked and enters a waiting state until a ready_flag = 1 signal is received.

[0018] As preferred, in step S4, data transmission between the source node and the backup node is performed through a transmission channel based on the TCP protocol; the transmission process adopts a CPU-to-CPU communication mode; the source node divides the Checkpoint data to be transmitted into a plurality of non-overlapping data block sets {c i,1 ,c i,2 ,…,c i,M}, each data block being disjoint from each other; the system independently schedules and sends each data block in parallel through a multi-thread mechanism.

[0019] As preferred, in step S4, after the receiving end completes the reception and writing of the data block set {c i,1 ,c i,2 ,…,c i,M}, the local state flag is updated to ckpt_state = True, indicating that the current backup task is successfully completed; the receiving end generates an acknowledgement signal ack_flag = 1 and feeds it back to the source node as the basis for the next round of transmission scheduling.

[0020] As preferred, S5, the system sends a heartbeat signal h i (t) to each training node at a fixed time interval Δt and requires the training node to return a response signal r i (t) within a specified timeout time τ; if the training node does not respond within the timeout time, the training node is determined to be faulty; the scheduling module selects an idle backup node from the backup node pool to replace it.

[0021] As preferred, S6, fast state recovery of non-faulty nodes is achieved through a shared storage path, and the Checkpoint backup scheme is used to guide the faulty node to asynchronously pull data from the backup node.

[0022] As preferred, step S6 includes:

[0023] S61, the system queries the shared memory path of the non-faulty node, directly loads the last completed model parameter copy, and completes state reconstruction; the loading operation is performed through a function Load(v i ,C i (t k )) execution;

[0024] S62, the system determines its corresponding backup node set V b ={v j ∣b ji =1} according to the cross-node parameter backup mapping relationship; the transmission function asynchronously pulls the model parameter copy from the node v i .

[0025] Compared with the prior art, the beneficial effects of the present application are reflected in:

[0026] Unlike the Checkpoint backup in the conventional technology, which only relies on remote storage and has a single backup mode, the present application dynamically constructs a multi-node backup strategy based on the large model parallel strategy awareness, in combination with the model parameter distribution characteristics and the cluster topology structure. The inter-node backup is realized through an efficient CPU-to-CPU transmission channel, effectively optimizing the backup copy number and distribution, significantly improving the Checkpoint backup speed and reliability, and enhancing the fault tolerance of the system.

[0027] In addition, the conventional technology usually needs to pull the Checkpoint from the remote storage when a node fails, and the recovery process is restricted by the network bandwidth and storage access bottleneck, resulting in low recovery efficiency. To solve this problem, the present application adopts an asynchronous recovery method based on a preset inter-node backup mapping relationship, supports the faulty node to directly pull the required Checkpoint from its corresponding backup node, realizes the localization of the recovery path and the high-concurrency execution of the recovery process. This method enables the rapid recovery of the training task, effectively ensuring the stability and efficiency of the distributed large-scale deep learning training. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 is the method architecture diagram of the embodiment 1 of the present application;

[0029] Figure 2 is the method flowchart of the embodiment 1 of the present application. DETAILED DESCRIPTION

[0030] In order to make the technical means, creative features, purposes and effects of the invention easy to understand, the present application will be further described in conjunction with specific drawings. However, the present application is not limited to the following embodiments.

[0031] It is to be understood that the structures, proportions, sizes, etc. shown in the drawings accompanying the present specification are merely intended to cooperate with the content disclosed in the specification for the understanding and reading of those skilled in the art, and are not intended to define the limiting conditions for the implementation of the present application, and therefore do not have technical substantive significance. Any modification of the structure, change of the proportion relationship or adjustment of the size, without affecting the effects that can be produced by the present application and the purposes that can be achieved, should still fall within the scope covered by the disclosed technology content of the present application.

[0032] Embodiment 1:

[0033] As shown in the checkpoint remote backup method based on large model parallel strategy perception, comprising the following steps: Figure 1 , 2 The checkpoint remote backup method based on large model parallel strategy perception comprises the following steps:

[0034] S1, design a multi-node backup strategy. Based on the mapping mechanism of topology perception, the backup mapping relationship between nodes is established, and combined with the redundancy mechanism and the training task characteristics, a multi-node collaborative backup strategy method is used to generate a complete Checkpoint backup scheme, covering the number of copies, node distribution and transmission path, etc. Key elements to achieve dynamic adaptive backup scheduling.

[0035] S11, construct a cross-node parameter backup mapping relationship based on the training task parallel strategy.

[0036] Collect and obtain the training task configuration, which includes parallel strategy type, node size, topology structure and other information. According to the model parameter distribution characteristics under different parallel modes, the backup mapping relationship of cross-node parameters is generated.

[0037] If the parallel mode is data parallel (DP), then each training node V={v1, v2, …, v N} on each training node V={v1, v2, …, v i} on each training node V={v1, v2, …, v N} on each training node V={v1, v2, …, v i} on each training node V={v1, v2, …, v

[0038] If the parallel mode is tensor parallel (TP), pipeline parallel (PP) or hybrid parallel, the model parameters will be divided into multiple non-overlapping subsets {M i} that satisfy ∪ i∈N M i Under such parallel strategies, each node only holds part of the parameter data, and needs to additionally build a cross-node backup relationship. For this purpose, an N×N backup mapping matrix B=[b ij ]∈{0,1} N×N , where:

[0039]

[0040] According to the matrix B, a parameter backup task set S = {(i, j, M i ) is constructed, and the task set is distributed to each backup sub-process for execution, thereby realizing off-site redundant backup of cross-node model parameters and enhancing the fault tolerance and recovery capability of the system.

[0041] S12, a model parameter fragmentation allocation scheme is set based on the checkpoint redundant storage, that is, the model parameters are divided into multiple subsets (fragments) and allocated to different computing nodes (such as GPU cards) for storage and calculation, thereby realizing efficient data division and transmission load balancing under a multi-copy structure.

[0042] According to the enabled state of the redundancy mechanism in the training task, the parameter load (the total number or total size of the model parameters allocated to each computing node) and the number of redundant copies are combined to generate a parameter copy distribution pattern consistent with the redundancy structure. Assuming that the model parameter quantity of each card is M and the number of data parallel redundant copies is K, each parameter copy is distributed on K cards, forming a K-fold redundant storage structure. The model parameters are divided into K equal subsets and allocated to each card, and only the corresponding subset According to the backup relationship, the parameter (cross-node parameter backup matrix B and corresponding transmission subset M i ) fragmentation backup and transmission tasks are scheduled, thereby reducing the single-card backup data volume and transmission overhead, and improving the transmission efficiency and network bandwidth utilization.

[0043] S13, a backup group construction method combining group division and ring link construction is adopted to realize redundant fault tolerance layout in a multi-node environment.

[0044] The system groups the training node set according to the number of backup copies set by the user, forms several full-mutual backup groups of equal size, realizes complete backup relationship between nodes in the group, and enhances the local fault recovery capability. For the remaining nodes that cannot be evenly divided, a ring topology structure is introduced to construct a one-way dependent circular backup chain, so that each node is the backup target of its predecessor node, thereby forming a closed loop path with a transfer redundancy capability. Assuming that the total number of nodes is N, the nodes are divided into groups of size C, and the remaining r = N - m × C nodes form a ring group. The nodes in the group backup each other, and the nodes in the ring group backup each other to form a closed loop structure, thereby ensuring data redundancy and recovery capability.

[0045] S2, based on memory reuse mechanism and asynchronous transmission process to achieve efficient management of resources and non-blocking data transmission in the checkpoint backup process. In each training node, according to the model parameter quantity, the checkpoint size S is estimated, and the static pre-allocation of the corresponding memory area is completed, ensuring that the data persistence operation has sufficient space support. The unified memory reuse mechanism is introduced to effectively avoid frequent application and release overhead, and improve the utilization of memory resources. At the same time, an independent checkpoint transmission sub-process is started in each node to complete the distribution and writing of checkpoint data blocks in a concurrent manner, effectively decoupling the computing and communication processes, and improving the system throughput and backup efficiency in the distributed training process.

[0046] S21, through the accurate calculation of model parameter quantity and byte number, the pre-allocation and reuse management of checkpoint memory space are realized. According to the parameter quantity P of the model to be backed up and the byte number b occupied by each parameter, the memory size S required by the checkpoint is calculated, and static allocation is completed in the host memory during the training initialization phase. The calculation formula is as follows:

[0047] S=P×b

[0048] A memory area with a size of copy_ckpt is pre-allocated in the host memory, satisfying copy_ckpt≥S. To avoid frequent memory application and release overhead, the memory reuse mechanism is introduced, and all subsequent checkpoint writing operations reuse this area.

[0049] S22, an independent process pre-starting method is used to realize the resource pre-allocation and data preparation management of the training node checkpoint backup. In each training node v i , the system pre-starts an independent checkpoint preparation process, denoted as The process is responsible for starting by the master program during the training initialization phase, and is resident on the CPU of the node, and is used to allocate resources and pre-load data for subsequent checkpoint backup operations. This method establishes a one-to-one correspondence between each node and an auxiliary processing unit, ensuring that the training main process and the checkpoint path are decoupled during runtime, thereby effectively improving the response speed of the backup operation and the parallel scheduling ability of the system as a whole. Specifically, let the training node set be V={v1,v2,…,v N} then for each node v i ∈V, there is a corresponding independent process to ensure that the checkpoint data can be transmitted and written in a timely and stable manner during the training process.

[0050] S3, through the round and time threshold combined triggering method, the timely writing of checkpoint to shared memory and data persistence management are realized. The system continuously tracks the iteration progress in the training process, and determines whether to trigger the checkpoint saving operation according to the preset round interval or time interval condition. When any triggering condition is met, the writing operation of state snapshot is executed, and the key information such as model parameters, optimizer state and training progress is stored in the shared storage path Pshared, providing consistency guarantee and data basis for subsequent cross-node backup and fault recovery.

[0051] Through the threshold judgment of saving frequency and time interval, the timely saving of training process checkpoint and data consistency guarantee are realized. In the training process, after completing the tth iteration, the system calculates the difference Δt=t-t last and time difference Δτ=τt-τ last When the conditions Δt≥k or Δτ≥ΔT are met, the writing operation of checkpoint data to shared storage is automatically triggered, covering model parameters, optimizer state and training progress, ensuring the integrity and consistency of backup data.

[0052] S4, based on the cooperative mechanism of CPU data transmission, the efficient parallel scheduling of checkpoint backup and training process is realized. After each round of iteration is completed and the checkpoint is written into the shared memory, according to the predefined backup strategy and the triggering condition of control flag, the inter-node checkpoint backup stage is entered. Through the direct communication mechanism between nodes, the asynchronous distribution and node writing operation of checkpoint data block are realized, and the multi-copy storage structure is constructed. This mechanism ensures the consistency and version synchronization of checkpoint data among multiple computing nodes, effectively improves the fault tolerance and system stability in the distributed training process.

[0053] S41, the condition judgment mechanism of configuration parameters is used to realize the orderly start and resource coordination management of inter-node checkpoint backup process. According to the ENABLE_NODE_CKPT_STORE∈{0,1} set in the training configuration and the backup frequency parameter f b Condition judgment is performed: if the current round t satisfies tmod f bCheckpoint backup flow. The backup node sends a wait_flag signal through the semaphore mechanism to detect whether the previous backup task is completed. If the previous task has not been completed, the system blocks the backup sub-process and enters a waiting state until it receives a ready_flag = 1 signal and then continues. After entering the backup flow, the system sends the current Checkpoint data size S ckpt to the target backup node through the network, and ensures the integrity and alignment requirements of the transmission according to the already applied memory area.

[0054] S42, high-performance block transmission and resource isolation of Checkpoint data are realized through the establishment of an efficient TCP transmission channel and a multi-thread scheduling mechanism. In a training node set V = {v1, v2, …, vn}, the source node vi and the target backup node vj are connected through a transmission channel T N based on the TCP protocol. i→j The connection is established to realize data transmission. The transmission process adopts a CPU-to-CPU communication mode to avoid the GPU channel to the greatest extent to reduce the interference on the computing resources. To improve the transmission efficiency, the source node divides the Checkpoint data C i to be transmitted into a plurality of non-overlapping data block sets {c i,1 ,c i,2 ,…, c i,M} that satisfy and each data block is disjointed. The system independently schedules and sends each data block in parallel through a multi-thread mechanism, thereby fully utilizing the network bandwidth, reducing the transmission delay, and optimizing the data transmission performance.

[0055] S43, the integrity verification and recovery preparation guarantee of the Checkpoint backup task are realized through the state synchronization and confirmation mechanism. After receiving and writing all the data block sets {c i,1 ,c i,2 ,…, c i,M}, the receiving end updates the local state flag to ckpt_state = True, indicating that the current backup task is successfully completed. Then, the receiving end generates an acknowledgement signal ack_flag = 1 and feeds it back to the sending node as the basis for the next round of transmission scheduling. The state information is also written into the shared control buffer for subsequent state reading and scheduling judgment in the recovery process. Based on the state, the backup node can safely participate in the model recovery, ensuring the reliability of data recovery after the training interruption.

[0056] S5, based on periodic health monitoring and dynamic scheduling mechanism to achieve rapid identification of training node failure and backup node replacement recovery. The system sends a heartbeat signal h to each node at a fixed time interval Δt i (t)), and requires the node to return a response signal r i (t) within a specified timeout time τ. If a node does not respond within the timeout time, i.e. satisfies the condition r i (t) = 0 and t - t last > τ, the node is determined to be faulty. After fault detection, the scheduling module selects an idle backup node from the backup node pool to replace it, and updates the node mapping relationship corresponding to the training task. The scheduling module then issues switching instructions to related components, triggering the system to enter the recovery process.

[0057] Through real-time running state monitoring and dynamic mapping update mechanism to realize the rapid replacement of faulty nodes and continuous guarantee of training tasks. Let the training node set be V = {v1, v2, …, v N}. When a node v i ∈ V fails, the scheduling module selects an idle node v idle ∈ V N+1 from the backup node pool V N+2 = {v N+m ,v r ,…v idle} to replace it, and updates the node mapping relationship as That is, the idle node v r acquires the backup information of the original node v i , then the scheduling module issues switching instructions to related components, and the system enters the fault recovery process, ensuring the continuity and stability of the training task.

[0058] S6, based on shared storage fast loading and backup node asynchronous data recovery mechanism, to realize efficient continuity guarantee of training tasks. In the recovery process, non-faulty nodes efficiently load the latest checkpoint copy through the shared storage path to complete state recovery, and faulty nodes asynchronously pull data from backup nodes according to the Checkpoint backup scheme to realize fast state reconstruction and ensure seamless connection and continuous execution of training tasks.

[0059] S61, using shared memory path loading mechanism to realize low-latency state recovery of non-faulty nodes. For non-faulty nodes v i ∈ V, the system queries its shared memory path Pshared, directly loads the last completed checkpoint copy C i , and completes state reconstruction. The loading operation is performed through the function Load(v i , C i (t k) is executed without cross-node communication, minimizing latency.

[0060] S62, through the mapping-driven redundancy recovery mechanism and efficient cross-node data scheduling, the quick state reconstruction of the replaced node and the seamless connection of the training task are realized. The system determines its corresponding backup node set V ij = {v b | b j = 1} according to the node mapping table and the backup matrix B = [b ji , initiates the parameter pulling operation. The transmission function asynchronously pulls the backup checkpoint from the node v i , and the data is received under the CPU-to-CPU channel through the communication process, and the deserialization process is recorded as D: bytes→Tensor, and loaded into the new node model cache area. After data recovery, the system sets the synchronization flag bit syncvr = 1, and after all nodes reach synchronization, the training task continues from round t k+1 without backtracking or restarting.

[0061] Embodiment 2:

[0062] Based on the above embodiment, the following comparative cases are provided:

[0063] This embodiment is deployed on the domestic MindSpore training framework and domestic Ascend computing device, and selects the method of pulling the checkpoint from the remote storage (OBS bucket of Huawei) of MindSpore as the Baseline for experimental comparison. The experimental results show that, compared with the baseline method, the present application avoids the checkpoint backup time in the inter-node data parallel scene; in other parallel strategies, the training process realizes the highest speedup of about 1.97 times in checkpoint saving time. At the same time, in the fault recovery scene, the average recovery time is accelerated by about 2.92 times. The above results fully verify the significant effect of the present application in improving the training efficiency and system reliability. The specific performance data is shown in Table 1 and Table 2.

[0064] Table 1 Performance comparison of checkpoint saving strategy

[0065]

[0066]

[0067] Table 2 Performance comparison of checkpoint loading (recovery)

[0068]

[0069] The above shows and describes the basic principles, main features and advantages of the present application. Those skilled in the art should understand that the present application is not limited to the above-mentioned embodiments, and the above-mentioned embodiments and descriptions in the specification are only preferred examples of the present application and are not intended to limit the present application. Various changes and improvements can be made to the present application without departing from the spirit and scope of the present application, and these changes and improvements all fall within the scope of the claimed present application. The scope of protection of the present application is defined by the appended claims and their equivalents.

Claims

1. A checkpoint remote backup method based on large model parallel strategy awareness, characterized in that, The method comprises the following steps: S1, constructing a Checkpoint backup scheme, comprising: S11, constructing different cross-node parameter backup mapping relationships according to different parallel modes; S12, constructing a model parameter sharding allocation scheme according to the parameter load of each card and the number of redundant copies; S13, grouping the training nodes according to the backup copy number set by the user to generate multiple full-mutual backup groups with equal size; the training nodes in the full-mutual backup group backup each other; S2, managing memory allocation in the Checkpoint backup process based on a memory reuse mechanism; managing the scheduling of CPU resources in the Checkpoint data transmission process according to an asynchronous transmission process; S3, realizing on-time writing of Checkpoint through a round and time threshold joint triggering method; S4, controlling Checkpoint backup starting and scheduling according to the Checkpoint backup scheme and triggering conditions.

2. The checkpoint remote backup method based on large model parallel strategy awareness according to claim 1, characterized in that, In the step S11, if the parallel mode is data parallel, the cross-node parameter backup mapping relationship is that a complete model parameter copy is saved on each training node; If the parallel mode is tensor parallel, pipeline parallel or hybrid parallel, the cross-node parameter backup mapping relationship is that the model parameter copy is divided into multiple non-overlapping subsets, and each training node holds part of the model parameter data.

3. The checkpoint remote backup method based on large model parallel strategy awareness according to claim 1, characterized in that, In the step S13, for the remaining training nodes that cannot be evenly divided, a one-way dependent circular backup chain is constructed through a ring topology structure, so that each training node serves as a backup target of its predecessor node.

4. The checkpoint remote backup method based on large model parallel strategy awareness according to claim 1, characterized in that, In the step S4, the backup node sends a wait_flag signal through a semaphore mechanism to detect whether the backup task of the previous round is completed; if the previous task has not been completed, the backup sub-process is blocked and enters a waiting state until a ready_flag=1 signal is received.

5. The checkpoint remote backup method based on large model parallel strategy awareness according to claim 4, characterized in that, In the step S4, data transmission is performed between the source node and the backup node through a transmission channel based on the TCP protocol; the transmission process adopts a CPU-to-CPU communication mode; the source node divides the Checkpoint data to be transmitted into multiple non-overlapping data block sets, and each data block is independent; the system independently schedules and sends each data block in parallel through a multi-thread mechanism.

6. The checkpoint remote backup method based on large model parallel strategy awareness according to claim 5, characterized in that, In the step S4, after receiving and writing the data block set, the receiving end updates the local state flag bit to ckpt_state=True, indicating that the current backup task is successfully completed; the receiving end generates an acknowledgement signal ack_flag=1 and feeds it back to the source node as the basis for the next round of transmission scheduling.

7. The checkpoint remote backup method based on large model parallel strategy awareness according to claim 1, characterized in that, Further comprising S5, the system sends a heartbeat signal to each training node at a fixed time interval, and the training node needs to return a response signal within a specified timeout period; if the training node does not respond within the timeout period, the training node is determined to be faulty; the scheduling module will select an idle backup node from the backup node pool for replacement.

8. The checkpoint remote backup method based on large model parallel strategy awareness according to claim 1, characterized in that, S6, the non-fault node realizes fast state recovery through a shared storage path, and uses the checkpoint backup scheme to guide the fault node to asynchronously pull data from the backup node.

9. The checkpoint remote backup method based on large model parallel strategy awareness according to claim 8, characterized in that, The step S6 comprises: S61, the system queries a shared memory path of the non-fault node, directly loads the latest completed model parameter copy, and completes state reconstruction; the loading operation is performed through a function; S62, the system determines a corresponding backup node set according to the cross-node parameter backup mapping relationship; and a transmission function asynchronously pulls the model parameter copy from the node.

Images