Checkpoint storage method and system for distributed training system

Abstract

The invention belongs to the technical field of computer storage, and particularly relates to a checkpoint storage method and system for a distributed training system, target gradient data is used as a snapshot target, and the transmission and storage overhead of each snapshot operation of a checkpoint process can be effectively reduced; according to the method, gradient synchronization and parallel snapshot are carried out on each piece of target gradient data, the computing power of each GPU and CPU in the training system can be effectively utilized, the execution efficiency of the check point process is improved, and meanwhile, the blocking of the training process can be effectively reduced. Besides, different data recovery strategies are set in the case of accidental interruption of different processes, so that the fault-tolerant capability of the training process can be effectively ensured while the data recovery efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer storage, and more particularly, relates to a checkpoint storage method and system for a distributed training system. BACKGROUND

[0002] A deep neural network contains a large number of model parameters. The process of training a neural network uses an optimizer to adjust the model parameters according to the gradient (i.e. the partial derivative of the loss function with respect to the parameters) so that the loss function of the model on a given task is minimized. To improve the training efficiency of large-scale models or data sets, distributed training systems are widely used, which divide the overall computation into multiple tasks and perform them in parallel on multiple GPUs. During training, the input data is divided and distributed to each node, and each node independently performs local forward and backward propagation. When the local backward propagation is completed, all nodes need to synchronize the gradient to ensure that the updates of the model replicas remain consistent, thereby achieving global collaboration.

[0003] In the multi-GPU distributed training process, due to the significant increase in system size and training duration, training interruption events occur more frequently. In particular, in large model training tasks, due to the high consumption of computing resources and the long training period, the probability of occurrence of unstable factors such as node failure and memory leakage is further increased. Therefore, checkpoint technology has been widely integrated into mainstream deep learning frameworks and has gradually become an important direction of fault tolerance research. This technology periodically snapshots and persists the model state during training to storage media, and supports recovery from the last checkpoint when a system failure occurs. With this mechanism, the cost of training rollback is strictly limited between two consecutive checkpoints, so higher frequency of checkpoints can improve the fault tolerance of distributed training.

[0004] However, the existing high-frequency checkpoint technology involves frequent state saving operations, which can block the normal training process. In addition, the existing checkpoint technology uses snapshots to copy all model state data corresponding to each checkpoint to CPU memory. This process involves a large number of checkpoints, which can cause large bandwidth and storage overheads for the training system. SUMMARY

[0005] In view of the above defects of the prior art, the present application provides a checkpoint storage method and system for a distributed training system, which aims to reduce the blocking of the training process by the checkpoint process of the training system, while ensuring the fault tolerance of the training and reducing the bandwidth and storage overheads of the checkpoint process.

[0006] In a first aspect, the present application provides a checkpoint storage method for a distributed training system, comprising: S1, obtain gradient data corresponding to a back propagation process of a network model in a distributed training system, the gradient data including a plurality of target gradient data, each target gradient data being obtained by sequentially performing gradient calculation on network model state data by different target GPUs; S2, construct a checkpoint process based on each target gradient data, and perform snapshot on target gradient data corresponding to each differential checkpoint in the checkpoint process; S3, based on the snapshot result of each differential checkpoint, update the network model state data in the target CPU corresponding to the snapshot result, and persist the network model state data in the target CPU after parameter update to a target storage device.

[0007] The training process is a single training iteration process, and the checkpoint process is a process parallel to the training process. The above training process is assigned to multiple target GPUs for pipeline processing. The checkpoint process is to reuse the gradient data of the back propagation process and is assigned to multiple target GPUs for parallel execution according to the number of target gradient data, which can effectively alleviate the memory bottleneck of single GPU training, enable the training system to train large-scale models, and improve the overall utilization of GPUs in the device.

[0008] Further, obtaining gradient data corresponding to a back propagation process of a network model in a distributed training system comprises: Based on the time sequence order of the back propagation process, each gradient generation node in the back propagation process is sequentially obtained, and the gradient data corresponding to each gradient generation node is taken as target gradient data.

[0009] Further, before constructing the checkpoint process based on each target gradient data, it further comprises: Based on the network model state data in the target CPU corresponding to each target gradient data, gradient synchronization processing is performed on each target gradient data through gradient communication, and the target gradient data that successfully synchronizes gradients is retained.

[0010] In the method of the present application, the checkpoint process and the training process are implemented by multiple target GPUs, and the network model state data in each target CPU must be consistent, otherwise the target gradient data obtained will deviate seriously, causing the training to not converge, and further causing the snapshot result obtained by the checkpoint process to be incorrect, thereby causing the subsequent training iteration to fail. Therefore, gradient synchronization processing is required to ensure the convergence and correctness of model training.

[0011] Further, constructing the checkpoint process based on each target gradient data comprises: Based on each target gradient data after gradient synchronization processing, obtain the gradient generation node corresponding to each target gradient data, and take the gradient generation node as a differential checkpoint.

[0012] The target gradient data is obtained directly from the training process using zero-copy technology. Therefore, in the parallel training and checkpointing processes, the target gradient data only needs to be read once, which can reduce the overall system's computational power consumption.

[0013] Traditional checkpointing mechanisms require taking a snapshot of the entire model state, including model parameters and optimizer parameters. This operation is particularly burdensome when using optimizers with a large number of parameters, such as Adam. As the model size increases and the frequency of checkpointing rises, the system overhead caused by checkpoint data transmission becomes increasingly significant.

[0014] A snapshot refers to the process of writing model state data from GPU memory to CPU memory during training. However, the method in this application stores the network model's state data in the CPU and takes snapshots of the gradient data. Compared to the model state, gradient generation is completed earlier in the iteration process, resulting in higher parallelism and scheduling flexibility. Since gradient data effectively reflects changes in the network model's state data in the GPU, taking snapshots of the gradient data first and then updating the parameters of the network model's state data in the target CPU based on the snapshot results can significantly reduce the amount of data that needs to be saved for each snapshot, and also reduce the transmission and storage overhead of the checkpoint process.

[0015] Furthermore, the snapshots of the target gradient data in this application are performed in parallel. That is, the snapshot operation performed at each differential checkpoint can be performed separately for each target gradient data. When some gradient data is large and its snapshot speed is slow, it does not affect the snapshot process of the subsequent operation points of the checkpoint process. This allows the idle computing power of each target GPU and target CPU in the system to be utilized, reducing the blocking of the training process.

[0016] Furthermore, after using the gradient generation node as the differential checkpoint, it also includes: Gradient compression is performed on the gradient data of each target. The gradient compression process includes: obtaining the gradient tensor of each target gradient data based on the Top-k sparse algorithm, and retaining the elements whose gradient tensor values ​​account for a predetermined percentage of the total values ​​in each target gradient data.

[0017] Furthermore, after step S3, the following steps are also included: When the backpropagation process is interrupted at the gradient generation node corresponding to the target gradient data, the target GPU corresponding to the gradient generation node is replaced by a backup GPU, and the network model state data of the target CPU corresponding to the target gradient data is stored in the backup GPU.

[0018] Further, after step S3, further comprising: When the checkpoint process is interrupted in the differential checkpoint, the target CPU corresponding to the differential checkpoint is replaced by a backup CPU, and the network model state data of the target CPU of the last differential checkpoint corresponding to the differential checkpoint in the checkpoint process is stored in the backup CPU.

[0019] In a second aspect, the present application also provides a checkpoint storage system for a distributed training system, for implementing the method of any one of the first aspect, comprising: a gradient data acquisition module, configured to acquire gradient data corresponding to the back propagation process, the gradient data comprising a plurality of target gradient data; a checkpoint process construction module, configured to construct a checkpoint process based on the target gradient data, the checkpoint process comprising a plurality of differential checkpoints; a data snapshot module, configured to take a snapshot of the target gradient data corresponding to each differential checkpoint, and perform parameter update on the network model state data in the target CPU based on the snapshot result of each differential checkpoint; a data persistence module, configured to persist the network model state data in each target CPU after parameter update to a target storage device.

[0020] In a third aspect, the present application also provides an electronic device, characterized in that comprising: at least one memory, configured to store a computer program; at least one processor, configured to execute the program stored in the memory, so that the processor executes the method described in the first aspect or any one of the possible implementation manners of the first aspect.

[0021] In a fourth aspect, the present application also provides a computer readable storage medium, which stores a computer program, and when the computer program runs on a processor, the processor executes the method described in the first aspect or any one of the possible implementation manners of the first aspect.

[0022] Overall, compared with the prior art, the above technical solutions conceived by the present application have the following beneficial effects: The checkpoint storage method and system for a distributed training system provided by the present application determine each differential checkpoint by reusing the target gradient data of the training process, construct a checkpoint process parallel to the training process, and can take a snapshot of the target gradient data corresponding to each differential checkpoint in parallel through different target GPUs, and then quickly update and persist the network model state data in the target CPU corresponding to each target gradient data according to each snapshot result.

[0023] The application uses target gradient data as a snapshot target, which can effectively reduce the transmission and storage overhead of each snapshot operation of the checkpoint process; gradient synchronization and parallel snapshot of each target gradient data can effectively utilize the computing power of each GPU and CPU in the training system, improve the execution efficiency of the checkpoint process, and also effectively reduce the blocking of the training process. In addition, the application has different data recovery strategies when facing unexpected interruptions of different processes, which can improve the data recovery efficiency while effectively ensuring the fault tolerance of the training process. BRIEF DESCRIPTION OF DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the application or the related art, the following will briefly introduce the drawings needed to be used in the embodiments or the related art description. Obviously, the drawings in the following description are some embodiments of the application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0025] Figure 1 is a flowchart of the checkpoint storage method provided by the embodiments of the application.

[0026] Figure 2 is a schematic diagram of the checkpoint process and the training process provided by the embodiments of the application.

[0027] Figure 3 is a structural schematic diagram of the checkpoint storage system provided by the embodiments of the application.

[0028] Figure 4 is a structural schematic diagram of an electronic device provided by the application. DETAILED DESCRIPTION

[0029] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application.

[0030] In the following description, the terms "first", "second" are only for the purpose of description, and cannot be understood as indicating or implying relative importance. The following description provides several embodiments of the application, and different embodiments can be replaced or combined, so the application can also be considered to include all possible combinations of the same and / or different embodiments described. Therefore, if one embodiment includes features A, B and C, and another embodiment includes features B and D, the application should also be considered to include one or more embodiments of all other possible combinations of A, B, C and D, although the embodiment may not be explicitly described in the following content.

[0031] The following description provides examples, and is not intended to limit the scope, applicability or example set forth in the claims. Alterations and further modifications of the described elements are possible without deviating from the scope of the application. Various examples can omit, substitute, or add various procedures or components as appropriate. For instance, the methods described can be performed in an order different than described, and various steps can be added, omitted, or combined. Also, features described with respect to some examples can be combined in other examples.

[0032] Figure 1 is a flowchart of a checkpoint storage method for a distributed training system provided by an embodiment of the present application, as shown in Figure 1 , the method comprises at least the following steps: S1, obtaining gradient data corresponding to a back propagation process of a network model in a distributed training system, the gradient data comprising a plurality of target gradient data, each target gradient data being obtained by sequentially performing gradient calculation on network model state data by different target GPUs.

[0033] In an embodiment of the present application, the execution subject is a control unit of the distributed training system, and the execution units of the training process and the checkpoint process are each target GPU and target CPU in the training system, while the checkpoint process mainly corresponds to the back propagation process.

[0034] In one implementation manner, the gradient data corresponding to the back propagation process of the network model in the distributed training system is obtained, comprising: Based on the time sequence order of the back propagation process, each gradient generation node in the back propagation process is sequentially obtained, and the gradient data corresponding to each gradient generation node is taken as target gradient data.

[0035] In an embodiment of the present application, as shown in Figure 2 , the gradient data of different gradient generation nodes in the back propagation process in a single data iteration is represented by B1 , B2 and B3 , respectively. B i represents a gradient output of the back propagation process. B i The number of

[0036] In addition, B1 , B2 andB3 The different target GPUs can be executed respectively, so that the GPU with low memory can also train the large model by combining the step-by-step execution. Therefore, the target gradient data can be obtained by sequentially performing gradient calculation on the network model state data by different target GPUs, and the target gradient data is zero-copy transferred, which prepares for the subsequent snapshot processing of the target gradient data. The zero-copy technology is directly reused in the checkpoint process when the target gradient data is generated by the target GPU, without reading and copying again, which can effectively reduce the algorithm consumption of data transmission in the system.

[0037] S2, constructing a checkpoint process based on the target gradient data, and snapshotting the target gradient data corresponding to each difference checkpoint in the checkpoint process.

[0038] In the embodiments of the present application, the checkpoint process is a checkpoint operation for a single data iteration process, that is, only for one forward propagation process and one backward propagation process. For example, if the network model is a target detection model, only the training process of a single image in the data set is performed, and the checkpoint operation is not performed after completing a complete training iteration (epoch) on the entire data set. This strategy can effectively shorten the rollback distance caused by a single fault, thereby improving the overall recovery speed and efficiency of the system. Figure 2 In the embodiments of the present application, the checkpoint process is a checkpoint operation for a single data iteration process, that is, only for one forward propagation process and one backward propagation process. For example, if the network model is a target detection model, only the training process of a single image in the data set is performed, and the checkpoint operation is not performed after completing a complete training iteration (epoch) on the entire data set. This strategy can effectively shorten the rollback distance caused by a single fault, thereby improving the overall recovery speed and efficiency of the system. F 、 B1 、 B2 、 B3 With U represents a single iteration process, C1 、 C2 and C3 represent a snapshot process of the target gradient data.

[0039] Compared with the conventional snapshot method taking the model state data as the core, the gradient does not change after synchronization. Specifically, after gradient calculation and synchronization are completed, the gradient can be snapshoted as the training result after a single iteration, without waiting for the parameter update phase to end. Thus, the decoupling of gradient snapshot and parameter update is realized, which can prevent the phenomenon that the GPU network model parameters in the training process have been updated twice, thereby causing conflict with the writing of the snapshot result. That is, using gradient data for snapshot can break the data dependency between snapshot and training process, and improve the fault tolerance rate in the system training process.

[0040] In addition, the snapshots of the target gradient data are performed in parallel, that is, the snapshots of the target gradient data are directly multiplexed from different target GPUs, so that each snapshot process is independent, and the idle computing power of the target GPUs and the target CPUs in the system can be utilized to reduce the blocking of the training process.

[0041] In an implementation manner, before the checkpoint process is constructed based on the target gradient data, the method further includes: Based on the network model state data in the target CPU corresponding to the target gradient data, gradient synchronization processing is performed on the target gradient data through gradient communication, and the target gradient data with successful gradient synchronization is retained.

[0042] In the embodiments of the present application, as shown in Figure 2 each time B1 , B2 and B3 is executed by different target GPUs, since different target gradient data corresponds to a part of parameter changes of the network model state data in the GPU, the parameters of other model state data need to be guaranteed to be consistent, otherwise data confusion will occur when the snapshot result is used to recover the damaged training process, resulting in data recovery failure and training divergence. Therefore, gradient synchronization processing is needed to ensure the convergence and correctness of model training. Gradient synchronization operation corresponds to Figure 2 in S1 , S2 and S3 is the basis for ensuring the smoothness of subsequent snapshot operations.

[0043] In an implementation manner, the checkpoint process is constructed based on the target gradient data, including: Based on the target gradient data after the gradient synchronization processing, the gradient generation node corresponding to each target gradient data is obtained, and the gradient generation node is taken as a differential checkpoint.

[0044] In an implementation manner, after the gradient generation node is taken as a differential checkpoint, the method further includes: Gradient compression processing is performed on the target gradient data. The gradient compression processing includes: obtaining the gradient tensor of each target gradient data based on the Top-k sparse algorithm, and retaining the elements whose gradient tensor values account for a front preset percentage in each target gradient data.

[0045] In the embodiments of the present application, the checkpoint, as a snapshot of the model state in the training process, also contains a large number of model parameters and optimizer states and other key tensors. Therefore, the checkpoint data can be compressed by referring to gradient compression or data transmission precision compression technology for targeted compression processing to reduce the storage bandwidth and transmission computing power requirements. In order to ensure the accurate recovery of the training data by using the snapshot results, the Top-k sparse algorithm in the gradient compression processing is used to achieve the above purpose in the embodiments of the present application, and the selection of the gradient compression processing related algorithm is not limited to this.

[0046] S3, based on the snapshot results of each differential checkpoint, updating the network model state data in the target CPU corresponding to the snapshot results, and persisting the network model state data in each target CPU after parameter updating to the target storage device.

[0047] In the embodiments of the present application, persistence refers to the process of writing snapshots in memory to persistent storage (such as SSD), which corresponds to Figure 2 In U 0 and P operation. The snapshot result of each differential checkpoint includes the change of the network model state data in the corresponding target CPU, so the model state data in each target CPU can be accurately updated through the snapshot result. In one training iteration, the snapshot results of the target gradient data corresponding to one target CPU are integrated, and the optimization of the network model state data of one target CPU is completed. The process in the embodiments of the present application can be realized in parallel by multiple target GPUs, so as to improve the execution efficiency of the checkpoint process and effectively reduce the blocking of the training process.

[0048] In one implementation manner, after step S3, the method further comprises: When the checkpoint process is interrupted at the differential checkpoint, the target CPU corresponding to the differential checkpoint is replaced by a backup CPU, and the network model state data of the target CPU of the last differential checkpoint corresponding to the differential checkpoint in the checkpoint process is stored in the backup CPU.

[0049] When the reverse propagation process is interrupted at the gradient generation node corresponding to the target gradient data, the target GPU corresponding to the gradient generation node is replaced by a backup GPU, and the network model state data of the target CPU corresponding to the target gradient data is stored in the backup GPU.

[0050] In this embodiment, the main reason for the checkpointing process interruption at the differential checkpoint may be a failure of the target CPU corresponding to that differential checkpoint (such as a training process crash, CPU resources being reclaimed by the operating system, or a temporary interruption). Since the checkpointing process in this application is executed asynchronously and in parallel, it can be repaired by replacing the CPU and calling the network model state data of the target CPU from the previous differential checkpoint. Similarly, when some target GPUs fail, the backpropagation process will be interrupted at the gradient generation node corresponding to that target GPU. Therefore, it is necessary to replace the target GPU and store the network model state data of the target CPU corresponding to the target gradient data into a spare GPU.

[0051] In addition, if the system experiences serious errors such as power failure, power outage, or operating system crash, the persistent network model state data of the target storage device can be used to repair the data of each GPU and CPU in the entire system.

[0052] like Figure 3 As shown, the structure of the checkpoint storage system provided in this application embodiment includes: The gradient data acquisition module is used to acquire gradient data corresponding to the backpropagation process of the network model within the distributed training system. The checkpoint process construction module is used to construct checkpoint processes based on the gradient data of each target. The data snapshot module is used to take snapshots of the target gradient data corresponding to each differential checkpoint in the checkpoint process, and update the parameters of the network model state data in the target CPU corresponding to the snapshot results based on the snapshot results of each differential checkpoint. The data persistence module is used to persist the updated network model state data of each target CPU to the target storage device.

[0053] like Figure 4 As shown, Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include: a processor 401, a communications interface 402, a memory 403, and a communication bus 404. The processor 401, communications interface 402, and memory 403 communicate with each other via the communication bus 404. The processor 401 can call software instructions in the memory 403 to execute the methods described in the above embodiments.

[0054] In addition, the logic instructions in the memory 403 described above can be implemented in the form of software function units and sold or used as independent products, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or say the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the embodiments of the present application.

[0055] Based on the method in the above embodiments, the embodiments of the present application provide a computer readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the method in the above embodiments.

[0056] Based on the method in the above embodiments, the embodiments of the present application provide a computer program product, which runs on a processor. When the computer program product runs on the processor, the processor executes the method in the above embodiments.

[0057] It can be understood that the processor in the embodiments of the present application can be a CPU (Central Processing Unit, central processing unit), and can also be other general-purpose processors, DSPs (Digital Signal Processors, digital signal processors), ASICs (Application Specific Integrated Circuits, application specific integrated circuits), FPGAs (Field Programmable Gate Arrays, field programmable gate arrays) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor can be a microprocessor, or any conventional processor.

[0058] The method steps in the embodiments of the present application can be implemented by means of hardware, or by means of a processor executing software instructions. The software instructions can be composed of corresponding software modules, which can be stored in a random RAM (Random Access Memory), a flash memory, a ROM (Read-only Memory), a PROM (Programmable ROM), an EPROM (Erasable PROM), an EEPROM (Electrically EPROM), a register, a hard disk, a mobile hard disk, a CD-ROM, or any other form of storage medium well known in the art. An exemplary storage medium is coupled to the processor, so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in an ASIC.

[0059] In the above embodiments, the implementation can be wholly or partially by software, hardware, firmware, or any combination thereof. When implemented by software, the implementation can be wholly or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the implementation wholly or partially produces the flow or function according to the embodiments of the present application. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in or transmitted by a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wire (for example, coaxial cable, optical fiber, digital subscriber line DSL) or wireless (for example, infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium accessible by a computer or a data storage device such as a server, data center, etc. containing one or more available media sets. The available media can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, an SSD (Solid State Disk)), etc.

[0060] It can be understood that the various numerical numbers involved in the embodiments of the present application are only for the convenience of differentiation, and do not limit the scope of the embodiments of the present application.

[0061] Those skilled in the art can easily understand that the above are only the preferred embodiments of the present application, and are not used to limit the present application, any modification, equivalent replacement and improvement made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A checkpoint storage method for a distributed training system, characterized by, Comprise: S1, obtain the gradient data corresponding to the back propagation process of the network model in the distributed training system, the gradient data comprises a plurality of target gradient data, each target gradient data is obtained by sequentially performing gradient calculation on network model state data by different target GPUs; S2, based on each target gradient data, a checkpoint process is constructed, and the target gradient data corresponding to each difference checkpoint in the checkpoint process is snapped; S3, based on the snapshot result of each difference checkpoint, the network model state data in the target CPU corresponding to the snapshot result is updated, and each network model state data in the target CPU after parameter update is persisted to the target storage device.

2. The checkpoint storage method for a distributed training system according to claim 1, wherein, The gradient data corresponding to the back propagation process of the network model in the distributed training system is obtained, comprising: Based on the time sequence order of the back propagation process, each gradient generation node in the back propagation process is obtained in turn, and the gradient data corresponding to each gradient generation node is taken as target gradient data.

3. The checkpoint storage method for a distributed training system according to claim 2, wherein, Before the checkpoint process is constructed based on each target gradient data, it further comprises: Based on the network model state data in the target CPU corresponding to each target gradient data, gradient synchronization processing is performed on each target gradient data through gradient communication, and the target gradient data with successful gradient synchronization is retained.

4. The checkpoint storage method for a distributed training system according to claim 3, wherein, The checkpoint process is constructed based on each target gradient data, comprising: Based on each target gradient data after gradient synchronization processing, the gradient generation node corresponding to each target gradient data is obtained, and the gradient generation node is taken as a difference checkpoint.

5. The checkpoint storage method for a distributed training system according to claim 4, wherein, After the gradient generation node is taken as a difference checkpoint, it further comprises: Gradient compression processing is performed on each target gradient data; Wherein, the gradient compression processing comprises: based on Top-k sparse algorithm, the gradient tensor of each target gradient data is obtained, and the elements with gradient tensor value accounting for the front preset percentage in each target gradient data are retained.

6. The checkpoint storage method for a distributed training system according to claim 5, wherein, After step S3, it further comprises: When the back propagation process is interrupted in the gradient generation node corresponding to the target gradient data, the target GPU corresponding to the gradient generation node is replaced by a backup GPU, and the network model state data of the target CPU corresponding to the target gradient data is stored in the backup GPU.

7. The checkpoint storage method for a distributed training system according to claim 5, wherein, After step S3, it further comprises: When the checkpoint process is interrupted in the difference checkpoint, the target CPU corresponding to the difference checkpoint is replaced by a backup CPU, and the network model state data of the target CPU of the previous difference checkpoint corresponding to the difference checkpoint in the checkpoint process is stored in the backup CPU.

8. A checkpoint storage system for a distributed training system for performing the method of any of claims 1-7, characterized in that, Comprise: Gradient data acquisition module, used for obtaining the gradient data corresponding to the back propagation process of the network model in the distributed training system; Checkpoint process construction module, used for constructing a checkpoint process based on each target gradient data; The data snapshot module is configured to snapshot the target gradient data corresponding to each differential checkpoint in the checkpoint process, and perform parameter update on the network model state data in the target CPU corresponding to the snapshot result of each differential checkpoint based on the snapshot result of each differential checkpoint. The data persistence module is configured to persist the network model state data in the target CPU after parameter update to a target storage device.

9. An electronic device, comprising: The computer program comprises: at least one memory for storing a computer program; at least one processor for executing the program stored in the memory, and when the program stored in the memory is executed, the processor is configured to execute the method according to any one of claims 1-7.

10. A computer readable storage medium having stored thereon a computer program, the computer readable storage medium having stored therein instructions which, when executed on a computer or processor, cause the computer or processor to perform the method according to any one of claims 1-7.

Images