Breakpoint saving method and device, storage medium and program product

Abstract

The embodiment of the invention provides a breakpoint storage method and device, a storage medium and a program product, relates to the technical field of artificial intelligence, and is used for effectively storing breakpoint data and improving the training efficiency of a large model. The method comprises the steps that when distributed model training reaches a preset breakpoint, model training is interrupted, breakpoint data of each training node are cached from a training node display memory to a training node main memory, and the breakpoint data comprise model state data and optimizer state data; backing up the optimizer state data in each training node main memory to a pre-configured backup node main memory and a distributed storage system, and backing up the model state data in a preset main training node main memory to the distributed storage system.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular to a breakpoint saving method, device, storage medium, and program product. Background Art

[0002] The training of large models often takes a long time, and some failures are inevitable during training. During the training of large models, some breakpoints are usually set. When the large model training reaches a breakpoint, the breakpoint data is saved. In the event of a failure, the training of the large model can be restored using the saved breakpoint data, thus avoiding the need to retrain from scratch every time a failure occurs, and improving training efficiency.

[0003] Therefore, saving and restoring breakpoints is crucial to ensuring the training efficiency of large models. How to effectively save and restore breakpoints to further improve the training efficiency of large models has become an urgent problem to be solved. Summary of the invention

[0004] The embodiments of the present application provide a breakpoint saving method, device, storage medium and program product for effectively saving breakpoint data and improving the training efficiency of large models.

[0005] On the one hand, an embodiment of the present application provides a breakpoint saving method, including:

[0006] When the distributed model training reaches a preset breakpoint, the model training is interrupted, and the breakpoint data of each training node is cached from the training node display memory to the training node main memory, wherein the breakpoint data includes model state data and optimizer state data;

[0007] The optimizer state data in the main memory of each training node is backed up to the pre-configured backup node main memory and distributed storage system, and the model state data in the main memory of the preset main training node is backed up to the distributed storage system.

[0008] On the one hand, an embodiment of the present application provides a breakpoint saving device, including:

[0009] A cache module, used to interrupt the model training when the distributed model training reaches a preset breakpoint, and cache the breakpoint data of each training node from the training node display memory to the training node main memory, wherein the breakpoint data includes model state data and optimizer state data;

[0010] A storage module is used to back up the optimizer state data in the main memory of each training node to a pre-configured backup node main memory and a distributed storage system, and to back up the model state data in the main memory of a preset master training node to the distributed storage system.

[0011] Optionally, the storage module is specifically used for:

[0012] Backing up the optimizer state data in the main memory of each training node to a pre-configured main memory of a backup node at a first frequency;

[0013] The optimizer state data in the main memory of each training node is backed up to the distributed storage system at a second frequency, and the first frequency is greater than the second frequency.

[0014] Optionally, the storage module is specifically used for:

[0015] After resuming model training, the optimizer state data in the main memory of each training node is backed up to the pre-configured backup node main memory and distributed storage system, and the model state data in the main memory of the preset main training node is backed up to the distributed storage system.

[0016] Optionally, the device further comprises:

[0017] A processing module, used for determining, when resuming training at a breakpoint, a target version of the breakpoint data to be restored through collective communication based on the meta information of the breakpoint data stored in each training node, wherein the meta information of the breakpoint data at least includes the version of the breakpoint data;

[0018] A data recovery module is used to recover the breakpoint data of the target version from the distributed storage system or the target storage corresponding to each training node. The target storage corresponding to each training node includes a training node main storage and a pre-configured backup node main storage.

[0019] Optionally, the processing module is specifically used to:

[0020] Obtaining meta information of breakpoint data stored in the corresponding target storage and the distributed storage system for each training node;

[0021] Instruct all training nodes to exchange meta information of their respective stored breakpoint data through collective communication, and instruct each training node to determine a target version of the breakpoint data to be restored.

[0022] Optionally, the data recovery module is specifically used to:

[0023] For a training node, perform the following operations to restore the breakpoint data of the target version:

[0024] When it is determined through integrity check that the breakpoint data of the target version is stored in the target memory corresponding to the one training node, recovering the breakpoint data of the target version from the target memory corresponding to the one training node; or

[0025] When it is determined through integrity verification that the breakpoint data of the target version is not stored in the target memory corresponding to the one training node, the breakpoint data of the target version is restored from the distributed storage system.

[0026] Optionally, the data recovery module is specifically used to:

[0027] When it is determined that the breakpoint data of the target version is stored in the main memory of the one training node, recovering the breakpoint data of the target version from the main memory of the one training node; or

[0028] When it is determined that the breakpoint data of the target version is stored in the backup node main memory corresponding to the one training node, the breakpoint data of the target version is restored from the backup node main memory.

[0029] On the one hand, an embodiment of the present application provides a computer device, including a memory, a processor chip, and a computer program stored in the memory and executable on the processor chip, wherein the processor chip implements the steps of the above-mentioned breakpoint saving method when executing the program.

[0030] On the one hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program executable by a computer device. When the program is run on the computer device, the computer device executes the steps of the above-mentioned breakpoint saving method.

[0031] On the one hand, an embodiment of the present application provides a computer program product, which includes a computer program stored on a computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer device, the computer device executes the steps of the above-mentioned breakpoint saving method.

[0032] In an embodiment of the present application, when the distributed model training reaches a preset breakpoint, the model training is interrupted, and the breakpoint data of each training node is cached from the training node display memory to the training node main memory, wherein the breakpoint data includes model state data and optimizer state data; then the optimizer state data in the main memory of each training node is backed up to the pre-configured backup node main memory and the distributed storage system, and the model state data in the preset main training node main memory is backed up to the distributed storage system.

[0033] By storing the optimizer state data of each training node in the training node main memory, the backup node main memory, and the distributed storage system, and storing the model state data in each training node main memory and the distributed storage system (the model state data of each training node is the same), the breakpoint data can be effectively saved. Even if part or all of the data in the training node main memory is lost or invalid, the breakpoint data can still be restored from the most recently saved breakpoint through the backup node main memory and the distributed storage system to continue the model training, thereby shortening the time overhead caused by breakpoint training and improving the efficiency of model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0035] Figure 1 A schematic diagram of an application scenario of an embodiment of the present application;

[0036] Figure 2 A schematic diagram of a process for saving breakpoints provided in an embodiment of the present application;

[0037] Figure 3 A schematic diagram of the principle of configuring a backup node provided in an embodiment of the present application;

[0038] Figure 4 A logical schematic diagram of a breakpoint saving method provided in an embodiment of the present application;

[0039] Figure 5 A schematic diagram of a process flow of implementing a breakpoint saving method provided in an embodiment of the present application;

[0040] Figure 6 A flowchart of an implementation process of a breakpoint resume training method provided in an embodiment of the present application;

[0041] Figure 7 A schematic diagram of the structure of a breakpoint saving device provided in an embodiment of the present application;

[0042] Figure 8 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the technical solution of the present application, rather than all of the embodiments. Based on the embodiments recorded in the application documents, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the technical solution of the present application.

[0044] It will be understood that in the following specific implementations of this application, data required for model training is involved. When the various embodiments of this application are applied to specific products or technologies, relevant licenses or consents need to be obtained, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0045] The following is an explanation of some of the terms involved in the embodiments of the present application.

[0046] 1. A breakpoint refers to a special point in the model training process. The model training progress before the breakpoint is known. By saving the breakpoint, the model training progress before the breakpoint can be saved. After the breakpoint is saved, the model training can continue until the next breakpoint is triggered.

[0047] 2. Breakpoint resume training means retraining from the location saved at the breakpoint. That is, after the breakpoint is saved, if an exception occurs during the model training process before the next breakpoint, causing the model training to be terminated or exited, when retraining the model, there is no need to start from the beginning. The data saved at the previous breakpoint can be restored and the model training can be continued from the location of the previous breakpoint.

[0048] The following is a brief introduction to the design concept of the embodiment of the present application:

[0049] The training of large models often takes a long time, and some failures are inevitable during training. During the training of large models, some breakpoints are usually set. When the large model training reaches a breakpoint, the breakpoint data is saved. In the event of a failure, the training of the large model can be restored using the saved breakpoint data, thus avoiding the need to retrain from scratch every time a failure occurs, and improving training efficiency.

[0050] Therefore, saving and restoring breakpoints is crucial to ensuring the training efficiency of large models. How to effectively save and restore breakpoints to further improve the training efficiency of large models has become an urgent problem to be solved.

[0051] In view of this, the embodiments of the present application provide a breakpoint saving method, device, storage medium and program product. When the distributed model training reaches a preset breakpoint, the model training is interrupted, and the breakpoint data of each training node is cached from the training node display memory to the training node main memory, wherein the breakpoint data includes model status data and optimizer status data; then the optimizer status data in the main memory of each training node is backed up to the pre-configured backup node main memory and the distributed storage system, and the model status data in the preset main training node main memory is backed up to the distributed storage system.

[0052] By storing the optimizer state data of each training node in the training node main memory, the backup node main memory, and the distributed storage system, and storing the model state data in each training node main memory and the distributed storage system (the model state data of each training node is the same), the breakpoint data can be effectively saved. Even if part or all of the data in the training node main memory is lost or invalid, the breakpoint data can still be restored from the most recently saved breakpoint through the backup node main memory and the distributed storage system to continue the model training, thereby shortening the time overhead caused by breakpoint training and improving the efficiency of model training.

[0053] The preferred embodiments of the present application are described below in conjunction with the drawings in the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application. In addition, the embodiments and features in the embodiments of the present application may be combined with each other if there is no conflict.

[0054] like Figure 1 As shown, it is a schematic diagram of an application scenario of an embodiment of the present application. The application scenario diagram includes any server 110 among multiple servers and multiple terminal devices 120.

[0055] In the embodiment of the present application, the terminal device 120 includes but is not limited to mobile phones, tablet computers, laptop computers, desktop computers and other devices; the terminal device 120 can be used as a training node for distributed model training in a data parallel manner, and trains the model based on the sample data provided by the server 110. The server 110 can accept the model training tasks issued by the user, divide the sample data into multiple parts, and perform model training in different training nodes.

[0056] Server 120 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), as well as big data and artificial intelligence platforms.

[0057] It should be noted that the breakpoint saving method in the embodiment of the present application can be executed by the central node or management device of the distributed model training. When the model training is performed, when the distributed model training reaches a preset breakpoint, the model training is interrupted, and the breakpoint data of each training node is cached from the training node display memory to the training node main memory. The breakpoint data includes model state data and optimizer state data. Then the model training is resumed, and the optimizer state data in the main memory of each training node is backed up to the pre-configured backup node main memory and distributed storage system, and the model state data in the preset main training node main memory is backed up to the distributed storage system. Among them, the training nodes mentioned in the embodiment of the present application refer to the training nodes belonging to different data parallel groups during the distributed training process of the model using data parallelism.

[0058] In an optional implementation, the terminal device 120 and the server 110 may communicate with each other via a communication network, which may be a wired network or a wireless network.

[0059] It should be noted that Figure 1 What is shown is just an example. In fact, the number and communication method of terminal devices and servers are not limited. When there are multiple servers, multiple servers can form a blockchain, and the server is a node on the blockchain, which is not specifically limited in the embodiments of the present application.

[0060] The following describes the breakpoint saving method provided by the exemplary embodiment of the present application in combination with the application scenarios described above and with reference to the accompanying drawings. It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principles of the present application, and the implementation of the present application is not limited in this respect.

[0061] It should be noted that the breakpoint saving solution provided in the embodiment of the present application not only includes the process of breakpoint data saving, but also includes the process of breakpoint resumption training when breakpoint resumption training is required. The implementation processes of breakpoint data saving and breakpoint resumption training are introduced below.

[0062] Implementation process 1. Save breakpoint data.

[0063] See also Figure 2As shown, it is an implementation flow chart of a breakpoint saving method in an embodiment of the present application, with the central node of the distributed model training as the execution body, and the specific implementation process of the method is as follows S201-S202:

[0064] S201: When the distributed model training reaches a preset breakpoint, the model training is interrupted, and the breakpoint data of each training node is cached from the training node display memory to the training node main memory, where the breakpoint data includes model status data and optimizer status data.

[0065] It should be noted that currently, large model training is limited by the display memory capacity of a single training node. Usually, when training a large model, the sample data needs to be divided into multiple data blocks, and each data block is trained on the graphics processing unit (GPU) of a training node, that is, the model is trained in a distributed manner using data parallelism. In distributed model training, the training process of each GPU is responsible for saving local training data, and the training progress can be synchronized between GPUs.

[0066] In practical applications, the breakpoints in the model training process can be set in an existing manner, for example, they can be set by the user. For example, a breakpoint is set every 100 steps of training, and a breakpoint is set every 2 hours of training. Of course, it can also be automatically configured by the central node according to the complexity of the training task, and the embodiments of the present application are not limited to this.

[0067] In specific implementation, model training is carried out in the GPU of the training node. When the model training reaches a preset breakpoint, the model training is interrupted, and the breakpoint data of each training node is cached from the training node display memory to the training node main memory, wherein the breakpoint data can be further divided into model state data and optimizer state data. Therefore, this step is to cache the model state data and optimizer state data when the model is trained to the breakpoint from the GPU display memory of each training node to the main memory of the central processing unit (CPU), so as to back up the model state data and optimizer state data of each training node and save the training progress of each training node, so that after the model training is interrupted due to a fault or other reasons, each training node can read the model state data and optimizer state data from the CPU main memory to resume the model training.

[0068] It should be noted that the embodiment of the present application further subdivides the breakpoint data into model state data and optimizer state data, wherein the model state data includes the model's weights, data types, shapes, BF16 type weights, and constant tensors, etc., and the optimizer state data includes BF32 type weights, optimizer hyperparameters (first-order momentum, second-order momentum), etc.

[0069] S202: Back up the optimizer state data in the main memory of each training node to the pre-configured backup node main memory and distributed storage system, and back up the model state data in the main memory of the preset master training node to the distributed storage system.

[0070] In specific implementation, after the breakpoint data of each training node is cached from the training node display memory to the training node main memory, in order to effectively save the breakpoint data, the breakpoint data in the training node main memory can be further stored in the pre-configured backup node main memory and distributed storage system. At the same time, considering that the model state data in each training node is the same, if each training node stores the model state data in the backup node main memory and the distributed storage system, it will cause data redundancy problems.

[0071] Therefore, after the embodiment of the present application caches the breakpoint data of each training node from the training node display memory to the training node main memory, the optimizer state data in the main memory of each training node is backed up to the pre-configured backup node main memory and distributed storage system, the optimizer state data is effectively preserved, and the model state data in the preset main training node main memory is backed up to the distributed storage system.

[0072] It should be noted that the model status data in the main memory of other training nodes except the preset main training node does not need to be backed up to the distributed storage system, and the model status data in the main memory of all training nodes does not need to be backed up to the main memory of the backup node to avoid redundant data storage.

[0073] Among them, the backup node pre-configured for each training node can be pre-configured by the user or the central node. The backup node configured for each training node can be other training nodes except itself among the multiple training nodes of the distributed model training. Each training node can be configured with one backup node or multiple backup nodes. The backup nodes configured for multiple training nodes can be the same, for example, the same backup node is configured for multiple training nodes, or different, for example, different backup nodes are configured for multiple training nodes respectively. This embodiment of the present application does not limit this.

[0074] It should be noted that the number of backup nodes configured for each training node can be flexibly designed according to the reliability requirements of model training. If the reliability requirements are high, two or more backup nodes are configured for each training node. If the reliability requirements are low, only one backup node can be configured for each training node.

[0075] For example, in distributed model training, assume that there are 4 training nodes, namely: training node A, training node B, training node C and training node D. When configuring the backup node of each training node, such as Figure 3 As shown, it is assumed that four training nodes form a ring, and each training node selects the next training node on the ring as a backup node. Specifically, training node B is configured as the backup node of training node A, training node C is configured as the backup node of training node B, training node D is configured as the backup node of training node C, and training node A is configured as the backup node of training node D.

[0076] Of course, in other examples, multiple backup nodes may be configured for each training node, for example, training node B and training node C may be configured as backup nodes for training node A, and one training node may be configured as a backup node for multiple training nodes, for example, training node B may be configured as a backup node for training node A and training node C. In practical applications, backup nodes may be flexibly set according to circumstances, and examples are not given one by one.

[0077] Of course, in other embodiments of the present application, the backup node pre-configured for each training node may also be an independently set backup memory, which is not limited in the embodiments of the present application.

[0078] In some embodiments, the process of backing up the optimizer state data in the main memory of each training node to the main memory of the backup node and the distributed storage system, and the process of backing up the model state data in the main memory of the preset master training node to the distributed storage system, can be performed asynchronously with the model training process to further save the breakpoint data preservation time and improve the model training efficiency.

[0079] Specifically, after interrupting the model training and caching the breakpoint data of each training node from the training node display memory to the training node main memory, the model training can be resumed first, and then the optimizer state data in the main memory of each training node can be backed up to the pre-configured backup node main memory and distributed storage system, and the model state data in the preset main training node main memory can be backed up to the distributed storage system.

[0080] In this way, since the time for caching the breakpoint data of each training node from the training node display memory to the training node main memory is generally in the order of seconds, and the process of backing up the optimizer state data in the main memory of each training node to the pre-configured backup node main memory and distributed storage system, and backing up the model state data in the preset main training node main memory to the distributed storage system is asynchronous with the model training, the time consumption has no effect on the corresponding model training. Therefore, when saving breakpoints, the embodiment of the present application can achieve high-frequency saving of breakpoints.

[0081] In some embodiments, in order to reduce storage overhead, when backing up the optimizer state data in each training node main memory to the pre-configured backup node main memory and the distributed storage system, the optimizer state data in each training node main memory may be backed up to the pre-configured backup node main memory at a first frequency, and the optimizer state data in each training node main memory may be backed up to the distributed storage system at a second frequency, and the first frequency is greater than the second frequency. For example, the optimizer state data in each training node main memory is backed up to the backup node main memory each time, and is backed up to the distributed storage system every predetermined number of times (for example, 5 times, 10 times, etc., which can be set based on experience).

[0082] The breakpoint saving solution provided in the embodiment of the present application is described below with reference to specific examples.

[0083] In one example, if Figure 4 As shown, the model adopts data parallel distributed training, the dimension of data parallel is 2, and two training nodes are used for distributed training, which are respectively denoted as: training node A and training node B. Training node A is the first data parallel group (Data Parallel, DP), denoted as DP0, and training node B is the second data parallel group, denoted as DP1. Training node A is the preset master node, and training node B is pre-configured as the backup node of training node A. Training node A is the backup node of training node B.

[0084] When the distributed model training reaches a preset breakpoint, the model training is interrupted, and training node A caches the breakpoint data, including the model status data and optimizer status data 1, from the display memory of training node A to the main memory of training node A, and training node B caches the breakpoint data, including the model status data and optimizer status data 2, from the display memory of training node B to the main memory of training node B.

[0085] After training node A and training node B cache the breakpoint data from their respective display memories to their respective main memories, model training is resumed. At the same time, the optimizer state data 1 in the main memory of training node A is backed up to the main memory of training node B, the optimizer state data 2 in the main memory of training node B is backed up to the main memory of training node A, and the model state data in the main memory of training node A is backed up to the distributed storage system.

[0086] At the same time, according to the frequency of backing up the optimizer state data to the distributed storage system, if the optimizer state data needs to be backed up to the distributed storage system, the optimizer state data 1 in the main memory of training node A and the optimizer state data 2 in the main memory of training node B can be backed up to the distributed storage system.

[0087] It should be noted that the frequency at which training node A backs up model state data to the distributed storage system may be the same as or different from the frequency at which optimizer state data is backed up to the distributed storage system, and this embodiment of the present application does not limit this.

[0088] Combine the following Figure 5 Taking the distributed model training, in which the optimizer state data and the model state data are backed up to the distributed storage system every 10 caches as an example, the specific implementation process of the breakpoint saving method provided in the embodiment of the present application is described in detail, such as Figure 5 As shown, the breakpoint saving method provided in the embodiment of the present application specifically includes:

[0089] Step 501, receive sample data and start model training.

[0090] Step 502: When the model training reaches a preset breakpoint, the model training is interrupted, and the breakpoint data of each training node is cached from the training node display memory to the training node main memory. The breakpoint data includes model state data and optimizer state data.

[0091] Step 503, resume model training until the next breakpoint, and execute step 502.

[0092] Step 504: back up the optimizer state data in the main memory of each training node to the main memory of the backup node.

[0093] Step 505: The counter recording the number of times the optimizer state data is backed up to the backup node is incremented by 1.

[0094] During specific implementation, counting can be performed through a system counter of the training node or a separately set counter.

[0095] Step 506, determine whether the count value of the counter is equal to 10, that is, determine whether the number of times the optimizer state data is backed up to the backup node reaches 10 times. If so, execute step 507, otherwise no processing is performed.

[0096] Step 507, when it is determined that the optimizer state data is backed up to the backup node 10 times, the optimizer state data in the main memory of each training node is backed up to the distributed storage system, and the model state data in the main memory of the preset master node is backed up to the distributed storage system.

[0097] Implementation process 2: Resume training from breakpoints.

[0098] During the actual model training process, model training may exit due to training anomalies or failures, or due to higher priority tasks. In this case, when the anomaly is eliminated or the model training task is re-executed, the model training can be continued by resuming training at the breakpoint because the breakpoint data is saved.

[0099] Taking distributed model training as an example, the model is trained in multiple distributed training nodes. When the embodiment of the present application resumes training at a breakpoint, the target version of the breakpoint data to be restored is first determined by collective communication based on the metadata of the breakpoint data stored in each training node, and then the breakpoint data of the target version is restored from the distributed storage system or the target storage corresponding to each training node. Among them, the target storage corresponding to each training node includes the training node main storage and the pre-configured backup node main storage, and the metadata of the breakpoint data includes at least the version of the breakpoint data.

[0100] The embodiment of the present application determines the target version of the breakpoint data to be restored by collective communication, that is, each training node can obtain the meta information of the breakpoint data stored by all training nodes, and then use the same decision strategy to determine the target version of the breakpoint data to be restored, and reduce the pressure of a single training node and improve stability in a decentralized manner. The specific collective communication method can adopt the method in the relevant technology, such as the all-gather method, which will not be repeated here.

[0101] It should be noted that the breakpoint data mentioned in the breakpoint resume training process of the embodiment of the present application also includes model state data and optimizer state data. Since breakpoint resume training requires the recovery of both model state data and optimizer state data, breakpoint data is still used to describe the two during breakpoint resume training. However, since the model state data of each training node is the same, when determining the target version of the breakpoint data, even if a certain version of the model state data is only stored in the main memory of one training node or in a distributed storage system, in this case, all training nodes can obtain this version of the model state data (distributed by the training node storing this version of the model state data), so it can be regarded as that this version of the model state data exists in each training node.

[0102] In specific implementation, based on the metadata of the breakpoint data stored in each training node, when determining the target version of the breakpoint data to be restored through collective communication, first obtain the metadata of the breakpoint data stored in the corresponding target memory and the distributed storage system of each training node, then instruct all training nodes to exchange the metadata of the breakpoint data stored by each training node through collective communication, and instruct each training node to determine the target version of the breakpoint data to be restored.

[0103] For example, the four training nodes in the above example are still used. When resuming training at a breakpoint, it is assumed that the version of the breakpoint data in the main memory of training node A is 5.0, the version of the breakpoint data in the main memory of the backup node is 4.0, and the version of the breakpoint data stored in the distributed storage system of training node A is 4.0; the version of the breakpoint data in the main memory of training node B is 6.0, the version of the breakpoint data in the main memory of the backup node is 5.0, and the version of the breakpoint data stored in the distributed storage system of training node B is 5.0; the version of the breakpoint data in the main memory of training node C is 5.0, the version of the breakpoint data in the main memory of the backup node is 5.0, and the version of the breakpoint data stored in the distributed storage system of training node C is 5.0; the version of the breakpoint data in the main memory of training node D is 5.0, the version of the breakpoint data in the main memory of the backup node is 5.0, and the version of the breakpoint data stored in the distributed storage system of training node D is 5.0.

[0104] In this case, the four training nodes exchange the version information of the breakpoint data with each other through collective communication, and each training node can obtain the above global information (version information of the breakpoint data of all training nodes). After that, each training node can use the same decision-making strategy to determine that the target version of the breakpoint data to be restored is 5.0.

[0105] In specific implementation, after determining the target version of the breakpoint data to be restored, each training node will give priority to restoring the breakpoint data of the target version from the target storage corresponding to each training node when restoring the breakpoint data of the target version. Specifically, for a training node, the following operations are performed to restore the breakpoint data of the target version: if the integrity check is passed and it is determined that the breakpoint data of the target version is stored in the target storage corresponding to the training node, the breakpoint data of the target version is restored from the target storage corresponding to the training node; if the integrity check is passed and it is determined that the breakpoint data of the target version is not stored in the target storage corresponding to the training node, the breakpoint data of the target version is restored from the distributed storage system.

[0106] Among them, the method of data integrity verification can adopt the method in the relevant technology, which will not be repeated here.

[0107] When restoring the breakpoint data of the target version, the embodiment of the present application preferentially restores the breakpoint data of the target version from the target storage corresponding to the training node when the breakpoint data of the target version is stored in both the target storage corresponding to the training node and the distributed storage system, thereby avoiding all training nodes from reading data from the distributed storage system and reducing the data transmission pressure of the distributed storage system.

[0108] Specifically, the target version of the breakpoint data is stored in the target memory corresponding to the training node. When restoring the breakpoint data of the target version from the target memory corresponding to the training node, the breakpoint data of the target version is restored from the main memory of the training node first. Specifically, if it is determined that the breakpoint data of the target version is stored in the main memory of the training node, the breakpoint data of the target version is restored from the main memory of the training node; if it is determined that the breakpoint data of the target version is stored in the main memory of the backup node corresponding to the training node, the breakpoint data of the target version is restored from the main memory of the backup node.

[0109] Combine the following Figure 6 Taking the distributed training of multiple training nodes in distributed model training as an example, the specific implementation process of breakpoint continuous training provided by the embodiment of the present application is described in detail, such as Figure 6 As shown, the breakpoint resume training method provided in the embodiment of the present application specifically includes:

[0110] Step 601, obtain the meta information of the breakpoint data stored in the target storage corresponding to each training node and the distributed storage system. The target storage corresponding to each training node includes the training node main storage and the pre-configured backup node main storage, and the meta information of the breakpoint data includes at least the version of the breakpoint data, and may also include the storage location of the breakpoint data, etc.

[0111] Step 602: All training nodes exchange meta information of breakpoint data in a collective communication manner, and each training node determines a target version of the breakpoint data to be restored.

[0112] It should be noted that the breakpoint data of this target version should be saved with complete data in all training nodes.

[0113] For each training node, perform the following operations from step 603 to step 607 respectively to restore the breakpoint data of the target version:

[0114] Step 603, perform integrity check to determine whether the target version of breakpoint data is stored in the target memory corresponding to the training node. If so, execute step 604; otherwise, execute step 607.

[0115] Step 604, determine whether the breakpoint data of the target version is stored in the main memory of the training node, if so, execute step 605, otherwise, execute step 606.

[0116] Step 605, restore the breakpoint data of the target version from the main memory of the training node.

[0117] Step 606, restore the breakpoint data of the target version from the main memory of the backup node corresponding to the training node.

[0118] Step 607: Restore the breakpoint data of the target version from the distributed storage system.

[0119] Based on the same technical concept, the present application embodiment provides a structural diagram of a breakpoint saving device, such as Figure 7 As shown, the breakpoint saving device 700 includes:

[0120] The cache module 701 is used to interrupt the model training when the distributed model training reaches a preset breakpoint, and cache the breakpoint data of each training node from the training node display memory to the training node main memory, where the breakpoint data includes model state data and optimizer state data;

[0121] Storage module 702 is used to back up the optimizer state data in the main memory of each training node to the pre-configured backup node main memory and distributed storage system, and to back up the model state data in the main memory of the preset master training node to the distributed storage system.

[0122] Optionally, the storage module 702 is specifically used for:

[0123] Backing up optimizer state data in each training node main memory to a pre-configured backup node main memory at a first frequency;

[0124] The optimizer state data in the main memory of each training node is backed up to the distributed storage system at a second frequency, and the first frequency is greater than the second frequency.

[0125] Optionally, the storage module 702 is specifically used for:

[0126] After resuming model training, the optimizer state data in the main memory of each training node is backed up to the pre-configured backup node main memory and distributed storage system, and the model state data in the main memory of the preset master training node is backed up to the distributed storage system.

[0127] Optionally, the device further comprises:

[0128] The processing module 703 is used to determine the target version of the breakpoint data to be restored by collective communication based on the meta information of the breakpoint data stored in each training node when resuming training at a breakpoint, where the meta information of the breakpoint data at least includes the version of the breakpoint data;

[0129] The data recovery module 704 is used to recover the breakpoint data of the target version from the distributed storage system or the target storage corresponding to each training node. The target storage corresponding to each training node includes the training node main storage and the pre-configured backup node main storage.

[0130] Optionally, the processing module 703 is specifically configured to:

[0131] Obtaining metadata of breakpoint data stored in the corresponding target storage and distributed storage system of each training node;

[0132] Instruct all training nodes to exchange meta information of their respective stored breakpoint data through collective communication, and instruct each training node to determine a target version of the breakpoint data to be restored.

[0133] Optionally, the data recovery module 704 is specifically used for:

[0134] For a training node, perform the following operations to restore the breakpoint data of the target version:

[0135] When it is determined through integrity check that the breakpoint data of the target version is stored in the target memory corresponding to a training node, the breakpoint data of the target version is restored from the target memory corresponding to a training node; or

[0136] When it is determined through integrity verification that the breakpoint data of the target version is not stored in the target storage corresponding to a training node, the breakpoint data of the target version is restored from the distributed storage system.

[0137] Optionally, the data recovery module 704 is specifically used for:

[0138] When it is determined that the breakpoint data of the target version is stored in a main memory of a training node, recover the breakpoint data of the target version from a main memory of a training node; or

[0139] When it is determined that the breakpoint data of the target version is stored in the backup node main memory corresponding to a training node, the breakpoint data of the target version is restored from the backup node main memory.

[0140] Based on the same technical concept, the embodiment of the present application provides a computer device, such as Figure 8 As shown, it includes at least one processor chip 801 and a memory 802 connected to the at least one processor chip. The specific connection medium between the processor chip 801 and the memory 802 is not limited in the embodiment of the present application. Figure 8 For example, the processor chip 801 and the memory 802 are connected via a bus. The bus can be divided into an address bus, a data bus, a control bus, and the like.

[0141] In the embodiment of the present application, the memory 802 stores instructions that can be executed by at least one processor chip 801, and the at least one processor chip 801 can perform the steps of the above-mentioned breakpoint saving method by executing the instructions stored in the memory 802.

[0142] Among them, the processor chip 801 is the control center of the computer device, and various interfaces and lines can be used to connect various parts of the computer device, and breakpoints can be saved by running or executing instructions stored in the memory 802 and calling data stored in the memory 802. Optionally, the processor chip 801 may include one or more processing units, and the processor chip 801 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor chip 801. In some embodiments, the processor chip 801 and the memory 802 may be implemented on the same chip, and in some embodiments, they may also be implemented separately on independent chips.

[0143] The processor chip 801 can be a general-purpose processor, such as a GPU, a general-purpose graphics processor (General-Purpose computing on Graphics Processing Units, GPGPU), a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly embodied as a hardware processor to be executed, or can be executed by a combination of hardware and software modules in the processor.

[0144] The memory 802 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 802 may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (Random Access Memory, RAM), a static random access memory (Static Random Access Memory, SRAM), a programmable read-only memory (Programmable Read Only Memory, PROM), a read-only memory (Read Only Memory, ROM), an electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), a magnetic memory, a disk, an optical disk, etc. The memory 802 is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer device, but is not limited thereto. The memory 802 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0145] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program executable by a computer device. When the program runs on the computer device, the computer device executes the steps of the above-mentioned breakpoint saving method.

[0146] Based on the same inventive concept, an embodiment of the present application provides a computer program product, which includes a computer program stored on a computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer device, the computer device executes the steps of the above-mentioned breakpoint saving method.

[0147] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0148] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable breakpoint saving device to produce a machine, so that the instructions executed by the processor of the computer device or other programmable breakpoint saving device generate instructions for implementing the process in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0149] These computer program instructions may also be stored in a computer readable memory capable of directing a computer device or other programmable breakpoint storage device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0150] These computer program instructions may also be loaded onto a computer device or other programmable breakpoint storage device so that a series of operating steps are executed on the computer device or other programmable device to produce a process implemented by the computer device, thereby providing instructions executed on the computer device or other programmable device for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0151] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0152] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A breakpoint saving method, characterized in that: include: When the distributed model training reaches a preset breakpoint, the model training is interrupted, and the breakpoint data of each training node is cached from the training node display memory to the training node main memory, wherein the breakpoint data includes model state data and optimizer state data; The optimizer state data in the main memory of each training node is backed up to the pre-configured backup node main memory and distributed storage system, and the model state data in the main memory of the preset main training node is backed up to the distributed storage system.

2. The method according to claim 1, characterized in that: The step of backing up the optimizer state data in the main memory of each training node to a pre-configured backup node main memory and a distributed storage system includes: Backing up the optimizer state data in the main memory of each training node to a pre-configured main memory of a backup node at a first frequency; The optimizer state data in the main memory of each training node is backed up to the distributed storage system at a second frequency, and the first frequency is greater than the second frequency.

3. The method according to claim 1, characterized in that: The step of backing up the optimizer state data in the main memory of each training node to a pre-configured backup node main memory and a distributed storage system, and backing up the model state data in the main memory of a preset master training node to the distributed storage system, comprises: After resuming model training, the optimizer state data in the main memory of each training node is backed up to the pre-configured backup node main memory and distributed storage system, and the model state data in the main memory of the preset main training node is backed up to the distributed storage system.

4. The method according to any one of claims 1 to 3, characterized in that The method further comprises: When resuming training at a breakpoint, based on the meta information of the breakpoint data stored in each training node, a target version of the breakpoint data to be restored is determined by collective communication, wherein the meta information of the breakpoint data at least includes the version of the breakpoint data; Restore the breakpoint data of the target version from the distributed storage system or the target storage corresponding to each training node, wherein the target storage corresponding to each training node includes a training node main storage and a pre-configured backup node main storage.

5. The method according to claim 4, characterized in that The step of determining the target version of the breakpoint data to be restored by collective communication based on the meta information of the breakpoint data stored in each training node includes: Obtaining meta information of breakpoint data stored in the corresponding target storage and the distributed storage system for each training node; Instruct all training nodes to exchange meta information of their respective stored breakpoint data through collective communication, and instruct each training node to determine a target version of the breakpoint data to be restored.

6. The method according to claim 4, characterized in that The recovering the breakpoint data of the target version from the distributed storage system or the target storage corresponding to each training node includes: For a training node, perform the following operations to restore the breakpoint data of the target version: When it is determined through integrity check that the breakpoint data of the target version is stored in the target memory corresponding to the one training node, recovering the breakpoint data of the target version from the target memory corresponding to the one training node; or When it is determined through integrity verification that the breakpoint data of the target version is not stored in the target memory corresponding to the one training node, the breakpoint data of the target version is restored from the distributed storage system.

7. The method according to claim 6, characterized in that The recovering the breakpoint data of the target version from the target memory corresponding to the one training node includes: When it is determined that the breakpoint data of the target version is stored in the main memory of the one training node, recovering the breakpoint data of the target version from the main memory of the one training node; or When it is determined that the breakpoint data of the target version is stored in the backup node main memory corresponding to the one training node, the breakpoint data of the target version is restored from the backup node main memory.

8. A computer device comprising a memory, a processor chip, and a computer program stored in the memory and executable on the processor chip, characterized in that: When the processor chip executes the program, the steps of any one of the methods of claims 1 to 7 are implemented.

9. A computer-readable storage medium, characterized in that: It stores a computer program executable by a computer device. When the program is run on the computer device, the computer device executes the steps of any method described in claims 1 to 7.

10. A computer program product, characterized in that The computer program product comprises a computer program stored on a computer-readable storage medium, wherein the computer program comprises program instructions, and when the program instructions are executed by a computer device, the computer device is caused to execute the steps of the method according to any one of claims 1 to 7.

Images