A method for recovering training data and a terminal

Abstract

The application provides a training data recovery method and a terminal, which comprise creating a cache center server, periodically pulling checkpoint data from each node and storing the checkpoint data, deleting the stored checkpoint data once every two periods by the cache center server, randomly selecting checkpoint data stored in one period in the remaining periods, and uploading the checkpoint data to a remote file server for storage, pulling corresponding checkpoint data from the cache center server when a node fails to train and needs to recover data, and pulling the checkpoint data from the remote file server if the cache center server fails or does not have corresponding cache, and recovering the training data. The application optimizes the storage of checkpoint data, effectively improves the recovery efficiency of training data, and further improves the data training efficiency.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method and terminal for recovering training data. Background Technology

[0002] Currently, various artificial intelligence (AI) functions have emerged, enabling various industries to perform different business processes through AI. The most common approach in AI is to input data for analysis, process it internally through a large AI model, and ultimately output the desired results. This large AI model requires input data from diverse corpora and training to obtain a relatively complete model. Because the amount of data used for model training is massive, large models not only consume significant GPU resources but also require substantial time for training. Therefore, to prevent training interruptions or to quickly recover training data after interruptions, checkpoints are typically saved during training. Saving data at different checkpoints allows for rapid recovery of the training data.

[0003] Because model training is distributed, the data generated at checkpoints during training is typically transmitted to a fixed machine. However, the entire training process needs to be paused to save this checkpoint data. Training can only resume after the checkpoint data has been successfully transmitted and saved remotely. This business logic is detrimental to improving the overall efficiency of data training. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a method and terminal for recovering training data, optimize the storage of checkpoint data, and effectively improve the efficiency of data training.

[0005] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0006] A method for recovering training data, comprising the steps of:

[0007] S1. Each node writes its own checkpoint data into its local memory in real time.

[0008] S2. The preset cache center server periodically retrieves the checkpoint data from the local memory and stores it.

[0009] S3. The cache center server uploads the checkpoint data to a remote file server for storage.

[0010] S4, when the node training fails and data needs to be recovered, the corresponding checkpoint data is pulled from the cache center server, if the cache center server fails or has no corresponding cache, the checkpoint data is pulled from the remote file server to recover the training data.

[0011] To solve the above technical problems, another technical solution adopted by the present application is:

[0012] A training data recovery terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program.

[0013] S1, each node writes its own checkpoint data into local memory in real time;

[0014] S2, the checkpoint data is periodically pulled from the local memory and stored by a preset cache center server;

[0015] S3, the cache center server uploads the checkpoint data to a remote file server for storage;

[0016] S4, when the node training fails and data needs to be recovered, the corresponding checkpoint data is pulled from the cache center server, if the cache center server fails or has no corresponding cache, the checkpoint data is pulled from the remote file server to recover the training data.

[0017] The present application has the advantages that a training data recovery method and terminal are provided, wherein the checkpoint data of all nodes is preferentially written into local memory in real time when generated, although the training process is paused during writing, but because it is written into local memory, the pause time of node training can be greatly reduced, and after the local memory is written, the training process can continue, then for all node checkpoint data, the checkpoint data is periodically pulled from the local memory and stored by a preset cache center server to avoid loss of checkpoint data; at the same time, during training recovery, the checkpoint data can be recovered through the cache center server first, without pulling data from the remote file server, and only when the cache center server fails or has no corresponding cache, the data is pulled from the remote file server, which can further improve the efficiency of training recovery, improve the efficiency of overall data training, and reduce time consumption. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 A flowchart of a training data recovery method according to an embodiment of the present application;

[0019] Figure 2A structure schematic diagram of a training data recovery terminal of an embodiment of the present application.

[0020] Label description:

[0021] 1. A training data recovery terminal; 2. a memory; 3. a processor. DETAILED DESCRIPTION

[0022] To make the technical content, the purposes and effects of the present application clear, the following will be described in conjunction with the embodiments and the accompanying drawings.

[0023] Please refer to Figure 1 A training data recovery method, comprising the steps of:

[0024] S1. Each node writes the respective checkpoint data into the local memory in real time;

[0025] S2. The checkpoint data is periodically pulled from the local memory and stored by a preset cache center server;

[0026] S3. The checkpoint data is uniformly uploaded to a remote file server for saving by the cache center server;

[0027] S4. When the node training fails and the data needs to be recovered, the corresponding checkpoint data is pulled from the cache center server, and if the cache center server fails or there is no corresponding cache, the checkpoint data is pulled from the remote file server for training data recovery.

[0028] From the above description, the beneficial effects of the present application are that a training data recovery method and terminal are provided, in which the checkpoint data of all nodes is preferentially written into the local memory in real time when the checkpoint data is generated. Although the training process is paused during the writing process, because the checkpoint data is written into the local memory, the pause time of the node training can be greatly reduced, and after the local memory is written, the training process can continue. Then, for all node checkpoint data, the checkpoint data is periodically pulled from the local memory and stored by a preset cache center server, so as to avoid the loss of checkpoint data. At the same time, during the training recovery process, the checkpoint data can be recovered through the cache center server, and there is no need to pull data from the remote file server. Only when the cache center server fails or there is no corresponding cache, the data needs to be pulled from the remote file server, which can further improve the efficiency of training recovery, improve the efficiency of overall data training, and reduce time consumption.

[0029] Further, in the step S2, the checkpoint data is periodically pulled from the local memory and stored by the preset cache center server, specifically:

[0030] The cache center server periodically pulls the checkpoint data from the local memory and performs persistent storage.

[0031] As described above, the cache center server avoids loss of checkpoint data by persistent caching.

[0032] Further, the step S3 is specifically:

[0033] The cache center server deletes the stored checkpoint data once every two cycles and randomly selects the checkpoint data stored in one of the remaining cycles to upload to the remote file server for storage.

[0034] As described above, since checkpoint data is not used every time, it is only used in the training interruption scenario, therefore, the cache center server deletes the stored data once every two cycles, and extracts one of the remaining cycles, and uploads the checkpoint data stored in this cycle to the remote file server, that is, the cache center server only retains the last two checkpoint data and uploads it to the remote file server for storage, effectively reducing data transmission and the amount of file data stored in the remote server.

[0035] Further, the step S3 further includes:

[0036] When the node training fails and data needs to be recovered, it is first determined whether the same batch of checkpoint data exists in the local cache of all nodes, if yes, the corresponding checkpoint data is pulled from the local memory, otherwise, the corresponding checkpoint data is pulled from the cache center server.

[0037] As described above, similarly, when the node training fails and data needs to be recovered, the corresponding checkpoint data is first pulled from the local memory, when the same batch of checkpoint data written in the local memory of all nodes exists in the checkpoint data of a certain node or multiple nodes, it is considered that the checkpoint data is incomplete, therefore, the cache center server can be pulled again to avoid failure to recover the training.

[0038] Further, the same batch of checkpoint data refers to checkpoint data generated at the same time.

[0039] As described above, the generation time is used as a criterion for judging the same batch of checkpoint data to ensure the accuracy of the determination of whether the same batch of checkpoint data exists in the local memory.

[0040] Please refer to Figure 2A recovery terminal of training data, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program;

[0041] S1, each node writes respective checkpoint data into local memory in real time;

[0042] S2, periodically pulling the checkpoint data from the local memory and storing by a preset cache center server;

[0043] S3, the cache center server uploads the checkpoint data to a remote file server for saving;

[0044] S4, when a node training fails and needs to recover data, the corresponding checkpoint data is pulled from the cache center server, if the cache center server fails or has no corresponding cache, the checkpoint data is pulled from the remote file server for training data recovery.

[0045] From the above description, the beneficial effects of the present application are that based on the same technical concept, in cooperation with the above-mentioned recovery method of training data, a recovery terminal of training data is provided, wherein the checkpoint data of all nodes is written into local memory in real time when generated, although the training process is paused during writing, but because it is written into local memory, the pause time of node training can be greatly reduced, and after the local memory is written, the training process can continue, then for all node checkpoint data, the checkpoint data is periodically pulled from the local memory and stored by a preset cache center server, to avoid loss of checkpoint data; at the same time, during the training recovery process, the checkpoint data can be recovered through the cache center server, without the need to pull data from the remote file server, and only when the cache center server fails or has no corresponding cache, the data needs to be pulled from the remote file server, which can further improve the efficiency of training recovery, improve the efficiency of overall data training, and reduce time consumption.

[0046] Further, in the step S2, the checkpoint data is periodically pulled from the local memory and stored by the preset cache center server, specifically:

[0047] The cache center server periodically pulls the checkpoint data from the local memory and performs persistent storage.

[0048] From the above description, the cache center server avoids loss of checkpoint data through persistent caching.

[0049] Further, the step S3 is specifically:

[0050] The cache center server deletes the stored checkpoint data once every two cycles, and uploads the checkpoint data stored in one of the remaining cycles to the remote file server for storage.

[0051] As can be seen from the above description, since the checkpoint data is not used every time, it is only used when the training is interrupted, therefore, the cache center server deletes the stored data once every two cycles, and extracts one of the checkpoint data stored in the remaining cycles, and uploads the checkpoint data stored in this cycle to the remote file server, that is, the cache center server only retains the last two checkpoint data and uploads it to the remote file server for storage, thereby effectively reducing the data transmission and the amount of file data stored in the remote server.

[0052] Further, the step S3 further comprises:

[0053] When the node training fails and data needs to be recovered, it is first determined whether the same batch of checkpoint data exists in the local cache of all nodes, if yes, the corresponding checkpoint data is pulled from the local memory, otherwise, the corresponding checkpoint data is pulled from the cache center server.

[0054] As can be seen from the above description, similarly, when the node training fails and data needs to be recovered, the corresponding checkpoint data is first pulled from the local memory, when the same batch of checkpoint data written in the local memory of all nodes is missing in the checkpoint data of a node or multiple nodes, it is considered that the checkpoint data is incomplete, therefore, the cache center server can be pulled again to avoid the failure of the training.

[0055] Further, the same batch of checkpoint data refers to the checkpoint data generated at the same time.

[0056] As can be seen from the above description, the generation time is used as a criterion for judging the same batch of checkpoint data, so as to ensure the accuracy of the determination of whether the same batch of checkpoint data exists in the local memory.

[0057] The present application provides a training data recovery method and terminal, which is mainly applied to the scene of training failure and recovery of training data in the process of distributed running training data, and will be described in detail below:

[0058] Please refer to Figure 1 The embodiment one of the present application is:

[0059] A training data recovery method, in the old way, the checkpoint data in the model training process is directly transmitted to the remote file server, the training process on the node is paused during the transmission process of the checkpoint data, and the training process continues after the transmission is completed.

[0060] And the embodiment of the present application improves the old way, as shown, comprising the steps of: Figure 1

[0061] S1, each node writes the respective checkpoint data into the local memory in real time.

[0062] S2, periodically pull checkpoint data from the local memory by the preset cache center server and store.

[0063] Wherein each node in generating the corresponding checkpoint data, will send a notification message to the cache center server, the cache center server by receiving notification message to periodically regularly to each node pull checkpoint data.

[0064] S3, the cache center server will checkpoint data uniformly uploaded to the remote file server for saving.

[0065] S4, when the node training failure needs to recover data, pull the corresponding checkpoint data from the cache center server, if the cache center server failure or no corresponding cache, from the remote file server to pull, training data recovery.

[0066] That is, in this embodiment, the checkpoint data of all nodes is written into the local memory in real time when it is generated, although the training process is also suspended during writing, but because it is written into the local memory, it can greatly reduce the suspension time of node training, and after the local memory is written, the training process can continue, and then for all node checkpoint data, the checkpoint data is periodically pulled from the local memory by a preset cache center server for storage, to avoid the loss of checkpoint data; At the same time, during the training recovery process, the checkpoint data can be recovered through the cache center server, without the need to pull data from the remote file server, only when the cache center server fails or there is no corresponding cache, it is necessary to pull from the remote file server, which can further improve the efficiency of training recovery, improve the efficiency of the whole data training, and reduce the time consumption.

[0067] Embodiment two of the present application is:

[0068] A training data recovery method, based on the above embodiment one, in this embodiment, the step S2 is that the checkpoint data is periodically pulled from the local memory by the preset cache center server and stored, specifically:

[0069] The cache center server periodically pulls checkpoint data from the local memory and stores persistently to avoid the loss of checkpoint data.

[0070] Meanwhile, in this embodiment, step S3 is specifically:​

[0071] The cache center server deletes the stored checkpoint data once every two cycles, and uploads the checkpoint data stored in one of the remaining cycles to the remote file server for storage.

[0072] That is, in the embodiment, since the checkpoint data is not used every time, it is only used when the training is interrupted, therefore, the cache center server deletes the stored data once every two cycles, and extracts one of the remaining cycles, and uploads the checkpoint data stored in the cycle to the remote file server, that is, the cache center server only retains the last two checkpoint data and uploads it to the remote file server for storage, thereby effectively reducing the data transmission and the amount of file data stored in the remote server.

[0073] In addition, in the embodiment, step S3 further includes:

[0074] When the node training fails and data needs to be recovered, it is first determined whether the same batch of checkpoint data exists in the local cache of all nodes, if yes, the corresponding checkpoint data is pulled from the local memory, otherwise, the corresponding checkpoint data is pulled from the cache center server.

[0075] That is, when the node training fails and data needs to be recovered, the corresponding checkpoint data is also pulled from the local memory first, when the same batch of checkpoint data written in the local memory of all nodes exists in the checkpoint data of a certain node or multiple nodes, it is considered that the checkpoint data is incomplete, therefore, the cache center server can be pulled again, thereby avoiding the failure of training recovery.

[0076] The same batch of checkpoint data refers to checkpoint data generated at the same time, that is, the generation time is used as a standard for judging the same batch of checkpoint data, so as to ensure the accuracy of the determination of whether the same batch of checkpoint data exists in the local memory.

[0077] Please refer to Figure 2 The fifth embodiment of the present application is:

[0078] A training data recovery terminal 1, as shown in Figure 2 The processor 3 executes the computer program to complete the steps of the training data recovery method in one of the first embodiment or the second embodiment.

[0079] In summary, the training data recovery method and terminal provided by the present application can improve the efficiency of training data recovery, thereby improving the efficiency of overall data training and reducing time consumption.

[0080] The above merely illustrates the embodiments of the present application, and is not intended to limit the patent scope of the present application. Any equivalent transformation or direct or indirect application in the related technical field based on the content of the present application specification and drawings is also included in the patent protection scope of the present application.

Claims

1. A method of recovering training data, characterized by, The method comprises the steps of: S1, each node writes the respective checkpoint data into the local memory in real time; S2, the preset cache center server periodically pulls the checkpoint data from the local memory and stores it; S3, the cache center server uploads the checkpoint data to the remote file server for saving; S4, when the node training fails and data needs to be recovered, the corresponding checkpoint data is pulled from the cache center server, if the cache center server fails or does not have corresponding cache, the remote file server is pulled for training data recovery; The step S3 is specifically: The cache center server deletes the stored checkpoint data once every two cycles, and randomly selects the checkpoint data stored in one of the remaining cycles to upload to the remote file server for saving; The step S4 further comprises: When the node training fails and data needs to be recovered, it is first determined whether the local cache of all nodes has the same batch of checkpoint data, if yes, the corresponding checkpoint data is pulled from the local memory, otherwise, the corresponding checkpoint data is pulled from the cache center server. 2.The method of claim 1, wherein, In the step S2, the preset cache center server periodically pulls the checkpoint data from the local memory and stores it, specifically: The cache center server periodically pulls the checkpoint data from the local memory and stores it persistently.

3. The method of claim 1, wherein, The same batch of checkpoint data refers to the checkpoint data generated at the same time.

4. A recovery terminal of training data, characterized by, A computer program product, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program: S1, each node writes the respective checkpoint data into the local memory in real time; S2, the preset cache center server periodically pulls the checkpoint data from the local memory and stores it; S3, the cache center server uploads the checkpoint data to the remote file server for saving; S4, when the node training fails and data needs to be recovered, the corresponding checkpoint data is pulled from the cache center server, if the cache center server fails or does not have corresponding cache, the remote file server is pulled for training data recovery; The step S3 is specifically: The cache center server deletes the stored checkpoint data once every two cycles, and randomly selects the checkpoint data stored in one of the remaining cycles to upload to the remote file server for saving; The step S4 further comprises: When the node training fails and data needs to be recovered, it is first determined whether the local cache of all nodes has the same batch of checkpoint data, if yes, the corresponding checkpoint data is pulled from the local memory, otherwise, the corresponding checkpoint data is pulled from the cache center server.

5. The training data recovery terminal of claim 4, wherein, In the step S2, the preset cache center server periodically pulls the checkpoint data from the local memory and stores it, specifically: The cache center server periodically pulls the checkpoint data from the local memory and performs persistent storage.

6. The training data recovery terminal of claim 4, wherein, The checkpoint data of the same batch refers to the checkpoint data generated at the same time.

Images