Model training checkpoint generation method, system, and storage medium

Abstract

The application provides a model training checkpoint generation method, system and storage medium. The method comprises: obtaining training data required for model training; performing full backup of the training data when processing of each batch of training data starts; periodically generating training state data according to a preset time interval during processing of each batch of training data; and performing merging operation on the training data and the training state data by an asynchronous task to generate checkpoint data. The application splits the checkpoint into two parts of training data and training state data, adopts one-time complete backup and periodic incremental backup strategies respectively, and then generates the final checkpoint by merging through an asynchronous task, thereby reducing the data volume and storage overhead of each backup, achieving more fine training state saving, and improving the overall training efficiency.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, system, and storage medium for generating model training checkpoints. Background Technology

[0002] The most crucial element in an artificial intelligence system is the large model, through which various analytical results are generated. The generation of large models requires training on massive amounts of data to obtain models suitable for specific domains.

[0003] During the training of large models, due to the massive amount of input training data involved, such as billions or tens of billions of data points, checkpoints are typically used in each training session to record the current batch of data being trained. When a system failure occurs, the checkpoint data can be reread and loaded, thereby resuming training from the point of failure and avoiding starting from scratch.

[0004] However, in existing technologies, a complete backup of the entire training process's data is required each time a checkpoint is generated. A complete checkpoint dataset typically includes both training data and training state data. Backing up both of these data each time a checkpoint is generated incurs additional resource consumption, including storage resource usage and backup time overhead, thus impacting overall training efficiency. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a method, system and storage medium for generating model training checkpoints, so as to solve the technical problem that in the prior art, each time a checkpoint is generated, a complete backup of the training data and training state data is required, which leads to high resource consumption and low training efficiency.

[0006] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: A method for generating model training checkpoints, comprising: Obtain the training data required for model training; When each batch of training data begins processing, the entire training data is backed up. During the processing of each batch of training data, training state data is generated periodically at preset time intervals. The training data and the training state data are merged through an asynchronous task to generate checkpoint data.

[0007] To solve the above-mentioned technical problems, another technical solution adopted by the present invention is as follows: A model training checkpoint generation system includes a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the steps of the method described above.

[0008] To solve the above-mentioned technical problems, another technical solution adopted by the present invention is as follows: A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above.

[0009] The beneficial effects of this invention are as follows: The entire checkpointing process is broken down. When a batch of training data has not been completed, that batch of training data is still used. Smaller time slices are used to store the training state data. Before each major checkpoint changes, only the state data of that smaller time slice needs to be updated. Other programs asynchronously merge the training data and training state data once each time training state data is generated, thus generating a checkpoint. This method significantly reduces the number of checkpoints generated, allows for more granular updates to checkpoint data, enables more accurate backup of training data, and improves overall training efficiency. Attached Figure Description

[0010] Figure 1 This is a flowchart of a model training checkpoint generation method according to an embodiment of the present invention; Figure 2 This is a schematic diagram of a model training checkpoint generation system according to an embodiment of the present invention. Detailed Implementation

[0011] Definitions:

[0012] To explain in detail the technical content, objectives, and effects of the present invention, the following description is provided in conjunction with the embodiments and accompanying drawings.

[0013] In existing technologies, training large models requires processing massive amounts of data, and the training process can last for days or even weeks. To ensure that the training task can be recovered after a failure, the industry generally adopts a checkpointing mechanism to periodically save the current training state, and load the latest checkpoint to resume training in case of a failure. Existing checkpoint generation methods typically perform a full snapshot at each preset saving period. A complete checkpoint includes two parts: training data and training state data. Training data is the raw data input to the model, and training state data is dynamically generated model parameters, optimizer states, etc. Each time a checkpoint is generated, both parts of data need to be backed up and uploaded to persistent storage. As the model size grows, the checkpoint file grows dramatically. Each full backup not only consumes a lot of storage resources and network bandwidth, but also blocks the main training process, resulting in a significant decrease in training efficiency.

[0014] To at least address the aforementioned issues, this invention provides a method for generating model training checkpoints. The checkpoint is split into two parts: training data and training state data. At the start of each batch of training, a full backup of the training data is performed. During the training process, training state data is generated periodically at preset time intervals. Subsequently, the training data and training state data are immediately merged through an asynchronous task to generate a complete checkpoint.

[0015] In this way, the amount of data generated at each checkpoint can be reduced from a full backup to backing up only the training data once and multiple incremental training state data, which significantly reduces the I / O overhead and storage resource consumption of a single backup. At the same time, asynchronous merging avoids blocking the main training process, enabling more granular saving of training state, thereby improving overall training efficiency.

[0016] The following describes in detail a method for generating model training checkpoints according to the present invention. Please refer to [link / reference]. Figure 1 The method 100 includes steps 101 to 104: Step 101: Obtain the training data required for model training, including: Step 1011: Transform the acquired raw training data into training data with a preset structure.

[0017] Specifically, the size of the training data is determined based on the size of the original training data; the size of the training state data is determined based on the size of the training data and the size of the preset checkpoint data.

[0018] Specifically, each model training session involves uploading a batch of raw training data. For example, to train a large model for an alarm domain, a batch of structured raw alarm data is transmitted. This raw training data is first converted into token data that the model can understand, resulting in the converted training data. The size of the training data can be determined based on the size of the raw training data. Simultaneously, the size of the training state data can be determined based on the size of the training data and the total data volume of a preset checkpoint. For example, assuming the total data volume of a checkpoint is 100GB, with 80GB of training data, the size of the training state data can be determined to be 20GB. In this way, by converting the raw training data into training data with a preset structure and determining the size of the training state data based on the size of the training data, the storage ratio between training data and training state data can be clearly distinguished. This provides a data foundation for subsequent differentiated backup strategies, ensuring the rationality and operability of checkpoint splitting.

[0019] Step 1012: Input the training data of the preset structure into the model.

[0020] Specifically, after the token data is obtained through conversion, the training data with these pre-defined structures is input into the model for training.

[0021] Step 102: When each batch of training data begins processing, a complete backup of the training data is performed.

[0022] Specifically, when each batch of training data begins processing, all training data is backed up and uploaded to a preset persistent data source asynchronously.

[0023] Specifically, at the start of processing each batch of training data, a complete backup of the current batch of training data is performed immediately. For example, with the aforementioned 80GB of training data, only this 80GB needs to be backed up during the first backup, and then uploaded to a persistent data source asynchronously. The persistent data source can be a database, file storage system, or disk, etc. Uploading asynchronously avoids blocking the main training process and does not affect the normal progress of training.

[0024] Step 103: During the processing of each batch of training data, training state data is generated periodically at preset time intervals, including: Step 1031: Divide the total time for processing each batch of training data into preset time slices using preset time intervals.

[0025] Specifically, an initial time interval is preset; during model training, the initial time interval is adjusted based on historical model training results to obtain the preset time interval.

[0026] Specifically, in addition to the original training data, the remaining temporary and intermediate data are all considered training state data. This data is temporarily stored in the cache or memory and needs to be read and used to generate training state data. Assuming that processing this batch of training data takes 1000 seconds, an initial time interval can be preset, such as 200 seconds, and the 1000 seconds can be divided into 5 time slices.

[0027] The initial time interval can be set to a fixed value. During model training, this interval is automatically adjusted based on historical training data, such as the frequency of changes in training state data and system load, resulting in an optimized time interval without constant manual setting. In this way, by dividing the total batch processing time into multiple time slices and periodically generating training state data, fine-grained preservation of the training state is achieved. Simultaneously, dynamically adjusting the time interval based on historical training results adapts to the characteristics of different training tasks, ensuring granular recovery while avoiding overly frequent backup operations, further optimizing system resource utilization.

[0028] Step 1032: Store the training state data in a preset temporary storage area according to the time slice.

[0029] Specifically, at the end of each 200-second time slice, the system reads the currently generated temporary and intermediate data from memory and cache, generates the training state data (e.g., 20G) corresponding to that time slice, and stores it in the temporary storage area.

[0030] Step 104: Merge the training data and the training state data using an asynchronous task to generate checkpoint data.

[0031] For the same batch of training data, after each training state data is generated, the training state data is merged with the currently generated training data through an asynchronous task to generate a checkpoint data.

[0032] Specifically, asynchronous tasks automatically merge the training data and training state data for a given checkpoint. For example, after generating the first 20GB of training state data at 200 seconds, the asynchronous task immediately merges it with the initial 80GB backup of training data to generate the first checkpoint. After generating the second 20GB of state data at 400 seconds, the asynchronous task again merges it with the same 80GB training data to generate the second, more real-time checkpoint. Asynchronous merging is used to improve execution efficiency without affecting the main process's time and resource consumption. Since the training data and training state data are already fixed during merging, they can be directly merged into the final checkpoint file. This method achieves finer-grained time and higher-frequency checkpoint data generation compared to traditional methods. For example, traditional methods may generate a complete 100G checkpoint every 20 minutes (each checkpoint includes 80G of training data and 20G of new training state data), while this embodiment can generate a complete 100G checkpoint every 5 minutes (asynchronously merging the newly generated 20G of training state data with the backed-up 80G of training data into a single checkpoint), thereby enabling training to be recovered from an updated and more accurate point in time in the event of a failure.

[0033] Following step 104, the method further includes: loading the checkpoint data to resume data training when a system failure occurs.

[0034] Specifically, when a system malfunctions, training can be resumed simply by loading the latest, more real-time checkpoint complete data generated in step 104, without having to retrain from the beginning, thus reducing training loss caused by the malfunction.

[0035] As described above, through the above steps, the present invention splits the complete checkpoint into two parts: training data and training state data. It adopts a strategy that combines a one-time complete backup of training data with periodic incremental backup of training state data. Then, the final checkpoint is generated by asynchronous tasks in stages. This reduces the amount of data and storage overhead for each backup, achieves more refined training state preservation, and improves overall training efficiency.

[0036] Please refer to Figure 2 The present invention also provides a model training checkpoint generation system 200, including a memory 201, a processor 202, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the various steps of a model training checkpoint generation method as described above.

[0037] Another technical solution adopted by the present invention is: a computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the various steps in the above-described model training checkpoint generation method.

[0038] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent modifications made based on the content of the present invention specification and drawings, or direct or indirect applications in related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A method for generating checkpoints for model training, characterized in that, include: Obtain the training data required for model training; When each batch of training data begins processing, the entire training data is backed up. During the processing of each batch of training data, training state data is generated periodically at preset time intervals. The training data and the training state data are merged through an asynchronous task to generate checkpoint data.

2. The method for generating model training checkpoints according to claim 1, characterized in that, Obtain the training data required for model training, including: The acquired raw training data is transformed into training data with a preset structure, and the training data with the preset structure is input into the model.

3. The method for generating model training checkpoints according to claim 2, characterized in that, The process of transforming the acquired raw training data into training data with a pre-defined structure also includes: The size of the training data is determined based on the size of the original training data. The size of the training state data is determined based on the size of the training data and the size of the preset checkpoint data.

4. The method for generating model training checkpoints according to claim 1, characterized in that, At the beginning of processing each batch of training data, a complete backup of the training data is performed, including: When each batch of training data begins processing, all training data is backed up and uploaded to a preset persistent data source asynchronously.

5. The method for generating model training checkpoints according to claim 3, characterized in that, During the processing of each batch of training data, training state data is generated periodically at preset time intervals, including: The total time for processing each batch of training data is divided into preset time slices by a preset time interval. The training state data is stored in a preset temporary storage area according to the time slice.

6. The method for generating model training checkpoints according to claim 5, characterized in that, The total time for processing each batch of training data is divided into preset time slices by a preset time interval, including: Preset initial time interval; During model training, the initial time interval is adjusted based on historical model training results to obtain a preset time interval.

7. The method for generating model training checkpoints according to claim 3, characterized in that, The training data and the training state data are merged through an asynchronous task to generate checkpoint data, including: For the same batch of training data, after each training state data is generated, the training state data is merged with the currently generated training data through an asynchronous task to generate a checkpoint data.

8. The method for generating model training checkpoints according to claim 1, characterized in that, Also includes: When the system fails, the checkpoint data is loaded to resume data training.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the model training checkpoint generation method according to any one of claims 1 to 8.

10. A model training checkpoint generation system, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement each step of the model training checkpoint generation method according to any one of claims 1 to 8.

Images