Management system of model check point file

Abstract

The invention relates to the technical field of artificial intelligence, and discloses a model check point file management system which comprises a data downloader, a metadata storage system, a local storage system and a remote storage system. The data downloader is used for receiving a check point file acquisition request and sending a metadata acquisition request for acquiring a storage address of a first check point file to the metadata storage system; the metadata storage system is used for querying first metadata corresponding to the first check point file according to the metadata acquisition request and sending a first storage address to the data downloader; the data downloader is used for acquiring the first check point file from the remote storage system according to the first storage address and storing the first check point file to the local storage system; the local storage system is used for returning the first check point file to the first training task. According to the method, massive check point files can be stored, the storage performance is optimized, and reliable operation of model training tasks can be guaranteed.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and more specifically to a management system for model checkpoint files. Background Technology

[0002] During model training, information such as model parameters, optimizer status, and training status is saved periodically, and corresponding checkpoints are generated so that training can continue when training is interrupted (such as by hardware failure) or needs to be resumed.

[0003] In scenarios involving large-scale AI (artificial intelligence) model training, such as autonomous driving, the large number of models necessitates the proper management and use of numerous checkpoints. Current checkpoint management methods are limited to single or small models and are unsuitable for large-scale training scenarios. Summary of the Invention

[0004] In view of this, the present invention provides a management system for model checkpoint files, which can efficiently manage checkpoint files for large-scale training tasks.

[0005] In a first aspect, the present invention provides a management system for model checkpoint files, comprising: a data downloader, a metadata storage system, a local storage system, and a remote storage system;

[0006] The data downloader is used to receive a checkpoint file acquisition request, which is used to acquire the first checkpoint file required for the first training task; and to send a metadata acquisition request to the metadata storage system to acquire the storage address of the first checkpoint file.

[0007] The metadata storage system is used to query the first metadata corresponding to the first checkpoint file according to the metadata acquisition request, and send the first storage address in the first metadata to the data downloader;

[0008] The data downloader is used to obtain a first checkpoint file stored at the first storage address from the remote storage system according to the first storage address, and store the first checkpoint file to the local storage system; the remote storage system is used to store checkpoint files for each training task;

[0009] The local storage system is used to return the first checkpoint file to the first training task.

[0010] In some optional implementations, the metadata acquisition request includes the task identifier and version information corresponding to the first training task; the metadata stored in the metadata storage system includes: task identifier, version information, and storage address;

[0011] The process by which the metadata storage system queries the first metadata corresponding to the first checkpoint file based on the metadata acquisition request includes:

[0012] Query at least one candidate metadata in the metadata storage system that has the same task identifier and the same version information as the first training task;

[0013] Select a first metadata from at least one of the candidate metadata; or send at least one of the candidate metadata to the data downloader to instruct the data downloader to select a first metadata from at least one of the candidate metadata.

[0014] In some optional implementations, the metadata stored in the metadata storage system may further include: time information and / or evaluation metrics;

[0015] The process by which the metadata storage system or the data downloader selects the first metadata from at least one of the candidate metadata includes:

[0016] Based on the time information and / or evaluation metrics of each candidate metadata, determine the latest candidate metadata or the candidate metadata with the best evaluation metrics;

[0017] The latest candidate metadata or the candidate metadata with the best evaluation metrics is used as the first metadata.

[0018] In some optional implementations, the process by which the metadata storage system sends the first storage address in the first metadata to the data downloader includes:

[0019] If the first checkpoint file is an incremental checkpoint file, determine the first base storage address of the first base checkpoint file corresponding to the first checkpoint file; the incremental checkpoint file is used to record the incremental checkpoint file information that has changed based on the base checkpoint file corresponding to the incremental checkpoint file.

[0020] Send the first storage address and the first base storage address from the first metadata to the data downloader;

[0021] The data downloader is configured to: obtain a first checkpoint file stored at the first storage address and a first benchmark checkpoint file stored at the first benchmark storage address from the remote storage system according to the first storage address and the first benchmark storage address, and store the first checkpoint file and the first benchmark checkpoint file to the local storage system;

[0022] The local storage system is used to: return the first checkpoint file and the first benchmark checkpoint file to the first training task.

[0023] In some alternative implementations, the local storage system includes: a main memory cache and multiple storage partitions;

[0024] The main memory cache is used to cache the data to be processed obtained by the local storage system and to divide the data to be processed into multiple partition data.

[0025] The storage partition is used to retrieve the corresponding partition data from the main memory cache.

[0026] In some optional implementations, the storage partition includes: a controller, a plurality of first storage devices, and a plurality of second storage devices; the first storage devices correspond one-to-one with the second storage devices;

[0027] The controller is used to divide the partition data corresponding to the storage partition into multiple data blocks, and write the multiple data blocks into multiple first storage devices in parallel.

[0028] The first storage device is used to store the corresponding data blocks;

[0029] The second storage device is used to back up the corresponding data blocks in the first storage device and to provide data read and write capabilities to the controller in the event of a failure of the first storage device.

[0030] In some alternative implementations, the management system includes: a data uploader;

[0031] The local storage system is also used to obtain the second checkpoint file to be stored sent by the second training task, and send the second checkpoint file to the data uploader;

[0032] The data uploader is used to upload the second checkpoint file to the second storage address of the remote storage system; generate second metadata containing the second storage address, and store the second metadata in the metadata storage system.

[0033] In some alternative implementations, the remote storage system is further used for:

[0034] Calculate the pending hash value of the second checkpoint file stored in the remote storage system, and obtain the standard hash value in the second metadata;

[0035] Determine whether the undetermined hash value is consistent with the standard hash value;

[0036] If the pending hash value matches the standard hash value, the second checkpoint file is determined to be stored successfully.

[0037] In some optional implementations, the process by which the data uploader uploads the second checkpoint file to the second storage address of the remote storage system includes:

[0038] If the second checkpoint file is generated based on the second baseline checkpoint file, the checkpoint file increment information between the second checkpoint file and the second baseline checkpoint file is uploaded to the second storage address of the remote storage system;

[0039] Determine the second baseline storage address of the second baseline checkpoint file, and add both the second storage address and the second baseline storage address to the second metadata of the second checkpoint file.

[0040] In some alternative implementations, the data uploader is also used for:

[0041] The upload frequency of checkpoint files is dynamically determined based on the upload bandwidth and concurrency limits of the remote storage system.

[0042] Upload checkpoint files to the remote storage system at the specified upload frequency.

[0043] The model checkpoint file management system provided by this invention enables a data downloader to obtain the storage address of the checkpoint files required for training tasks from a metadata storage system. This allows the checkpoint files in the remote storage system to be stored in the local storage system for use by training tasks. The management system can combine local and remote storage systems to store massive amounts of checkpoint files and optimize storage performance. Based on the metadata of the checkpoint files, each checkpoint file can be effectively managed. Even when there are a large number of training tasks and a large number of checkpoint files, the data downloader can quickly obtain the required checkpoint files based on the metadata and provide them to each training task to resume model training, minimize training downtime, and ensure the reliable operation of model training tasks. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the specific embodiments or related technologies of the present invention, the drawings used in the description of the specific embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0045] Figure 1 This is a schematic diagram of a management system for model checkpoint files according to an embodiment of the present invention;

[0046] Figure 2This is another structural diagram of the management system for model checkpoint files according to an embodiment of the present invention;

[0047] Figure 3 This is a schematic diagram of a local storage system according to an embodiment of the present invention.

[0048] Explanation of reference numerals in the attached figures:

[0049] 10. Data downloader; 20. Metadata storage system; 30. Local storage system; 40. Remote storage system; 50. Data uploader; 31. Main memory cache; 32. Storage partition; 321. Controller; 322. First storage device; 323. Second storage device. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0051] In the training of large-scale AI models, effective management of model checkpoint files is crucial for ensuring training continuity, fault tolerance, and distributed scalability. As the complexity of deep learning models continues to increase, model training typically requires terabytes (TB) of storage space and thousands of GPUs (graphics processing units). Model training is computationally expensive, time-consuming, and resource-constrained, making checkpoint files essential.

[0052] Specifically, large-scale models require significant GPU / TPU (Graphics Processing Unit / Tensor Processing Unit) resources, which can fail due to hardware or cloud outages. Checkpoint files ensure recovery without requiring a restart, thus reducing wasted computation time and costs.

[0053] Renting GPU clusters is expensive, but saving checkpoint files allows training to be paused and resumed without leaving resources idle. Large models are often trained on multiple nodes, and checkpoint files help synchronize model weights across different nodes and allow for recovery in case of node failure.

[0054] Furthermore, intermediate checkpoint files help analyze model behavior and diagnose problems such as overfitting, underfitting, or catastrophic forgetting. Uploaded checkpoint files allow for the execution of different tasks using pre-trained large models, thereby reducing training time and data requirements.

[0055] Large-scale distributed training clusters involve numerous training tasks, resulting in a massive number of model checkpoint files. This necessitates a checkpoint file management system capable of efficiently storing, retrieving, and synchronizing these files. However, current checkpoint file management methods are limited to single models or a small number of models, making them unsuitable for large-scale training scenarios.

[0056] The present invention provides a model checkpoint file management system that combines local storage and remote storage to store massive amounts of checkpoint files; furthermore, based on the metadata of the checkpoint files, it can effectively manage each checkpoint file and provide fast and accurate read and write capabilities for checkpoint files.

[0057] Figure 1 The diagram shown is a structural schematic of a management system for model checkpoint files according to an embodiment of the present invention. Figure 1 As shown, the management system includes: a data downloader 10, a metadata storage system 20, a local storage system 30, and a remote storage system 40.

[0058] The data downloader 10 is used to receive checkpoint file acquisition requests, which are used to acquire the first checkpoint file required for the first training task; and to send a metadata acquisition request to the metadata storage system 20 to acquire the storage address of the first checkpoint file.

[0059] The metadata storage system 20 is used to query the first metadata corresponding to the first checkpoint file according to the metadata acquisition request, and send the first storage address in the first metadata to the data downloader 10.

[0060] The data downloader 10 is used to obtain the first checkpoint file stored at the first storage address from the remote storage system 40 according to the first storage address, and store the first checkpoint file to the local storage system 30; the remote storage system 40 is used to store the checkpoint files of each training task.

[0061] Local storage system 30 is used to return the first checkpoint file to the first training task.

[0062] In this management system, the various nodes cooperate with each other to complete the process of restoring the training task. For example... Figure 1 As shown, the working process of the management system may include the following steps S101 to S106.

[0063] Step S101: Data downloader 10 receives a request to obtain checkpoint files for the first training task.

[0064] In this embodiment, when a training task needs to resume training, it can send a corresponding request to the data downloader 10. For ease of description, this training task is referred to as the first training task. Furthermore, since training can only be resumed based on the corresponding checkpoint file at this time, the request initiated based on the first training task is referred to as the checkpoint file acquisition request.

[0065] In the event of a failure during model training or when a phase of training is completed, if it is necessary to continue training the model, a corresponding warmup request can be initiated. This warmup request is a type of checkpoint file retrieval request.

[0066] It is understandable that in a distributed training scenario, due to the large number of training tasks, each training task may need to resume training. Therefore, the data downloader 10 may receive multiple requests from training teams or trainers to obtain checkpoint files for their respective training tasks. For ease of description, this embodiment only takes the first training task as an example. The recovery process of the remaining training tasks is similar and will not be described in detail.

[0067] In step S102, the data downloader 10 sends a metadata retrieval request to the metadata storage system 20. This metadata retrieval request is used to obtain the storage address of the first checkpoint file.

[0068] In this embodiment, after receiving the checkpoint file retrieval request, the data downloader 10 can determine the first checkpoint file required for the first training task. For example, the checkpoint file retrieval request includes relevant information about the first checkpoint file, such as its unique identifier. Based on this, the data downloader 10 can generate a request to obtain the storage address of the first checkpoint file, i.e., a metadata retrieval request, and send this metadata retrieval request to the metadata storage system 20 to instruct the metadata storage system 20 to return the storage address of the first checkpoint file.

[0069] In step S103, the metadata storage system 20 queries the first metadata corresponding to the first checkpoint file according to the metadata acquisition request, and sends the first storage address in the first metadata to the data downloader 10.

[0070] In this embodiment, the metadata storage system 20 is used to store the metadata of each checkpoint file; wherein, the metadata storage system 20 can be a storage cluster, or since the amount of metadata information is small, the metadata storage system 20 can also be a single storage device, or multiple storage devices with redundancy. This embodiment does not limit the implementation form of the metadata storage system 20.

[0071] After receiving the metadata retrieval request sent by the data downloader 10, the metadata storage system 20 can query the locally stored metadata to determine the metadata corresponding to the first checkpoint file, i.e., the first metadata.

[0072] The checkpoint file itself is stored in the remote storage system 40, while the metadata storage system 20 is only used to store the metadata of the checkpoint file. This metadata includes the storage address of the corresponding checkpoint file in the remote storage system 40, which can specifically be the URL (Uniform Resource Locator) of the checkpoint file. Correspondingly, the first metadata also includes the storage address of the first checkpoint file in the remote storage system 40, i.e., the first storage address.

[0073] For example, the metadata stored in the metadata storage system 20 includes a task identifier and a storage address, with the task identifier corresponding one-to-one with the training task. Furthermore, the metadata retrieval request includes the task identifier corresponding to the first training task. Based on this task identifier, the metadata storage system 20 can query the first metadata corresponding to the first training task and then extract the first storage address from the first metadata.

[0074] After the metadata storage system 20 retrieves the first metadata, it can determine the first storage address within it and then send the first storage address to the data downloader 10 so that the data downloader 10 can obtain the first checkpoint file from the remote storage system 40.

[0075] In this process, while sending the first storage address to the data downloader 10, the metadata storage system 20 can also send other information from the first metadata to the data downloader 10 as needed, such as sending the entire first metadata to the data downloader 10 for its use. For example, the metadata may include the hash value of the corresponding checkpoint file, and the data downloader 10 can verify the integrity of the first checkpoint file obtained from the remote storage system 40 based on the hash value in the first metadata.

[0076] In step S104, the data downloader 10 retrieves the first checkpoint file stored at the first storage address from the remote storage system 40 based on the first storage address.

[0077] In this embodiment, the remote storage system 40 can specifically store the information corresponding to each checkpoint in the form of files, and the corresponding checkpoint files are recorded at their respective storage addresses. As described above, the remote storage system 40 is used to store the checkpoint files for each training task. Each checkpoint file includes information such as model parameters, optimizer status, and training status.

[0078] After obtaining the first storage address, the data downloader 10 can initiate a request to the remote storage system 40 to obtain the first checkpoint file based on the first storage address, thereby reading the checkpoint file stored in the remote storage system 40 at the first storage address, i.e., the first checkpoint file.

[0079] In step S105, the data downloader 10 stores the first checkpoint file to the local storage system 30.

[0080] In this embodiment, after the data downloader 10 obtains the first checkpoint file from the remote storage system 40, it stores the first checkpoint file in the local storage system 30 for use by the first training task.

[0081] Both the local storage system 30 and the remote storage system 40 can be in the form of a storage cluster to store the required checkpoint files. Furthermore, there can be multiple local storage systems 30. The data downloader 10 can select the local storage system 30 most suitable for the first training task, such as the local storage system 30 closest to the first training task, and store the first checkpoint file in the local storage system 30. For example, the local storage system 30 can be the storage system within an AI training node.

[0082] In step S106, the local storage system 30 returns the first checkpoint file to the first training task.

[0083] In this embodiment, after the local storage system 30 obtains the first checkpoint file from the remote storage system 40, it can provide checkpoint file service to the first training task, that is, provide the first checkpoint file to the first training task so that the first training task can restore the model to be trained to the state corresponding to the first checkpoint file. For example, the parameters of the model corresponding to the first training task can be modified to the relevant parameters in the first checkpoint file, and the model training can continue.

[0084] The model checkpoint file management system provided in this embodiment uses a data downloader 10 to obtain the storage address of the checkpoint files required for training tasks from the metadata storage system 20. This allows the checkpoint files in the remote storage system 40 to be stored in the local storage system 30 for use by training tasks. The management system can combine the local storage system 30 and the remote storage system 40 to store massive amounts of checkpoint files and optimize storage performance. Based on the metadata of the checkpoint files, each checkpoint file can be effectively managed. When there are a large number of training tasks and a large number of checkpoint files, the data downloader 10 can also quickly obtain the required checkpoint files based on the metadata and provide them to each training task to resume model training, minimize training downtime, and ensure that the model training tasks can run reliably.

[0085] This embodiment provides another management system for model checkpoint files. Figure 2 A schematic diagram of the management system is shown, as follows: Figure 2 As shown, the management system includes: a data downloader 10, a metadata storage system 20, a local storage system 30, a remote storage system 40, and a data uploader 50.

[0086] The working principles of the data downloader 10, metadata storage system 20, local storage system 30, and remote storage system 40 can be found in [link to relevant documentation]. Figure 1 The illustrated embodiment is described in detail below. Furthermore, this management system can also support real-time storage of checkpoint files for training tasks.

[0087] In this embodiment, the local storage system 30 is also used to obtain the second checkpoint file to be stored sent by the second training task, and send the second checkpoint file to the data uploader 50.

[0088] The data uploader 50 is used to upload the second checkpoint file to the second storage address of the remote storage system 40; generate second metadata containing the second storage address, and store the second metadata in the metadata storage system 20.

[0089] Specifically, such as Figure 2 As shown, the working process of the management system may include the following steps S201 to S211.

[0090] In step S201, the data downloader 10 receives a request to obtain the checkpoint file for the first training task.

[0091] In step S202, the data downloader 10 sends a metadata retrieval request to the metadata storage system 20. This metadata retrieval request is used to obtain the storage address of the first checkpoint file.

[0092] In step S203, the metadata storage system 20 queries the first metadata corresponding to the first checkpoint file according to the metadata acquisition request, and sends the first storage address in the first metadata to the data downloader 10.

[0093] In some optional implementations, the metadata acquisition request sent by the data downloader 10 may specifically include: the task identifier and version information corresponding to the first training task.

[0094] Each training task corresponds to a unique task identifier, such as a task ID. The checkpoint file retrieval request initiated based on the first training task may include the task identifier corresponding to the first training task, enabling the data downloader 10 to generate a metadata retrieval request containing the task identifier corresponding to the first training task.

[0095] Furthermore, each training task also has version information, which may include data version, code version, etc. Generally, a model to be trained can correspond to multiple versions. The checkpoint file retrieval request can also include the version information corresponding to the first training task, enabling the data downloader 10 to generate a metadata retrieval request containing the task identifier and version information corresponding to the first training task.

[0096] In this embodiment, the metadata stored in the metadata storage system 20 includes at least the task identifier, version information, and storage address corresponding to the checkpoint file. The metadata stored in the metadata storage system 20 is uploaded by the data uploader 50. The data uploader 50 can determine the task identifier, version information, and storage address of each checkpoint file, thereby generating the metadata of the checkpoint file and storing it in the metadata storage system 20.

[0097] In step S203 above, the process of the metadata storage system 20 querying the first metadata corresponding to the first checkpoint file according to the metadata acquisition request includes the following steps A1 to A2.

[0098] Step A1: Query at least one candidate metadata in the metadata storage system 20 that has the same task identifier and the same version information as the first training task.

[0099] Step A2: Select the first metadata from at least one candidate metadata; or, send at least one candidate metadata to the data downloader 10 to instruct the data downloader 10 to select the first metadata from at least one candidate metadata.

[0100] In this embodiment, after the metadata storage system 20 obtains the task identifier and version information corresponding to the first training task, it can match the task identifier and version information corresponding to the first training task with the task identifiers and version information in various metadata stored locally to determine at least one metadata that has the same task identifier and version information as the first training task, i.e., candidate metadata. Generally, the metadata storage system 20 records the metadata of multiple checkpoint files corresponding to a certain training task, and correspondingly, the number of candidate metadata determined is also multiple.

[0101] If the metadata storage system 20 does not contain metadata that matches the task identifier and version information corresponding to the first training task, it can return a matching failure message to the data downloader 10.

[0102] After filtering out candidate metadata, the metadata storage system 20 can select the candidate metadata required for the first training task and use it as the first metadata; alternatively, the metadata storage system 20 can send each candidate metadata to the data downloader 10, for example, sending a list of metadata containing these candidate metadata to the data downloader 10, and the data downloader 10 can select the first metadata from these candidate metadata. It can be understood that the first metadata is one of the candidate metadata.

[0103] Optionally, the metadata stored in the metadata storage system 20 may also include: time information and / or evaluation metrics.

[0104] The time information could be, for example, the timestamp of the checkpoint file, or information indicating the age of the checkpoint file, such as whether it is the most recent checkpoint file. Evaluation metrics are used to represent the performance of the model corresponding to the checkpoint file; a higher evaluation metric indicates higher model performance, and the more likely the model parameters and other information recorded in the corresponding checkpoint file are optimal.

[0105] As shown in step A2 above, the first metadata can be determined by the metadata storage system 20 or the data downloader 10, and the process of “selecting the first metadata from at least one candidate metadata” in step A2 above may include steps A21 to A22.

[0106] Step A21: Based on the time information and / or evaluation metrics of each candidate metadata, determine the latest candidate metadata or the candidate metadata with the best evaluation metrics.

[0107] Step A22: Select the latest candidate metadata or the candidate metadata with the best evaluation metric as the first metadata.

[0108] In this embodiment, the latest candidate metadata can be determined based on the time information of each candidate metadata, such as the candidate metadata with the largest timestamp or the candidate metadata with the "latest" label, and the latest candidate metadata can be used as the first metadata.

[0109] Alternatively, the candidate metadata with the best evaluation index can be determined based on the evaluation index of each candidate metadata, and then the candidate metadata with the best evaluation index can be used as the first metadata.

[0110] In this embodiment, version information is set for the training task, and corresponding checkpoint files are recorded, which is beneficial for tracking the progress of different versions and comparing performance. Setting version information in the metadata enables tracking multiple versions of checkpoint files, and querying appropriate metadata based on the version information of the training task can provide the necessary checkpoint files to the training task, avoiding the selection of incorrect checkpoint files. Based on time information and / or evaluation metrics, the latest or the first metadata with the best evaluation metrics can be selected, ensuring model performance when the training task continues to train and avoiding waste of computing resources.

[0111] In step S204, the data downloader 10 retrieves the first checkpoint file stored at the first storage address from the remote storage system 40. The remote storage system 40 stores checkpoint files for each training task.

[0112] In step S205, the data downloader 10 stores the first checkpoint file to the local storage system 30.

[0113] In step S206, the data downloader 10 sends a notification message to the first training task indicating that the first checkpoint file has been prepared.

[0114] In this embodiment, the data downloader 10 can use parallel downloading to accelerate the transmission of large checkpoint files. Furthermore, after obtaining the first checkpoint file from the remote storage system 40, the data downloader 10 can perform file verification based on the hash value of the first checkpoint file, etc., to verify the integrity of the checkpoint file and prevent file corruption.

[0115] Furthermore, after the first checkpoint file is stored in the local storage system 30, if the first checkpoint file is available for use, for example, if the local storage system 30 returns a message to the data downloader 10 that the first checkpoint file has been stored, the data downloader 10 can send a notification message to the first training task indicating that the first checkpoint file has been prepared, so that the first training task can read the first checkpoint file from the local storage system 30.

[0116] Step S207: The first training task retrieves the first checkpoint file from the local storage system 30.

[0117] In this embodiment, the notification message may include the storage address of the first checkpoint file in the local storage system 30, so that the first training task can read the local storage system 30 based on the storage address to obtain the first checkpoint file from the local storage system 30.

[0118] In this embodiment, the training task actively reads the required checkpoint files, so that the local storage system 30 only needs to provide data read and write functions, and the division of labor among the nodes is clear.

[0119] In some alternative implementations, see [link to implementation details]. Figure 3 As shown, the local storage system 30 includes: a main memory cache 31 and multiple storage partitions 32.

[0120] The main memory cache 31 is used to cache the data to be processed obtained by the local storage system 30 and to divide the data to be processed into multiple partitions.

[0121] Storage partition 32 is used to retrieve the corresponding partition data from the main memory cache 31.

[0122] In this embodiment, in the scenario of large-scale training, the local storage system 30 needs to store checkpoint files with a large amount of data. In order to ensure the reliability of data reading and writing, the local storage system 30 uses the main memory cache 31 to cache data and uses a partitioning method to partition the data, storing multiple copies of data into the corresponding storage partitions 32 respectively, thereby realizing partition-level data persistence.

[0123] like Figure 3 As shown, for the data to be processed acquired by the local storage system 30, the main memory cache 31 caches the data to be processed. This data to be processed includes a first checkpoint file provided by the data downloader 10. Additionally, the data to be processed may also include checkpoint files generated during training tasks (i.e., subsequent second checkpoint files), thereby providing fast, temporary storage services for each checkpoint file during training.

[0124] Furthermore, the main memory cache 31 divides the data to be processed into multiple parts, each part being called a partition. For example, the main memory cache 31 can use horizontal partitioning to divide the data. Generally, the number of partitions is the same as the number of storage partitions 32, so that each partition's data is stored in its corresponding storage partition 32.

[0125] like Figure 3 As shown, the local storage system 30 has N storage partitions 32. The main memory cache 31 can divide the data to be processed into N partition data, and the partition data corresponds one-to-one with the storage partition 32.

[0126] Optionally, such as Figure 3 As shown, the storage partition 32 includes: a controller 321, a plurality of first storage devices 322 and a plurality of second storage devices 323; the first storage devices 322 and the second storage devices 323 correspond one-to-one, that is, the number of first storage devices 322 and the number of second storage devices 323 are the same. Figure 3 The example shown is that storage partition 32 contains two first storage devices 322.

[0127] The controller 321 is used to divide the partition data corresponding to the storage partition 32 into multiple data blocks and write the multiple data blocks in parallel to multiple first storage devices 322.

[0128] The first storage device 322 is used to store the corresponding data blocks.

[0129] The second storage device 323 is used to back up the data blocks in the corresponding first storage device 322 and to provide data read and write capabilities to the controller 321 when the first storage device 322 fails.

[0130] In this embodiment, the storage partition 32 includes multiple storage devices and is managed by the controller 321. Each storage partition 32 can be configured with its own controller 321, or multiple or all storage partitions 32 can share the same controller 321, that is, all storage devices are managed by a unified controller 321.

[0131] For partition data that needs to be stored in a certain storage partition 32, it can be divided into multiple data blocks, and each data block can be written to the corresponding first storage device 322 in parallel. The number of data blocks can be the same as the number of first storage devices 322, meaning there is a one-to-one correspondence between data blocks and first storage devices 322; alternatively, the number of data blocks can be different from the number of first storage devices 322, for example, the number of data blocks can be greater than the number of first storage devices 322, and the data blocks are written to the corresponding first storage devices 322 sequentially.

[0132] For example, if the number of first storage devices 322 is m, then each data block can be assigned a corresponding index i (i = 0, 1, 2, ...). For a data block with index i satisfying i mod m = k, it is stored in the first storage device 322 with index k; where mod is the modulo function, k represents the index of the first storage device 322, and k = 0, 1, 2, ..., m-1. For example, firstly, data blocks with indices from 0 to m-1 are written in parallel to m first storage devices 322, then data blocks with indices from m to 2m-1 are written in parallel to m first storage devices 322, and so on, until all data blocks are written.

[0133] When data needs to be read from storage partition 32, controller 321 can also read data from multiple first storage devices 322 in parallel, realizing parallel read and write capabilities, which can provide faster data transfer speeds and higher system performance.

[0134] Storage partition 32 also includes a second storage device 323 corresponding to the first storage device 322. The second storage device 323 can back up the data blocks in the corresponding first storage device 322, providing data redundancy. If the first storage device 322 fails, the corresponding second storage device 323 can provide temporary data read and write capabilities to the controller 321, avoiding service terminal failure and reminding the user to perform fault repair.

[0135] In this embodiment, data caching via main memory cache 31 improves data read / write response efficiency. Dividing the data to be processed into multiple parts and storing them in corresponding storage partitions 32 ensures reliable data persistence even with large data volumes, achieving distributed storage. Furthermore, data block partitioning enables parallel read / write operations on multiple first storage devices 322, improving data transfer speed. Utilizing multiple second storage devices 323 provides data redundancy, ensuring the reliability of checkpoint file data.

[0136] In step S208, the second training task sends the second checkpoint file to be stored to the local storage system 30.

[0137] When the training task is executed again, a corresponding checkpoint file can be generated at regular intervals to record the latest model parameters and other information. In this embodiment, the training task corresponding to the training period is called the second training task, and the checkpoint file generated by it is called the second checkpoint file. Furthermore, the second training task can send the generated second checkpoint file to the local storage system 30 in real time to achieve fast storage of the checkpoint file.

[0138] In step S209, the local storage system 30 sends the second checkpoint file to the data uploader 50.

[0139] In this embodiment, when a new second checkpoint file exists, the local storage system 30 can immediately or periodically send the second checkpoint file to the data uploader 50. Alternatively, the local storage system 30 can send a notification message to the data uploader 50 to inform it that a new checkpoint file exists and needs to be uploaded; or the data uploader 50 can periodically retrieve new checkpoint files from the local storage system 30, including the aforementioned second checkpoint file.

[0140] For example, the second checkpoint file can be stored in the first storage device 322 and backed up by the second storage device 323. The data uploader 50 periodically retrieves new checkpoint files, including the second checkpoint file, from the local storage system 30; furthermore, if the second checkpoint file in the local storage system 30 has already been sent to the data uploader 50, the second checkpoint file in the local storage system 30 can be deleted, or the second checkpoint file in the local storage system 30 can be marked as deletable and deleted when the storage space of the local storage system 30 is insufficient.

[0141] In step S210, the data uploader 50 uploads the second checkpoint file to the second storage address of the remote storage system 40.

[0142] In this embodiment, after the data uploader 50 obtains the second checkpoint file, it can determine the storage address in the remote storage system 40 used to store the second checkpoint file, namely the second storage address, and upload the second checkpoint file to the second storage address of the remote storage system 40 to realize the cloud storage of the checkpoint file.

[0143] In some optional implementations, the process of the data uploader 50 uploading the second checkpoint file to the second storage address of the remote storage system 40 in step S210 may specifically include steps B1 to B2.

[0144] Step B1: If the second checkpoint file is generated based on the second baseline checkpoint file, upload the checkpoint file increment information between the second checkpoint file and the second baseline checkpoint file to the second storage address of the remote storage system 40.

[0145] Step B2: Determine the second baseline storage address of the second baseline checkpoint file, and add both the second storage address and the second baseline storage address to the second metadata of the second checkpoint file.

[0146] In this embodiment, since checkpoint files need to record a large amount of data, especially for checkpoint files of large-scale AI models, incremental processing is performed on checkpoint files to improve the resource utilization of the storage system. Specifically, during the training of a model, training is generally based on a baseline checkpoint file. Subsequent checkpoint files are all newly generated based on the baseline checkpoint file. Since most of the information in the new checkpoint file is likely the same as that in the baseline checkpoint file, only the difference information between the new checkpoint file and the baseline checkpoint file can be recorded, greatly reducing the storage space occupied by the new checkpoint file. When it is necessary to restore training based on the new checkpoint file later, the state corresponding to the new checkpoint file can be restored based on the recorded difference information and the baseline checkpoint file.

[0147] Specifically, if the second checkpoint file is generated based on a baseline checkpoint file, incremental processing can be performed. For ease of description, the baseline checkpoint file corresponding to the second checkpoint file is referred to as the second baseline checkpoint file. The differences between the second checkpoint file and the second baseline checkpoint file can be determined; this difference information is the checkpoint file increment information between the two. This checkpoint file increment information is used as the checkpoint file to be stored and uploaded to the second storage address of the remote storage system 40. That is, this second storage address only needs to store the checkpoint file increment information; this checkpoint file increment information is also a type of checkpoint file, hereinafter referred to as the incremental checkpoint file.

[0148] Furthermore, to ensure that the training task can use the second checkpoint file correctly, the data uploader 50 also associates the second metadata corresponding to the second checkpoint file with the second baseline storage address of the second baseline checkpoint file. Specifically, both the second storage address and the second baseline storage address are added to the second metadata of the second checkpoint file, so that when the second checkpoint file is needed later, the corresponding checkpoint file incremental information can be obtained based on the second storage address, and the second baseline checkpoint file can be obtained based on the second baseline storage address. Finally, by combining the second baseline checkpoint file and its incremental information, the complete second checkpoint file can be obtained.

[0149] Optionally, the first checkpoint file used for recovery training may also be obtained based on incremental processing. Accordingly, the process of the metadata storage system 20 sending the first storage address in the first metadata to the data downloader 10 may include steps C1 to C2.

[0150] Step C1: If the first checkpoint file is an incremental checkpoint file, determine the first base storage address of the first base checkpoint file corresponding to the first checkpoint file; the incremental checkpoint file is used to record the incremental information of the checkpoint file that has changed based on the base checkpoint file corresponding to the incremental checkpoint file.

[0151] Step C2: Send the first storage address and the first base storage address from the first metadata to the data downloader 10.

[0152] Furthermore, the data downloader 10 is specifically used to: obtain the first checkpoint file stored at the first storage address and the first benchmark checkpoint file stored at the first benchmark storage address from the remote storage system 40 according to the first storage address and the first benchmark storage address, and store the first checkpoint file and the first benchmark checkpoint file to the local storage system 30.

[0153] The local storage system 30 is specifically used to: return the first checkpoint file and the first baseline checkpoint file to the first training task.

[0154] In this embodiment, similar to steps B1 to B2 above, if, when recording the first checkpoint file, the incremental information of the changed checkpoint file is determined based on the corresponding first baseline checkpoint file, and incremental processing is then implemented, in other words, the first checkpoint file only includes the incremental information of the checkpoint file that has changed compared to the first baseline checkpoint file, i.e., the first checkpoint file is an incremental checkpoint file, then the metadata storage system 20 needs to determine not only the first storage address of the first checkpoint file, but also the first baseline storage address of the first baseline checkpoint file, and simultaneously send the first storage address and the first baseline storage address to the data downloader 10. For example, the first metadata of the first checkpoint file includes the first storage address and the first baseline storage address.

[0155] After obtaining the first storage address and the first reference storage address, the data downloader 10 needs to retrieve the first checkpoint file stored at the first storage address from the remote storage system 40, and also the first reference checkpoint file stored at the first reference storage address from the remote storage system 40. Then, it stores the first checkpoint file and the first reference checkpoint file in the local storage system 30. The data downloader 10 can restore the first checkpoint file and the first reference checkpoint file to a complete original checkpoint file, and store this original checkpoint file in the local storage system 30. It can be understood that the difference information between this original checkpoint file and the first reference checkpoint file constitutes the first checkpoint file.

[0156] Once the local storage system 30 obtains the first checkpoint file and the first baseline checkpoint file, it can be read by the first training task. For example, the local storage system 30 can directly store the complete original checkpoint file. After the first training task reads the original checkpoint file, it can determine the model parameters and other information based on the original checkpoint file and resume model training.

[0157] It is understood that if the first training task and the second training task belong to the same training task, for example, if their task identifiers are the same, then after saving the second checkpoint file, this second checkpoint file can be provided to the first training task as the first checkpoint file; that is, the second checkpoint file and the first checkpoint file can be the same checkpoint file. Of course, the second checkpoint file and the first checkpoint file can also be different checkpoint files, and this embodiment does not limit this.

[0158] In step S211, the data uploader 50 generates second metadata containing the second storage address and stores the second metadata in the metadata storage system 20.

[0159] In this embodiment, in addition to storing the second checkpoint file to the remote storage system 40, the data uploader 50 also needs to synchronously store the corresponding second metadata to the metadata storage system 20.

[0160] Specifically, the data uploader 50 can obtain some metadata from the second checkpoint file, such as the task identifier and version information corresponding to the second checkpoint file. In addition, the data uploader 50 can also determine additional information such as the second storage address, thereby generating second metadata containing the second storage address and information such as the task identifier and version information.

[0161] In some alternative implementations, the second metadata generated by the data uploader 50 may also include a corresponding hash value, and the metadata stored in the metadata storage system 20 may also include a hash value to ensure data integrity.

[0162] In this embodiment, the remote storage system 40 is further configured to: calculate the pending hash value of the second checkpoint file stored in the remote storage system 40, and obtain the standard hash value in the second metadata; determine whether the pending hash value is consistent with the standard hash value; and determine that the second checkpoint file has been stored if the pending hash value is consistent with the standard hash value.

[0163] Specifically, the data uploader 50 generates second metadata for the second checkpoint file, which includes the hash value corresponding to the second checkpoint file, i.e., the standard hash value; in addition to storing the second checkpoint file, the remote storage system 40 also obtains the standard hash value from the second metadata from the data uploader 50 or the metadata storage system 20.

[0164] After the remote storage system 40 stores the second checkpoint file, it calculates the hash value of the second checkpoint file, i.e., the pending hash value. This pending hash value is the hash value of the second checkpoint file stored in the remote storage system 40. Normally, this pending hash value is the same as the standard hash value. However, if an error occurs during data transmission in the process of storing the second checkpoint file, the stored second checkpoint file will be different from the original second checkpoint file, i.e., the pending hash value will be different from the standard hash value. In this case, the data uploader 50 needs to re-upload the second checkpoint file to the remote storage system 40, or report an error.

[0165] The remote storage system 40 can also adopt a similar approach. Figure 3The storage architecture of the local storage system 30 shown is based on a load balancing strategy to store each checkpoint file to the corresponding storage node, and data redundancy backup is performed within the storage node to improve data reliability.

[0166] In addition, the first metadata corresponding to the first checkpoint file may also include the hash value of the first checkpoint file. Accordingly, when the data downloader 10 obtains the first checkpoint file, or when the local storage system 30 stores the first checkpoint file, it may also perform verification based on the hash value to ensure the integrity of the checkpoint file and avoid the problem that the checkpoint file will be corrupted due to excessive data volume or interruption during the download / upload process, thus making it unusable.

[0167] In this embodiment, specific information in the metadata can be set as needed. As mentioned above, the metadata may include: task identifier, version information, storage address, hash value, etc. In addition, the metadata may also include team ID, personnel ID related to the training task, as well as node ID or number of nodes involved in the model, and resource information in each node (such as GPU information in the node), so that when the model training is resumed based on the checkpoint file, the model can be trained again based on the same nodes.

[0168] Optionally, the data uploader 50 is also used to: dynamically determine the upload frequency of checkpoint files based on the upload bandwidth and concurrency limits of the remote storage system 40; and upload checkpoint files to the remote storage system 40 according to the upload frequency.

[0169] In this embodiment, the data uploader 50 can determine the upload bandwidth limit between itself and the remote storage system 40 in real time, and determine the concurrency limit. The concurrency limit is used to limit the concurrency between the data uploader 50 and the remote storage system 40. It can be understood that the higher the upload bandwidth and the higher the concurrency, the higher the upload frequency can be set.

[0170] In addition, the data uploader 50 can adjust the upload frequency based on the model's training speed.

[0171] The model checkpoint file management system provided in this embodiment uses a data downloader 10 and a data uploader 50 to download and upload checkpoint files. Parallel downloading and uploading methods improve the speed and efficiency of checkpoint file transmission. The local storage system 30 stores checkpoint files using data segmentation and data block partitioning, and backs up checkpoint files, improving read / write efficiency while ensuring data reliability. Incremental processing of checkpoint files reduces the storage space required for incremental checkpoint files, improving the utilization of the remote storage system 40's storage space. Setting the upload frequency of the data uploader 50 on demand enables on-demand scheduling of data uploads.

[0172] Although embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations should all be covered within the protection scope of the present invention.

Claims

1. A management system for model checkpoint files, characterized in that, The management system includes: a data downloader (10), a metadata storage system (20), a local storage system (30), and a remote storage system (40); The data downloader (10) is used to receive a checkpoint file acquisition request, which is used to acquire the first checkpoint file required for the first training task; and to send a metadata acquisition request to the metadata storage system (20) to acquire the storage address of the first checkpoint file. The metadata storage system (20) is used to query the first metadata corresponding to the first checkpoint file according to the metadata acquisition request, and send the first storage address in the first metadata to the data downloader (10); The data downloader (10) is used to obtain the first checkpoint file stored at the first storage address from the remote storage system (40) according to the first storage address, and store the first checkpoint file to the local storage system (30); the remote storage system (40) is used to store the checkpoint files of each training task; The local storage system (30) is used to return the first checkpoint file to the first training task, instructing the first training task to restore to the state corresponding to the first checkpoint file.

2. The management system according to claim 1, characterized in that, The metadata acquisition request includes the task identifier and version information corresponding to the first training task; the metadata stored in the metadata storage system (20) includes: task identifier, version information and storage address; The process by which the metadata storage system (20) queries the first metadata corresponding to the first checkpoint file according to the metadata acquisition request includes: Query at least one candidate metadata in the metadata storage system (20) that has the same task identifier and the same version information as the first training task; Select a first metadata from at least one of the candidate metadata; or send at least one of the candidate metadata to the data downloader (10) to instruct the data downloader (10) to select a first metadata from at least one of the candidate metadata.

3. The management system according to claim 2, characterized in that, The metadata stored in the metadata storage system (20) also includes: time information and / or evaluation metrics; The process by which the metadata storage system (20) or the data downloader (10) selects the first metadata from at least one of the candidate metadata includes: Based on the time information and / or evaluation metrics of each candidate metadata, determine the latest candidate metadata or the candidate metadata with the best evaluation metrics; The latest candidate metadata or the candidate metadata with the best evaluation metrics is used as the first metadata.

4. The management system according to claim 1, characterized in that, The process by which the metadata storage system (20) sends the first storage address in the first metadata to the data downloader (10) includes: If the first checkpoint file is an incremental checkpoint file, determine the first base storage address of the first base checkpoint file corresponding to the first checkpoint file; the incremental checkpoint file is used to record the incremental checkpoint file information that has changed based on the base checkpoint file corresponding to the incremental checkpoint file. Send the first storage address and the first base storage address in the first metadata to the data downloader (10); The data downloader (10) is used to: obtain the first checkpoint file stored at the first storage address and the first reference checkpoint file stored at the first reference storage address from the remote storage system (40) according to the first storage address and the first reference storage address, and store the first checkpoint file and the first reference checkpoint file to the local storage system (30); The local storage system (30) is used to: return the first checkpoint file and the first benchmark checkpoint file to the first training task.

5. The management system according to claim 1, characterized in that, The local storage system (30) includes: a main memory cache (31) and multiple storage partitions (32); The main memory cache (31) is used to cache the data to be processed obtained by the local storage system (30) and to divide the data to be processed into multiple partition data. The storage partition (32) is used to obtain the corresponding partition data from the main memory cache (31).

6. The management system according to claim 5, characterized in that, The storage partition (32) includes: a controller (321), a plurality of first storage devices (322) and a plurality of second storage devices (323); the first storage devices (322) correspond one-to-one with the second storage devices (323); The controller (321) is used to divide the partition data corresponding to the storage partition (32) into multiple data blocks, and write the multiple data blocks in parallel to multiple first storage devices (322); The first storage device (322) is used to store the corresponding data blocks; The second storage device (323) is used to back up the corresponding data blocks in the first storage device (322) and to provide data read and write capabilities to the controller (321) when the first storage device (322) fails.

7. The management system according to any one of claims 1 to 6, characterized in that, The management system includes: a data uploader (50); The local storage system (30) is also used to obtain the second checkpoint file to be stored sent by the second training task, and send the second checkpoint file to the data uploader (50); The data uploader (50) is used to upload the second checkpoint file to the second storage address of the remote storage system (40); generate second metadata containing the second storage address, and store the second metadata in the metadata storage system (20).

8. The management system according to claim 7, characterized in that, The remote storage system (40) is also used for: Calculate the pending hash value of the second checkpoint file stored in the remote storage system (40), and obtain the standard hash value in the second metadata; Determine whether the undetermined hash value is consistent with the standard hash value; If the pending hash value matches the standard hash value, the second checkpoint file is determined to be stored successfully.

9. The management system according to claim 7, characterized in that, The process by which the data uploader (50) uploads the second checkpoint file to the second storage address of the remote storage system (40) includes: If the second checkpoint file is generated based on the second baseline checkpoint file, the checkpoint file increment information between the second checkpoint file and the second baseline checkpoint file is uploaded to the second storage address of the remote storage system (40); Determine the second baseline storage address of the second baseline checkpoint file, and add both the second storage address and the second baseline storage address to the second metadata of the second checkpoint file.

10. The management system according to claim 7, characterized in that, The data uploader (50) is also used for: The upload frequency of checkpoint files is dynamically determined based on the upload bandwidth and concurrency limits of the remote storage system (40). Upload checkpoint files to the remote storage system (40) at the specified upload frequency.

Images