Model checkpoint file management system, method, equipment and storage medium
By constructing a hierarchical storage structure and a unified file access path in the computing cluster, and asynchronously managing checkpoint files, the problem of training blockage in large model training is solved, achieving efficient checkpoint management and reducing training overhead and time.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHEJIANG LAB
- Filing Date
- 2026-04-21
- Publication Date
- 2026-07-17
AI Technical Summary
Existing synchronous checkpointing methods result in high runtime overhead and block the training process during large model training, and traditional methods require modifications to the code and configuration of the model training framework.
By deploying a distributed cache file system in the computing cluster, a hierarchical storage structure of memory layer and persistent storage layer is constructed. The efficiency of the memory layer is used to asynchronously migrate checkpoint files, and transparent checkpoint management is achieved through a unified file access path, thereby reducing training blocking time.
It improves the throughput of checkpoint saving and loading processes, reduces training overhead, shortens the end-to-end training time of large models, and does not require modification of the training framework and model code.
Smart Images
Figure CN122064535B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, and in particular to a model checkpoint file management system, method, device and storage medium. Background Technology
[0002] With the widespread application of large language models across various fields, their training process typically consumes significant resources and time due to the rapid growth in training data and parameter size. During training sessions that can last for weeks or even months, various failures and unsatisfactory training trajectories can occur. Checkpointing mechanisms, by saving intermediate results during training, can be used to restart training or roll back to a previous training state when faced with training failures caused by the aforementioned issues. Therefore, an efficient checkpointing mechanism is crucial for ensuring the successful training of large models.
[0003] Large model checkpoints primarily involve model parameters and optimizer states. Large models themselves have a massive number of parameters, and their adaptive learning rate optimizers (such as Adam) require storing additional state information, causing the size of the optimizer states to expand dramatically. Therefore, the overall size of the checkpoints becomes extremely large. Currently, the default synchronous checkpointing method blocks training until the model state is written to persistent storage, which results in high runtime overhead. Summary of the Invention
[0004] To overcome the problems existing in related technologies, this specification provides a model checkpoint file management system, method, device and storage medium.
[0005] Firstly, a model checkpoint file management system, the system comprising: Storage resources include local storage clusters, which consist of distributed cache file systems deployed on each computing node in each computing cluster, including a hierarchical storage structure formed by a memory layer and a persistent storage layer. The local storage cluster forms a unified file access path through mounting, which is accessed by the model training framework. The checkpoint management module, deployed between the training framework and the storage resources, is used to perform hierarchical asynchronous management of checkpoint files based on the unified file access path when receiving checkpoint operation requests from model training tasks. When the checkpoint operation request is a checkpoint save request, the checkpoint file is written to the unified file access path so that the checkpoint file is preferentially stored in the memory layer through the distributed cache file system. During the execution of the training task, the checkpoint file in the memory layer is asynchronously migrated to the persistent storage layer.
[0006] According to the model checkpoint file management system provided in this application, the storage cluster further includes a remote storage cluster deployed with a distributed parallel file system for storing the checkpoint files. If the checkpoint management module is used to perform hierarchical asynchronous management of the checkpoint files based on the unified file access path, the checkpoint management module is configured to: During the execution of the training task, the checkpoint files stored in the persistent storage layer are asynchronously migrated to the remote storage cluster.
[0007] According to the model checkpoint file management system provided in this application, the remote storage cluster is deployed by mounting it to each computing node of the computing cluster via client mounting, and the mount point of the local storage cluster is soft-linked to the mount directory of the remote storage cluster in the form of a directory, forming a unified file access path accessible to the model training framework.
[0008] According to the model checkpoint file management system provided in this application, if the checkpoint management module is further configured to asynchronously migrate checkpoint files stored in the persistent storage layer to the remote storage cluster, the checkpoint management module is configured to: Real-time monitoring of the storage status of the persistent storage layer; When the storage state meets the preset conditions, the checkpoint file to be migrated is determined; The checkpoint file to be migrated is migrated from the persistent storage layer to the remote storage cluster.
[0009] According to the model checkpoint file management system provided in this application, when the checkpoint operation request is a checkpoint loading request, if the checkpoint management module is also used to perform hierarchical asynchronous management of checkpoint files based on the unified file access path, the checkpoint management module is configured to: When a follow-up training task is executed based on the current training task, the required checkpoint files are preloaded to the corresponding storage levels in sequence according to the priority order of the remote storage cluster, the persistent storage layer of the local storage cluster, and the memory layer of the local storage cluster, based on the unified file access path. The checkpoint files are then read from the memory layer of the local storage cluster to resume training.
[0010] Secondly, a model checkpoint file management method is provided, applied to a model checkpoint file management system as described in any of the first aspects, the method comprising: Receive checkpoint operation requests from the model training task; Based on the unified file access path, hierarchical asynchronous management is performed on checkpoint files; When the checkpoint operation request is a checkpoint save request, the checkpoint file is written to the unified file access path so that the checkpoint file is preferentially stored in the memory layer through the distributed cache file system. During the training task execution, the checkpoint files in the memory layer are asynchronously migrated to the persistent storage layer.
[0011] According to the model checkpoint file management method provided in this application, the storage cluster further includes a remote storage cluster deployed with a distributed parallel file system for storing the checkpoint files. The step of performing hierarchical asynchronous management of the checkpoint files based on the unified file access path includes: During the execution of the training task, the checkpoint files stored in the persistent storage layer are asynchronously migrated to the remote storage cluster.
[0012] According to the model checkpoint file management method provided in this application, the asynchronous migration of checkpoint files stored in the persistent storage layer to the remote storage cluster includes: Real-time monitoring of the storage status of the persistent storage layer; When the storage state meets the preset conditions, the checkpoint file to be migrated is determined; The checkpoint file to be migrated is migrated from the persistent storage layer to the remote storage cluster.
[0013] According to the model checkpoint file management method provided in this application, determining the checkpoint file to be migrated includes: Based on the order of checkpoint file generation time, the checkpoint file with the earliest generation time is determined as the checkpoint file to be migrated.
[0014] According to the model checkpoint file management method provided in this application, when the checkpoint operation request is a checkpoint loading request, the hierarchical asynchronous management of the checkpoint file based on the unified file access path includes: When a follow-up training task is executed based on the current training task, the required checkpoint files are preloaded to the corresponding storage levels in parallel according to the priority order of the remote storage cluster, the persistent storage layer of the local storage cluster, and the memory layer of the local storage cluster, based on the unified file access path. The required checkpoint files are read from the memory layer of the local storage cluster to resume training.
[0015] Thirdly, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, it implements the model checkpoint file management method as described in the second aspect.
[0016] Fourthly, a computer-readable storage medium is provided, wherein a model checkpoint file management program is stored on the computer-readable storage medium, and the model checkpoint file management program, when executed, implements any of the model checkpoint file management methods described in the second aspect above.
[0017] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the model checkpoint file management method as described in any of the second aspects above.
[0018] The model checkpoint file management system, method, device, and storage medium described in this specification have the following beneficial effects: On the one hand, local storage clusters are deployed on each computing node of the computing cluster, making full use of the local storage resources such as memory and solid-state drives of each node in the computing cluster to construct a distributed cache file system with a hierarchical storage structure including a memory layer and a persistent storage layer. From the perspective of the file system, this provides efficient hierarchical storage management for checkpoints in model training tasks. Specifically, when a model training task issues a checkpoint save request, the checkpoint file is preferentially stored in the memory layer through the distributed cache file system. During the execution of the training task, the checkpoint file stored in the memory layer is asynchronously migrated to the persistent storage layer. This process of reading and writing from memory improves the throughput of the checkpoint save and load process, effectively reducing training blocking time, lowering the overhead of checkpoint operations during large model training, and shortening the end-to-end training runtime of large models.
[0019] On the other hand, from a file system perspective, using the mount point of the local storage cluster as a unified file access path accessible to the model training framework allows the training model to access data distributed across different physical storage media through a single file path. Users do not need to modify any model or model training framework code or configuration, achieving transparency at the training framework level. For the model training framework, the generated checkpoint files are simply written to this unified file access path. The backend uses a distributed cache file system to implement hierarchical asynchronous storage management of the checkpoint files, thus achieving checkpoint storage optimization that is transparent to the training framework at the file system level.
[0020] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this specification. Attached Figure Description
[0021] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this specification and, together with the description, serve to explain the principles of this specification.
[0022] Figure 1This is a schematic diagram of the architecture of a model checkpoint file management system illustrated in this specification according to an exemplary embodiment.
[0023] Figure 2 This is a schematic diagram illustrating the construction process of storage resources in a model checkpoint file management system provided in this specification according to an exemplary embodiment.
[0024] Figure 3 This is a flowchart illustrating a model checkpoint file management method provided in this specification according to an exemplary embodiment.
[0025] Figure 4 This is another flowchart illustrating a model checkpoint file management method provided in this specification according to an exemplary embodiment.
[0026] Figure 5 This is a schematic block diagram of a model checkpoint file management device illustrated in this specification according to an exemplary embodiment. Detailed Implementation
[0027] The technical solutions in the embodiments (or "implementations") of this application will be clearly and completely described herein with reference to the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements.
[0028] If the embodiments of this application contain terms relating to directional indications or positional relationships (such as up, down, left, right, front, back, inside, outside, top, bottom, center, vertical, horizontal, longitudinal, transverse, length, width, counterclockwise, clockwise, axial, radial, circumferential, etc.), such terms are only used to explain the relative positional relationships and movement of the components in a specific posture (as shown in the attached figures); if the specific posture changes, the directional indications or positional relationships will also change accordingly. Furthermore, the terms "first" and "second" used in the embodiments of this application are only for descriptive convenience and should not be construed as indicating or implying relative importance.
[0029] This application provides a model checkpoint file management system, method, apparatus, and storage medium. The following detailed description, in conjunction with the accompanying drawings, illustrates this application. The features described in the embodiments and implementations can be combined with each other.
[0030] To address the aforementioned technical issues, this specification provides a method for managing model checkpoint files.
[0031] This solution aims to improve the efficiency of hierarchical checkpoint storage management from a file system perspective. It fully utilizes the memory and SSD storage resources of each node in the computing cluster while maintaining transparency at the training framework level. Users do not need to modify any model or model training framework code and configuration. Reading and writing from memory increases the throughput of checkpoint saving and loading processes, reduces the overhead of checkpoint operations, and shortens the end-to-end training runtime of large models.
[0032] In this embodiment, the large model checkpoints include model parameters and optimizer state.
[0033] This application provides a first embodiment of a model checkpoint file management system.
[0034] The architecture of the model checkpoint file management system includes the computing cluster deployment structure and the storage resource structure.
[0035] On the computing cluster side, a model training framework and a checkpoint management module are deployed. The model training framework performs large language model training tasks based on the computing resources in the computing cluster.
[0036] The checkpoint management module is located between the training framework and storage resources. It includes monitoring and control components, and is used to perform hierarchical asynchronous management of checkpoint files based on the unified file access path when receiving checkpoint operation requests from model training tasks. This includes, but is not limited to, monitoring, scheduling, and migrating checkpoint files during the training process.
[0037] On the storage resource side, there is a local storage cluster, which consists of a distributed cache file system deployed on each compute node. It includes a hierarchical storage structure formed by a memory layer and a persistent storage layer. The memory layer corresponds to the memory resources of the compute nodes, and the persistent storage layer corresponds to the local persistent storage resources such as solid-state drives of the compute nodes.
[0038] Furthermore, the local storage cluster forms a unified file access path through mounting, which is accessible to the model training framework and the checkpoint management module. This includes, but is not limited to, performing operations such as saving, recording, migrating, and archiving checkpoint files through the unified file access path.
[0039] Through the aforementioned model checkpoint management system, a unified file access path is provided to the training framework with a checkpoint access interface using a standard file system path. The checkpoint storage process optimization and hierarchical storage scheduling are implemented at the file system layer, which is transparent to the training framework and requires no modification to the training framework and model code and configuration.
[0040] Based on the first embodiment described above, this application provides a second embodiment of a model checkpoint file management system. Figure 1This is a schematic diagram of the architecture of a model checkpoint file management system provided in the embodiments of this specification.
[0041] In some embodiments, considering that the storage space of the local storage cluster is limited by the local storage resources of the computing nodes, making it impossible to save all checkpoints for a long time and to support checkpoint sharing and archiving across training tasks, the model checkpoint management system also includes a remote storage cluster for archiving, storing and sharing checkpoint files.
[0042] Reference Figure 1 The architecture of the model checkpoint file management system includes the computing cluster deployment structure and the storage resource structure.
[0043] On the storage resource side, storage resources also include remote storage clusters, which are deployed with distributed parallel file systems. The integration of the remote storage clusters with the local storage clusters constitutes a unified storage resource and forms a unified file access path accessible for model training.
[0044] In this embodiment, the remote storage cluster is deployed by mounting it to each computing node of the computing cluster via client mounting, and the mount point of the local storage cluster is soft-linked to the mount directory of the remote storage cluster in the form of a directory, forming a unified file access path that is accessible to the model training framework.
[0045] Specifically, refer to Figure 2 , Figure 2 This is a schematic diagram illustrating the construction process of storage resources in a model checkpoint file management system provided in the embodiments of this specification. The construction process of storage resources in the above embodiments is as follows: Step 1.1: Deploy a distributed parallel file system on the remote storage cluster and mount it to the preset mount directory of each node in the computing cluster through client mounting.
[0046] Step 1.2: Deploy a distributed cache file system on each node in the computing cluster to form a unified storage cluster, i.e., a local storage cluster, from the local storage resources of each computing node. Local storage resources include, but are not limited to, memory and solid-state drives (SSDs).
[0047] The local storage cluster consists of two storage tiers: a memory tier and a persistent storage tier. The memory tier corresponds to the memory resources of the compute nodes, while the persistent storage tier corresponds to the local persistent storage resources of the compute nodes, such as solid-state drives.
[0048] Step 1.3: Integrate the distributed cache file system of the local storage cluster and the distributed parallel file system of the remote storage cluster together using symbolic links. The mount point of the local storage cluster serves as a directory under the remote storage cluster, thereby constructing a globally unified namespace. The unified namespace is presented externally as a single, contiguous file system path, allowing the training model to access data distributed across different physical storage media through a single file path.
[0049] Step 1.4: Set the save and load path for the checkpoints of the large model training to the unified namespace of Step 3.
[0050] Through the above process, both the training framework and the checkpoint management module control the checkpoint operations of the entire training task through the path under this unified namespace, including but not limited to performing checkpoint file saving, loading, migration and archiving operations, and finally managing the storage location of checkpoint files in a reasonable and efficient manner.
[0051] In some embodiments, the storage resources include at least one of a local storage cluster and a unified storage resource formed by the local storage cluster and a remote storage cluster.
[0052] In some embodiments, based on the system description above, when the checkpoint operation request is a checkpoint save request, the checkpoint file is written to the unified file access path so that the checkpoint file is preferentially stored in the memory layer through the distributed cache file system; and during the execution of the training task, the checkpoint file in the memory layer is asynchronously migrated to the persistent storage layer.
[0053] Specifically, refer to Figure 3 , Figure 3 This is a flowchart illustrating a model checkpoint file management method provided in the embodiments of this specification.
[0054] Step 2.1: Deploy the remote storage cluster and the local storage cluster to form a unified namespace, deploy the framework and checkpoint management module required for the training task, and start the training task.
[0055] Step 2.2: The training task executes an iterative process, generating a checkpoint save request.
[0056] Step 2.3: When a checkpoint save request is received, the control component of the checkpoint management module writes all the latest checkpoint files to the directory of the local storage cluster under the unified file access path, and the checkpoint request returns a success.
[0057] In this embodiment, the local storage cluster will preferentially store these checkpoint files in the memory layer, making full use of the high I / O speed and high read / write bandwidth of the local storage cluster's memory layer to shorten the training blocking time caused by saving checkpoints.
[0058] Step 2.4 During training, before the next checkpoint save request arrives or after the memory layer write is completed, the control component of the checkpoint management module asynchronously stores the previous checkpoint file stored in the local storage cluster memory layer to the persistent storage layer.
[0059] In this way, the memory layer ensures that it always stores the latest checkpoint file, while also ensuring that there is enough space to accommodate the next checkpoint write.
[0060] Through the above embodiments, from a file system perspective, the mount point of the local storage cluster is used as a unified file access path accessible to the model training framework. This allows the training model to access data distributed across different physical storage media through a single file path, without requiring users to modify any model or model training framework code or configuration, achieving transparency at the training framework level. Simultaneously, from a file system perspective, efficient checkpoint hierarchical storage management is provided for model training tasks. This enables read / write operations from memory, improving the throughput of checkpoint saving and loading processes, effectively reducing training blocking time, lowering the overhead of checkpoint operations during large model training, and shortening the end-to-end training runtime of large models.
[0061] In some embodiments, the storage cluster further includes a remote storage cluster deployed with a distributed parallel file system for storing the checkpoint files. Based on the above description, when the checkpoint management module performs hierarchical asynchronous management of checkpoint files based on the unified file access path, it can be configured to: During the execution of the training task, checkpoint files stored in the persistent storage layer are asynchronously migrated to the remote storage cluster.
[0062] In this embodiment, asynchronous means that the migration operation of copying or moving the checkpoint file from one storage level to another is not executed synchronously with the critical path of the training task. That is, the background thread / process performs the actual data transfer while the training continues. Compared with the training process having to wait for the remote write to be completed, asynchronous migration allows slow operations to be hidden in the background, so that the training always runs at the fastest storage level (memory / SSD) speed, achieving zero-blocking migration.
[0063] Through the above embodiments, checkpoint data is reliably saved to a remote, high-capacity storage facility, achieving persistent archiving and providing shared access capabilities for subsequent tasks. Meanwhile, asynchronous migration does not affect training performance.
[0064] In some embodiments, the triggering conditions for asynchronously migrating checkpoint files stored in the persistent storage layer to a remote storage cluster may be, but are not limited to, asynchronous migration based on storage status, file lifecycle, etc.
[0065] The following explanation uses storage state as an example of asynchronous migration.
[0066] In some embodiments, if the checkpoint management module is further configured to asynchronously migrate checkpoint files stored in the persistent storage layer to the remote storage cluster, the checkpoint management module is configured to: The storage status of the persistent storage layer is monitored in real time; when the storage status meets preset conditions, the checkpoint file to be migrated is determined; and the checkpoint file to be migrated is migrated from the persistent storage layer to the remote storage cluster.
[0067] In this embodiment, the storage status includes, but is not limited to, storage capacity utilization rate, and the corresponding preset conditions include the storage capacity utilization rate reaching a set threshold.
[0068] Continuing with the aforementioned examples and references Figure 3 Step 2.5: During the training task execution, the monitoring component of the checkpoint management module continuously monitors whether the storage capacity utilization rate of the persistent storage layer of the local storage cluster reaches a set threshold. When the storage capacity utilization rate reaches the set threshold, the control component of the checkpoint management module determines the checkpoint file to be migrated and migrates it from the persistent storage layer to the remote storage cluster. Conversely, if the storage capacity utilization rate of the persistent storage layer does not reach the set threshold, the storage location remains unchanged.
[0069] In this embodiment, the method for determining the checkpoint files to be migrated includes determining the checkpoint files to be migrated according to at least one of the following: the order of the checkpoint files' generation time, importance score, or file size.
[0070] In some embodiments, determining the checkpoint file to be migrated includes: Based on the order of checkpoint file generation time, the checkpoint file with the earliest generation time is determined as the checkpoint file to be migrated.
[0071] In other words, the control component of the checkpoint management module stores the earliest checkpoint file in the remote storage cluster.
[0072] This embodiment achieves automated space management, ensuring that the local persistent layer always has enough space to receive new data; it also eliminates the oldest files in chronological order and retains the newest checkpoints, meeting the needs of training recovery.
[0073] Step 2.6: If the training task encounters an unexpected situation that causes the training to fail, restart the training task.
[0074] In some embodiments, the checkpoint management module is further configured to: During the training restart process, the latest checkpoint file is loaded directly from the local storage cluster memory layer through the control component of the checkpoint management module.
[0075] Then, return to step 2.2 to continue the training task; otherwise, proceed to the next step, 2.7.
[0076] Step 2.7: Determine if the training task has ended. If it has not ended, repeat the process until step 2.2; otherwise, proceed to step 2.8.
[0077] Step 2.8: After training is complete, store and archive the checkpoint files.
[0078] In some embodiments, the checkpoint management module is further configured to: After training is completed, the control component of the checkpoint management module will store all checkpoint files that were still stored in the local storage cluster for this training task to the remote storage cluster for storage archiving.
[0079] A checkpoint operation request includes at least one of a checkpoint save request and a checkpoint load request. When training is complete and it is necessary to continue training based on the current training task, the checkpoint operation request is a checkpoint load request.
[0080] In some embodiments, when the checkpoint operation request is a checkpoint loading request, if the checkpoint management module is further configured to perform hierarchical asynchronous management of checkpoint files based on the unified file access path, the checkpoint management module is configured to: When a follow-up training task is executed based on the current training task, the required checkpoint files are preloaded to the corresponding storage levels in sequence according to the priority order of the remote storage cluster, the persistent storage layer of the local storage cluster, and the memory layer of the local storage cluster, based on the unified file access path. The checkpoint files are then read from the memory layer of the local storage cluster to resume training.
[0081] Specifically, continue with the examples and references above. Figure 3 If the next training task is initiated, the control component of the checkpoint management module will load the required checkpoint files in batches in parallel, following the order of remote storage cluster, local storage cluster persistent storage layer, and local storage cluster memory layer.
[0082] The training task reads the checkpoint file from the local storage cluster memory layer and returns to step 2.2 to continue the training task; otherwise, the entire checkpoint operation control flow ends.
[0083] In this embodiment, the storage location scheduling, hierarchical storage, anomaly recovery, and post-training archiving processes of checkpoint files are all implemented based on the aforementioned mounting structure and unified namespace. The unified namespace provides checkpoint access interfaces to the training framework using a standard file system path. The logic related to checkpoint storage process optimization and hierarchical storage scheduling is executed at the file system level and within the checkpoint management module, which is transparent to the training framework and requires no modification to the training framework and model code and configuration.
[0084] Based on the aforementioned model checkpoint file management system, this application provides a first embodiment of a model checkpoint file management method. (Refer to...) Figure 4 , Figure 4 This is a flowchart illustrating a model checkpoint file management method provided in an embodiment of this specification. The model checkpoint file management method is applied to the checkpoint management module of a model checkpoint file management system.
[0085] In some embodiments, the method specifically includes the following steps 101 to 102: Step 101: Receive the checkpoint operation request from the model training task; Step 102: Based on the unified file access path, perform hierarchical asynchronous management of checkpoint files; When the checkpoint operation request is a checkpoint save request, the checkpoint file is written to the unified file access path so that the checkpoint file is preferentially stored in the memory layer through the distributed cache file system. During the training task execution, the checkpoint files in the memory layer are asynchronously migrated to the persistent storage layer.
[0086] The storage cluster also includes a remote storage cluster deployed with a distributed parallel file system for storing the checkpoint files. The hierarchical asynchronous management of the checkpoint files based on the unified file access path includes: During the execution of the training task, the checkpoint files stored in the persistent storage layer are asynchronously migrated to the remote storage cluster.
[0087] In some embodiments, the asynchronous migration of checkpoint files stored in the persistent storage layer to the remote storage cluster includes: Real-time monitoring of the storage status of the persistent storage layer; When the storage state meets the preset conditions, the checkpoint file to be migrated is determined; The checkpoint file to be migrated is migrated from the persistent storage layer to the remote storage cluster.
[0088] In some embodiments, determining the checkpoint file to be migrated includes: Based on the order of checkpoint file generation time, the checkpoint file with the earliest generation time is determined as the checkpoint file to be migrated.
[0089] In some embodiments, when the checkpoint operation request is a checkpoint loading request, the hierarchical asynchronous management of the checkpoint file based on the unified file access path includes: When a follow-up training task is executed based on the current training task, the required checkpoint files are preloaded to the corresponding storage levels in parallel according to the priority order of the remote storage cluster, the persistent storage layer of the local storage cluster, and the memory layer of the local storage cluster, based on the unified file access path. The required checkpoint files are read from the memory layer of the local storage cluster to resume training.
[0090] For a detailed description of the model checkpoint file management method, please refer to the relevant description of the checkpoint management module in the aforementioned model checkpoint file management system. The model checkpoint file management method has the same beneficial effects as the model checkpoint file management system, and will not be elaborated upon here.
[0091] In this embodiment, the checkpoint management module is typically located in the computing cluster or on a computing node of the computing cluster.
[0092] In this embodiment, the checkpoint management module includes routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This model checkpoint file management scheme is typically implemented in large model training environments, where tasks are performed by the monitoring and execution components of the added checkpoint file manager.
[0093] The aforementioned model checkpoint file management system deploys a distributed cached file system on compute nodes, utilizing the memory and SSDs of each node to build a local storage cluster. It constructs a memory layer and a local persistent storage layer, prioritizing writes to the memory layer before asynchronously migrating to the persistent layer, fully leveraging the performance advantages of both media types. Simultaneously, from a file system perspective, storage management is decentralized to the file system layer. The storage, layering, and asynchronous migration of all checkpoint files are handled internally by the file system. By setting a unified mount point path as the storage entry point accessible to the training framework, the framework only needs to read and write to this path through a standard file interface, without needing to be aware of the internal storage hierarchy and asynchronous migration logic. Therefore, performance optimization can be achieved without modifying the framework code or configuration.
[0094] Figure 5 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 5As shown, the electronic device may include a processor 510, a communications interface 520, a memory 530, and a communication bus 540. The processor 510, communications interface 520, and memory 530 communicate with each other via the communication bus 540. The processor 510 can call logical instructions from the memory 530 to execute the model checkpoint file management method.
[0095] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0096] On the other hand, this application also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the model checkpoint file management method provided by the above methods.
[0097] In another aspect, this application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the model checkpoint file management methods provided by the above methods.
[0098] It should be noted that the technical solutions or features described in the above embodiments can be combined or supplemented with each other without conflict. The scope of protection of this application is not limited to the precise structures described in the above embodiments and shown in the accompanying drawings; all modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.
Claims
1. A model checkpoint file management system, characterized in that, The system includes: Storage resources include local storage clusters, which consist of distributed cache file systems deployed on each computing node in each computing cluster. The local storage cluster includes a hierarchical storage structure formed by a memory layer and a persistent storage layer. The local storage cluster forms a unified file access path through mounting, which is accessed by the model training framework. The checkpoint management module, deployed between the training framework and the storage resources, is used to perform hierarchical asynchronous management of checkpoint files based on the unified file access path when receiving checkpoint operation requests from model training tasks. When the checkpoint operation request is a checkpoint save request, the checkpoint file is written to the unified file access path so that the checkpoint file is preferentially stored in the memory layer through the distributed cache file system. During the execution of the training task, the checkpoint file in the memory layer is asynchronously migrated to the persistent storage layer.
2. The model checkpoint file management system as described in claim 1, characterized in that, The storage cluster also includes a remote storage cluster deployed with a distributed parallel file system for storing the checkpoint files. If the checkpoint management module is used to perform hierarchical asynchronous management of the checkpoint files based on the unified file access path, the checkpoint management module is configured to: During the execution of the training task, the checkpoint files stored in the persistent storage layer are asynchronously migrated to the remote storage cluster.
3. The model checkpoint file management system as described in claim 2, characterized in that, The remote storage cluster is deployed by mounting it to each computing node of the computing cluster via client mounting, and the mount point of the local storage cluster is soft-linked to the mount directory of the remote storage cluster in the form of a directory, forming a unified file access path that is accessible to the model training framework.
4. The model checkpoint file management system as described in claim 2, characterized in that, If the checkpoint management module is also used to asynchronously migrate checkpoint files stored in the persistent storage layer to the remote storage cluster, the checkpoint management module is configured to: Real-time monitoring of the storage status of the persistent storage layer; When the storage state meets the preset conditions, the checkpoint file to be migrated is determined; The checkpoint file to be migrated is migrated from the persistent storage layer to the remote storage cluster.
5. The model checkpoint file management system as described in claim 4, characterized in that, When the checkpoint operation request is a checkpoint loading request, if the checkpoint management module is also used to perform hierarchical asynchronous management of checkpoint files based on the unified file access path, the checkpoint management module is configured to: When a follow-up training task is executed based on the current training task, the required checkpoint files are preloaded to the corresponding storage levels in sequence according to the priority order of the remote storage cluster, the persistent storage layer of the local storage cluster, and the memory layer of the local storage cluster, based on the unified file access path. The checkpoint files are then read from the memory layer of the local storage cluster to resume training.
6. A method for managing model checkpoint files, characterized in that, Applied to a model checkpoint file management system as described in any one of claims 1 to 5, the method comprises: Receive checkpoint operation requests from the model training task; Based on the unified file access path, hierarchical asynchronous management is performed on checkpoint files; When the checkpoint operation request is a checkpoint save request, the checkpoint file is written to the unified file access path so that the checkpoint file is preferentially stored in the memory layer through the distributed cache file system. During the training task execution, the checkpoint files in the memory layer are asynchronously migrated to the persistent storage layer.
7. The model checkpoint file management method as described in claim 6, characterized in that, The storage cluster also includes a remote storage cluster deployed with a distributed parallel file system for storing the checkpoint files. The hierarchical asynchronous management of the checkpoint files based on the unified file access path includes: During the execution of the training task, the checkpoint files stored in the persistent storage layer are asynchronously migrated to the remote storage cluster.
8. The model checkpoint file management method as described in claim 7, characterized in that, The asynchronous migration of checkpoint files stored in the persistent storage layer to the remote storage cluster includes: Real-time monitoring of the storage status of the persistent storage layer; When the storage state meets the preset conditions, the checkpoint file to be migrated is determined; The checkpoint file to be migrated is migrated from the persistent storage layer to the remote storage cluster.
9. The model checkpoint file management method as described in claim 8, characterized in that, The file for determining the checkpoints to be migrated includes: Based on the order of checkpoint file generation time, the checkpoint file with the earliest generation time is determined as the checkpoint file to be migrated.
10. The model checkpoint file management method as described in claim 7, characterized in that, When the checkpoint operation request is a checkpoint loading request, the hierarchical asynchronous management of the checkpoint file based on the unified file access path includes: When a follow-up training task is executed based on the current training task, the required checkpoint files are preloaded to the corresponding storage levels in sequence according to the priority order of the remote storage cluster, the persistent storage layer of the local storage cluster, and the memory layer of the local storage cluster, based on the unified file access path. The required checkpoint files are read from the memory layer of the local storage cluster to resume training.
11. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that, when executed by the processor, implements the model checkpoint file management method as described in any one of claims 6 to 9.
12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed, implements the model checkpoint file management method as described in any one of claims 6 to 9.