Checkpoint file storage method, device, electronic device and storage medium

Abstract

The present invention provides a checkpoint file saving method, device, electronic device and storage medium, belonging to the field of data storage technology, including: obtaining a target container group, wherein the target container group includes a main container, and the main container is used to perform a model training task; adding an auxiliary container to the target container group, wherein the auxiliary container and the main container share a cache directory; monitoring the cache directory through the auxiliary container to obtain a checkpoint file, wherein the checkpoint file is used to save the parameters generated by the main container during the model training process; and saving the checkpoint file to a shared storage. The checkpoint file saving method provided by the present invention solves the technical problem that the checkpoint file saving method in the related art has a technical problem that the model training efficiency is affected due to the simultaneous model training and checkpoint file saving.

Description

Technical Field

[0001] The present invention relates to the technical field of data storage, and in particular to a checkpoint file storage method, device, electronic equipment and storage medium. Background Art

[0002] During the model training process, a checkpoint file is generated. The checkpoint file is used to save the state of the model at a specific time point during the training process, including model network parameters, optimizer status, etc.; in related technologies, checkpoint files are usually saved to shared storage, and model training and checkpoint file saving are carried out simultaneously, which occupies computer computing power, resulting in a slower checkpoint file saving speed, affecting model training efficiency.

[0003] It can be seen that the checkpoint file saving method in the related art has a technical problem that the model training efficiency is affected because the model training and the checkpoint file saving are carried out simultaneously. Summary of the invention

[0004] The present invention provides a checkpoint file saving method, device, electronic device and storage medium, which are used to solve the technical problem that the checkpoint file saving method in the related art has the problem that the model training efficiency is affected due to the simultaneous execution of model training and checkpoint file saving.

[0005] The present invention provides a checkpoint file saving method, comprising the following steps: obtaining a target container group, wherein the target container group includes a main container, and the main container is used to execute a model training task; adding an auxiliary container to the target container group, wherein the auxiliary container shares a cache directory with the main container; monitoring the cache directory through the auxiliary container to obtain a checkpoint file, wherein the checkpoint file is used to save parameters generated by the main container during the model training process; and saving the checkpoint file to a shared storage.

[0006] According to a checkpoint file saving method provided by the present invention, the method of obtaining the checkpoint file by monitoring the cache directory through the auxiliary container includes: monitoring the checkpoint file to be confirmed in the cache directory through the auxiliary container, and determining whether the checkpoint file to be confirmed has changed within a target time period; if the determination is no, determining the checkpoint file to be confirmed as the checkpoint file.

[0007] According to a checkpoint file saving method provided by the present invention, the auxiliary container monitors the checkpoint file to be confirmed in the cache directory to determine whether the checkpoint file to be confirmed has changed within a target time period, including: detecting the file size of the checkpoint file to be confirmed in the cache directory through the auxiliary container, and when the file size of the checkpoint file to be confirmed has changed within the target time period, determining that the checkpoint file to be confirmed has changed; or detecting the MD5 value of the checkpoint file to be confirmed in the cache directory through the auxiliary container, and when the MD5 value of the checkpoint file to be confirmed has changed within the target time period, determining that the checkpoint file to be confirmed has changed.

[0008] According to a checkpoint file saving method provided by the present invention, the checkpoint file is saved to a shared storage, comprising: saving the checkpoint file to a target snapshot, wherein the target snapshot is a snapshot corresponding to the target time period; comparing the target snapshot with a historical snapshot for similarity, and judging whether a target checkpoint file exists in the target snapshot based on the comparison result, wherein the historical snapshot is a snapshot corresponding to a historical time period, and the historical time period is a previous time period adjacent to the target time period; if the target checkpoint file exists, saving the target checkpoint file to the shared storage.

[0009] According to a checkpoint file saving method provided by the present invention, the target snapshot is compared with the historical snapshot for similarity, and based on the comparison result, it is determined whether the target checkpoint file exists in the target snapshot, including: when the target snapshot is inconsistent with the historical snapshot, it is determined that the target checkpoint file exists in the target snapshot, and the incremental file of the target snapshot compared to the historical snapshot is determined as the target checkpoint file.

[0010] According to a checkpoint file saving method provided by the present invention, the target snapshot is compared with the historical snapshot for similarity, and based on the comparison result, it is determined whether the target checkpoint file exists in the target snapshot, including: when the target snapshot is consistent with the historical snapshot, determining that the target checkpoint file does not exist in the target snapshot.

[0011] According to a checkpoint file saving method provided by the present invention, after saving the target checkpoint file to the shared storage when the target checkpoint file exists, the method further includes: determining whether the target checkpoint file is successfully saved to the shared storage; if the determination is no, deleting the target snapshot.

[0012] According to a checkpoint file saving method provided by the present invention, the obtaining of the target container group includes: obtaining the target container group from multiple container groups based on a preset interception condition through a network hook, wherein the preset interception condition is to intercept a container group with a target label, and the container group with the target label is a container group that needs to execute a model training task.

[0013] According to a checkpoint file saving method provided by the present invention, before obtaining the target container group, the method also includes: building a target image, wherein the target image includes running the auxiliary container to listen to the cache directory, and saving the obtained checkpoint file to all components of the shared storage.

[0014] According to a checkpoint file saving method provided by the present invention, adding an auxiliary container in the target container group includes: adding the auxiliary container in the target container group; setting the startup image of the auxiliary container to the target image; and mounting the auxiliary container in the cache directory.

[0015] According to a checkpoint file saving method provided by the present invention, after the checkpoint file is saved to the shared storage, the method further includes: monitoring the target container group through a container group listener; when the main container completes the model training task, querying whether the checkpoint file in the cache directory has been saved to the shared storage; and when the checkpoint file has been saved to the shared storage, switching the target container group to an end state.

[0016] The present invention also provides a checkpoint file saving device, comprising the following modules: an acquisition module, used to obtain a target container group, wherein the target container group includes a main container, and the main container is used to execute a model training task; an adding module, used to add an auxiliary container in the target container group, wherein the auxiliary container shares a cache directory with the main container; an execution module, used to monitor the cache directory through the auxiliary container to obtain a checkpoint file, wherein the checkpoint file is used to save the parameters generated by the main container during the model training process; and a saving module, used to save the checkpoint file to a shared storage.

[0017] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the checkpoint file saving methods described above is implemented.

[0018] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the checkpoint file saving method described in any one of the above is implemented.

[0019] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the checkpoint file saving method described in any one of the above is implemented.

[0020] The checkpoint file saving method, device, electronic device and storage medium provided by the present invention obtain a target container group; add an auxiliary container to the target container group. Since the auxiliary container shares a cache directory with the main container, the checkpoint file can be obtained by monitoring the cache directory through the auxiliary container, and the checkpoint file is saved in the shared storage; since the model training process and the checkpoint file saving process are respectively executed by different containers, the checkpoint file saving process will not affect the efficiency of the model training, thereby solving the technical problem of the checkpoint file saving method in the related art that the model training and the checkpoint file saving are carried out simultaneously, which affects the model training efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced one by one below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0022] Figure 1 It is a flow chart of the checkpoint file saving method provided by the present invention.

[0023] Figure 2 This is one of the schematic diagrams of the checkpoint file saving method provided by the present invention.

[0024] Figure 3 This is the second schematic diagram of the checkpoint file saving method provided by the present invention.

[0025] Figure 4 This is the third schematic diagram of the checkpoint file saving method provided by the present invention.

[0026] Figure 5 It is a structural schematic diagram of the checkpoint file storage device provided by the present invention.

[0027] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] It should be noted that in the description of the present invention, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "include one..." do not exclude the existence of other identical elements in the process, method, article or device including the elements. The orientation or position relationship indicated by the terms "upper", "lower" and the like is based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation of the present invention. Unless otherwise clearly specified and limited, the terms "installed", "connected" and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be a connection between the two elements. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0030] The terms "first", "second", etc. in the present invention are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present invention can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" means at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0031] With the rapid development of artificial intelligence technology, large models have become a key force in promoting intelligent applications. These models have achieved remarkable results in image recognition, natural language processing and other fields with their huge parameter scale and powerful learning ability. As an advanced container orchestration system, Kubernetes has greatly improved the efficiency of model training and deployment in large model scenarios. Kubernetes' elastic scaling, automated deployment and efficient resource management capabilities provide a stable and scalable operating environment for large models.

[0032] The checkpoint preservation problem generated during the training process of large models is still a technical challenge. Checkpoint is the key data in the model training process. It records the parameter status of the model and is used for model recovery and transfer learning. However, due to the large parameter scale of large models, the checkpoint preservation and loading process is often time-consuming and occupies a lot of storage resources. In related technologies, in order to speed up the checkpoint preservation speed, it is necessary to invade the user code and implement it by modifying or extending the model training framework. This approach not only increases the user's development burden, but may also cause code compatibility issues and maintenance difficulties. In addition, the frequent read and write operations of checkpoints put higher requirements on the input / output (I / O) performance of the storage system, and traditional storage solutions often find it difficult to meet such high concurrency and high throughput requirements.

[0033] In Kubernetes, container groups (pods) can apply for certain resources to carry deep learning training. Therefore, it is feasible to use pods to encapsulate large model training logic. Pods can mount local directories, shared memory, or shared storage for pod containers to use. Large models can save checkpoints in these directories. In related technologies, checkpoint files are usually saved to shared storage, and model training and checkpoint file saving are performed simultaneously. Since shared storage is generally limited by network bandwidth and the speed of storage server downloading, saving checkpoints directly to shared storage is relatively inefficient and occupies computer computing power, resulting in a lower checkpoint file saving speed, affecting model training efficiency.

[0034] In order to at least solve some of the above problems, the following Figure 1-Figure 6 The present invention describes a method, device, electronic device and storage medium for saving checkpoint files.

[0035] The checkpoint file saving method provided in this embodiment can be applied to the scenario of saving checkpoint files in the model training process; the checkpoint file saving method provided in this embodiment can be executed by a container orchestration system; the checkpoint file saving method in this embodiment can also be executed by a server, wherein the server runs a container orchestration system, and the container orchestration system can be Kubernetes.

[0036] Figure 1 : is a flow chart of the checkpoint file saving method provided by the present invention, such as Figure 1 As shown, including but not limited to the following steps:

[0037] Step 101: Obtain a target container group, wherein the target container group includes a main container, and the main container is used to execute a model training task.

[0038] The target container group can be selected from multiple container groups. The target container group includes the main container, which is used to perform model training tasks. The container group is the smallest deployable unit in the container orchestration system and the basic unit for management and scheduling. A container group can contain one or more containers. These containers share the same network namespace and storage volume and can communicate with each other.

[0039] Step 102: Add an auxiliary container to the target container group, wherein the auxiliary container shares a cache directory with the main container.

[0040] Figure 2 It is one of the schematic diagrams of the checkpoint file saving method provided by the present invention, such as Figure 2 As shown, Figure 2 The pod in the example is the target container group in this embodiment, which includes the main container (corresponding to Figure 2 The training task container in the example above is a checkpoint management service container. By adding an auxiliary container (checkpoint management service container) to the initial pod, the final pod can be obtained. The auxiliary container mounts the same cache directory as the main container.

[0041] Step 103, obtain a checkpoint file by monitoring the cache directory through the auxiliary container, wherein the checkpoint file is used to save the parameters generated by the main container during the model training process.

[0042] When the main container is executing the model training task, the cache directory will continuously update the checkpoint file. Since the auxiliary container shares the cache directory with the main container, the checkpoint file can be obtained by monitoring the cache directory through the auxiliary container.

[0043] During the model training process, checkpoint files are an important mechanism for saving the model status during the training process. These files usually contain key information of the model so that training can be resumed after interruption, or the model can be reloaded for inference or prediction when needed; specifically, checkpoint files include model parameters, optimizer status, and training status.

[0044] Step 104, save the checkpoint file to the shared storage.

[0045] After obtaining the checkpoint file, save the checkpoint file to shared storage; in this embodiment, shared storage refers to a mechanism that allows multiple container groups to share the same persistent storage volume. This shared storage implementation provides many advantages, such as data sharing, data persistence, data backup and recovery, etc., thereby enhancing the flexibility and reliability of containerized applications.

[0046] Through the embodiments provided by the present application, a target container group is obtained, wherein the target container group includes a main container, and the main container is used to execute a model training task; an auxiliary container is added to the target container group, wherein the auxiliary container shares a cache directory with the main container; a checkpoint file is obtained by monitoring the cache directory through the auxiliary container, wherein the checkpoint file is used to save the parameters generated by the main container during the model training process; the checkpoint file is saved to a shared storage; the checkpoint file saving method in the related art is solved, and there is a technical problem that the model training efficiency is affected due to the simultaneous model training and checkpoint file saving, thereby improving the model training efficiency.

[0047] As an optional solution, the auxiliary container monitors the cache directory to obtain the checkpoint files, including:

[0048] S11, monitoring the checkpoint files to be confirmed in the cache directory through the auxiliary container, and determining whether the checkpoint files to be confirmed have changed within the target time period;

[0049] S12: If the judgment result is negative, the checkpoint file to be confirmed is confirmed as a checkpoint file.

[0050] In this embodiment, the monitoring unit is a time period, the length of the monitoring time period is configured, and the auxiliary container monitors the checkpoint files to be confirmed in the cache directory to determine whether the checkpoint files to be confirmed are changed within the target time period.

[0051] It should be noted that during the execution of the training task by the main container, the checkpoint files to be confirmed in the cache directory will be continuously updated, and different checkpoint files will be updated for different training progress. When it is detected that the checkpoint files to be confirmed have not changed within the target time period, it means that the checkpoint files to be confirmed have been cached, and the checkpoint files to be confirmed can be confirmed as checkpoint files; when it is detected that the checkpoint files to be confirmed have changed within the target time period, it means that the checkpoint files to be confirmed have not been completed, and the auxiliary container continues to perform the monitoring detection steps on the cache directory.

[0052] Through this embodiment, the checkpoint file is determined by judging whether the checkpoint file to be confirmed has changed within the target time period, which can improve the efficiency and accuracy of obtaining the checkpoint file.

[0053] As an optional solution, the auxiliary container monitors the checkpoint files to be confirmed in the cache directory to determine whether the checkpoint files to be confirmed have changed within the target time period, including:

[0054] S21, detecting the file size of the checkpoint file to be confirmed in the cache directory through the auxiliary container, and determining that the checkpoint file to be confirmed has changed when the file size of the checkpoint file to be confirmed changes within the target time period; or,

[0055] S22, detecting the MD5 value of the checkpoint file to be confirmed in the cache directory through the auxiliary container, and determining that the checkpoint file to be confirmed has changed when the MD5 value of the checkpoint file to be confirmed changes within the target time period.

[0056] In this embodiment, it can be determined whether the checkpoint file to be confirmed has changed based on the file size or MD5 value of the checkpoint file to be confirmed; the file size of the checkpoint file to be confirmed refers to the size of the storage space occupied by the amount of data contained in the confirmation checkpoint file; the MD5 value of the checkpoint file to be confirmed refers to the unique hash value calculated by the MD5 algorithm (Message-DigestAlgorithm 5) and used to identify the file. Different files have different MD5 values. Therefore, it can be determined whether the checkpoint file to be confirmed has changed based on the MD5 value.

[0057] Optionally, determine whether the checkpoint file to be confirmed has changed based on the file modification time of the checkpoint file to be confirmed; since the file system records the last modification time of each file, by comparing the last modification time of the file with the start and end time of the target time period, it can be determined whether the file has been modified within the target time period.

[0058] Optionally, it is determined whether the checkpoint file to be confirmed has changed based on changes in the file attributes of the checkpoint file to be confirmed; specifically, in addition to the file content and size, the file attributes (such as permissions, owner, number of links, etc.) may also change. By comparing the current attributes of the file with the attributes at the beginning of the target time period, it can be determined whether the file has changed in these aspects.

[0059] Through this embodiment, whether the checkpoint file to be confirmed has changed is identified based on the file size or MD5 value of the checkpoint file to be confirmed, which can improve the accuracy of judging whether the checkpoint file to be confirmed has changed within the target time period.

[0060] As an optional solution, save the checkpoint files to shared storage, including:

[0061] S31, saving the checkpoint file to a target snapshot, wherein the target snapshot is a snapshot corresponding to a target time period;

[0062] S32, comparing the target snapshot with the historical snapshot for similarity, and judging whether the target checkpoint file exists in the target snapshot based on the comparison result, wherein the historical snapshot is a snapshot corresponding to a historical time period, and the historical time period is a previous time period adjacent to the target time period;

[0063] S33: If the target checkpoint file exists, save the target checkpoint file to the shared storage.

[0064] After obtaining the checkpoint file, the target checkpoint file in the checkpoint file is saved to the shared storage through snapshot technology; snapshot technology is a data backup and recovery technology that allows a fully available copy of a specific data set at a specified point in time, thereby retaining the image of the data at that point in time.

[0065] In this embodiment, the checkpoint file is saved in the target snapshot, the target snapshot is compared with the historical snapshot for similarity, and it is determined whether the target checkpoint file exists in the target snapshot based on the comparison result; here, it is possible to determine whether the target checkpoint file exists in the target snapshot based on the similarity between the target snapshot and the historical snapshot, and then determine the incremental file of the target snapshot compared to the historical snapshot as the target checkpoint file, and if the target checkpoint file exists, the target checkpoint file is saved to the shared storage.

[0066] Through this embodiment, the increments of the checkpoint files can be found through snapshot comparison, and only these incremental files need to be moved, thus avoiding repeated movement of checkpoint files and effectively reducing computing power loss.

[0067] As an optional solution, the target snapshot is compared with the historical snapshot for similarity, and based on the comparison result, it is determined whether the target checkpoint file exists in the target snapshot, including:

[0068] S41, when the target snapshot is inconsistent with the historical snapshot, it is determined that there is a target checkpoint file in the target snapshot, and an incremental file of the target snapshot compared with the historical snapshot is determined as the target checkpoint file.

[0069] Figure 3 This is a second schematic diagram of the checkpoint file storage method provided by the present invention, such as Figure 3 As shown, version n corresponds to the historical snapshot in this embodiment, and version n+1 corresponds to the target snapshot in this embodiment.

[0070] In this embodiment, when the target snapshot is inconsistent with the historical snapshot, it means that the checkpoint file has been updated within the target time period. Therefore, it is determined that the target checkpoint file exists in the target snapshot, and the incremental file of the target snapshot compared to the historical snapshot is determined as the target checkpoint file.

[0071] Exemplarily, the historical snapshot records file a and file c, and the target snapshot records file a, file b, and file c, indicating that file b is an incremental file, and file b is determined as the target checkpoint file.

[0072] Through this embodiment, whether the target checkpoint file exists in the target snapshot is determined based on the consistency between the target snapshot and the historical snapshot, so that the accuracy of the obtained target checkpoint file can be improved.

[0073] As an optional solution, the target snapshot is compared with the historical snapshot for similarity, and based on the comparison result, it is determined whether the target checkpoint file exists in the target snapshot, including:

[0074] S51: When the target snapshot is consistent with the historical snapshot, it is determined that the target checkpoint file does not exist in the target snapshot.

[0075] In this embodiment, when the target snapshot is consistent with the historical snapshot, it means that the checkpoint file has not changed within the target time period, that is, there is no cached checkpoint file in the cache directory within the target time period. Therefore, it is determined that the target checkpoint file does not exist in the target snapshot, and the auxiliary container continues to listen to the cache directory in the time period after the target time period.

[0076] Through this embodiment, whether the target checkpoint file exists in the target snapshot is determined based on the consistency between the target snapshot and the historical snapshot, so that the accuracy of the obtained target checkpoint file can be improved.

[0077] As an optional solution, if the target checkpoint file exists, after saving the target checkpoint file to the shared storage, the following steps are also included:

[0078] S61, determining whether the target checkpoint file is successfully saved to the shared storage;

[0079] S62: If the determination is no, the target snapshot is deleted.

[0080] In this embodiment, a fault-tolerant mechanism is added. Specifically, after the target checkpoint file is saved to the shared storage, it is determined whether the target checkpoint file is successfully saved to the shared storage. If the judgment is no, the target snapshot is deleted. If the target checkpoint file fails to be saved to the shared storage, the snapshot is restored and the target checkpoint file is removed from the snapshot, that is, the target snapshot is deleted and the next snapshot comparison logic is waited for.

[0081] Through this embodiment, by adding a fault tolerance mechanism after saving the target checkpoint file to the shared storage, the reliability and stability of the checkpoint file saving method in this embodiment are increased.

[0082] As an optional solution, obtain the target container group, including:

[0083] S71, obtaining a target container group from multiple container groups based on a preset interception condition through a network hook, wherein the preset interception condition is to intercept a container group with a target label, and the container group with the target label is a container group that needs to execute a model training task.

[0084] It should be noted that the webhook in this embodiment can be triggered at a specific stage in the life cycle of the container and allows the execution of custom operations, see Figure 2 , the target container group can be obtained from multiple container groups based on preset interception conditions through network hooks.

[0085] In this embodiment, Webhook is described by taking Kubernetes as the container orchestration system. Webhook is a HyperText Transfer Protocol (HTTP) callback mechanism used to trigger remote operations or custom logic when a specific event occurs. It is part of the Kubernetes Application Programming Interface (API) and allows users to interact with external services through HTTP requests when key lifecycle events occur in the Kubernetes cluster.

[0086] In this embodiment, the multiple container groups are multiple container groups included in the container orchestration system. The container group that needs to execute the model training task among the multiple container groups is pre-labeled with a target label, and the preset interception condition is to intercept the container group with the target label. The target container group can be accurately obtained from the multiple container groups based on the preset interception condition through the network hook.

[0087] According to this embodiment, the target container group is obtained from multiple container groups based on the preset interception condition through the network hook, so that the accuracy of obtaining the target container group can be improved.

[0088] As an optional solution, before obtaining the target container group, the method further includes:

[0089] S81, building a target image, wherein the target image includes all components that run an auxiliary container to listen to a cache directory and save the obtained checkpoint file to a shared storage.

[0090] It should be noted that an image is a lightweight, executable, independent software package that contains everything needed to run an application, including code, runtime environment, libraries, environment variables, and configuration files. That is, for different environments, you can quickly start and run applications based on the image without worrying about environmental differences or dependency conflicts.

[0091] In this embodiment, before obtaining the target container group, it is necessary to build a target image, which includes all components that run the auxiliary container to listen to the cache directory and save the obtained checkpoint file to the shared storage, that is, based on the target image, the auxiliary container can listen to the cache directory and save the obtained checkpoint file to the shared storage.

[0092] Through this embodiment, a target image for assisting in building an auxiliary container is provided, thereby improving the integrity and reliability of the checkpoint file saving method in this embodiment.

[0093] As an optional solution, add auxiliary containers to the target container group, including:

[0094] S91, adding an auxiliary container to the target container group;

[0095] S92, setting the startup image of the auxiliary container as the target image;

[0096] S93, mounting the auxiliary container in the cache directory.

[0097] In this embodiment, after obtaining the target container group, an auxiliary container is added to the target container group; the startup image of the auxiliary container is set as the target image, that is, the auxiliary container can implement subsequent monitoring cache directories based on the startup image, and the obtained checkpoint file is saved to the shared storage step; the auxiliary container is mounted on the cache directory, that is, the auxiliary container and the main container are mounted on the same cache directory, so that the auxiliary container can determine the checkpoint file by monitoring the cache directory.

[0098] Through this embodiment, a specific method for adding an auxiliary container to a target container group is provided, thereby improving the integrity and reliability of the checkpoint file saving method in this embodiment.

[0099] As an optional solution, after saving the checkpoint file to the shared storage, the method further includes:

[0100] S111, monitoring the target container group through the container group listener;

[0101] S112, when the main container completes the model training task, query whether the checkpoint file in the cache directory has been saved to the shared storage;

[0102] S113: When the checkpoint file has been saved to the shared storage, the target container group is switched to the end state.

[0103] In this embodiment, the target container group is monitored by the container group listener. Figure 4 Schematic diagram of the checkpoint file storage method provided by the present invention is shown in FIG. Figure 4 As shown, the container group listener can monitor the container status of the target container group based on the interface server. When the end event of the main container model training task is monitored, it can query whether the checkpoint files in the cache directory have all been saved to the shared storage; if the checkpoint files have all been saved to the shared storage, the target container group is switched to the end state; if the checkpoint files have not all been saved to the shared storage, wait until all the checkpoint files are saved to the shared storage, then switch the target container group to the end state to release the resources of the target container group.

[0104] Through this embodiment, by setting a container group listener to listen to the container status of the target container group and performing subsequent management based on the container status of the target container group, the reliability and stability of the checkpoint file saving method in this embodiment can be improved.

[0105] The following is an explanation of the checkpoint file saving method of the embodiment of the present application in conjunction with an optional example. In this optional example, the checkpoint file saving method in this embodiment is executed by a server, and the server runs Kubernetes.

[0106] Specifically, using the webhook mechanism of Kubernetes, before creating the pod that carries the large model training task, the pod that needs to be intercepted (the pod for training the large model) is filtered out through the logic of the webhook hook, and the sidecar container configuration is added to the original pod (the sidecar container is a commonly used design pattern used to provide auxiliary functions for the main container in the Pod). Compared with the original pod that only contains the container configuration for training the large model, the pod configuration modified by the webhook contains two container configurations, one is the container configuration for training the large model, and the other is the checkpoint management service container configuration, and the two containers have the same mount directory configuration. The training script in the model training container is not modified. The original training script is still used, and the checkpoints in the training process are saved in the agreed mount directory. The checkpoint management service container monitors the changes in the mount cache directory where the checkpoint is saved in real time, and uses snapshot technology to identify the checkpoint sub-files that have been saved (the checkpoint management service container monitors the files that are constantly changing, indicating that these files are still being written. When it is found that the files no longer change within a certain period of time, it means that the file data is written). These checkpoint files that have completed writing data will be recorded in a snapshot metadata, and then the checkpoint management service container will move these checkpoint files saved in the cache directory to shared storage. Before each move, a snapshot comparison will be performed (that is, the current record snapshot is compared with the previous record snapshot). Through the snapshot comparison, the increment of this checkpoint can be found, and only these incremental files need to be moved, avoiding the repeated movement of checkpoint files. The user (algorithm engineer) has no perception of the entire process, and there is no need to modify the user's training script. This not only speeds up the synchronization and saving of checkpoints, but also does not invade the script, greatly improving the training efficiency.

[0107] Figure 5 Schematic diagram of the structure of the checkpoint file storage device provided by the present invention. Figure 5 As shown, including but not limited to the following modules:

[0108] An acquisition module 501 is used to acquire a target container group, wherein the target container group includes a main container, and the main container is used to execute a model training task;

[0109] An adding module 502 is used to add an auxiliary container to the target container group, wherein the auxiliary container shares a cache directory with the main container;

[0110] An execution module 503 is used to monitor the cache directory through the auxiliary container to obtain a checkpoint file, wherein the checkpoint file is used to save the parameters generated by the main container during the model training process;

[0111] The saving module 504 is used to save the checkpoint file to the shared storage.

[0112] Through the embodiments of the present application, a target container group is obtained, wherein the target container group includes a main container, and the main container is used to execute a model training task; an auxiliary container is added to the target container group, wherein the auxiliary container shares a cache directory with the main container; the cache directory is monitored by the auxiliary container to obtain a checkpoint file, wherein the checkpoint file is used to save the parameters generated by the main container during the model training process; the checkpoint file is saved to a shared storage; the checkpoint file saving method in the related art is solved, and there is a technical problem that the model training efficiency is affected due to the simultaneous model training and checkpoint file saving, thereby improving the model training efficiency.

[0113] It should be noted that the checkpoint file saving device provided by the present invention can execute the checkpoint file saving method of any of the above embodiments during specific operation, which will not be described in detail in this embodiment.

[0114] Figure 6 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 6 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the checkpoint file saving method, which includes: obtaining a target container group, wherein the target container group includes a main container, and the main container is used to perform a model training task; adding an auxiliary container to the target container group, wherein the auxiliary container shares a cache directory with the main container; monitoring the cache directory through the auxiliary container to obtain a checkpoint file, wherein the checkpoint file is used to save the parameters generated by the main container during the model training process; and saving the checkpoint file to a shared storage.

[0115] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0116] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions, and when the program instructions are executed by a computer, the computer can execute the checkpoint file saving method provided in the above embodiments, the method including: obtaining a target container group, wherein the target container group includes a main container, and the main container is used to execute a model training task; adding an auxiliary container to the target container group, wherein the auxiliary container shares a cache directory with the main container; obtaining a checkpoint file by monitoring the cache directory through the auxiliary container, wherein the checkpoint file is used to save the parameters generated by the main container during the model training process; and saving the checkpoint file to shared storage.

[0117] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to execute the checkpoint file saving method provided in the above-mentioned embodiments, the method comprising: obtaining a target container group, wherein the target container group includes a main container, and the main container is used to execute a model training task; adding an auxiliary container to the target container group, wherein the auxiliary container shares a cache directory with the main container; obtaining a checkpoint file by monitoring the cache directory through the auxiliary container, wherein the checkpoint file is used to save the parameters generated by the main container during the model training process; and saving the checkpoint file to a shared storage.

[0118] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, i.e., they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.

[0119] Through the description of the above implementation modes, those skilled in the art can clearly understand that each implementation mode can be implemented by means of software plus a necessary general hardware platform, or of course by hardware. Based on such an understanding, the above technical solution can essentially or in other words be embodied in the form of a software product that contributes to the prior art. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.

[0120] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A checkpoint file storage method, characterized in that: include: Acquire a target container group, wherein the target container group includes a main container, and the main container is used to execute a model training task; Adding an auxiliary container to the target container group, wherein the auxiliary container shares a cache directory with the main container; The auxiliary container monitors the cache directory to obtain a checkpoint file, wherein the checkpoint file is used to save the parameters generated by the main container during the model training process; Saving the checkpoint file to shared storage; The step of monitoring the cache directory through the auxiliary container to obtain a checkpoint file includes: Monitoring the to-be-confirmed checkpoint file in the cache directory by the auxiliary container, and determining whether the to-be-confirmed checkpoint file changes within a target time period; If the judgment is no, confirming the to-be-confirmed checkpoint file as the checkpoint file; The monitoring the to-be-confirmed checkpoint file in the cache directory by the auxiliary container and determining whether the to-be-confirmed checkpoint file changes within a target time period includes: Detecting the file size of the checkpoint file to be confirmed in the cache directory by the auxiliary container, and determining that the checkpoint file to be confirmed has changed if the file size of the checkpoint file to be confirmed has changed within the target time period; or The auxiliary container detects the MD5 value of the checkpoint file to be confirmed in the cache directory, and when the MD5 value of the checkpoint file to be confirmed changes within the target time period, it is determined that the checkpoint file to be confirmed changes.

2. The checkpoint file storage method according to claim 1, characterized in that: Saving the checkpoint file to a shared storage includes: Saving the checkpoint file to a target snapshot, wherein the target snapshot is a snapshot corresponding to the target time period; Compare the target snapshot with the historical snapshot for similarity, and determine whether the target checkpoint file exists in the target snapshot based on the comparison result, wherein the historical snapshot is a snapshot corresponding to a historical time period, and the historical time period is a previous time period adjacent to the target time period; If the target checkpoint file exists, the target checkpoint file is saved to the shared storage.

3. The checkpoint file storage method according to claim 2, characterized in that: The comparing the target snapshot with the historical snapshots for similarity, and judging whether the target checkpoint file exists in the target snapshot based on the comparison result, includes: When the target snapshot is inconsistent with the historical snapshot, it is determined that the target checkpoint file exists in the target snapshot, and an incremental file of the target snapshot compared with the historical snapshot is determined as the target checkpoint file.

4. The checkpoint file storage method according to claim 2, characterized in that: The comparing the target snapshot with the historical snapshots for similarity, and judging whether the target checkpoint file exists in the target snapshot based on the comparison result, includes: When the target snapshot is consistent with the historical snapshot, it is determined that the target checkpoint file does not exist in the target snapshot.

5. The checkpoint file storage method according to claim 2, characterized in that: After saving the target checkpoint file to the shared storage when the target checkpoint file exists, the method further includes: Determining whether the target checkpoint file is successfully saved to the shared storage; If the judgment is no, the target snapshot is deleted.

6. The checkpoint file storage method according to claim 1, characterized in that: The step of obtaining the target container group includes: The target container group is obtained from multiple container groups based on a preset interception condition through a network hook, wherein the preset interception condition is to intercept the container group with the target label, and the container group with the target label is the container group that needs to execute the model training task.

7. The checkpoint file storage method according to claim 1, characterized in that: Before obtaining the target container group, the method further includes: A target image is constructed, wherein the target image includes all components that run the auxiliary container to listen to the cache directory and save the obtained checkpoint file to the shared storage.

8. The checkpoint file storage method according to claim 7, characterized in that: The adding of an auxiliary container in the target container group includes: Adding the auxiliary container to the target container group; Setting the startup image of the auxiliary container to the target image; The auxiliary container is mounted on the cache directory.

9. The checkpoint file storage method according to any one of claims 1 to 8, characterized in that: After saving the checkpoint file to the shared storage, the method further includes: Monitor the target container group through a container group listener; When the main container completes the model training task, query whether the checkpoint file in the cache directory has been saved to the shared storage; When the checkpoint file has been saved to the shared storage, the target container group is switched to an end state.

10. A checkpoint file storage device, characterized in that: include: An acquisition module, used to acquire a target container group, wherein the target container group includes a main container, and the main container is used to execute a model training task; An adding module, configured to add an auxiliary container to the target container group, wherein the auxiliary container shares a cache directory with the main container; An execution module, configured to monitor the cache directory through the auxiliary container to obtain a checkpoint file, wherein the checkpoint file is used to save parameters generated by the main container during the model training process; A saving module, used for saving the checkpoint file to a shared storage; The step of monitoring the cache directory through the auxiliary container to obtain a checkpoint file includes: Monitoring the to-be-confirmed checkpoint file in the cache directory by the auxiliary container, and determining whether the to-be-confirmed checkpoint file changes within a target time period; If the judgment is no, confirming the to-be-confirmed checkpoint file as the checkpoint file; The monitoring the to-be-confirmed checkpoint file in the cache directory by the auxiliary container and determining whether the to-be-confirmed checkpoint file changes within a target time period includes: Detecting the file size of the checkpoint file to be confirmed in the cache directory by the auxiliary container, and determining that the checkpoint file to be confirmed has changed if the file size of the checkpoint file to be confirmed has changed within the target time period; or The auxiliary container detects the MD5 value of the checkpoint file to be confirmed in the cache directory, and when the MD5 value of the checkpoint file to be confirmed changes within the target time period, it is determined that the checkpoint file to be confirmed changes.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the checkpoint file saving method according to any one of claims 1 to 9 is implemented.

12. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the checkpoint file saving method according to any one of claims 1 to 9 is implemented.

13. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the checkpoint file saving method according to any one of claims 1 to 9 is implemented.

Images