Checkpoint cache control method and device, electronic equipment and readable medium

Abstract

The invention provides a check point cache control method and device, electronic equipment and a readable medium, and the method comprises the steps: when a check point request is generated, an acceleration service receives current check point data of a corresponding training task main control container, applies for a memory space corresponding to the capacity of the current check point data from a host machine, and sends the memory space to the host machine; the method comprises the following steps: storing current check point data into a memory space through a storage queue, mapping the latest current check point data in the memory space into a loading queue, asynchronously loading the current check point data into a storage, and after an asynchronous loading process is finished, releasing memory occupation except the latest check point data of the loading queue. According to the method, the hit rate of the cache is increased, the time overhead and the memory overhead are greatly reduced, the storage efficiency of the AI training task Checkpoint is improved, and the fault tolerance of the training task is indirectly improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence computing technology, and in particular to a checkpoint cache control method, a checkpoint cache control device, an electronic device and a computer-readable medium. Background Art

[0002] In the field of AI computing, checkpoint (ckpt) technology is widely used to save and restore the intermediate state of computing tasks to cope with interruptions and failures in long-running tasks. Some optimization solutions for ckpt in the industry consider the acceleration of saveCheckpoint and loadCheckpoint respectively. For example, the acceleration of loadCheckpoint mostly uses Fluid-based distributed cache clusters, while the acceleration of saveCheckpoint mostly uses asynchronous disk acceleration based on memory mapping.

[0003] Based on the above technical status, the use of ckpt acceleration cache has the following problems:

[0004] Performance limitation: The connection point between the saving and loading phases is the storage, so the minimum time for saving and restoring a ckpt includes the time loss of storage and reading, and the time overhead of storage is relatively large.

[0005] Low cache hit rate: The limitation of distributed cache cluster is that the cache is discretely distributed in different hosts. When loading the checkpoint cache, a miss may occur.

[0006] Large memory overhead: When using a distributed cache cluster as a cache pool, for the sake of high availability and scalability, the architecture becomes cumbersome and the cache is redundant, resulting in more memory overhead. Summary of the invention

[0007] In view of the above problems, the present invention is proposed to provide a checkpoint cache control method and a corresponding checkpoint cache control device, an electronic device and a computer-readable medium that overcome the above problems or at least partially solve the above problems.

[0008] The present invention discloses a checkpoint cache control method, wherein each training task master control container is bound to a corresponding acceleration service, the acceleration service manages a save queue and a load queue, and is used to provide a cache service for saving and loading checkpoint data for the corresponding training task master control container, the acceleration service is configured with a preset initial memory and a preset maximum memory, and the method comprises:

[0009] When a checkpoint request is generated, the acceleration service receives the current checkpoint data of the corresponding training task master container;

[0010] Apply to the host for memory space corresponding to the capacity of the current checkpoint data;

[0011] The current checkpoint data is saved to the memory space through the save queue, and the latest current checkpoint data in the memory space is mapped to the load queue, and the current checkpoint data is asynchronously written to the disk to the storage; the load queue continuously maintains a copy of the latest checkpoint data;

[0012] After the asynchronous disk flushing process is completed, the memory occupied by the loading queue except for the latest checkpoint data is released.

[0013] Optionally, the steps of saving the current checkpoint data to the memory space through the save queue, mapping the latest current checkpoint data in the memory space to the load queue, and asynchronously dropping the current checkpoint data to the storage include:

[0014] The current checkpoint data is saved to the memory space through the save queue, and the latest current checkpoint data in the memory space is mapped to the load queue, and the current checkpoint data is asynchronously stored in the disk in a first-in-first-out manner.

[0015] Optionally, the method further includes:

[0016] If a second checkpoint request occurs before the asynchronous disk flushing process ends, the acceleration service determines whether the sum of the currently requested memory space size and the already requested memory space size exceeds the preset maximum memory of the acceleration service;

[0017] If the sum of the currently applied memory space size and the already applied memory space size does not exceed the preset maximum memory of the acceleration service, the acceleration service continues to apply for memory space from the host machine;

[0018] If the sum of the currently requested memory space size and the already requested memory space size exceeds the preset maximum memory of the acceleration service, the checkpoint request is rejected.

[0019] Optionally, the method further comprises:

[0020] When a training task fails, the acceleration service responds to the call of the training task master container and the request to obtain the current checkpoint data, and checks whether the loading queue saves the current checkpoint data;

[0021] If the loading queue stores the current checkpoint data, the current checkpoint data stored in the loading queue is sent to the training task master container;

[0022] If the loading queue does not save the current checkpoint data, the current checkpoint data is queried from the storage through the loading queue and loaded into the memory cache, and the current checkpoint data cached in the memory is sent to the training task master container.

[0023] Optionally, the acceleration service corresponding to the training task master container is configured with an alternative acceleration service, and the method further includes:

[0024] When caching the current checkpoint data, the acceleration service corresponding to the training task master container synchronizes the current checkpoint data to the loading queue of the alternative acceleration service;

[0025] When the cluster node where the training task master container is located goes down, the cluster management system schedules the training task master container to the disaster recovery node where the alternative acceleration service is located and starts the training task master container on the disaster recovery node;

[0026] The alternative acceleration service sends the current checkpoint data saved by the loading queue to the training task master container on the disaster recovery node for resuming the execution of the training task.

[0027] The present invention also discloses a checkpoint cache control device, each training task master container is bound to a corresponding acceleration service, the acceleration service manages and controls the save queue and the load queue, and is used to provide a cache service for saving and loading checkpoint data for the corresponding training task master container, the acceleration service is configured with a preset initial memory and a preset maximum memory, and the method includes:

[0028] A receiving module, used for accelerating the service to receive the current checkpoint data of the corresponding training task master container when a checkpoint request is generated;

[0029] A memory application module is used to apply to the host machine for memory space corresponding to the capacity of the current checkpoint data;

[0030] The memory mapping and asynchronous disk write module is used to save the current checkpoint data to the memory space through the save queue, and map the latest current checkpoint data in the memory space to the load queue, and asynchronously write the current checkpoint data to the storage; the load queue continuously maintains a copy of the latest checkpoint data;

[0031] The memory release module is used to release the memory occupied by the loading queue except for the latest checkpoint data after the asynchronous disk placement process is completed.

[0032] Optionally, the memory mapping and asynchronous disk storage module includes:

[0033] The memory mapping and asynchronous disk placement module is used to save the current checkpoint data to the memory space through the save queue, map the latest current checkpoint data in the memory space to the load queue, and asynchronously store the current checkpoint data in a first-in-first-out manner.

[0034] Optionally, the device further comprises:

[0035] The used memory judgment module is used to judge whether the sum of the currently applied memory space size and the applied memory space size exceeds the preset maximum memory of the acceleration service if a second checkpoint request occurs before the asynchronous disk placement process ends;

[0036] The checkpoint request module is executed to ensure that if the sum of the currently applied memory space size and the already applied memory space size does not exceed the preset maximum memory of the acceleration service, the acceleration service continues to apply for memory space from the host machine;

[0037] The checkpoint request rejection module is used to reject the checkpoint request if the sum of the currently applied memory space size and the applied memory space size exceeds the preset maximum memory of the acceleration service.

[0038] Optionally, the device further comprises:

[0039] The check module is used to accelerate the service response to the call of the training task master container and the current checkpoint data acquisition request when a training task fails, and check whether the loading queue saves the current checkpoint data;

[0040] The loading queue loading module is used to send the current checkpoint data saved in the loading queue to the training task master container if the loading queue saves the current checkpoint data;

[0041] The storage loading module is used to query the current checkpoint data from the storage through the loading queue and load it into the memory cache if the loading queue does not save the current checkpoint data, and send the current checkpoint data cached in the memory to the training task master container.

[0042] Optionally, the acceleration service corresponding to the training task master container is configured with an alternative acceleration service, and the device further includes:

[0043] The synchronization module is used to synchronize the current checkpoint data to the loading queue of the alternative acceleration service when the current checkpoint data is cached.

[0044] The alternative acceleration service startup module is used to schedule the training task master container to the disaster recovery node where the alternative acceleration service is located and start the training task master container on the disaster recovery node when the cluster node where the training task master container is located goes down;

[0045] The loading data recovery training module is used for the alternative acceleration service to send the current checkpoint data saved in the loading queue to the training task master container on the disaster recovery node for resuming the execution of the training task.

[0046] The present invention also discloses an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0047] The memory is used to store computer programs;

[0048] The processor is used to implement the checkpoint cache control method of the present invention when executing the program stored in the memory.

[0049] The present invention also discloses one or more computer-readable media, on which instructions are stored. When executed by one or more processors, the processors are enabled to execute the checkpoint cache control method of the present invention.

[0050] The present invention includes the following advantages:

[0051] The checkpoint cache control method of the present invention, when a checkpoint request is generated, accelerates the service to receive the current checkpoint data of the corresponding training task master container, applies to the host machine for memory space corresponding to the capacity of the current checkpoint data, saves the current checkpoint data to the memory space through the save queue, and maps the latest current checkpoint data in the memory space to the loading queue, and asynchronously writes the current checkpoint data to the storage. After the asynchronous write process is completed, the memory occupancy other than the latest checkpoint data of the loading queue is released. This method improves the cache hit rate and greatly reduces time overhead and memory overhead, improves the saving efficiency of the AI ​​training task Checkpoint, and indirectly improves the fault tolerance of the training task. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 is a flow chart of the steps of a checkpoint cache control method provided by an embodiment of the present invention;

[0053] Figure 2 is a flow chart of controlling checkpoint saving and recovery provided by an embodiment of the present invention;

[0054] Figure 3 It is a structural block diagram of a checkpoint cache control device provided by an embodiment of the present invention;

[0055] Figure 4 is a block diagram of an electronic device provided by an embodiment of the present invention;

[0056] Figure 5 It is a schematic diagram of a computer-readable medium provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0057] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0058] Reference Figure 1 , shows a flow chart of the steps of a checkpoint cache control method provided in an embodiment of the present invention, which may specifically include the following steps:

[0059] Step 101, when a checkpoint request is generated, the acceleration service receives the current checkpoint data of the corresponding training task master container;

[0060] Step 102, applying to the host machine for memory space corresponding to the capacity of the current checkpoint data;

[0061] Step 103, save the current checkpoint data to the memory space through the save queue, map the latest current checkpoint data in the memory space to the load queue, and asynchronously write the current checkpoint data to the storage; the load queue continuously maintains a copy of the latest checkpoint data;

[0062] Step 104, after the asynchronous disk write process is completed, the memory occupied by the loading queue except for the latest checkpoint data is released.

[0063] The embodiment of the present invention implements an integrated checkpoint data cache control technology, which realizes cache reuse and optimal path of checkpoint data ckpt by jointly maintaining cache space of the saving link and the loading link; and by binding the training task master control container Master Pod and the acceleration service one by one, it is scheduled in groups to form a dedicated cache provided for the training task, and the cache pool and the business binding using the cache can reduce cache redundancy and improve cache hit rate. And through a flexible memory allocation method, on-demand memory allocation can be achieved, greatly reducing memory overhead.

[0064] Specifically, refer to Figure 1The present invention configures an exclusive acceleration service for each training task master container to provide a cache service for saving and loading ckpt. The acceleration service maintains a memory acceleration queue and dynamically maintains the number of ckpt requests in the queue according to the memory size. That is, the acceleration service can control the number of checkpoint requests for saving the queue within its initial memory and maximum memory range. When a checkpoint request is generated, the acceleration service determines whether its maximum memory has been reached. If it has not been reached, the checkpoint request can be added to the save queue for execution. If it has been reached, the checkpoint request can be rejected. The user only needs to configure the initial memory and maximum memory for the acceleration service, and the acceleration service will adaptively adjust the memory footprint within the memory range. Specifically, the user declares the acceleration service, the kubernetes cluster deploys the acceleration service, and sets the Limit value to be greater than the Request value. When a ckpt request is generated, the acceleration service applies for the corresponding memory space from the host machine, saves the current checkpoint data to the memory space through the save queue, and maps the latest current checkpoint data in the memory space to the loading queue, so that the loading queue always maintains an up-to-date ckpt resource, and starts the asynchronous disk placement process. After the asynchronous disk placement process ends, only a copy of the latest checkpoint data in the loading queue is retained as cache data for subsequent recovery, releasing all other memory usage. Therefore, the method of the present invention greatly reduces the time overhead, referring to Figure 1 , the storage time of checkpoint data is directly shortened from a1+a2 to a1. There is no need to wait until the checkpoint data is stored on disk before resuming the execution of the training task. It only takes a1 time to save the mapping before loading and resuming the execution of the training task from the loading queue, reducing the interruption time of the business. And normalization only occupies the memory cache space size of one ckpt, and the hit rate is 100%. Compared with traditional distributed caching solutions, it can achieve memory savings and accurate hits. In addition, the acceleration service can allocate memory on demand, which also reduces memory overhead.

[0065] In one embodiment of the present invention, the steps of saving the current checkpoint data to the memory space through the save queue, mapping the latest current checkpoint data in the memory space to the load queue, and asynchronously dropping the current checkpoint data to the storage include:

[0066] The current checkpoint data is saved to the memory space through the save queue, and the latest current checkpoint data in the memory space is mapped to the load queue, and the current checkpoint data is asynchronously stored in the disk in a first-in-first-out manner.

[0067] Due to the placement time (refer to Figure 1The a2 link) is relatively slow in writing to the memory. In the embodiment of the present invention, the preservation queue asynchronously writes the memory data to the storage in a first-in-first-out manner. Through the first-in-first-out asynchronous writing, data can be processed continuously and efficiently without waiting for the response of the storage device, thereby reducing time delays. The loading of checkpoint data can also be carried out at the same time, which can reduce the delay caused by waiting for all data to be written to the disk.

[0068] In one embodiment of the present invention, the method further includes:

[0069] If a second checkpoint request occurs before the asynchronous disk flushing process ends, the acceleration service determines whether the sum of the currently requested memory space size and the already requested memory space size exceeds the preset maximum memory of the acceleration service;

[0070] If the sum of the currently applied memory space size and the already applied memory space size does not exceed the preset maximum memory of the acceleration service, the acceleration service continues to apply for memory space from the host machine;

[0071] If the sum of the currently requested memory space size and the already requested memory space size exceeds the preset maximum memory of the acceleration service, the checkpoint request is rejected.

[0072] In an embodiment of the present invention, if a second ckpt request occurs before the asynchronous disk drop process ends, the acceleration service can continue to apply for memory from the host machine, but before that, it can first determine whether the sum of the currently applied memory space size and the already applied memory space size exceeds the preset maximum memory of the acceleration service. If the sum of the currently applied memory space size and the already applied memory space size does not exceed the preset maximum memory of the acceleration service, the acceleration service can continue to apply for memory space from the host machine; if the sum of the currently applied memory space size and the already applied memory space size exceeds the preset maximum memory of the acceleration service, the checkpoint request can be rejected to achieve control of the acceleration service to adaptively adjust the memory occupied space size within the memory range.

[0073] In one embodiment of the present invention, the method further includes:

[0074] When a training task fails, the acceleration service responds to the call of the training task master container and the request to obtain the current checkpoint data, and checks whether the loading queue saves the current checkpoint data;

[0075] If the loading queue stores the current checkpoint data, the current checkpoint data stored in the loading queue is sent to the training task master container;

[0076] If the loading queue does not save the current checkpoint data, the current checkpoint data is queried from the storage through the loading queue and loaded into the memory cache, and the current checkpoint data cached in the memory is sent to the training task master container.

[0077] In the embodiment of the present invention, when a training task fails, it is necessary to load the checkpoint data ckpt to recover the task. The training task master container calls the interface of the acceleration service to obtain ckpt. The acceleration service first returns the latest checkpoint data saved in the loading queue. Therefore, the method of the embodiment of the present invention greatly reduces the time overhead. Figure 1 , the time for saving and restoring checkpoint data is directly shortened from a1+a2+b1+b2 to a1+b1, that is, the time required to resume the execution of the training task is directly shortened to a1+b1. There is no need to wait until the checkpoint data is stored on disk before resuming the execution of the training task. It only takes a1 time to save the mapping before loading and resuming the execution of the training task from the loading queue, thus reducing the interruption time to the business. If there is no latest checkpoint data cache in the loading queue, the loading queue automatically queries the latest checkpoint data from the storage, loads it into the memory for cache, and then returns it to the training task master container, and provides acceleration capabilities for subsequent loading requests.

[0078] In one embodiment of the present invention, the method further includes:

[0079] When caching the current checkpoint data, the acceleration service corresponding to the training task master container synchronizes the current checkpoint data to the loading queue of the alternative acceleration service;

[0080] When the cluster node where the training task master container is located goes down, the cluster management system schedules the training task master container to the disaster recovery node where the alternative acceleration service is located and starts the training task master container on the disaster recovery node;

[0081] The alternative acceleration service sends the current checkpoint data saved by the loading queue to the training task master container on the disaster recovery node for resuming the execution of the training task.

[0082] For the scenario of node downtime, the present invention maintains the latest ckpt data by deploying an alternative acceleration service of a disaster recovery node. When the node where the training task master container Master Pod is located fails, the migration and ckpt loading can be completed at the fastest speed. In other words, an alternative acceleration service paired with the acceleration service can be maintained in other available nodes. The alternative acceleration service is dedicated to maintaining the latest cached data of a cross-machine checkpoint for efficient recovery in the event of downtime.

[0083] Specifically, the acceleration service will select an alternative acceleration service from other nodes, and always synchronize the latest checkpoint data to the loading queue of the alternative acceleration service. When the original node goes down, Kubernetes schedules the training task master container Master Pod to the node where the alternative acceleration service is located. After the new training task master container MasterPod is started, it obtains the latest cached checkpoint data from the local alternative acceleration service and resumes training. The embodiment of the present invention can improve the ckpt recovery time after downtime migration and achieve accurate recovery by pre-setting disaster recovery cache nodes.

[0084] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0085] Reference Figure 3 , shows a structural block diagram of a checkpoint cache control device provided in an embodiment of the present invention, which may specifically include the following modules:

[0086] The receiving module 301 is used for the acceleration service to receive the current checkpoint data of the corresponding training task master container when a checkpoint request is generated;

[0087] A memory application module 302 is used to apply to the host machine for memory space corresponding to the capacity of the current checkpoint data;

[0088] The memory mapping and asynchronous disk write module 303 is used to save the current checkpoint data to the memory space through the save queue, and map the latest current checkpoint data in the memory space to the load queue, and asynchronously write the current checkpoint data to the storage; the load queue continuously maintains a copy of the latest checkpoint data;

[0089] The memory release module 304 is used to release the memory occupied by the loading queue except for the latest checkpoint data after the asynchronous disk placement process is completed.

[0090] Optionally, the memory mapping and asynchronous disk storage module includes:

[0091] The memory mapping and asynchronous disk placement module is used to save the current checkpoint data to the memory space through the save queue, map the latest current checkpoint data in the memory space to the load queue, and asynchronously store the current checkpoint data in a first-in-first-out manner.

[0092] Optionally, the device further comprises:

[0093] The used memory judgment module is used to judge whether the sum of the currently applied memory space size and the applied memory space size exceeds the preset maximum memory of the acceleration service if a second checkpoint request occurs before the asynchronous disk placement process ends;

[0094] The checkpoint request module is executed to ensure that if the sum of the currently applied memory space size and the already applied memory space size does not exceed the preset maximum memory of the acceleration service, the acceleration service continues to apply for memory space from the host machine;

[0095] The checkpoint request rejection module is used to reject the checkpoint request if the sum of the currently applied memory space size and the applied memory space size exceeds the preset maximum memory of the acceleration service.

[0096] Optionally, the device further comprises:

[0097] The check module is used to accelerate the service response to the call of the training task master container and the current checkpoint data acquisition request when a training task fails, and check whether the loading queue saves the current checkpoint data;

[0098] The loading queue loading module is used to send the current checkpoint data saved in the loading queue to the training task master container if the loading queue saves the current checkpoint data;

[0099] The storage loading module is used to query the current checkpoint data from the storage through the loading queue and load it into the memory cache if the loading queue does not save the current checkpoint data, and send the current checkpoint data cached in the memory to the training task master container.

[0100] Optionally, the acceleration service corresponding to the training task master container is configured with an alternative acceleration service, and the device further includes:

[0101] The synchronization module is used to synchronize the current checkpoint data to the loading queue of the alternative acceleration service when the current checkpoint data is cached.

[0102] The alternative acceleration service startup module is used to schedule the training task master container to the disaster recovery node where the alternative acceleration service is located and start the training task master container on the disaster recovery node when the cluster node where the training task master container is located goes down;

[0103] The loading data recovery training module is used for the alternative acceleration service to send the current checkpoint data saved in the loading queue to the training task master container on the disaster recovery node for resuming the execution of the training task.

[0104] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0105] In addition, an embodiment of the present invention further provides an electronic device, such as Figure 4 As shown, it includes a processor 401, a communication interface 402, a memory 403 and a communication bus 404, wherein the processor 401, the communication interface 402, and the memory 403 communicate with each other through the communication bus 404.

[0106] Memory 403, used for storing computer programs;

[0107] The processor 401 is used to implement the checkpoint cache control method described in the above embodiment when executing the program stored in the memory 403.

[0108] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0109] The communication interface is used for communication between the above terminal and other devices.

[0110] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0111] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0112] like Figure 5 As shown, in another embodiment provided by the present invention, a computer-readable storage medium 501 is also provided, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer executes the checkpoint cache control method described in the above embodiment.

[0113] In another embodiment of the present invention, a computer program product including instructions is provided. When the computer program product is executed on a computer, the computer executes the checkpoint cache control method described in the above embodiment.

[0114] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0115] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0116] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0117] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A checkpoint cache control method, characterized in that: Each training task master container is bound to a corresponding acceleration service, the acceleration service manages the save queue and the load queue, and is used to provide a cache service for saving and loading checkpoint data for the corresponding training task master container, the acceleration service is configured with a preset initial memory and a preset maximum memory, and the method includes: When a checkpoint request is generated, the acceleration service receives the current checkpoint data of the corresponding training task master container; Apply to the host for memory space corresponding to the capacity of the current checkpoint data; The current checkpoint data is saved to the memory space through the save queue, and the latest current checkpoint data in the memory space is mapped to the load queue, and the current checkpoint data is asynchronously written to the disk to the storage; the load queue continuously maintains a copy of the latest checkpoint data; After the asynchronous disk flushing process is completed, the memory occupied by the loading queue except for the latest checkpoint data is released.

2. The method according to claim 1, characterized in that The steps of saving the current checkpoint data to the memory space through the save queue, mapping the latest current checkpoint data in the memory space to the load queue, and asynchronously dropping the current checkpoint data to the storage include: The current checkpoint data is saved to the memory space through the save queue, and the latest current checkpoint data in the memory space is mapped to the load queue, and the current checkpoint data is asynchronously stored in the disk in a first-in-first-out manner.

3. The method according to claim 1, characterized in that The method further comprises: If a second checkpoint request occurs before the asynchronous disk flushing process ends, the acceleration service determines whether the sum of the currently requested memory space size and the already requested memory space size exceeds the preset maximum memory of the acceleration service; If the sum of the currently applied memory space size and the already applied memory space size does not exceed the preset maximum memory of the acceleration service, the acceleration service continues to apply for memory space from the host machine; If the sum of the currently requested memory space size and the already requested memory space size exceeds the preset maximum memory of the acceleration service, the checkpoint request is rejected.

4. The method according to claim 1, characterized in that: The method further comprises: When a training task fails, the acceleration service responds to the call of the training task master container and the request to obtain the current checkpoint data, and checks whether the loading queue saves the current checkpoint data; If the loading queue stores the current checkpoint data, the current checkpoint data stored in the loading queue is sent to the training task master container; If the loading queue does not save the current checkpoint data, the current checkpoint data is queried from the storage through the loading queue and loaded into the memory cache, and the current checkpoint data cached in the memory is sent to the training task master container.

5. The method according to claim 1, characterized in that The acceleration service corresponding to the training task master container is configured with an alternative acceleration service, and the method further includes: When caching the current checkpoint data, the acceleration service corresponding to the training task master container synchronizes the current checkpoint data to the loading queue of the alternative acceleration service; When the cluster node where the training task master container is located goes down, the cluster management system schedules the training task master container to the disaster recovery node where the alternative acceleration service is located and starts the training task master container on the disaster recovery node; The alternative acceleration service sends the current checkpoint data saved by the loading queue to the training task master container on the disaster recovery node for resuming the execution of the training task.

6. A checkpoint cache control device, characterized in that: Each training task master container is bound to a corresponding acceleration service, the acceleration service manages the save queue and the load queue, and is used to provide a cache service for saving and loading checkpoint data for the corresponding training task master container, the acceleration service is configured with a preset initial memory and a preset maximum memory, and the method includes: A receiving module, used for accelerating the service to receive the current checkpoint data of the corresponding training task master container when a checkpoint request is generated; A memory application module is used to apply to the host machine for memory space corresponding to the capacity of the current checkpoint data; The memory mapping and asynchronous disk write module is used to save the current checkpoint data to the memory space through the save queue, and map the latest current checkpoint data in the memory space to the load queue, and asynchronously write the current checkpoint data to the storage; the load queue continuously maintains a copy of the latest checkpoint data; The memory release module is used to release the memory occupied by the loading queue except for the latest checkpoint data after the asynchronous disk placement process is completed.

7. The device according to claim 6, characterized in that The memory mapping and asynchronous disk storage module includes: The memory mapping and asynchronous disk placement module is used to save the current checkpoint data to the memory space through the save queue, map the latest current checkpoint data in the memory space to the load queue, and asynchronously store the current checkpoint data in a first-in-first-out manner.

8. The device according to claim 6, characterized in that The device also includes: The used memory judgment module is used to judge whether the sum of the currently applied memory space size and the applied memory space size exceeds the preset maximum memory of the acceleration service if a second checkpoint request occurs before the asynchronous disk placement process ends; The checkpoint request module is executed to ensure that if the sum of the currently applied memory space size and the already applied memory space size does not exceed the preset maximum memory of the acceleration service, the acceleration service continues to apply for memory space from the host machine; The checkpoint request rejection module is used to reject the checkpoint request if the sum of the currently applied memory space size and the applied memory space size exceeds the preset maximum memory of the acceleration service.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; The memory is used to store computer programs; The processor is used to implement the checkpoint cache control method according to any one of claims 1 to 5 when executing the program stored in the memory.

10. One or more computer-readable media having instructions stored thereon, which, when executed by one or more processors, enable the processors to execute the checkpoint cache control method according to any one of claims 1 to 5.

Images