Method for backing up training snapshot of model and method for migrating training snapshot of model
By backing up training snapshots of the model in the memory of the training node, the problem of synchronous operations affecting training performance is solved, fast recovery and efficient training are achieved, and repeated calculation overhead is significantly reduced.
Patent Information
- Application Number
- CN202411855824.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-16
- Publication Date
- 2025-04-15
AI Technical Summary
In the prior art, the training performance is affected due to synchronization operations during the training snapshot backup process of the model, which extends the training interrupt time.
By backing up training snapshots in the memory of the training node, the impact of synchronization operations on training performance is avoided, and rapid migration is achieved when the training node fails, shortening the training interrupt time.
Improves the training efficiency of the model, significantly reduces the overhead of repeated calculations, and achieves rapid recovery when training nodes fail.
Smart Images
Figure CN119829330A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of computer technology, specifically to deep learning and artificial intelligence technology, and in particular to a training snapshot backup method of a model and a training snapshot migration method of a model. Background Art
[0002] In the related art, the training snapshot backup method for the model uses a device to host (Device to Host, abbreviated as D2H) synchronization method and a persistent asynchronous two-stage method to perform training snapshot backup. However, since a large number of operations for saving training snapshots are synchronous, the above method often affects the training performance of the model and prolongs the training interruption time. Summary of the invention
[0003] The present disclosure provides a model training snapshot backup method, a model training snapshot migration method, a model training snapshot backup device, a model training snapshot migration device, an electronic device, a storage medium and a computer program product.
[0004] According to a first aspect of the present disclosure, a method for backing up a training snapshot of a model is provided, which is executed by a first training node and includes: obtaining dynamic parameters to be backed up of a current training step of the model; determining a training snapshot of the current training step based on the dynamic parameters; and backing up the training snapshot to the memory of the first training node.
[0005] According to a second aspect of the present disclosure, a method for migrating training snapshots of a model is provided, which is executed by a first training node, and includes: in response to identifying that the first training node has a fault, determining a target training snapshot to be migrated in the memory of the first training node; and migrating the target training snapshot to the memory of a scheduled second training node.
[0006] According to a third aspect of the present disclosure, a method for migrating a training snapshot of a model is provided, which is executed by a second training node and includes: receiving a target training snapshot migrated from a failed first training node, wherein the target training snapshot is stored in the memory of the first training node; and storing the target training snapshot in the memory of the second training node.
[0007] According to a fourth aspect of the present disclosure, a training snapshot backup device for a model is provided, which is applied to a first training node and includes: an acquisition module for acquiring dynamic parameters to be backed up of a current training step of the model; a determination module for determining the training snapshot of the current training step based on the dynamic parameters; and a backup module for backing up the training snapshot to the memory of the first training node.
[0008] According to a fifth aspect of the present disclosure, a training snapshot migration device for a model is provided, which is applied to a first training node and includes: an identification module for determining a target training snapshot to be migrated in the memory of the first training node in response to identifying that the first training node has a fault; and a migration module for migrating the target training snapshot to the memory of a scheduled second training node.
[0009] According to a sixth aspect of the present disclosure, a training snapshot migration device for a model is provided, which is applied to a second training node and includes: a receiving module for receiving a target training snapshot migrated from a first training node that has a fault, wherein the target training snapshot is stored in the memory of the first training node; and a storage module for storing the target training snapshot in the memory of the second training node.
[0010] According to the seventh aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the training snapshot backup method of the model described in the first aspect of the present disclosure, the training snapshot migration method of the model described in the second aspect, or the training snapshot migration method of the model described in the third aspect.
[0011] According to an eighth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the training snapshot backup method of the model described in the first aspect of the present disclosure, the training snapshot migration method of the model described in the second aspect, or the training snapshot migration method of the model described in the third aspect.
[0012] According to a ninth aspect of the present disclosure, a computer program product is provided, comprising a computer program, characterized in that when the computer program is executed by a processor, the computer program implements the training snapshot backup method of the model described in the first aspect of the present disclosure, the training snapshot migration method of the model described in the second aspect, or the training snapshot migration method of the model described in the third aspect.
[0013] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.
[0015] Figure 1 A flowchart of a training snapshot backup method for a model according to an embodiment of the present disclosure;
[0016] Figure 2 A flowchart of a training snapshot backup method for a model according to an embodiment of the present disclosure;
[0017] Figure 3 A flowchart of a training snapshot backup method for a model according to an embodiment of the present disclosure;
[0018] Figure 4 A flowchart of a training snapshot migration method for a model according to an embodiment of the present disclosure;
[0019] Figure 5 A flowchart of a training snapshot migration method for a model according to an embodiment of the present disclosure;
[0020] Figure 6 A flowchart of a training snapshot migration method for a model according to an embodiment of the present disclosure;
[0021] Figure 7 A schematic diagram of the structure of a training snapshot backup device of a model according to an embodiment of the present disclosure;
[0022] Figure 8 It is a structural schematic diagram of a training snapshot migration device of a model according to an embodiment of the present disclosure;
[0023] Fig. 9 It is a structural schematic diagram of a training snapshot migration device of a model according to an embodiment of the present disclosure;
[0024] Fig.10 A schematic block diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION
[0025] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.
[0026] The following is a brief description of the technical field involved in the solution of the present disclosure:
[0027] The content of computer technology is very broad and can be roughly divided into several aspects such as computer system technology, computer device technology, computer component technology and computer assembly technology. Computer technology includes: the basic principles of operation methods and the design of operators, instruction system, central processing unit design, pipeline principle and its application in the design of central processing unit (CPU), storage system, bus and input and output. Computer is a modern intelligent electronic device with data storage and modification functions, and realizes the calculation of related logic and data. It is an electronic device that integrates network, computing, media and other technologies. Computer technology refers to the technical methods and technical means used in the computer field, or refers to its hardware technology, software technology and application technology. Computer technology has obvious comprehensive characteristics. It is closely integrated with electronic engineering, applied physics, mechanical engineering, modern communication technology, mathematics, etc., and has developed rapidly.
[0028] Artificial Intelligence (AI) is a discipline that studies how computers can simulate certain thought processes and intelligent behaviors of human beings (such as learning, reasoning, thinking, planning, etc.). It includes both hardware-level technologies and software-level technologies. AI hardware technologies generally include computer vision technology, speech recognition technology, natural language processing technology, as well as learning / deep learning, big data processing technology, knowledge graph technology, and other aspects.
[0029] Deep Learning (DL) is a new research direction in the field of Machine Learning (ML). It is introduced into machine learning to make it closer to its original goal - artificial intelligence. Deep learning is the internal laws and representation levels of learning sample data. The information obtained in the learning process is of great help in the interpretation of data such as text, images and sounds. Its ultimate goal is to enable machines to have analytical learning capabilities like humans and to be able to recognize data such as text, images and sounds. Deep learning is a complex machine learning algorithm that has achieved results in speech and image recognition that far exceed previous related technologies.
[0030] The following describes a model training snapshot backup method and a model training snapshot migration method according to an embodiment of the present disclosure with reference to the accompanying drawings.
[0031] Figure 1 FIG. 1 is a flow chart of a training snapshot backup method of a model according to an embodiment of the present disclosure. Figure 1 As shown, the training snapshot backup method of the model proposed in this embodiment is executed by the first training node and includes the following steps:
[0032] S101. Obtain the dynamic parameters to be backed up in the current training step of the model.
[0033] It should be noted that if a training task fails during the training of the model, the dynamic parameters of the unsaved training steps cannot contribute effective computing time. The dynamic parameters of the unsaved training steps will be recalculated after the training task returns to normal, which generates repeated computing overhead. Therefore, the present invention can obtain the backup dynamic parameters of the current training step of the model to realize the training snapshot backup of the subsequent model, thereby minimizing the repeated computing overhead of the model.
[0034] It should be noted that during the training process of the model, each training step of the model will generate multiple dynamic parameters. Based on the multiple dynamic parameters generated by the current training step of the model, the dynamic parameters to be backed up in the current training step of the model can be determined.
[0035] It should be noted that the present disclosure does not limit the type of dynamic parameters to be backed up in the current training step of the model, and can be selected according to actual conditions.
[0036] Optionally, the dynamic parameters to be backed up in the current training step of the model may be model parameters; optionally, the dynamic parameters to be backed up in the current training step of the model may be model parameters and model training status.
[0037] For example, the model parameters may be the weight values of each network layer in the model and the bias values of each network layer in the model, and the model training status may be the current training steps and learning rate of the model.
[0038] S102: Determine a training snapshot of a current training step according to the dynamic parameters.
[0039] The training snapshot of the current training step refers to the recording and saving of the state of the current training step of the model during the model training process.
[0040] In the disclosed embodiment, after the dynamic parameters are acquired, a training snapshot of the current training step may be constructed according to the dynamic parameters.
[0041] Optionally, the dynamic parameters of the current training step may be summarized by type to determine a training snapshot of the current training step.
[0042] S103: Back up the training snapshot to the memory of the first training node.
[0043] In an embodiment of the present disclosure, a target backup process corresponding to a current training step may be determined, and the target backup process may be executed to back up a training snapshot to a memory of a first training node, wherein the target backup process is one of a first backup process and a second backup process.
[0044] For example, if the target backup process is the first backup process, a single backup is performed on the dynamic parameters to obtain the first training snapshot, and the first training snapshot is written to the mount abstraction layer of the first training node; if the target backup process is the second backup process, a double backup is performed on the dynamic parameters to obtain the first training snapshot and the second training snapshot, the mount abstraction layer of the first training node writes the first training snapshot, and the second training snapshot is written to the remote storage cluster through the persistent medium of the first training node.
[0045] According to the training snapshot backup method of the model in the embodiment of the present disclosure, the dynamic parameters to be backed up of the current training step of the model are obtained, and the training snapshot of the current training step is determined based on the dynamic parameters, and the training snapshot is backed up to the memory of the first training node. Therefore, the present disclosure can avoid the impact on the training performance by backing up the training snapshot to the memory of the training node, so that the model training node can be restored in the most efficient way when a failure occurs, shortening the time of training interruption, significantly reducing the overhead of repeated calculations, and improving the training efficiency of the model.
[0046] Figure 2 It is a flowchart of a training snapshot backup method of a model according to an embodiment of the present disclosure.
[0047] like Figure 2 As shown, the training snapshot backup method of the model proposed in this embodiment includes the following steps:
[0048] S201. Obtain the dynamic parameters to be backed up in the current training step of the model.
[0049] For the relevant contents of step S201, please refer to the above embodiment, which will not be described again here.
[0050] S202: Determine a target backup process corresponding to the current training step, where the target backup process is one of a first backup process and a second backup process.
[0051] The first backup process may be a high-speed checkpoint (Flash Checkpoint, FC for short), which backs up the training snapshots to the memory of the first training node one training step at a time.
[0052] The second backup process may be a stable checkpoint (SC for short), which backs up the training snapshot to the memory of the first training node at regular training steps.
[0053] The first backup process and the second backup process are compatible and complementary.
[0054] It should be noted that the present disclosure does not limit the specific method for determining the target backup process corresponding to the current training step, and it can be selected according to actual conditions.
[0055] Optionally, a preset training step interval of the second backup process may be obtained, and according to the preset training step interval, the training steps to be adopted in the second backup process are determined, and the remaining training steps adopt the first backup process.
[0056] For example, the second backup process backs up the training snapshot to the memory of the first training node every 50 training steps. The target backup process corresponding to the 1st to 49th training steps is the first backup process, the target backup process corresponding to the 50th training step is the second backup process, and so on.
[0057] S203: The target backup process is the first backup process, and a single backup is performed on the dynamic parameters to obtain a first training snapshot.
[0058] In the disclosed embodiment, when it is determined that the target backup process corresponding to the current training step is the first backup process, a single backup is performed on the dynamic parameters to obtain a first training snapshot.
[0059] S204. Write the first training snapshot to the mount abstraction layer of the first training node.
[0060] It should be noted that the present disclosure abstracts a layer of file system expression, namely, the mount abstraction layer, so that the training snapshot (file) written to a specific memory block can continue to be accessed through the standard Portable Operating System Interface of UNIX (POSIX). At the same time, the mount abstraction layer can also ensure that the training snapshot is in a state of migration preparation.
[0061] In an embodiment of the present disclosure, after the first training snapshot is acquired, the first training snapshot may be written to the mount abstraction layer of the first training node.
[0062] In an embodiment of the present disclosure, in order to more accurately obtain the status of each link of the backup process, each link of the first backup process can be monitored for abnormalities, and in response to the failure of one of the links, a first indication message is sent to the fault-tolerant platform, wherein the first indication message is used to instruct the fault-tolerant platform to perform at least one of the following operations: marking the failure of the first backup process of the current training step of the model and issuing a first alarm message.
[0063] It should be noted that the present disclosure does not limit the first warning information. Optionally, the first warning information may be text warning information.
[0064] In the present embodiment, after writing the first training snapshot to the mount abstraction layer of the first training node, a third indication message is sent to the fault-tolerant platform, and the third indication message is used to instruct the fault-tolerant platform to perform at least one of the following operations: updating the first backup list corresponding to the first backup process, clearing the expired first training snapshot and the second training snapshot corresponding to the expired first training snapshot.
[0065] S205 , the target backup process is the second backup process, and a double backup is performed on the dynamic parameters to obtain a first training snapshot and a second training snapshot.
[0066] In the disclosed embodiment, when it is determined that the target backup process corresponding to the current training step is the second backup process, a double backup is performed on the dynamic parameters to obtain a first training snapshot and a second training snapshot.
[0067] S206. Write the first training snapshot to the mount abstraction layer of the first training node.
[0068] S207: Write the second training snapshot to the remote storage cluster through the persistent medium of the first training node.
[0069] In the embodiment of the present disclosure, after the second training snapshot is obtained, the second training snapshot can be written to the persistent medium of the first training node, and the second training snapshot can be written to the remote storage cluster through the persistent medium of the first training node.
[0070] In the embodiment of the present disclosure, in order to more accurately obtain the status of each link of the backup process, each link of the second backup process can be monitored for abnormalities. In response to the failure of one of the links, at least one of the following operations is performed: exiting the training process, sending a second indication message to the fault-tolerant platform, and the second indication message is used to instruct the second backup process to send a second alarm message.
[0071] It should be noted that the present disclosure does not limit the second alarm information. Optionally, the second alarm information may be telephone alarm information.
[0072] In an embodiment of the present disclosure, after writing the second training snapshot to the remote storage cluster, a fourth indication information may be sent to the fault-tolerant platform, and the fourth indication information is used to instruct the fault-tolerant platform to perform at least one of the following operations: updating the second backup list corresponding to the second backup process, storing the second backup list to the remote server, and clearing the expired first training snapshot and the second training snapshot corresponding to the expired first training snapshot.
[0073] It should be noted that in the related art, when backing up training snapshots, the training snapshot process backup is often implemented through the Vanilla Checkpoint (VC) backup process, wherein the training snapshots in the training container are often saved and backed up at certain time intervals. There are a large number of synchronous operations in the backup process, which introduces time consumption. At the same time, in order to migrate the training snapshots when the training node fails, the backup process needs to be persisted through the disk and the back-end storage cluster. Through asynchronous transformation, the time-consuming overhead limits the upper limit of the triggering frequency. In particular, when the back-end storage cluster uses the Transmission Control Protocol (TCP) service, the persistence of a training snapshot will take nearly 10 minutes. After the failed training node goes offline and the new normal training node goes online, the training process needs to pull the corresponding training snapshot from the back-end storage cluster through the platform for deployment, which introduces time consumption.
[0074] The specific process of the training snapshot backup method of the model in the embodiment of the present disclosure is explained below.
[0075] For example, Figure 3As shown, the dynamic parameters to be backed up of the current training step of the model are obtained, and it is determined whether the target backup process corresponding to the current training step is the second backup process SC. If the target backup process corresponding to the current training step is not the second backup process SC, it is the first backup process FC. A single backup of the dynamic parameters is performed to obtain a first training snapshot, and the first training snapshot is written to the mount abstraction layer of the first training node. Each link of the first backup process is monitored for abnormalities. In response to the failure of one of the links, a first indication message is sent to the fault-tolerant platform, wherein the first indication message is used to instruct the fault-tolerant platform to perform at least one of the following operations: marking the failure of the first backup process of the current training step of the model and issuing a first alarm message, after writing the first training snapshot to the mount abstraction layer of the first training node, a third indication message is sent to the fault-tolerant platform, and the third indication message is used to instruct the fault-tolerant platform to perform at least one of the following operations: updating the first backup list corresponding to the first backup process, clearing expired first training snapshots and expired first training The second training snapshot corresponding to the snapshot, if the target backup process corresponding to the current training step is the second backup process SC, perform a double backup of the dynamic parameters to obtain the first training snapshot and the second training snapshot, write the first training snapshot to the mount abstraction layer of the first training node, write the second training snapshot to the remote storage cluster through the persistent medium of the first training node, monitor each link of the second backup process for abnormalities, and perform at least one of the following operations in response to the failure of one of the links: exit the training process, send a second indication message to the fault-tolerant platform, the second indication message is used to instruct the second backup process to send a second alarm message, after writing the second training snapshot to the remote storage cluster, send a fourth indication message to the fault-tolerant platform, the fourth indication message is used to instruct the fault-tolerant platform to perform at least one of the following operations: update the second backup list corresponding to the second backup process, store the second backup list to the remote server, and clear the expired first training snapshot and the second training snapshot corresponding to the expired first training snapshot.
[0076] In summary, the training snapshot backup method of the model of the embodiment of the present invention obtains the dynamic parameters to be backed up of the current training step of the model, determines the target backup process corresponding to the current training step, wherein the target backup process is one of the first backup process and the second backup process, the target backup process is the first backup process, performs a single backup on the dynamic parameters, obtains the first training snapshot, writes the first training snapshot to the mounting abstraction layer of the first training node, and the target backup process is the second backup process, performs a double backup on the dynamic parameters, obtains the first training snapshot and the second training snapshot, writes the first training snapshot to the mounting abstraction layer of the first training node, and writes the second training snapshot to the remote storage cluster through the persistent medium of the first training node. Therefore, the present invention improves the efficiency of training snapshot backup by asynchronously setting the first backup process and the second backup process, can avoid the impact of training snapshot backup on model training performance, so that the model training node can be restored in the most efficient way when a failure occurs, shortens the training interruption time, significantly reduces the repeated calculation overhead, and improves the training efficiency of the model.
[0077] Figure 4 FIG. 1 is a flow chart of a training snapshot migration method of a model according to an embodiment of the present disclosure. Figure 4 As shown, the training snapshot migration method of the model proposed in this embodiment is executed by the first training node and includes the following steps:
[0078] S401: In response to identifying that a first training node fails, determine a target training snapshot to be migrated in a memory of the first training node.
[0079] In an embodiment of the present disclosure, during the model training process, fault identification can be performed on the first training node, and in response to identifying that a fault occurs on the first training node, a target training snapshot to be migrated in the memory of the first training node is determined.
[0080] S402: Migrate the target training snapshot to the memory of the scheduled second training node.
[0081] It should be noted that in a distributed training scenario, when a failure of the first training node is identified, in order to ensure the continuity of training and reduce repeated computing overhead, the target training snapshot to be migrated in the memory of the first training node can be determined, and the target training snapshot can be migrated to the memory of the scheduled second training node (a normally available training node).
[0082] In an embodiment of the present disclosure, in response to identifying that a fault has occurred in a first training node, an exception indication may be sent to a fault-tolerant platform, wherein the exception indication is used to instruct the fault-tolerant platform to send a node scheduling indication to a scheduling server, and the node scheduling indication is used to instruct the scheduling server to schedule a second training node from available training nodes and wake up the second training node.
[0083] In an embodiment of the present disclosure, the target training snapshot stored in the mount abstraction layer of the first training node can be migrated to the mount abstraction layer of the second training node through the memory service container.
[0084] It should be noted that the present disclosure provides a service that can be deployed on a training cluster, namely a memory service container. The memory service container enables the training nodes to make full use of the remote direct memory access (RDMA) network bandwidth. When scheduling the second training node from the available training nodes, the target training snapshot stored in the mount abstraction layer is migrated at high speed, that is, it is migrated to the mount abstraction layer of the second training node through the memory service container at high speed. The migration time can be completely covered by the scheduling time of the second training node. Therefore, the migration cost of the training snapshot is minimized.
[0085] In an embodiment of the present disclosure, the target training snapshot stored in the mount abstraction layer may be encoded to obtain a continuous memory space, and the continuous memory space may be sent to a second memory service container in a second training node through a first memory service container in a first training node.
[0086] It should be noted that by encoding the target training snapshot stored in the mount abstraction layer, a continuous memory space buffer is obtained, and the network parameters of the first memory service container are configured, that is, high-speed network transmission is initialized for the first memory service container, and a security check is performed on the target training snapshot. After the check passes, the continuous memory space buffer is sent to the second memory service container based on the configured network.
[0087] Optionally, the integrity of the target training snapshot may be verified by a hash algorithm (Message Digest Algorithm 5, MD5 for short) to implement security verification of the target training snapshot.
[0088] In an embodiment of the present disclosure, in order to more accurately obtain the status of each link of the migration process, the migration process of the target training snapshot can be monitored, and in response to an abnormality, a fifth indication information is sent to the fault-tolerant platform, and the fifth indication information is used to instruct the fault-tolerant platform to issue an alarm message.
[0089] In the disclosed embodiment, in order to more accurately obtain the status of each link of the migration process, the migration process of the target training snapshot can be monitored, and in response to an abnormality, the relevant information of the second training node in the mounting abstraction layer can be cleared.
[0090] To summarize, according to the training snapshot migration method of the model in the embodiment of the present disclosure, in response to identifying that a first training node has failed, a target training snapshot to be migrated in the memory of the first training node is determined, and the target training snapshot is migrated to the memory of the scheduled second training node. As a result, the present disclosure shortens the training snapshot migration time of the model, reduces the training snapshot migration cost of the model, significantly improves the efficiency of the training snapshot migration of the model, reduces the time of model training interruption, and is conducive to improving the training efficiency of the model.
[0091] Figure 5 It is a flowchart of a training snapshot migration method of a model according to an embodiment of the present disclosure.
[0092] like Figure 5 As shown, the training snapshot migration method of the model proposed in this embodiment is executed by the second training node and includes the following steps:
[0093] S501: Receive a target training snapshot for migration of a first training node that has a fault, where the target training snapshot is stored in a memory of the first training node.
[0094] In the embodiment of the present disclosure, the target training snapshot is migrated from the first training node to the memory of the scheduled second training node, and accordingly, the second training node receives the target training snapshot migrated from the failed first training node.
[0095] S502: Store the target training snapshot in the memory of the second training node.
[0096] In an embodiment of the present disclosure, a target training snapshot sent by the first training node through the first memory service container is received through the second memory service container in the second training node, and the target training snapshot is written into the mount abstraction layer of the second training node through the second memory service container.
[0097] Optionally, a second memory service container in a second training node receives a continuous memory space buffer sent by the first memory service container, and decodes the continuous memory space to obtain a target training snapshot, wherein the continuous memory space buffer is obtained by encoding the target training snapshot by the first memory service container.
[0098] In the disclosed embodiment, after obtaining the target training snapshot, the target training snapshot may be verified and the directory restored.
[0099] In an embodiment of the present disclosure, a snapshot status perception request can be sent to the fault-tolerant platform, and the snapshot status perception request is used to request the fault-tolerant platform to determine whether there is an available training snapshot in the target training snapshot, receive loading indication information sent by the fault-tolerant platform, and load the available training snapshot according to the loading indication information for training recovery.
[0100] Optionally, in response to the loading indication information instructing the second training node to load an available training snapshot (SC backup) from the remote storage cluster, the remote storage cluster is accessed, and the available training snapshot is obtained from the remote storage cluster for training recovery.
[0101] Optionally, in response to the loading indication information instructing the second training node to load an available training snapshot (FC backup) from the mount abstraction layer, the available training snapshot is loaded from the mount abstraction layer for training recovery.
[0102] According to the training snapshot migration method of the model in the embodiment of the present disclosure, a target training snapshot migrated from a first training node that has failed is received, the target training snapshot is stored in the memory of the first training node, and the target training snapshot is stored in the memory of the second training node. As a result, the present disclosure shortens the training snapshot migration time of the model, reduces the training snapshot migration cost of the model, significantly improves the efficiency of the training snapshot migration of the model, reduces the time of model training interruption, and is conducive to improving the training efficiency of the model.
[0103] The specific process of the training snapshot migration method of the model in the embodiment of the present disclosure is explained below.
[0104] For example, Figure 6As shown, in response to identifying that a first training node has a fault, an abnormal indication is sent to the fault-tolerant platform, wherein the abnormal indication is used to instruct the fault-tolerant platform to send a node scheduling indication to the scheduling server, and the node scheduling indication is performed by instructing the scheduling server to schedule a second training node from available training nodes and wake up the second training node, encode the target training snapshot stored in the mount abstraction layer of the first training node, obtain continuous memory space, configure network parameters for the first memory service container through the first memory service container in the first training node, and perform security verification on the target training snapshot. After the verification passes, the continuous memory space is sent to the second memory service container based on the configured network, and the migration process of the target training snapshot may be abnormal, and fifth indication information is sent to the fault-tolerant platform, and the fifth indication information is used to instruct the fault-tolerant platform to issue an alarm message and clear the relevant information of the second training node in the mount abstraction layer, that is, to achieve the first training node. The training node migrates the target training snapshot to the memory of the scheduled second training node. The second training node receives the continuous memory space sent by the first memory service container through the second memory service container in the second training node, and decodes the continuous memory space to obtain the target training snapshot. The target training snapshot is written into the mount abstraction layer of the second training node through the second memory service container, and the target training snapshot is verified and the directory is restored. A snapshot status perception request is sent to the fault-tolerant platform, and the loading indication information sent by the fault-tolerant platform is received. In response to the loading indication information instructing the second training node to load the available training snapshot from the remote storage cluster, the remote storage cluster is accessed, and the available training snapshot is obtained from the remote storage cluster for training recovery. In response to the loading indication information instructing the second training node to load the available training snapshot from the mount abstraction layer, the available training snapshot is loaded from the mount abstraction layer for training recovery.
[0105] To sum up, the training snapshot migration method of the model according to the embodiment of the present invention shortens the training snapshot migration time of the model, reduces the training snapshot migration cost of the model, significantly improves the efficiency of the training snapshot migration of the model, reduces the time of model training interruption, and increases the time of effective model training, which is conducive to improving the training efficiency of the model.
[0106] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.
[0107] Corresponding to the training snapshot backup methods of the model provided in the above-mentioned embodiments, an embodiment of the present disclosure also provides a training snapshot backup device of the model. Since the training snapshot backup device of the model provided in the embodiment of the present disclosure corresponds to the training snapshot backup methods of the model provided in the above-mentioned embodiments, the implementation method of the training snapshot backup method of the model is also applicable to the training snapshot backup device of the model provided in this embodiment, and will not be described in detail in this embodiment.
[0108] Figure 7 It is a structural diagram of a training snapshot backup device of a model according to an embodiment of the present disclosure.
[0109] like Figure 7 As shown, the training snapshot backup device 700 of the model is applied to the first training node, including: an acquisition module 710, a determination module 720 and a backup module 730, wherein:
[0110] An acquisition module 710 is used to acquire the dynamic parameters to be backed up in the current training step of the model;
[0111] A determination module 720, configured to determine a training snapshot of the current training step according to the dynamic parameters;
[0112] The backup module 730 is used to back up the training snapshot to the memory of the first training node.
[0113] Among them, the backup module 730 is also used to: determine the target backup process corresponding to the current training step; execute the target backup process to back up the training snapshot to the memory of the first training node; wherein the target backup process is one of the first backup process and the second backup process.
[0114] The target backup process is a first backup process, and the backup module 730 is further used to: perform a single backup of the dynamic parameters to obtain a first training snapshot; and write the first training snapshot to the mount abstraction layer of the first training node.
[0115] Among them, the target backup process is the second backup process, and the backup module 730 is also used to: perform double backup of the dynamic parameters to obtain a first training snapshot and a second training snapshot; write the first training snapshot to the mount abstraction layer of the first training node; and write the second training snapshot to the remote storage cluster through the persistent medium of the first training node.
[0116] Among them, the device 700 is also used to: perform abnormal monitoring on each link of the first backup process, and in response to the failure of one of the links, send a first indication information to the fault-tolerant platform, and the first indication information is used to instruct the fault-tolerant platform to perform at least one of the following operations: mark the failure of the first backup process of the current training step of the model; and issue a first alarm message.
[0117] Among them, the device 700 is also used to: monitor each link of the second backup process for abnormalities, and in response to the failure of one of the links, perform at least one of the following operations: exit the training process; send a second indication information to the fault-tolerant platform, and the second indication information is used to instruct the second backup process to send a second alarm information.
[0118] Among them, after writing the first training snapshot to the mounting abstraction layer of the first training node, the device 700 is also used to: send a third indication information to the fault-tolerant platform, and the third indication information is used to instruct the fault-tolerant platform to perform at least one of the following operations: update the first backup list corresponding to the first backup process; clear the expired first training snapshot and the second training snapshot corresponding to the expired first training snapshot.
[0119] Among them, after writing the second training snapshot to the remote storage cluster, the device 700 is also used to: send a fourth indication information to the fault-tolerant platform, and the fourth indication information is used to instruct the fault-tolerant platform to perform at least one of the following operations: update the second backup list corresponding to the second backup process; store the second backup list to the remote server; clear the expired first training snapshot and the second training snapshot corresponding to the expired first training snapshot.
[0120] According to the training snapshot backup device of the model in the embodiment of the present disclosure, the dynamic parameters to be backed up of the current training step of the model are obtained, and the training snapshot of the current training step is determined based on the dynamic parameters, and the training snapshot is backed up to the memory of the first training node. Therefore, the present disclosure can avoid the impact on the training performance by backing up the training snapshot to the memory of the training node, so that the model training node can be restored in the most efficient way when a failure occurs, shortening the time of training interruption, significantly reducing the overhead of repeated calculations, and improving the training efficiency of the model.
[0121] Corresponding to the training snapshot migration method of the model provided in the above-mentioned embodiments, an embodiment of the present disclosure also provides a training snapshot migration device of the model. Since the training snapshot migration device of the model provided in the embodiment of the present disclosure corresponds to the training snapshot migration method of the model provided in the above-mentioned embodiments, the implementation method of the training snapshot migration method of the model is also applicable to the training snapshot migration device of the model provided in this embodiment, and will not be described in detail in this embodiment.
[0122] Figure 8 It is a structural diagram of a training snapshot migration device of a model according to an embodiment of the present disclosure.
[0123] like Figure 8 As shown, the training snapshot migration device 800 of the model is applied to the first training node, and includes: an identification module 810 and a migration module 820. Among them:
[0124] An identification module 810, configured to determine a target training snapshot to be migrated in a memory of the first training node in response to identifying that the first training node has a fault;
[0125] The migration module 820 is used to migrate the target training snapshot to the memory of the scheduled second training node.
[0126] Among them, the migration module 820 is used to: migrate the first training snapshot stored in the mount abstraction layer of the first training node to the mount abstraction layer of the second training node through the memory service container.
[0127] Among them, the migration module 820 is used to: encode the first training snapshot stored in the mounting abstraction layer to obtain a continuous memory space; and send the continuous memory space to the second memory service container in the second training node through the first memory service container in the first training node.
[0128] Among them, the migration module 820 is used to: configure network parameters for the first memory service container, and perform a security check on the target training snapshot; after the check passes, send the continuous memory space to the second memory service container based on the configured network.
[0129] Among them, the device 800 is used to: in response to identifying that the first training node has a fault, send an exception indication to the fault-tolerant platform; wherein the exception indication is used to instruct the fault-tolerant platform to send a node scheduling indication to the scheduling server; the node scheduling indication is used to instruct the scheduling server to schedule the second training node from the available training nodes and wake up the second training node.
[0130] Among them, the device 800 is used to: monitor the migration process of the target training snapshot, and in response to an abnormality, send fifth indication information to the fault-tolerant platform, wherein the fifth indication information is used to instruct the fault-tolerant platform to issue an alarm message.
[0131] Among them, the device 800 is used to: monitor the migration process of the target training snapshot, and in response to an exception, clear the relevant information of the second training node in the mounting abstraction layer.
[0132] According to the training snapshot migration device of the model in the embodiment of the present disclosure, in response to identifying that a first training node has failed, a target training snapshot to be migrated in the memory of the first training node is determined, and the target training snapshot is migrated to the memory of the scheduled second training node. As a result, the present disclosure shortens the training snapshot migration time of the model, reduces the training snapshot migration cost of the model, significantly improves the efficiency of the training snapshot migration of the model, reduces the time of model training interruption, and is conducive to improving the training efficiency of the model.
[0133] Corresponding to the training snapshot migration method of the model provided in the above-mentioned embodiments, an embodiment of the present disclosure also provides a training snapshot migration device of the model. Since the training snapshot migration device of the model provided in the embodiment of the present disclosure corresponds to the training snapshot migration method of the model provided in the above-mentioned embodiments, the implementation method of the training snapshot migration method of the model is also applicable to the training snapshot migration device of the model provided in this embodiment, and will not be described in detail in this embodiment.
[0134] Fig. 9 It is a structural diagram of a training snapshot migration device of a model according to an embodiment of the present disclosure.
[0135] like Fig. 9 As shown, the training snapshot migration device 900 of the model is applied to the second training node, and includes: a receiving module 910 and a storage module 920. Among them:
[0136] A receiving module 910 is configured to receive a target training snapshot of a first training node that has a fault, wherein the target training snapshot is stored in a memory of the first training node;
[0137] The storage module 920 is used to store the target training snapshot in the memory of the second training node.
[0138] Among them, the device 900 is also used to: receive the target training snapshot sent by the first training node through the first memory service container through the second memory service container in the second training node; and write the target training snapshot into the mount abstraction layer of the second training node through the second memory service container.
[0139] Among them, the device 900 is also used to: receive the continuous memory space sent by the first memory service container through the second memory service container in the second training node, and decode the continuous memory space to obtain the target training snapshot; wherein the continuous memory space is obtained by encoding the target training snapshot by the first memory service container.
[0140] The device 900 is further used to verify and restore the directory of the target training snapshot.
[0141] Among them, after the target training snapshot is verified and the directory is restored, the device 900 is also used to: send a snapshot status perception request to the fault-tolerant platform, and the snapshot status perception request is used to request the fault-tolerant platform to determine whether there is an available training snapshot in the target training snapshot; receive loading indication information sent by the fault-tolerant platform; and load the available training snapshot according to the loading indication information for training recovery.
[0142] Among them, the device 900 is also used to: in response to the loading indication information instructing the second training node to load the available training snapshot from the remote storage cluster, access the remote storage cluster and obtain the available training snapshot from the remote storage cluster for training recovery; in response to the loading indication information instructing the second training node to load the available training snapshot from the mounting abstraction layer, load the available training snapshot from the mounting abstraction layer for training recovery.
[0143] According to the training snapshot migration device of the model in the embodiment of the present disclosure, a target training snapshot migrated from a first training node that has failed is received, the target training snapshot is stored in the memory of the first training node, and the target training snapshot is stored in the memory of the second training node. Therefore, the present disclosure shortens the training snapshot migration time of the model, reduces the training snapshot migration cost of the model, significantly improves the efficiency of the training snapshot migration of the model, reduces the time of model training interruption, and is conducive to improving the training efficiency of the model.
[0144] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.
[0145] Fig.10 A schematic block diagram of an example electronic device 1000 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.
[0146] like Fig.10As shown, the device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the device 1000 can also be stored. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.
[0147] A number of components in the device 1000 are connected to the I / O interface 1005, including: an input unit 1006, such as a keyboard, a mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a disk, an optical disk, etc.; and a communication unit 1009, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1009 allows the device 1000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0148] The computing unit 1001 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1001 performs the various methods and processes described above, such as a training snapshot backup method of a model or a training snapshot migration method of a model. For example, in some embodiments, a training snapshot backup of a model or a training snapshot migration method of a model may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded into the RAM 1003 and executed by the computing unit 1001, one or more steps of the training snapshot backup of the model or the training snapshot migration method of the model described above may be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to execute a training snapshot backup method of a model or a training snapshot migration method of a model in any other appropriate manner (for example, by means of firmware).
[0149] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0150] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0151] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0152] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0153] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.
[0154] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.
[0155] The present disclosure also provides a computer program product, including a computer program, which, when executed by a processor, implements the training snapshot backup method of the model or the training snapshot migration method of the model as described above.
[0156] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.
[0157] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.
Claims
1. A training snapshot backup method for a model, wherein: Executed by a first training node, the method includes: Get the dynamic parameters to be backed up for the current training step of the model; Determining a training snapshot of the current training step according to the dynamic parameters; The training snapshot is backed up to the memory of the first training node.
2. The method according to claim 1, wherein: The method further comprises: Determine the target backup process corresponding to the current training step; The target backup process is executed to back up the training snapshot to the memory of the first training node; wherein the target backup process is one of the first backup process and the second backup process.
3. The method according to claim 2, wherein: The target backup process is a first backup process, and the method further includes: Performing a single backup of the dynamic parameters to obtain a first training snapshot; The first training snapshot is written to the mount abstraction layer of the first training node.
4. The method according to claim 2, wherein: The target backup process is a second backup process, and the method further includes: Performing a double backup of the dynamic parameters to obtain a first training snapshot and a second training snapshot; Writing the first training snapshot to the mount abstraction layer of the first training node; The second training snapshot is written to the remote storage cluster through the persistent medium of the first training node.
5. The method according to claim 3, wherein: The method further comprises: Perform abnormal monitoring on each link of the first backup process, and in response to failure of one of the links, send first indication information to the fault-tolerant platform, where the first indication information is used to instruct the fault-tolerant platform to perform at least one of the following operations: Marking the first backup process of the current training step of the model as failed; Issue the first warning message.
6. The method according to claim 4, wherein: The method further comprises: Each link of the second backup process is monitored for abnormalities, and in response to a failure of one link, at least one of the following operations is performed: Exit the training process; Sending second indication information to the fault-tolerant platform, where the second indication information is used to instruct the second backup process to send second alarm information.
7. The method according to claim 3, wherein: After writing the first training snapshot to the mount abstraction layer of the first training node, the method further includes: Sending third indication information to the fault-tolerant platform, where the third indication information is used to instruct the fault-tolerant platform to perform at least one of the following operations: Updating a first backup list corresponding to the first backup process; The expired first training snapshot and the second training snapshot corresponding to the expired first training snapshot are cleared.
8. The method according to claim 4, wherein: After writing the second training snapshot to the remote storage cluster, the method further includes: Sending fourth indication information to the fault-tolerant platform, where the fourth indication information is used to instruct the fault-tolerant platform to perform at least one of the following operations: Updating the second backup list corresponding to the second backup process; storing the second backup list in a remote server; The expired first training snapshot and the second training snapshot corresponding to the expired first training snapshot are cleared.
9. A training snapshot migration method for a model, wherein: Executed by a first training node, the method includes: In response to identifying that the first training node has failed, determining a target training snapshot to be migrated in a memory of the first training node; Migrate the target training snapshot to the memory of the scheduled second training node.
10. The method according to claim 9, wherein: The migrating the first training snapshot to the scheduled second training node includes: The first training snapshot stored in the mount abstraction layer of the first training node is migrated to the mount abstraction layer of the second training node through the memory service container.
11. The method according to claim 10, wherein: The step of migrating the first training snapshot stored in the mount abstraction layer of the first training node to the mount abstraction layer of the second training node through the memory service container includes: Encoding the first training snapshot stored in the mount abstraction layer to obtain a continuous memory space; The continuous memory space is sent to a second memory service container in the second training node through a first memory service container in the first training node.
12. The method according to claim 10, wherein: The sending the target cache to the second memory service container in the second training node through the first memory service container in the first training node also includes: Performing network parameter configuration on the first memory service container and performing a security check on the target training snapshot; After the verification is passed, the continuous memory space is sent to the second memory service container based on the configured network.
13. The method according to claim 9, wherein: The method further comprises: In response to identifying that the first training node has a fault, sending an abnormality indication to the fault-tolerant platform; Wherein, the abnormal indication is used to instruct the fault-tolerant platform to send a node scheduling indication to the scheduling server; The node scheduling indication is used to instruct the scheduling server to schedule the second training node from available training nodes and wake up the second training node.
14. The method according to claim 9, wherein: The method further comprises: The migration process of the target training snapshot is monitored, and in response to an abnormality, fifth indication information is sent to the fault-tolerant platform, where the fifth indication information is used to instruct the fault-tolerant platform to issue an alarm message.
15. The method according to claim 10, wherein: The method further comprises: The migration process of the target training snapshot is monitored, and in response to an exception, the relevant information of the second training node in the mounting abstraction layer is cleared.
16. A training snapshot migration method for a model, wherein: Executed by the second training node, the method includes: receiving a target training snapshot migrated from a first training node that has failed, the target training snapshot being stored in a memory of the first training node; The target training snapshot is stored in the memory of the second training node.
17. The method according to claim 16, wherein: The method further comprises: Receiving, through the second memory service container in the second training node, the target training snapshot sent by the first training node through the first memory service container; The target training snapshot is written into the mount abstraction layer of the second training node through the second memory service container.
18. The method according to claim 17, wherein: The method further comprises: Receiving, through the second memory service container in the second training node, the continuous memory space sent by the first memory service container, and decoding the continuous memory space to obtain the target training snapshot; The continuous memory space is obtained by encoding the target training snapshot by the first memory service container.
19. The method according to claim 18, wherein: The method further comprises: The target training snapshot is verified and catalog restored.
20. The method according to claim 19, wherein: After verifying and restoring the directory of the target training snapshot, the method further includes: Sending a snapshot status perception request to the fault-tolerant platform, wherein the snapshot status perception request is used to request the fault-tolerant platform to determine whether there is an available training snapshot in the target training snapshot; Receiving loading instruction information sent by the fault-tolerant platform; According to the loading instruction information, available training snapshots are loaded to perform training recovery.
21. The method according to claim 20, wherein: The step of loading the available training snapshot according to the loading indication information to perform training recovery includes: In response to the loading indication information instructing the second training node to load an available training snapshot from a remote storage cluster, accessing the remote storage cluster and obtaining the available training snapshot from the remote storage cluster for training recovery; In response to the loading indication information instructing the second training node to load an available training snapshot from the mount abstraction layer, the available training snapshot is loaded from the mount abstraction layer for training recovery.
22. A training snapshot backup device for a model, wherein: Applied to a first training node, the device comprises: The acquisition module is used to obtain the dynamic parameters to be backed up in the current training step of the model; A determination module, configured to determine a training snapshot of the current training step according to the dynamic parameters; A backup module is used to back up the training snapshot to the memory of the first training node.
23. The device according to claim 22, wherein: The backup module is further used for: Determine the target backup process corresponding to the current training step; Executing the target backup process to back up the training snapshot to the memory of the first training node; The target backup process is one of the first backup process and the second backup process.
24. The device according to claim 23, wherein: The target backup process is a first backup process, and the backup module is further used for: Performing a single backup of the dynamic parameters to obtain a first training snapshot; The first training snapshot is written to the mount abstraction layer of the first training node.
25. The device according to claim 23, wherein: The target backup process is a second backup process, and the backup module is further used for: Performing a double backup of the dynamic parameters to obtain a first training snapshot and a second training snapshot; Writing the first training snapshot to the mount abstraction layer of the first training node; The second training snapshot is written to the remote storage cluster through the persistent medium of the first training node.
26. A training snapshot migration device for a model, wherein: Applied to a first training node, the device comprises: an identification module, configured to determine a target training snapshot to be migrated in a memory of the first training node in response to identifying that the first training node has a fault; A migration module is used to migrate the target training snapshot to the memory of the scheduled second training node.
27. A training snapshot migration device for a model, wherein: Applied to a second training node, the device comprises: A receiving module, configured to receive a target training snapshot of a first training node that has a fault, wherein the target training snapshot is stored in a memory of the first training node; A storage module is used to store the target training snapshot in the memory of the second training node.
28. An electronic device, characterized in that: including a processor and a memory; The processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, so as to implement the method as claimed in any one of claims 1-8, claims 9-15, or claims 16-21.
29. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 8, claims 9 to 15, or claims 16 to 21 is implemented.
30. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1-8 or claims 9-15 or claims 16-21.