Distributed training fault recovery method and device, medium and computer program product

Abstract

The present invention discloses a distributed training fault recovery method and device, medium and computer program product, which relates to the field of computer technology. The method comprises: starting a first thread, a second thread and a third thread at the beginning of distributed training; wherein the first thread is used for data calculation, the second thread is used for synchronizing gradient data, and the third thread is used for storing checkpoint data; during the distributed training process, when the second thread starts gradient data synchronization, the gradient data synchronization flag is set to a first preset value, and when the gradient data synchronization is ended, the gradient data synchronization flag is set to a second preset value; when the third thread detects that the gradient data synchronization flag is the second preset value, the checkpoint data is stored; when the distributed training system fails, the fault recovery is performed based on the stored checkpoint data. The present invention avoids the influence of stored checkpoint data on training efficiency.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and more specifically, to a distributed training fault recovery method and device, medium and computer program product. Background Art

[0002] Deep learning models are widely used in various industries. As the scale of deep learning models continues to expand and the training process becomes more complicated, erroneous exits are difficult to avoid, so a fault recovery mechanism is needed when errors occur.

[0003] In the related art, the state of the model is periodically set to a checkpoint after a certain iteration of training, and the checkpoint data is stored on a persistent hard disk. When an error occurs, the fault recovery program searches for the most recent checkpoint data for fault recovery. In the above scheme, the checkpoint data is stored after a certain iteration of training. The storage process of the checkpoint data is in the critical path of training, which prolongs the normal training process and reduces the training efficiency.

[0004] Therefore, how to avoid the impact of storing checkpoint data on training efficiency is a technical problem that technicians in this field need to solve. Summary of the invention

[0005] The object of the present invention is to provide a distributed training fault recovery method and device, medium and computer program product, which avoids the influence of storage checkpoint data on training efficiency.

[0006] To achieve the above object, the present invention provides a distributed training fault recovery method, which is applied to a server in a distributed training system, and the method comprises:

[0007] At the beginning of distributed training, the first thread, the second thread, and the third thread are started; wherein the first thread is used for data calculation, the second thread is used for synchronizing gradient data, and the third thread is used for storing checkpoint data;

[0008] During the distributed training, when the second thread starts gradient data synchronization, the gradient data synchronization flag is set to a first preset value, and when the gradient data synchronization is ended, the gradient data synchronization flag is set to a second preset value;

[0009] When the third thread detects that the gradient data synchronization flag is the second preset value, storing the checkpoint data;

[0010] When the distributed training system fails, failure recovery is performed based on the stored checkpoint data.

[0011] The storage checkpoint data includes:

[0012] The checkpoint data is divided into a plurality of data blocks, and the plurality of data blocks are stored in sequence.

[0013] The sequentially storing the plurality of data blocks comprises:

[0014] After the current data block is stored, detecting the gradient data synchronization flag to determine whether the gradient data synchronization flag is the second preset value;

[0015] If yes, continue to store the next data block;

[0016] If not, the step of detecting the gradient data synchronization flag is re-entered.

[0017] Among them, it also includes:

[0018] The storage frequency is determined according to the interval time of synchronizing gradient data, the data volume of checkpoint data, the transmission bandwidth between the computing device in the server and the shared extended memory, and the transmission bandwidth between servers; wherein the storage frequency is used to describe the interval of the number of iterations of storing checkpoint data;

[0019] Correspondingly, when the third thread detects that the gradient data synchronization flag is the second preset value, storing the checkpoint data includes:

[0020] The third thread determines whether the current iteration number is an integer multiple of the storage frequency, and if so, detects the gradient data synchronization flag, and stores the checkpoint data when it is detected that the gradient data synchronization flag is the second preset value, and stops storing the checkpoint data when it is detected that the gradient data synchronization flag is the first preset value.

[0021] The distributed training system includes an upper switch, multiple bottom switches and multiple servers, the upper switch is connected to multiple bottom switches, and each bottom switch is connected to multiple different servers;

[0022] Accordingly, the storage frequency is determined according to the interval time of synchronizing gradient data, the amount of checkpoint data, the transmission bandwidth between the computing device in the server and the shared extended memory, and the transmission bandwidth between servers, including:

[0023] The storage frequency is determined according to the interval time of synchronizing gradient data, the amount of checkpoint data, the transmission bandwidth between the computing device in the server and the shared extended memory, the transmission bandwidth between servers under the same underlying switch, and the transmission bandwidth between underlying switches.

[0024] The step of determining the storage frequency according to the interval time of synchronizing gradient data, the amount of checkpoint data, the transmission bandwidth between the computing device in the server and the shared extended memory, the transmission bandwidth between servers under the same underlying switch, and the transmission bandwidth between underlying switches includes:

[0025] During the training process of a preset number of iterations, counting the average time interval of the synchronized gradient data within the preset number of iterations;

[0026] After training for a preset number of iterations, a first ratio between the amount of checkpoint data and a first transmission bandwidth is calculated; wherein the first transmission bandwidth is a transmission bandwidth between a computing device in a server and a shared extended memory;

[0027] Calculate a first product between the first number, the second number, and the data volume of the checkpoint data, and calculate a second ratio between the first product and the second transmission bandwidth; wherein the first number is the number of servers connected to each of the underlying switches, the second number is the first number minus one, and the second transmission bandwidth is the transmission bandwidth between servers under the same underlying switch;

[0028] Calculate a second product between the first quantity, the third quantity, and the data volume of the checkpoint data, and calculate a third ratio between the second product and a third transmission bandwidth; wherein the third quantity is the number of underlying switches included in the distributed training system, and the third transmission bandwidth is the transmission bandwidth between underlying switches;

[0029] A sum value of the first ratio, the second ratio, and the third ratio is calculated, and a ratio between the sum value and the average time interval is used as the storage frequency.

[0030] The storage checkpoint data includes:

[0031] storing the checkpoint data of the computing device in the server in the shared extended memory of the server;

[0032] Storing the checkpoint data stored in the shared extended memory of the server into the shared extended memory of other servers connected to the same underlying switch as the server;

[0033] One of the servers under each of the underlying switches stores the checkpoint data stored in the shared extended memory in the shared extended memory of one of the servers under the other underlying switches.

[0034] The step of storing the checkpoint data stored in the shared extended memory in one of the servers under each of the underlying switches to the shared extended memory of one of the servers under the other underlying switches includes:

[0035] The last server under each of the underlying switches stores the checkpoint data stored in the shared extended memory into the shared extended memory of the first server under the adjacent underlying switch.

[0036] Wherein, when the distributed training system fails, fault recovery is performed based on the stored checkpoint data, including:

[0037] When the distributed training system fails, if the node corresponding to the server is a faulty node, the fault type of the faulty node is determined, checkpoint data is obtained from the shared extended memory of the corresponding server according to the fault type, and the faulty node is restored according to the obtained checkpoint data.

[0038] The determining of the fault type of the faulty node, acquiring checkpoint data from a shared extended memory of a corresponding server according to the fault type, and restoring the faulty node according to the acquired checkpoint data includes:

[0039] If the fault type of the faulty node is a computing device fault, determining whether all computing devices in the server corresponding to the faulty node are faulty;

[0040] If all fail, determine whether there is a non-faulty computing device among other servers connected to the same underlying switch as the server corresponding to the failed node; if so, obtain the checkpoint data from the shared extended memory in the server where the non-faulty computing device is located; if not, obtain the checkpoint data from the shared extended memory of the first server under the adjacent underlying switch;

[0041] If not all nodes fail, obtaining the checkpoint data from the shared extended memory in the server corresponding to the failed node;

[0042] The failed node is restored according to the acquired checkpoint data.

[0043] The determining of the fault type of the faulty node, acquiring checkpoint data from a shared extended memory of a corresponding server according to the fault type, and restoring the faulty node according to the acquired checkpoint data includes:

[0044] If the fault type of the faulty node is a server fault, determining whether all other servers connected to the same underlying switch as the server corresponding to the faulty node are faulty;

[0045] If all fail, the checkpoint data is obtained from the shared extended memory of the first server under the adjacent underlying switch;

[0046] If not all fail, obtaining the checkpoint data from the shared extended memory of other non-failure servers connected to the same underlying switch by the server corresponding to the failed node;

[0047] The failed node is restored according to the acquired checkpoint data.

[0048] Wherein, when the distributed training system fails, it also includes:

[0049] If the node corresponding to the server is a non-faulty node, the checkpoint data is obtained from the shared extended memory of the server corresponding to the non-faulty node, and the non-faulty node is restored according to the obtained checkpoint data.

[0050] To achieve the above object, the present invention provides a distributed training fault recovery device, which is applied to a server in a distributed training system, and the device includes:

[0051] A startup module, used to start the first thread, the second thread, and the third thread when the distributed training starts; wherein the first thread is used to perform data calculation, the second thread is used to synchronize gradient data, and the third thread is used to store checkpoint data;

[0052] A setting module, configured to set the gradient data synchronization flag to a first preset value when the second thread starts gradient data synchronization during distributed training, and to set the gradient data synchronization flag to a second preset value when the gradient data synchronization ends;

[0053] A storage module, configured to store checkpoint data when the third thread detects that the gradient data synchronization flag is the second preset value;

[0054] A fault recovery module is used to perform fault recovery based on the stored checkpoint data when the distributed training system fails.

[0055] To achieve the above object, the present invention provides an electronic device, comprising:

[0056] Memory for storing computer programs;

[0057] A processor is used to implement the steps of the above-mentioned distributed training fault recovery method when executing the computer program.

[0058] To achieve the above-mentioned purpose, the present invention provides a non-volatile storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned distributed training fault recovery method are implemented.

[0059] To achieve the above objectives, the present invention provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the above-mentioned distributed training fault recovery method.

[0060] It can be seen from the above scheme that a distributed training fault recovery method provided by the present invention includes: at the beginning of distributed training, starting a first thread, a second thread, and a third thread; wherein the first thread is used to perform data calculation, the second thread is used to synchronize gradient data, and the third thread is used to store checkpoint data; during the distributed training process, when the second thread starts gradient data synchronization, the gradient data synchronization flag is set to a first preset value, and when the gradient data synchronization is ended, the gradient data synchronization flag is set to a second preset value; when the third thread detects that the gradient data synchronization flag is the second preset value, the checkpoint data is stored; when the distributed training system fails, fault recovery is performed based on the stored checkpoint data.

[0061] The beneficial effect of the present invention is that the distributed training fault recovery method provided by the present invention realizes parallel processing of data calculation, gradient data synchronization and checkpoint data storage by introducing a multi-threading mechanism in distributed training, thereby improving the training efficiency. Among them, the first thread is responsible for data calculation, the second thread is responsible for synchronizing gradient data, and the third thread is responsible for storing checkpoint data. This design allows each thread to perform its task independently, reduces mutual interference between threads, and improves the overall training performance. In the training process, gradient data synchronization is a key step because it involves communication between different training nodes. In order to minimize the impact of storing checkpoint data on training efficiency, the present invention uses the communication gap of gradient data synchronization to store checkpoint data, which minimizes the impact of storing checkpoint data on training efficiency. When a distributed training system fails, the most recently stored checkpoint data can be used for rapid recovery, which not only improves the robustness of the system, but also reduces the training interruption time caused by the failure. The present invention also discloses a distributed training fault recovery device, an electronic device, a non-volatile storage medium and a computer program product, which can also achieve the above technical effects.

[0062] It is to be understood that the foregoing general description and the following detailed description are exemplary only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. The drawings are used to provide a further understanding of the present disclosure and constitute a part of the specification. Together with the following specific implementation methods, they are used to explain the present disclosure, but do not constitute a limitation to the present disclosure. In the drawings:

[0064] Figure 1 is a flow chart of a distributed training fault recovery method according to an exemplary embodiment;

[0065] Figure 2 is a flowchart of another distributed training fault recovery method according to an exemplary embodiment;

[0066] Figure 3 A three-layer network topology cluster architecture diagram is shown according to an exemplary embodiment;

[0067] Figure 4 A schematic diagram of a checkpoint backup method for intermittent transmission according to an exemplary embodiment;

[0068] Figure 5 The present invention is an overall flow chart of distributed training with checkpoint backup according to an exemplary embodiment;

[0069] Figure 6 is a flowchart showing a checkpoint storage frequency and checkpoint data backup process in thread 3 according to an exemplary embodiment;

[0070] Figure 7 The present invention is a schematic diagram showing a checkpoint data backup in the same server according to an exemplary embodiment;

[0071] Figure 8 The figure is a schematic diagram showing a checkpoint data backup in a same bottom layer switch according to an exemplary embodiment;

[0072] Fig. 9 The figure is a schematic diagram showing a checkpoint data backup between different bottom switches according to an exemplary embodiment;

[0073] Fig.10 The following is a flowchart of restoring a failed node according to an exemplary embodiment;

[0074] Fig.11 is a structural diagram of a distributed training fault recovery device according to an exemplary embodiment;

[0075] Fig.12 The figure is a structural diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0076] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. In addition, in the embodiments of the present invention, "first", "second", etc. are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0077] The embodiment of the present invention discloses a distributed training fault recovery method, which avoids the influence of storage checkpoint data on training efficiency.

[0078] See also Figure 1 , a flowchart of a distributed training fault recovery method according to an exemplary embodiment is shown as follows Figure 1 As shown, including:

[0079] S101: When distributed training starts, start a first thread, a second thread, and a third thread; wherein the first thread is used for data calculation, the second thread is used for synchronizing gradient data, and the third thread is used for storing checkpoint data;

[0080] This embodiment is applied to a distributed training system, which includes multiple servers, and the executor of this embodiment is each server of the distributed training system. In this step, in the initialization phase of distributed training, the server will start three key threads to process different tasks in parallel. The first thread is specifically responsible for processing data calculations, which is the basic work in the training process and involves a large number of mathematical operations. The second thread is responsible for synchronizing gradient data to ensure that all nodes participating in the training can update and share the latest gradient information. The third thread is responsible for storing checkpoint data, which may include parameters of each layer in the neural network, global parameters during training, etc. This is a backup mechanism for periodically saving the state of the model during training so that training can be restored from the most recent checkpoint in the event of a failure.

[0081] S102: during the distributed training process, when the second thread starts gradient data synchronization, the gradient data synchronization flag is set to a first preset value, and when the gradient data synchronization is ended, the gradient data synchronization flag is set to a second preset value;

[0082] In this step, when distributed training is in progress, the second thread will set a specific flag to the first preset value when starting to synchronize gradient data. This operation is a signal that the gradient data synchronization process is about to begin. When the gradient data synchronization is completed, the flag will be updated to the second preset value, which is a signal that the gradient data synchronization has ended. Such a flag mechanism ensures that at different stages of gradient data synchronization, other threads in the server can accurately know the current status and coordinate their behavior.

[0083] S103: When the third thread detects that the gradient data synchronization flag is the second preset value, storing checkpoint data;

[0084] In a specific implementation, the third thread is responsible for monitoring the status of the gradient data synchronization flag. When it detects that the flag has been set to the second preset value, it means that the gradient data synchronization has been completed and the system is in a relatively stable stage. In this case, the third thread will perform the operation of storing the checkpoint data, which ensures that the storage of the checkpoint data does not interfere with the synchronization process of the gradient data. By storing the checkpoint after the gradient data synchronization is completed, the impact on the training efficiency can be minimized, and it also ensures that when a system failure occurs, the training can be quickly resumed from the latest checkpoint.

[0085] As a feasible implementation mode, this embodiment also includes: determining the storage frequency according to the interval time of synchronizing gradient data, the data volume of checkpoint data, the transmission bandwidth between the computing device in the server and the shared extended memory, and the transmission bandwidth between servers; wherein the storage frequency is used to describe the interval of the number of iterations for storing checkpoint data; accordingly, when the third thread detects that the gradient data synchronization flag is the second preset value, storing the checkpoint data includes: the third thread determines whether the current number of iterations is an integer multiple of the storage frequency, and if so, detecting the gradient data synchronization flag, storing the checkpoint data when it is detected that the gradient data synchronization flag is the second preset value, and stopping storing the checkpoint data when it is detected that the gradient data synchronization flag is the first preset value.

[0086] In a specific implementation, by comprehensively considering the time interval for synchronizing gradient data, the size of checkpoint data, the transmission bandwidth between the computing device in the server and the shared extended memory, and the transmission bandwidth between servers, the frequency of storing checkpoint data is intelligently determined, that is, how many iterations are used to store the checkpoint data. Specifically, when the third thread performs the task of storing checkpoint data, it will first determine whether the current number of iterations meets the predetermined storage frequency. If so, it will further check the state of the gradient data synchronization flag. If the gradient data synchronization flag is the second preset value, indicating that the gradient synchronization has been completed, the third thread will store the checkpoint data; if the gradient data synchronization flag is the first preset value, indicating that the gradient synchronization is in progress, the third thread will suspend the storage operation to avoid interfering with the gradient synchronization process, ensuring the continuity of training and the consistency of data. This implementation method determines the number of iterations required to store a checkpoint through an online analysis method, and sets the number of iterations as the checkpoint storage cycle to maximize its storage frequency.

[0087] As a feasible implementation mode, the distributed training system includes an upper switch, multiple underlying switches and multiple servers, the upper switch is connected to multiple underlying switches, and each underlying switch is connected to multiple different servers; accordingly, the storage frequency is determined according to the interval time of synchronizing gradient data, the data volume of checkpoint data, the transmission bandwidth between the computing device in the server and the shared extended memory, and the transmission bandwidth between servers, including: determining the storage frequency according to the interval time of synchronizing gradient data, the data volume of checkpoint data, the transmission bandwidth between the computing device in the server and the shared extended memory, the transmission bandwidth between servers under the same underlying switch, and the transmission bandwidth between underlying switches.

[0088] In a specific implementation, the distributed training system consists of an upper-layer switch, multiple lower-layer switches, and multiple servers. The upper-layer switch is responsible for connecting multiple lower-layer switches, and each lower-layer switch is connected to different server groups. Multiple computing devices are installed in each server, such as a GPU (Graphics Processing Unit) accelerator. Preferably, the server is a server device with memory expanded using CXL (Compute Express Link) technology, which means that each server has a large amount of extended memory available, that is, the shared extended memory expanded by CXL technology stores checkpoint data, which improves storage efficiency. The lower-layer switch is responsible for processing data exchange and communication between machines, ensuring that different machines can communicate and transmit data with each other. The upper-layer switch is connected to different lower-layer switches and to the external network, and is responsible for the flow control, routing management and other functions of the entire computer room network.

[0089] In this architecture, the frequency of storing checkpoint data is not determined by a single factor, but by a combination of multiple factors, including the time interval for gradient data synchronization, the size of the checkpoint data, the data transmission speed between the server's internal computing device and the shared extended memory, the data transmission speed between servers under the same underlying switch, and the data transmission speed between different underlying switches. Through the comprehensive consideration of these factors, the system can intelligently determine an optimal storage frequency to ensure that while ensuring training efficiency, data backup can also be performed effectively, so that the training state can be quickly restored when a system failure occurs.

[0090] As a feasible implementation method, the storage frequency is determined according to the interval time of synchronized gradient data, the data volume of checkpoint data, the transmission bandwidth between the computing device in the server and the shared extended memory, the transmission bandwidth between the servers under the same underlying switch, and the transmission bandwidth between the underlying switches, including: during the training process of a preset number of iterations, counting the average time interval of synchronized gradient data within the preset number of iterations; after the training of the preset number of iterations, calculating the first ratio between the data volume of the checkpoint data and the first transmission bandwidth; wherein the first transmission bandwidth is the transmission bandwidth between the computing device in the server and the shared extended memory; calculating the first product between the first quantity, the second quantity, and the data volume of the checkpoint data, and calculating a second ratio between the first product and the second transmission bandwidth; wherein, the first number is the number of servers connected to each of the underlying switches, the second number is the first number minus one, and the second transmission bandwidth is the transmission bandwidth between servers under the same underlying switch; calculating the second product between the first number, the third number, and the amount of data of the checkpoint data, and calculating the third ratio between the second product and the third transmission bandwidth; wherein, the third number is the number of underlying switches included in the distributed training system, and the third transmission bandwidth is the transmission bandwidth between the underlying switches; calculating the sum of the first ratio, the second ratio, and the third ratio, and taking the ratio between the sum and the average time interval as the storage frequency.

[0091] In a specific implementation, assuming that the average time interval for synchronizing gradient data within a preset number of iterations is t, the amount of checkpoint data is X, the number of servers under each underlying switch is p, there are q underlying switches in total, the transmission bandwidth from the computing device to the local server shared extended memory is W0, the transmission bandwidth between servers under the same underlying switch is W1, and the transmission bandwidth between the underlying switches is W2, then the calculation formula for the storage frequency m is: m=[X / W0+p(p-1)×X / W1+p×X×q / W2] / t, and the calculation result can be rounded up.

[0092] As a preferred implementation, a machine learning algorithm can be introduced to dynamically adjust the storage strategy of checkpoint data. Specifically, historical training data and system performance indicators can be used to predict the optimal checkpoint storage frequency and timing through a machine learning model to adapt to different training loads and network conditions. Through the prediction of the machine learning model, the system can adaptively adjust the checkpoint storage strategy to cope with changing training needs and network status and improve storage efficiency. Dynamic adjustment of checkpoint storage can reduce unnecessary storage operations, thereby saving storage resources and network bandwidth and improving overall training efficiency. In the event of a system failure, since the checkpoint data more accurately reflects the latest status of the system, it can be restored to the training state before the failure more quickly, reducing the training interruption time.

[0093] As a feasible implementation, the storing of checkpoint data includes: dividing the checkpoint data into a plurality of data blocks, and sequentially storing the plurality of data blocks. In a specific implementation, the third thread processes the checkpoint data in blocks, each data block is of size blk, for example, 10M, and stores one data block at a time.

[0094] As a feasible implementation manner, the sequentially storing the plurality of data blocks includes: after the storage of the current data block is completed, detecting the gradient data synchronization flag, and determining whether the gradient data synchronization flag is the second preset value; if so, continuing to store the next data block; if not, re-entering the step of detecting the gradient data synchronization flag.

[0095] In a specific implementation, after storing each data block, the third thread first checks the value of the gradient data synchronization flag. Once it is found to be the first preset value, indicating that the gradient synchronization communication process has started, the checkpoint storage operation is immediately stopped, and the occupied bandwidth is returned to the gradient data synchronization communication process. At the same time, it is checked whether the current checkpoint data backup has been completed. If not, the value of the gradient data synchronization flag is continuously checked to wait for the gap to perform the checkpoint backup. If it has been completed, the current iteration number is obtained to start a new round of checkpoint storage.

[0096] S104: When the distributed training system fails, failure recovery is performed based on the stored checkpoint data.

[0097] In specific implementations, when a distributed training system encounters a failure, the server will use the checkpoint data previously stored at a specific frequency to recover from the failure. These checkpoint data contain the parameters of each layer in the neural network and the global parameters during the training process. They represent the key state information during the training process and allow the system to restart training from the most recent stable state instead of starting from scratch. This can minimize the loss of training progress caused by the failure and quickly restore to the training state before the failure, ensuring the continuity and efficiency of training.

[0098] The distributed training fault recovery method provided by the embodiment of the present invention realizes parallel processing of data calculation, gradient data synchronization and checkpoint data storage by introducing a multi-threading mechanism in distributed training, thereby improving training efficiency. Among them, the first thread is responsible for data calculation, the second thread is responsible for synchronizing gradient data, and the third thread is responsible for storing checkpoint data. This design allows each thread to perform its task independently, reduces mutual interference between threads, and improves the overall training performance. In the training process, gradient data synchronization is a key step because it involves communication between different training nodes. In order to minimize the impact of storing checkpoint data on training efficiency, the embodiment of the present invention uses the communication gap of gradient data synchronization to store checkpoint data, which minimizes the impact of storing checkpoint data on training efficiency. When a distributed training system fails, the most recently stored checkpoint data can be used for rapid recovery, which not only improves the robustness of the system, but also reduces the training interruption time caused by the failure.

[0099] The embodiment of the present invention discloses a distributed training fault recovery method. Compared with the previous embodiment, this embodiment further explains and optimizes the technical solution. Specifically:

[0100] See also Figure 2 , a flowchart of another distributed training fault recovery method according to an exemplary embodiment is shown, such as Figure 2 As shown, including:

[0101] S201: When distributed training starts, start a first thread, a second thread, and a third thread; wherein the first thread is used for data calculation, the second thread is used for synchronizing gradient data, and the third thread is used for storing checkpoint data;

[0102] This embodiment is applied to a distributed training system, which is composed of an upper-layer switch, multiple bottom-layer switches and multiple servers. The upper-layer switch is responsible for connecting multiple bottom-layer switches, and each bottom-layer switch is connected to different server groups. Multiple computing devices, such as GPU accelerators, are installed in each server. Preferably, the server is a server device with memory expanded using CXL technology, which means that each server has a large amount of extended memory available for use, that is, the shared extended memory expanded by CXL technology stores checkpoint data, thereby improving storage efficiency. The bottom-layer switch is responsible for processing data exchange and communication between machines, ensuring that different machines can communicate and transmit data with each other. The upper-layer switch is connected to different bottom-layer switches and to the external network, and is responsible for functions such as flow control and routing management of the entire computer room network.

[0103] S202: during the distributed training process, when the second thread starts gradient data synchronization, the gradient data synchronization flag is set to a first preset value, and when the gradient data synchronization is ended, the gradient data synchronization flag is set to a second preset value;

[0104] S203: When the third thread detects that the gradient data synchronization flag is the second preset value, the checkpoint data of the computing device in the server is stored in the shared extended memory of the server, the checkpoint data stored in the shared extended memory of the server is stored in the shared extended memory of other servers connected to the same underlying switch as the server, and one of the servers under each of the underlying switches stores the checkpoint data stored in the shared extended memory in the shared extended memory of one of the servers under the other underlying switches.

[0105] In a specific implementation, when the third thread detects that the gradient data synchronization flag is the second preset value, it indicates that the gradient data synchronization has been completed. At this time, the third thread will first store the checkpoint data generated by the computing device in the server in the shared extended memory of the server to ensure immediate backup of the data. Subsequently, these checkpoint data will be copied and stored in the shared extended memory of other servers connected to the same underlying switch to achieve local redundancy of the data. Finally, a designated server under each underlying switch will further store these checkpoint data in the shared extended memory of the servers connected to other underlying switches, thereby achieving cross-switch data backup in the entire distributed system, enhancing data reliability and system fault tolerance. This process ensures that in a distributed training system, even if a part fails, the server can use these distributed stored checkpoint data for rapid recovery.

[0106] As a feasible implementation manner, the last server under each of the underlying switches stores the checkpoint data stored in the shared extended memory into the shared extended memory of the first server under the adjacent underlying switch.

[0107] S204: When the distributed training system fails, if the node corresponding to the server is a non-faulty node, obtain checkpoint data from the shared extended memory of the server corresponding to the non-faulty node, and restore the non-faulty node according to the obtained checkpoint data;

[0108] When a distributed training system fails, step S204 involves retrieving checkpoint data from the shared extended memory of the server corresponding to the non-faulty node. These checkpoint data were previously stored when the system was running normally, and contain the state information of the training model. By using this data, the system can reconstruct the state of the non-faulty nodes, thereby resuming the training process of these nodes and ensuring that the continuity of training is not affected.

[0109] S205: If the node corresponding to the server is a faulty node, determine the fault type of the faulty node, obtain checkpoint data from the shared extended memory of the corresponding server according to the fault type, and restore the faulty node according to the obtained checkpoint data.

[0110] In the specific implementation, for a failed node, the specific fault type of the failed node is first determined. Understanding the nature of the fault is crucial because it determines which server's shared extended memory needs to be retrieved from. Once the fault type is determined, the server obtains the necessary checkpoint data from the corresponding server and uses this data to restore the state of the failed node. This approach allows the system to adopt customized recovery strategies for different types of faults, improving the efficiency and accuracy of fault recovery.

[0111] As a feasible implementation mode, the method of determining the fault type of the faulty node, obtaining checkpoint data from the shared extended memory of the corresponding server according to the fault type, and restoring the faulty node according to the obtained checkpoint data includes: if the fault type of the faulty node is a computing device failure, determining whether all computing devices in the server corresponding to the faulty node are faulty; if all are faulty, determining whether there are non-faulty computing devices in other servers connected to the same underlying switch as the server corresponding to the faulty node; if so, obtaining checkpoint data from the shared extended memory in the server where the non-faulty computing device is located; if not, obtaining checkpoint data from the shared extended memory of the first server under the adjacent underlying switch; if not all are faulty, obtaining checkpoint data from the shared extended memory in the server corresponding to the faulty node; and restoring the faulty node according to the obtained checkpoint data.

[0112] In a specific implementation, when the system detects that the fault type of the failed node is a computing device failure, it will first check whether all computing devices in the server corresponding to the failed node have failed. If all computing devices have failed, the system will further check whether there are available non-faulty computing devices in other servers connected to the same underlying switch. If so, the system will extract the checkpoint data from the shared extended memory of the server where these non-faulty computing devices are located; if not, the system will obtain the checkpoint data from the shared extended memory of the first server under the adjacent underlying switch. If the computing devices of the failed node are not all faulty, the system will directly obtain the checkpoint data from the shared extended memory of the server corresponding to the failed node. In either case, after obtaining the checkpoint data, the system will use this data to restore the state of the failed node to ensure the continuity of the training process and the stability of the system.

[0113] As another feasible implementation mode, the method of determining the fault type of the faulty node, obtaining checkpoint data from the shared extended memory of the corresponding server according to the fault type, and restoring the faulty node according to the obtained checkpoint data includes: if the fault type of the faulty node is a server fault, determining whether all other servers connected to the same underlying switch as the server corresponding to the faulty node are faulty; if all are faulty, obtaining checkpoint data from the shared extended memory of the first server under the adjacent underlying switch; if not all are faulty, obtaining checkpoint data from the shared extended memory of other non-faulty servers connected to the same underlying switch as the server corresponding to the faulty node; and restoring the faulty node according to the obtained checkpoint data.

[0114] In the specific implementation, when the fault type of the failed node is determined to be a server failure, the system will evaluate whether all other servers connected to the same underlying switch as the failed server have also failed. If all servers under the same switch have failed, the system will obtain the checkpoint data from the shared extended memory of the first server of the adjacent underlying switch. If there are other non-faulty servers under the same switch, the system will obtain the checkpoint data from the shared extended memory of these non-faulty servers. In either case, once the checkpoint data is obtained, the system will use this data to recover the failed node and ensure that the training process can continue, thereby improving the system's fault tolerance and recovery efficiency.

[0115] It can be seen that this embodiment uses CXL technology to expand the memory of each server in the three-layer network topology cluster. Then, according to the characteristics of the three-layer network topology cluster with expanded memory, an efficient and reliable distributed training fault fast recovery method is formulated, including a hierarchical redundant checkpoint backup strategy, a backup execution strategy for checkpoint data in training, and a fault recovery strategy after a fault occurs.

[0116] The following describes an application embodiment provided by the present invention. The three-layer network topology cluster architecture is as follows: Figure 3 As shown, multiple GPU accelerators are installed in each server, and the server uses CXL technology to expand the memory. The bottom switch S1 connects multiple servers, and the upper switch S2 connects different bottom switches and is connected to the external network. In this embodiment, a checkpoint backup method for intermittent transmission is proposed, that is, the checkpoint data is transmitted in the gap between the gradient synchronization data transmission of distributed training, such as Figure 4 As shown, it does not affect the normal training process.

[0117] The overall process of distributed training with checkpoint backup is as follows Figure 5As shown, three threads are started simultaneously during the training process. Thread 1 is used for calculations during the training process, thread 2 is used for gradient synchronization during the training process, and thread 3 is used for checkpoint backup. The three threads are executed concurrently. The calculation process of thread 1 includes: forward calculation, reverse calculation, obtaining gradient data, obtaining aggregated gradients, updating weights, judging whether the training is finished, if so, ending the calculation, if not, returning to the step of forward calculation. The gradient data synchronization flag (allreduce_flag) is set to indicate whether data synchronization communication is currently being performed. The gradient synchronization process of thread 2 includes: obtaining gradient data, starting gradient data synchronization operation, setting the gradient data synchronization flag to 1, transmitting gradient data, calculating aggregated gradients, setting the gradient data synchronization flag to 0, judging whether the training is finished, if so, ending gradient synchronization, if not, returning to the step of obtaining gradient data. When the value of allreduce_flag is 0, it indicates that it is in the gap of gradient data synchronization and checkpoint storage can be performed. The checkpoint backup process of thread 3 includes: checkpoint storage frequency calculation and checkpoint data backup, see for details Figure 6 Initialize the data block size blk, the number of communication gaps for each iteration to 0, obtain the current number of iterations, determine whether the current number of iterations is greater than 100, if the previous number of iterations is less than or equal to 100, obtain the gradient data synchronization flag, determine whether the gradient data synchronization flag is 0, if not, re-enter the step of obtaining the gradient data synchronization flag, if so, record the current time point as the starting time point, determine whether the gradient data synchronization flag is 1, if so, calculate the gradient communication gap for each iteration += current time point - starting time point, if not, re-enter the step of determining whether the gradient data synchronization flag is 1. If the previous iteration number is greater than 100, determine whether the storage frequency is zero. If the storage frequency is zero, calculate the average communication gap of each iteration and calculate the checkpoint storage frequency. If the storage frequency is not zero, determine whether the remainder of the current iteration number and the storage frequency is 0. If the remainder is not 0, re-enter the step of obtaining the current iteration number. If the remainder is 0, obtain the gradient data synchronization flag and determine whether the gradient data synchronization flag is 0. If the gradient data synchronization flag is not 0, re-enter the step of obtaining the gradient data synchronization flag. If the gradient data synchronization flag is 0, transmit a checkpoint data block of size blk. Determine whether the gradient data synchronization flag is 1. If the gradient data synchronization flag is not 1, re-enter the step of transmitting a checkpoint data block of size blk. If the gradient data synchronization flag is 1, stop the checkpoint data transmission, store the transmission position, determine whether the current checkpoint backup is completed, if completed, re-enter the step of obtaining the current iteration number. If not completed, re-enter the step of obtaining the gradient data synchronization flag.

[0118] The specific explanation is as follows: First, the checkpoint storage frequency during the training process is determined using the first 100 iterations of training, that is, how many training iterations are used to perform a complete checkpoint storage. Thread 3 uses the allreduce_flag flag to count the gradient synchronization time gap ti during each iteration, and calculates the average value t of the gradient synchronization gap in 100 iterations of training. Then, based on the amount of data X required to be transmitted for checkpoint storage and the topological bandwidth of each layer, the number of iterations m required for a complete backup is calculated. Assuming that the average time interval for synchronizing gradient data within the preset number of iterations is t, the amount of checkpoint data is X, the number of servers under each underlying switch is p, there are q underlying switches in total, the transmission bandwidth from the computing device to the local server shared extended memory is W0, the transmission bandwidth between servers under the same underlying switch is W1, and the transmission bandwidth between underlying switches is W2, then the calculation formula for the storage frequency m is: m=[X / W0+p(p-1)×X / W1+p×X×q / W2] / t, and the calculation result can be rounded up.

[0119] After getting the value of m, the checkpoint backup storage officially starts after 100 iterations. Thread 3 starts a new round of checkpoint backup only when the current number of iterations is divisible by m. After starting, it periodically checks the value of allreduce_flag. Once it finds that its value is false, it immediately starts the process checkpoint storage communication. This thread divides the checkpoint data to be stored into blocks, each data block size is blk, and one data block is transmitted each time. After each database is transmitted, the value of allreduce_flag is rechecked. Once its value is true, it means that the gradient synchronization communication process has started, then the checkpoint storage operation is immediately stopped, and the occupied bandwidth is returned to the gradient data synchronization communication process. At the same time, it checks whether the current checkpoint data backup has been completed. If not, it continues to check the value of allreduce_flag and waits for the gap to perform checkpoint backup. If it has been completed, it obtains the current iteration number and starts a new round of checkpoint storage.

[0120] According to the cluster network topology hierarchy, the checkpoint data of each GPU accelerator is backed up in a hierarchical and redundant manner to the CXL extended memory of each server in the cluster. The checkpoint backup efficiency is improved by storing it in the CXL extended memory, and the fault tolerance of the system is increased through hierarchical redundant backup.

[0121] 1. Checkpoint data backup within the same server: Each GPU within the same server stores its checkpoint data in the shared extended memory; for example, Figure 7 As shown, accelerator 1 (GPU1) and accelerator 2 (GPU2) in server 0 (Sever0) store their checkpoint data 0-1 and 0-2 in the shared extended memory of the current server respectively.

[0122] 2. Checkpoint data backup in the same underlying switch: Each server broadcasts the checkpoint data in the shared extended memory to the shared memory of other servers under the same underlying switch for backup storage; for example, Figure 8 As shown in the figure, Sever0 broadcasts data 0-1 and 0-2 in its shared extended memory to other server nodes under the same underlying switch, here Server 1; at the same time, Sever1 also performs the same broadcast operation.

[0123] 3. Checkpoint data backup between different bottom switches: The last server under each bottom switch transfers the data in its shared memory to the first server shared extended memory of the adjacent switch for backup again. Fig. 9 As shown, server 3 (Sever3) broadcasts the data 3-1, 3-2, 2-1, and 2-2 in its shared extended memory to Server0; at the same time, Sever1 broadcasts the data 1-1, 1-2, 0-1, and 0-2 in its shared extended memory to Server2.

[0124] When a failure occurs, the fault recovery program is triggered. The fault recovery program first analyzes the failure and implements different recovery strategies according to different failure types. When a failure occurs, all GPUs need to be restored to the same training moment, that is, the same checkpoint. For non-faulty nodes, it is only necessary to obtain its most recent checkpoint data from the current server shared memory for recovery. For the recovery of faulty nodes, it is necessary to handle it according to the specific situation.

[0125] The recovery of the failed node is as follows Fig.10 As shown, the fault type is obtained. If the fault type is an accelerator fault, that is, the fault is in the accelerator, then determine whether all accelerators in the current server are faulty: (1) If not, then only some processes are faulty at this time, and the data in the shared memory will not be lost, so the GPU can be re-acquired and the latest checkpoint data can be quickly obtained from the shared memory of the current server; (2) If yes, then all threads on the server will exit with an error, and the shared memory data will be lost. At this time, it is necessary to further determine whether there are still non-faulty accelerators under the current underlying switch: ① If yes, that is, not all accelerators under the same underlying switch are faulty, then the latest checkpoint data is obtained from the extended memory of other servers under the same underlying switch (servers where non-faulty GPUs are located) for recovery; ② If not, that is, all accelerators under the same underlying switch are faulty, so at this time it is necessary to obtain the latest checkpoint data from the extended memory of the first server of the adjacent underlying switch for recovery.

[0126] If the fault type is a server fault, that is, the server fails, the entire server is down, and the data in the current server shared memory is lost. At this time, it is necessary to re-acquire the server and further determine whether all servers under the same underlying switch are faulty: ① If so, obtain the most recent checkpoint data from the first server extended memory of the adjacent underlying switch for recovery; ② If not, obtain the most recent checkpoint data from other servers (non-faulty servers) under the same underlying switch for recovery.

[0127] A distributed training fault recovery device provided in an embodiment of the present invention is introduced below. The distributed training fault recovery device described below and the distributed training fault recovery method described above can be referenced to each other.

[0128] See also Fig.11 , according to an exemplary embodiment, a structural diagram of a distributed training fault recovery device is shown, such as Fig.11 As shown, including:

[0129] A startup module 100 is used to start a first thread, a second thread, and a third thread when distributed training starts; wherein the first thread is used to perform data calculation, the second thread is used to synchronize gradient data, and the third thread is used to store checkpoint data;

[0130] A setting module 200, configured to set the gradient data synchronization flag to a first preset value when the second thread starts gradient data synchronization during distributed training, and to set the gradient data synchronization flag to a second preset value when the gradient data synchronization ends;

[0131] A storage module 300, configured to store checkpoint data when the third thread detects that the gradient data synchronization flag is the second preset value;

[0132] The fault recovery module 400 is used to perform fault recovery based on the stored checkpoint data when the distributed training system fails.

[0133] The distributed training fault recovery device provided by the embodiment of the present invention realizes parallel processing of data calculation, gradient data synchronization and checkpoint data storage by introducing a multi-threading mechanism in distributed training, thereby improving the training efficiency. Among them, the first thread is responsible for data calculation, the second thread is responsible for synchronizing gradient data, and the third thread is responsible for storing checkpoint data. This design allows each thread to perform its task independently, reduces mutual interference between threads, and improves the overall training performance. In the training process, gradient data synchronization is a key step because it involves communication between different training nodes. In order to minimize the impact of storing checkpoint data on training efficiency, the embodiment of the present invention uses the communication gap of gradient data synchronization to store checkpoint data, which minimizes the impact of storing checkpoint data on training efficiency. When a distributed training system fails, the most recently stored checkpoint data can be used for rapid recovery, which not only improves the robustness of the system, but also reduces the training interruption time caused by the failure.

[0134] Based on the above embodiment, as a preferred implementation mode, the storage module 300 is specifically used to: divide the checkpoint data into multiple data blocks, and store the multiple data blocks in sequence.

[0135] Based on the above embodiment, as a preferred implementation mode, the storage module 300 is specifically used to: after the current data block is stored, detect the gradient data synchronization flag, and determine whether the gradient data synchronization flag is the second preset value; if so, continue to store the next data block; if not, re-enter the step of detecting the gradient data synchronization flag.

[0136] Based on the above embodiment, as a preferred implementation, it also includes:

[0137] A determination module, used to determine the storage frequency according to the interval time of synchronization gradient data, the data volume of checkpoint data, the transmission bandwidth between the computing device and the shared extended memory in the server, and the transmission bandwidth between servers; wherein the storage frequency is used to describe the interval of the number of iterations of storing checkpoint data;

[0138] Correspondingly, the storage module 300 is specifically used to: determine whether the current number of iterations is an integer multiple of the storage frequency through the third thread, and if so, detect the gradient data synchronization flag, and when it is detected that the gradient data synchronization flag is the second preset value, store the checkpoint data, and when it is detected that the gradient data synchronization flag is the first preset value, stop storing the checkpoint data.

[0139] Based on the above embodiment, as a preferred implementation, the distributed training system includes an upper switch, multiple lower switches and multiple servers, the upper switch is connected to multiple lower switches, and each lower switch is connected to different multiple servers;

[0140] Correspondingly, the determination module is specifically used to determine the storage frequency according to the interval time of synchronizing gradient data, the data volume of checkpoint data, the transmission bandwidth between the computing device in the server and the shared extended memory, the transmission bandwidth between servers under the same underlying switch, and the transmission bandwidth between the underlying switches.

[0141] Based on the above embodiments, as a preferred implementation mode, the determination module is specifically used to: count the average time interval of synchronized gradient data within a preset number of iterations; calculate a first ratio between the data volume of checkpoint data and a first transmission bandwidth; wherein the first transmission bandwidth is the transmission bandwidth between the computing device in the server and the shared extended memory; calculate a first product between the first number and the data volume of the checkpoint data, and calculate a second ratio between the first product and a second transmission bandwidth; wherein the first number is the number of underlying switches included in the distributed training system, and the second transmission bandwidth is the transmission bandwidth between servers under the same underlying switch; calculate a second product between the first number, the second number, and the data volume of the checkpoint data, and calculate a third ratio between the second product and a third transmission bandwidth; wherein the second number is the number of servers connected to each of the underlying switches, and the third transmission bandwidth is the transmission bandwidth between the underlying switches; calculate the sum of the first ratio, the second ratio, and the third ratio, and use the ratio between the sum and the average time interval as the storage frequency.

[0142] Based on the above embodiments, as a preferred implementation mode, the storage module 300 is specifically used to: store the checkpoint data of the computing device in the server in the shared extended memory of the server; store the checkpoint data stored in the shared extended memory of the server in the shared extended memory of other servers connected to the same underlying switch as the server; and store the checkpoint data stored in the shared extended memory of one of the servers under each of the underlying switches in the shared extended memory of one of the servers under the other underlying switches.

[0143] Based on the above embodiment, as a preferred implementation, the storage module 300 is specifically used for: the last server under each of the underlying switches stores the checkpoint data stored in the shared extended memory into the shared extended memory of the first server under the adjacent underlying switch.

[0144] Based on the above embodiments, as a preferred implementation mode, the fault recovery module 400 is specifically used to: when the distributed training system fails, determine the fault type of the faulty node, obtain checkpoint data from the shared extended memory of the corresponding server according to the fault type, and restore the non-faulty node according to the obtained checkpoint data.

[0145] On the basis of the above embodiments, as a preferred implementation mode, the fault recovery module 400 is specifically used for: if the fault type of the faulty node is a computing device failure, then determining whether all computing devices in the server corresponding to the faulty node are faulty; if all are faulty, then determining whether there are non-faulty computing devices in other servers connected to the same underlying switch as the server corresponding to the faulty node; if so, obtaining checkpoint data from the shared extended memory in the server where the non-faulty computing device is located; if not, obtaining checkpoint data from the shared extended memory of the first server under the adjacent underlying switch; if not all are faulty, obtaining checkpoint data from the shared extended memory in the server corresponding to the faulty node; and recovering the non-faulty node according to the obtained checkpoint data.

[0146] On the basis of the above embodiment, as a preferred implementation mode, the fault recovery module 400 is specifically used for: if the fault type of the faulty node is a server fault, determining whether all other servers connected to the same underlying switch as the server corresponding to the faulty node are faulty; if all are faulty, obtaining checkpoint data from the shared extended memory of the first server under the adjacent underlying switch; if not all are faulty, obtaining checkpoint data from the shared extended memory of other non-faulty servers connected to the same underlying switch as the server corresponding to the faulty node; and restoring the non-faulty node according to the obtained checkpoint data.

[0147] Based on the above embodiment, as a preferred implementation, the fault recovery module 400 is also used to: obtain checkpoint data from the shared extended memory of the server corresponding to the non-faulty node, and restore the non-faulty node according to the obtained checkpoint data.

[0148] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0149] Based on the hardware implementation of the above program modules and in order to implement the method of the embodiment of the present invention, the embodiment of the present invention further provides an electronic device, Fig.12 FIG. 1 is a structural diagram of an electronic device according to an exemplary embodiment. Fig.12 As shown, the electronic equipment includes:

[0150] Communication interface 1, capable of exchanging information with other devices such as network devices;

[0151] The processor 2 is connected to the communication interface 1 to implement information interaction with other devices, and is used to execute the distributed training fault recovery method provided by one or more technical solutions when running a computer program. The computer program is stored in the memory 3.

[0152] Of course, in actual application, the various components in the electronic device are coupled together through the bus system 4. It can be understood that the bus system 4 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 4 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, Fig.12 Various buses are labeled as bus system 4 .

[0153] The memory 3 in the embodiment of the present invention is used to store various types of data to support the operation of the electronic device. Examples of such data include: any computer program used to operate on the electronic device.

[0154] It can be understood that the memory 3 can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM, SyncLink Dynamic Random Access Memory), and direct RAMbus random access memory (DRRAM, Direct Rambus Random Access Memory).The memory 3 described in the embodiments of the present invention is intended to include but is not limited to these and any other suitable types of memories.

[0155] The method disclosed in the above embodiment of the present invention can be applied to the processor 2, or implemented by the processor 2. The processor 2 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 2 or the instruction in the form of software. The above processor 2 can be a general-purpose processor, a DSP, or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor 2 can implement or execute the methods, steps and logic block diagrams disclosed in the embodiment of the present invention. The general-purpose processor can be a microprocessor or any conventional processor, etc. In combination with the steps of the method disclosed in the embodiment of the present invention, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium, which is located in the memory 3. The processor 2 reads the program in the memory 3 and completes the steps of the above method in combination with its hardware.

[0156] When the processor 2 executes the program, the corresponding processes in the various methods of the embodiments of the present invention are implemented, which will not be described here for the sake of brevity.

[0157] In an exemplary embodiment, the embodiment of the present invention further provides a non-volatile storage medium storing a computer program, which can be executed by the processor 2 to complete the steps of the aforementioned method.

[0158] In an exemplary embodiment, the present invention further provides a computer program product, including a computer program, which is executed by the processor 2 to complete the steps of the aforementioned method.

[0159] A person of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiments can be completed by hardware related to computer program instructions, and the aforementioned computer program can be stored in a non-volatile storage medium. When the computer program is executed, it executes the steps of the above method embodiments. Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a non-volatile storage medium. Based on this understanding, the technical solution of the embodiment of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a non-volatile storage medium and includes a number of instructions for an electronic device (which can be a personal computer, a server, a network device, etc.) to execute all or part of the methods described in each embodiment of the present invention.

[0160] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A distributed training fault recovery method, characterized in that: Applied to a server in a distributed training system, the method comprises: At the beginning of distributed training, the first thread, the second thread, and the third thread are started; wherein the first thread is used for data calculation, the second thread is used for synchronizing gradient data, and the third thread is used for storing checkpoint data; During the distributed training, when the second thread starts gradient data synchronization, the gradient data synchronization flag is set to a first preset value, and when the gradient data synchronization is ended, the gradient data synchronization flag is set to a second preset value; When the third thread detects that the gradient data synchronization flag is the second preset value, the checkpoint data is stored, so as to store the checkpoint data by utilizing the communication gap of the gradient data synchronization; When the distributed training system fails, failure recovery is performed based on the stored checkpoint data.

2. The distributed training fault recovery method according to claim 1, characterized in that: The storage checkpoint data includes: The checkpoint data is divided into a plurality of data blocks, and the plurality of data blocks are stored in sequence.

3. The distributed training fault recovery method according to claim 2, characterized in that: The sequentially storing the plurality of data blocks comprises: After the current data block is stored, detecting the gradient data synchronization flag to determine whether the gradient data synchronization flag is the second preset value; If yes, continue to store the next data block; If not, the step of detecting the gradient data synchronization flag is re-entered.

4. The distributed training fault recovery method according to claim 1, characterized in that: Also includes: The storage frequency is determined according to the interval time of synchronizing gradient data, the data volume of checkpoint data, the transmission bandwidth between the computing device in the server and the shared extended memory, and the transmission bandwidth between servers; wherein the storage frequency is used to describe the interval of the number of iterations of storing checkpoint data; Correspondingly, when the third thread detects that the gradient data synchronization flag is the second preset value, storing the checkpoint data includes: The third thread determines whether the current iteration number is an integer multiple of the storage frequency, and if so, detects the gradient data synchronization flag, and stores the checkpoint data when it is detected that the gradient data synchronization flag is the second preset value, and stops storing the checkpoint data when it is detected that the gradient data synchronization flag is the first preset value.

5. The distributed training fault recovery method according to claim 4, characterized in that: The distributed training system includes an upper switch, multiple bottom switches and multiple servers, the upper switch is connected to multiple bottom switches, and each bottom switch is connected to multiple different servers; Accordingly, the storage frequency is determined according to the interval time of synchronizing gradient data, the amount of checkpoint data, the transmission bandwidth between the computing device in the server and the shared extended memory, and the transmission bandwidth between servers, including: The storage frequency is determined according to the interval time of synchronizing gradient data, the amount of checkpoint data, the transmission bandwidth between the computing device in the server and the shared extended memory, the transmission bandwidth between servers under the same underlying switch, and the transmission bandwidth between underlying switches.

6. The distributed training fault recovery method according to claim 5, characterized in that: The storage frequency is determined according to the interval time of synchronizing gradient data, the amount of checkpoint data, the transmission bandwidth between the computing device in the server and the shared extended memory, the transmission bandwidth between servers under the same underlying switch, and the transmission bandwidth between underlying switches, including: During the training process of a preset number of iterations, counting the average time interval of the synchronized gradient data within the preset number of iterations; After training for a preset number of iterations, a first ratio between the amount of checkpoint data and a first transmission bandwidth is calculated; wherein the first transmission bandwidth is a transmission bandwidth between a computing device in a server and a shared extended memory; Calculate a first product between the first number, the second number, and the data volume of the checkpoint data, and calculate a second ratio between the first product and the second transmission bandwidth; wherein the first number is the number of servers connected to each of the underlying switches, the second number is the first number minus one, and the second transmission bandwidth is the transmission bandwidth between servers under the same underlying switch; Calculate a second product between the first quantity, the third quantity, and the data volume of the checkpoint data, and calculate a third ratio between the second product and a third transmission bandwidth; wherein the third quantity is the number of underlying switches included in the distributed training system, and the third transmission bandwidth is the transmission bandwidth between underlying switches; A sum value of the first ratio, the second ratio, and the third ratio is calculated, and a ratio between the sum value and the average time interval is used as the storage frequency.

7. The distributed training fault recovery method according to claim 5, characterized in that: The storage checkpoint data includes: storing the checkpoint data of the computing device in the server in the shared extended memory of the server; Storing the checkpoint data stored in the shared extended memory of the server into the shared extended memory of other servers connected to the same underlying switch as the server; One of the servers under each of the underlying switches stores the checkpoint data stored in the shared extended memory in the shared extended memory of one of the servers under the other underlying switches.

8. The distributed training fault recovery method according to claim 7, characterized in that: One of the servers under each of the underlying switches stores the checkpoint data stored in the shared extended memory into the shared extended memory of one of the servers under the other underlying switches, including: The last server under each of the underlying switches stores the checkpoint data stored in the shared extended memory into the shared extended memory of the first server under the adjacent underlying switch.

9. The distributed training fault recovery method according to claim 8, characterized in that: When the distributed training system fails, failure recovery is performed based on the stored checkpoint data, including: When the distributed training system fails, if the node corresponding to the server is a faulty node, the fault type of the faulty node is determined, checkpoint data is obtained from the shared extended memory of the corresponding server according to the fault type, and the faulty node is restored according to the obtained checkpoint data.

10. The distributed training fault recovery method according to claim 9, characterized in that: The determining the fault type of the faulty node, acquiring checkpoint data from a shared extended memory of a corresponding server according to the fault type, and restoring the faulty node according to the acquired checkpoint data includes: If the fault type of the faulty node is a computing device fault, determining whether all computing devices in the server corresponding to the faulty node are faulty; If all fail, determine whether there is a non-faulty computing device among other servers connected to the same underlying switch as the server corresponding to the failed node; if so, obtain the checkpoint data from the shared extended memory in the server where the non-faulty computing device is located; if not, obtain the checkpoint data from the shared extended memory of the first server under the adjacent underlying switch; If not all nodes fail, obtaining the checkpoint data from the shared extended memory in the server corresponding to the failed node; The failed node is restored according to the acquired checkpoint data.

11. The distributed training fault recovery method according to claim 9, characterized in that: The determining the fault type of the faulty node, acquiring checkpoint data from a shared extended memory of a corresponding server according to the fault type, and restoring the faulty node according to the acquired checkpoint data includes: If the fault type of the faulty node is a server fault, determining whether all other servers connected to the same underlying switch as the server corresponding to the faulty node are faulty; If all fail, the checkpoint data is obtained from the shared extended memory of the first server under the adjacent underlying switch; If not all fail, obtaining the checkpoint data from the shared extended memory of other non-failure servers connected to the same underlying switch by the server corresponding to the failed node; The failed node is restored according to the acquired checkpoint data.

12. The distributed training fault recovery method according to claim 9, characterized in that: When the distributed training system fails, the method further includes: If the node corresponding to the server is a non-faulty node, the checkpoint data is obtained from the shared extended memory of the server corresponding to the non-faulty node, and the non-faulty node is restored according to the obtained checkpoint data.

13. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the distributed training fault recovery method as claimed in any one of claims 1 to 12 when executing the computer program.

14. A non-volatile storage medium, characterized in that: The non-volatile storage medium stores a computer program, which, when executed, implements the steps of the distributed training fault recovery method as described in any one of claims 1 to 12.

15. A computer program product, characterized in that It comprises a computer program, which, when executed, implements the steps of the distributed training fault recovery method as claimed in any one of claims 1 to 12.

Images