Fault process recovery method and electronic equipment

Abstract

The invention provides a fault process recovery method and electronic equipment. The method comprises the following steps: receiving first state information of a working process in at least one node; the first state information comprises abnormal events and training parameter versions of at least two working processes of at least one node; analyzing the first state information, and determining a fault process with a fault according to the working process with the abnormal event; and determining a globally available training parameter version based on the fault process, and recovering the fault process by utilizing the globally available training parameter version.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing, and in particular to a fault process recovery method and electronic device. Background Art

[0002] As the size of the cluster required for training models increases, large clusters bring more frequent failures. In related technologies, the fault resolution method is to periodically save the model training parameters, but high-frequency model training parameter saving will increase a large amount of job blocking time, and low-frequency model training parameter saving will extend the retraining time after a failure occurs, resulting in the inability to quickly return to normal when a model training failure occurs. Summary of the invention

[0003] The present disclosure provides a fault process recovery method and an electronic device to at least solve the above technical problems existing in the prior art.

[0004] According to a first aspect of the present disclosure, a method for recovering a failed process is provided, the method comprising:

[0005] Receive first status information of a working process in at least one node; the first status information includes abnormal events and training parameter versions occurring in at least two working processes of at least one node;

[0006] Analyze the first state information, and determine the faulty process where the fault occurs according to the working process where the abnormal event occurs;

[0007] A globally available training parameter version is determined based on the faulty process, and the globally available training parameter version is used to restore the faulty process.

[0008] In one possible implementation, the abnormal event that occurs in the work process includes at least one of the following:

[0009] The worker process exits;

[0010] There is a work process that exceeds the preset training time; the preset training time is the maximum value of a single iteration time of model training.

[0011] In one possible implementation, before receiving the first status information of the working process in the working process, the method further includes:

[0012] The second status information sent by the main process in the receiving node includes heartbeat information and fault process information; the working process is created by the main process;

[0013] The fault location is determined according to the heartbeat information or the fault process information; the fault location includes a fault node and a fault process.

[0014] In one possible implementation manner, determining the fault location according to the heartbeat information or the fault process information includes:

[0015] According to the heartbeat information, determining that a node that has not received the heartbeat has a node failure;

[0016] The fault process is determined according to the fault process information; the fault process information includes a fault error code.

[0017] In one possible implementation, determining a globally available training parameter version based on the fault process includes:

[0018] Determine the order of iterations in which the faulty process occurs and the stage of the optimization step;

[0019] The training parameter version of the healthy process and the training parameter version of the faulty process are determined respectively according to the iteration order and the stage of the optimization step, and the training parameter version of the healthy process or the training parameter version of the faulty process is determined to be the globally available training parameter version.

[0020] In one possible implementation manner, determining that the training parameter version of the healthy process or the training parameter version of the faulty process is a globally available training parameter version includes:

[0021] Depending on the stage of the optimization step in different work processes, the training parameter version is updated synchronously or asynchronously to be the globally available training parameter version.

[0022] In one possible implementation, according to the stage of the optimization step in different work processes, the training parameter version is synchronously updated to a globally available training parameter version, including:

[0023] If the faulty process occurs in the first phase of the iteration, the training parameter version of the faulty process is the same as the training parameter version of the healthy process;

[0024] If the faulty process occurs in the second phase of the iteration, the training parameter version of the faulty process is determined as the globally available training parameter version.

[0025] In one possible implementation, according to the stage of the optimization step in different work processes, the training parameter version is asynchronously updated to a globally available training parameter version, including:

[0026] Determine the versions of training parameters for different stages of the optimization step of the working process;

[0027] The minimum value among the training parameter versions at different stages in the worker process is determined as the globally available training parameter version.

[0028] In one possible implementation manner, after determining the faulty process where the fault occurs, the method further includes:

[0029] Query the data parallel group corresponding to the faulty process, and determine whether there is a healthy process in the data parallel group;

[0030] If it exists, determine the globally available training parameter version; otherwise send an exit command to the main process.

[0031] According to a second aspect of the present disclosure, there is provided an electronic device, including:

[0032] at least one processor; and

[0033] a memory communicatively connected to the at least one processor; wherein,

[0034] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the present disclosure.

[0035] According to a third aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method described in the present disclosure.

[0036] The fault process recovery method and electronic device disclosed in the present invention can perform an inspection after a fault occurs, determine a training parameter version that can restore training, and use the training parameter version to restore the fault process, thereby retraining, shortening the retraining time and reducing the fault inspection cost.

[0037] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, in which:

[0039] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts.

[0040] Figure 1 The following is a schematic diagram showing the implementation process of the fault process recovery method of the present disclosure. Figure 1 ;

[0041] Figure 2 A 3D parallel communication grouping diagram provided by an embodiment of the present disclosure is shown;

[0042] Figure 3The figure shows the agent architecture diagram provided by the embodiment of the present disclosure;

[0043] Figure 4 A diagram showing a parameter synchronization mode provided by an embodiment of the present disclosure is shown;

[0044] Figure 5 A parameter asynchronous mode diagram provided by an embodiment of the present disclosure is shown;

[0045] Figure 6 A parameter rollback process flow chart of an embodiment of the present disclosure is shown;

[0046] Figure 7 A schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0047] In order to make the purpose, features, and advantages of the present disclosure more obvious and easy to understand, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0048] In the related art, the existing fault solution method is generally to minimize the checkpoint storage time through multi-level cache and input and output IO optimization technology, but long-term high-frequency checkpoints will bring additional time costs and a large number of IO operations will bring greater pressure on the processor CPU memory and disk, which is easy to introduce new storage device failures. In addition, the solution to faults in the related art can also be to take an immediate inspection method after the fault occurs, but the immediate inspection after the fault occurs can only be used for data parallel training tasks and has great limitations. The current large model training basically adopts 3D parallel methods, namely data parallelism, pipeline parallelism, and tensor parallelism, and the existing instant inspection method after the fault occurs cannot be applied to the current large model training method.

[0049] A fault process recovery method and electronic device provided by the present application are described below in conjunction with the accompanying drawings.

[0050] like Figure 1 As shown, the present application provides a method for recovering a faulty process, comprising:

[0051] S101, receiving first status information of a working process in at least one node; the first status information includes abnormal events and training parameter versions occurring in at least two working processes of at least one node;

[0052] First of all, it should be noted that the technical solution provided by this application can be used in 3D parallel large model training. 3D parallelism refers to data parallelism, pipeline parallelism, and tensor parallelism.

[0053] For example, Figure 2 As shown in the figure, assume that there are two nodes Node1 and Node2, each node has 8 GPUs, a total of 16 GPUs. The 16 GPUs are numbered Rank0, Rank1, ..., Rank15. In addition, assume that the user sets the pipeline parallelism to 4 and the tensor parallelism to 2.

[0054] For example, pipeline parallelism will divide the entire model into four parts, which are called sub-models. Each of the four consecutive GPUs is responsible for one sub-model.

[0055] Tensor parallelism will split tensors for sub-models in pipeline parallelism. That is, Rank0 and Rank1 are responsible for one sub-model sub_model_1, Rank2 and Rank3 are responsible for another sub-model sub_model_1; Rank4 and Rank5 are responsible for sub-model sub_model_2, Rank6 and Rank7 are responsible for another sub-model sub_model_2; and so on. Each tensor parallel group is jointly responsible for a specific sub-model sub_model.

[0056] The purpose of data parallelism is to ensure that the same model parameters in parallel read the same data. After pipeline parallelism and tensor parallelism, Rank0 and Rank2 are responsible for the same model parameters, so Rank0 and Rank2 are the same data parallel group.

[0057] The model training process requires loop iterations to continuously update model parameters. Each round of iteration includes forward calculation (Forward), reverse calculation (Backwad), and parameter update (Optimizer step). Forward calculation and reverse calculation will not modify the parameters of the model. The probability of single-point failure in large-scale distributed training is high, but the probability of multi-point failure is very low. When a process failure or node failure occurs, the faulty process will fall into a deadlock or fault exit state. At this time, other healthy processes will not immediately perceive the failure but continue to execute until a timeout error occurs when executing collective communication.

[0058] The fault process recovery method provided in this application can be applied to the Nginx (engine x) proxy. Nginx is a high-performance HTTP and reverse proxy web server, such as Figure 3As shown, it includes multiple nodes Node, each node is provided with a main process (Master process) and multiple worker processes (Worker processes), the master process is used to receive instructions issued by the control component (Brain) and respond, the Worker process receives processing and coordination commands from the Master process, and is responsible for processing various network events. The Master process is the parent process of the Worker process and is responsible for creating and training the Worker process. Each Worker process can be a user-defined python training script, and the training script will call the collective communication (NVIDIA Collective Communications Library, NCCL) library when performing communication operations. This application sets an NCCL interception library for each Worker process of each node to intercept all python training scripts from calling the NCCL interface API. Specifically, the NCCL interception library intercepts all training scripts from calling collective communications, and determines whether the execution time of the collective communication operator exceeds the maximum time of the iteration. If it exceeds, it is determined to be a timeout event (TIMEOUT event), that is, an abnormal event, and the timeout event is sent to the Brain through the remote procedure call protocol rpc.

[0059] like Figure 3 As shown, in the present application, Brain can receive the first state information sent by the Worker process in at least one node, and the first state information includes abnormal events and training parameter versions occurring in at least two worker processes of at least one node. Among them, the abnormal events occurring in at least two worker processes refer to the single iteration time of at least two worker processes being different.

[0060] Exemplarily, the Brain component counts the single iteration time collected from the training process and calculates the maximum value T. The NCCL interception library in the Worker process sends the captured TIMEOUT event to the Brain. At the same time, the Worker process also sends the training parameter version to the Brain.

[0061] S102, analyzing the first state information, and determining a faulty process where a fault occurs according to the working process where the abnormal event occurs;

[0062] Specifically, the first state information is analyzed to determine which work processes sent the abnormal event, thereby determining the faulty process that failed. It should be noted that analyzing the first state information mainly analyzes what the abnormal event that occurred in the work process is, and determining the faulty process that failed according to the specific abnormal event that occurred.

[0063] Exemplarily, the abnormal events of the working process include multiple types, which may be error code exit or deadlock, etc. If the abnormal event is error code exit, because the Worker process is created by the Master process, the Master process can directly locate the faulty process and report the faulty process information to the Brain component. If the abnormal event is a deadlock, the NCCL interception library captures the TIMEOUT event. It should be noted that when a deadlock occurs, the healthy process is blocked by communication. Therefore, with the help of the TIMEOUT event captured by the NCCL interception library, the Brain component can obtain the TIMEOUT event reported by all healthy processes, so the remaining process is the faulty process. It should be noted that the Brain component will count the single iteration time collected from the training process and calculate the maximum value T as the waiting time window after capturing the TIMEOUT event to wait for other healthy processes to send TIMEOUT events.

[0064] S103: Determine a globally available training parameter version based on the faulty process, and use the globally available training parameter version to restore the faulty process.

[0065] After determining the faulty process, the training parameter version of the healthy process and the training parameter version of the faulty process can be determined. It can be understood that, generally, the training parameter version of the healthy process may have one more iteration than the training parameter version of the faulty process, or it may be the same number of iterations. If the training parameter version of the healthy process may have one more iteration than the training parameter version of the faulty process, the training parameter version of the faulty process shall be used as the globally available training parameter version. If the training parameter versions of the healthy process and the faulty process are the same, either the training parameter version of the faulty process or the training parameter version of the healthy process may be used as the globally available training parameter version.

[0066] It should be noted that the operations that the training script needs to perform include registering the pre_hook function before the optimization step Optimizer step to set the environment variable PARAM_VERSION parameter version to the current iteration round i; registering the post_hook function after the optimization step Optimizer step to set the environment variable PARAM_VERSION parameter version to i+1, and counting the execution intervals of two adjacent post_hooks to obtain the execution time of one iteration and sending it to the Brain component through rpc.

[0067] Exemplarily, if the failure occurs in the forward reverse phase of the iteration, the training parameter versions PARAM_VERSION of all processes are i, so the training parameter versions of healthy processes and faulty processes are globally consistent and are both i. At this time, the globally available training parameter version is i.

[0068] If the failure occurs during the Optimizer step, the training parameter version PARAM_VERSION of the failed process is i, and the healthy process will continue to execute after the Optimizer step until a collective communication operation is required, so the PARAM_VERSION of the healthy process is globally consistent i+1. At this time, the globally available training parameter version is i.

[0069] The fault process recovery method provided by the present application, after receiving the first state information sent by the working process of at least one node, the Brain can analyze the first state confidence, determine the faulty process that has failed, and then determine the globally available training parameter version based on the acquired training parameter version and the faulty process, so as to roll back the faulty process with the globally available training parameter version. The present application can detect the faulty process after the fault occurs and determine the globally available training parameter version, so as to gray out the faulty process, and only need to check after the fault occurs, which reduces the detection frequency, saves time cost, and ensures the consistency of the training parameter version after the fault.

[0070] In some embodiments, the abnormal event that occurs in the work process includes at least one of the following:

[0071] The worker process exits;

[0072] There is a work process that exceeds the preset training time; the preset training time is the maximum value of a single iteration time of model training.

[0073] Specifically, the faulty process may be an abnormal exit of a working process, or a working process that exceeds a preset training time, namely, a TIMEOUT event.

[0074] For example, the abnormal exit of the worker process can be an error code exit, because the Worker process, that is, the training process, is created by the Master process, so the Master process can perceive the exit of its child process. Therefore, if a training process sends a fault and returns an error code to exit, the Master process can directly locate the faulty process and report the faulty process information to the Brain component. For a worker process that exceeds the preset training time, it may be a deadlock. On the surface, all processes will be stuck in waiting due to deadlock, but the healthy process is blocked by communication. Therefore, with the help of the TIMEOUT event captured by the NCCL interception library, the Brain component can obtain the TIMEOUT event reported by all healthy processes sent by the NCCL interception library, so the remaining process is the faulty process.

[0075] In some embodiments, before receiving the first status information of the working process in the working process, the method further includes:

[0076] The second status information sent by the main process in the receiving node includes heartbeat information and fault process information; the working process is created by the main process;

[0077] The fault location is determined according to the heartbeat information or the fault process information; the fault location includes a fault node and a fault process.

[0078] It can be understood that the fault process recovery method provided in the present application can also detect node failures. Before the Brain component determines whether a process has failed, it must first determine whether there is a node failure. If a node failure already exists, there is no need to determine whether the working process has failed. The node failure must be resolved first. In addition, the Master process in the present application can also capture the abnormal exit of the worker process in real time.

[0079] In some embodiments, determining the fault location according to the heartbeat information or the fault process information includes:

[0080] According to the heartbeat information, determining that a node that has not received the heartbeat has a node failure;

[0081] The fault process is determined according to the fault process information; the fault process information includes a fault error code.

[0082] In this application, if the Brain component does not receive the heartbeat information, it is determined that there is no heartbeat node, and this failure is a node failure; if an error code exits during the training process, the master process can directly locate the faulty process and send the faulty process information to the Brain component.

[0083] For example, the Brain component receives the heartbeat information regularly reported by the Master process. If the Brain component does not receive the heartbeat of a Master process, it is considered that the node has failed, so all training processes in the node have failed. Alternatively, if the Master process senses the exit of its child process, for example, the training process sends a fault return error code to exit, the Master process can directly locate the faulty process and report the faulty process information to the Brain component.

[0084] In some embodiments, after determining the faulty process where the fault occurs, the method further includes:

[0085] Query the data parallel group corresponding to the faulty process, and determine whether there is a healthy process in the data parallel group;

[0086] If it exists, determine the globally available training parameter version; otherwise send an exit command to the main process.

[0087] After the faulty process is determined, the application queries the data parallel group corresponding to the faulty process to determine whether there is a healthy process in the data parallel group. If not, it means that checkpoint cannot be saved after the failure. The Brain component sends an exit to the Master process assigned to each node. The Master process sends a SIGINT signal to terminate all training processes. If it exists, the globally available training parameter version VALIDATE_VERSION is determined next.

[0088] It is understandable that after determining the faulty process that has failed, first determine the data parallel group of the faulty process, and determine whether there is a healthy process in the data parallel group. Only with a healthy process can there be a corresponding training parameter version. If there is no healthy process, it means that all processes may fail, and the globally available training parameter version cannot be determined. At this time, the faulty process cannot be saved, and the Brain component will issue an exit assigned to the Master process. If there is a healthy process in the data parallel group of the faulty process, the training parameter version of the healthy process and the training parameter version of the faulty process are determined to determine which training parameter version can be globally applicable. It should be noted that, in general, the training parameter version with the previous iteration is selected as the globally available training parameter version. For example, the training parameter version of the faulty process is i, and the training parameter version of the healthy process is i+1, then the training parameter i is used as the globally available training parameter version. In this way, not only can the training process be rolled back, but also the training parameters closest to the process without failure are selected, saving training time.

[0089] In some embodiments, determining a globally available training parameter version based on the fault process includes:

[0090] Determine the order of iterations in which the faulty process occurs and the stage of the optimization step;

[0091] The training parameter version of the healthy process and the training parameter version of the faulty process are determined respectively according to the iteration order and the stage of the optimization step, and the training parameter version of the healthy process or the training parameter version of the faulty process is determined to be the globally available training parameter version.

[0092] Specifically, based on the number of iterations of the faulty process, and depending on whether the optimization step occurs, it can be determined whether the healthy process has normally performed the next iteration.

[0093] Exemplarily, assuming that the failure occurs in the i-th iteration, if the failure occurs in the forward reverse stage of the iteration, that is, optimization has not yet been performed, then the PARAM_VERSION of all processes is i, so the model parameter version of the healthy process is globally consistent as i, and the globally available training parameter version is i; if the failure occurs in the optimizer step, the PARAM_VERSION of the failed process is i, and the healthy process will continue to execute after the optimizer step until a collective communication operation is required, so the PARAM_VERSION of the healthy process is globally consistent as i+1, and the globally available training parameter version is i.

[0094] In some embodiments, determining that the training parameter version of the healthy process or the training parameter version of the faulty process is a globally available training parameter version includes:

[0095] Depending on the stage of the optimization step in different work processes, the training parameter version is updated synchronously or asynchronously to be the globally available training parameter version.

[0096] It is understandable that in this application, different optimization steps may lead to different training parameter versions for different work processes. For example, parameter updates may be synchronous. In most pipeline parallel practices, synchronization between pipeline stages is usually performed in the optimizer step; or parameter updates are asynchronous, because the optimization steps of different processes in pipeline parallelism are executed sequentially, so that the previous stage receives the partially reduced global state, combines it with the local state of the current stage, and then passes it to the next stage. Finally, after receiving the global state, each stage verifies whether the previous optimization step is valid and determines the globally available training parameter version.

[0097] In some embodiments, according to the stage of the optimization step in different work processes, the training parameter version is synchronously updated to a globally available training parameter version, including:

[0098] If the faulty process occurs in the first phase of the iteration, the training parameter version of the faulty process is the same as the training parameter version of the healthy process;

[0099] If the faulty process occurs in the second phase of the iteration, the training parameter version of the faulty process is determined as the globally available training parameter version.

[0100] Among them, the synchronous update training parameter version is a globally available training parameter version. Specifically, the synchronization between pipeline stages is performed in the optimizer step to ensure the stability of the values, such as Figure 4As shown in the figure, if the faulty process occurs in the forward reverse phase of the iteration, all processes have not yet updated their parameters. Therefore, the training parameter versions of the faulty process and the healthy process have not been updated and are still the current training parameter versions. If the faulty process occurs in the optimization step of the iteration, the faulty process cannot iterate due to the fault, while the healthy process iterates normally to update the training parameter version. Therefore, the training parameter version that has not been updated by the faulty process is the globally available training parameter version.

[0101] For example, if the failure occurs in the forward and backward phase of the iteration, the PARAM_VERSION of all processes is i, so the model parameter version of the healthy process is globally consistent as i; if the failure occurs in the optimization step Optimizerstep, the training parameter version PARAM_VERSION of the failed process is i, and the globally available training parameter version is i. The healthy process will continue to execute after the optimization step Optimizerstep until a collective communication operation is required, so the training parameter version PARAM_VERSION of the healthy process is globally consistent as i+1, and the globally available training parameter version is i+1.

[0102] In some embodiments, according to the stage of the optimization step in different work processes, the training parameter version is asynchronously updated to a globally available training parameter version, including:

[0103] Determine the versions of training parameters for different stages of the optimization step of the working process;

[0104] The minimum value among the training parameter versions at different stages in the worker process is determined as the globally available training parameter version.

[0105] It is understandable that the parameter update in this application can also be asynchronous, such as Figure 5 As shown, the optimization steps of different processes in the pipeline parallel are executed sequentially, where: Figure 5 1-4 in it means propagating from 1 to 4 to reduce the local value, and 5-8 means propagating the global reduced value to each stage; this application must first determine whether the optimization step of each stage in the working process is executed. Before the optimization step of each stage, the global state of partial reduction is received from the previous stage, combined with the local state of the current stage, and then passed to the next stage. It should be noted that the optimizer step of each stage is controlled by the partial reduction state, and finally the minimum value of the training parameter versions of different stages in the working process is determined as the globally available training parameter version.

[0106] Exemplarily, the optimization steps of different processes in the pipeline are executed in sequence, such as Figure 4 As shown, before the optimization step of each stage, the partially reduced global state is received from the previous stage, combined with the local state of the current stage, and then passed to the next stage. The optimization step of each stage is controlled by the partially reduced state. For example, the update is skipped when it is detected that the norm of the partially reduced gradient exceeds the clipping threshold. In the warmup phase of the next iteration, the fully reduced global state is propagated back to the first stage from the last stage. After receiving the global state, each stage will perform verification to determine whether the previous optimization step is valid. If the gradient needs to be modified, a parameter rollback operation will be performed.

[0107] As an example, assume that device3 fails. Depending on the time of the failure, there are several possible situations:

[0108] (1) The fault occurs in the weight gradient calculation stage, that is, Figure 4 In the stages before 1, 2, 3, and 4: At this time, the process of stage 1 and stage 3 will continue to execute parameter updates and enter the next iteration. Stage 2 will fall into send blocking when passing the local state after the specification before the optimization step Optimizer step. Similarly, stage 4 will fall into receive blocking when waiting for stage 3 to send itself the global state of the partial specification before the optimization step Optimizer step. Therefore, the training parameter version PARAM_VERSION of stage 2 and stage 3 is i when the TIMEOUT event is captured. However, stage 1 will wait to receive the global specification state sent by stage 2 after the warmup phase of the i+1th iteration, so the PARAM_VERSION of stage 1 is i+1 when the TIMEOUT event is captured.

[0109] (2) The failure occurs in the parameter update phase, which is the phase after 1, 2, 3, and 4 in the figure. At this time, except for phase 3, phases 1, 2, and 4 will complete the parameter update and enter the i+1th iteration. Phase 2 will fall into the send blocking after executing a micro-batch forward. Phase 4 will fall into the receive blocking of the forward calculation after updating the parameters. Phase 1 will fall into the receive blocking of receiving the global protocol status after warmup is executed. Therefore, when all healthy processes capture the TIMEOUT event, PARAM_VERSION is i+1.

[0110] (3) The failure occurs in the warmup phase: At this time, all healthy processes have completed the parameter update phase, so when the TIMEOUT event is captured, PARAM_VERSION is i+1.

[0111] In the fault process recovery method provided in this application, the Brain component takes the smallest PARAM_VERSION from the TIMEOUT events of all collected healthy processes as the globally available training parameter version VALIDATE_VERSION.

[0112] After the Brain component determines the globally available training parameter version, it sends the save instruction and the globally available training parameter version to the Master process of each node. The Master process then sends the processing signal SIGTERM to all healthy processes and sends the parameter version VALIDATE_VERSION through the pipeline. The signal processing function of the healthy process performs the rollback operation. Figure 6 As shown, the working process reads the model parameter version in the pipeline file. It should be noted that the model parameter version is a globally available training parameter version. Each working process determines whether the current training parameter version is a globally available training parameter version. If not, parameter rollback is performed, so that when the working process is rolled back to the globally available training parameter version, the training parameter version of the current process is saved. If so, the training parameter version of the current process is directly saved, and then the result is saved and written to the pipeline file, so that all processes execute the globally available training parameter version, thereby realizing fault recovery.

[0113] An embodiment of the present application provides an electronic device, including: at least one processor; and

[0114] a memory communicatively connected to the at least one processor; wherein,

[0115] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform

[0116] Receive first status information of a working process in at least one node; the first status information includes abnormal events and training parameter versions occurring in at least two working processes of at least one node;

[0117] Analyze the first state information, and determine the faulty process where the fault occurs according to the working process where the abnormal event occurs;

[0118] A globally available training parameter version is determined based on the faulty process, and the globally available training parameter version is used to restore the faulty process.

[0119] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device and a readable storage medium.

[0120] Figure 7A schematic block diagram of an example electronic device 700 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0121] like Figure 7 As shown, the device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0122] A number of components in the device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0123] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above, such as the fault process recovery method. For example, in some embodiments, the fault process recovery method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the fault process recovery method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform the fault process recovery method in any other appropriate manner (e.g., by means of firmware).

[0124] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0125] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0126] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0127] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0128] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0129] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0130] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0131] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present disclosure, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0132] The above is only a specific embodiment of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present disclosure, which should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A method for recovering a failed process, the method comprising: Receiving first status information of a working process in at least one node; The first state information includes abnormal events and training parameter versions occurring in at least two working processes of at least one node; Analyze the first state information, and determine the faulty process where the fault occurs according to the working process where the abnormal event occurs; A globally available training parameter version is determined based on the faulty process, and the globally available training parameter version is used to restore the faulty process.

2. According to the method of claim 1, the abnormal event occurring in the work process includes at least one of the following: The worker process exits; There is a work process that exceeds the preset training time; the preset training time is the maximum value of a single iteration time of model training.

3. The method according to claim 1, before receiving the first status information of the working process in the working process, further comprises: The second status information sent by the main process in the receiving node; The second status information includes heartbeat information and fault process information; The working process is created by the main process; The fault location is determined according to the heartbeat information or the fault process information; the fault location includes a fault node and a fault process.

4. The method according to claim 3, wherein determining the fault location according to the heartbeat information or the fault process information comprises: According to the heartbeat information, determining that a node that has not received the heartbeat has a node failure; determining a fault process according to the fault process information; The fault process information includes a fault error code.

5. The method according to claim 1, wherein determining a globally available training parameter version based on the fault process comprises: Determine the order of iterations in which the faulty process occurs and the stage of the optimization step; The training parameter version of the healthy process and the training parameter version of the faulty process are determined respectively according to the iteration order and the stage of the optimization step, and the training parameter version of the healthy process or the training parameter version of the faulty process is determined to be the globally available training parameter version.

6. The method according to claim 5, wherein determining the training parameter version of the healthy process or the training parameter version of the faulty process as a globally available training parameter version comprises: Depending on the stage of the optimization step in different work processes, the training parameter version is updated synchronously or asynchronously to be the globally available training parameter version.

7. The method according to claim 6, wherein according to the stage of the optimization step in different work processes, the training parameter version is synchronously updated to a globally available training parameter version, comprising: If the faulty process occurs in the first phase of the iteration, the training parameter version of the faulty process is the same as the training parameter version of the healthy process; If the faulty process occurs in the second phase of the iteration, the training parameter version of the faulty process is determined as the globally available training parameter version.

8. The method according to claim 6, wherein the training parameter version is asynchronously updated to a globally available training parameter version according to the stage of the optimization step in different work processes, comprising: Determine the versions of training parameters for different stages of the optimization step of the working process; The minimum value among the training parameter versions at different stages in the worker process is determined as the globally available training parameter version.

9. The method according to claim 3, after determining the faulty process where the fault occurs, further comprises: Query the data parallel group corresponding to the faulty process, and determine whether there is a healthy process in the data parallel group; If it exists, determine the globally available training parameter version; otherwise send an exit command to the main process.

10. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform Receive first status information of a working process in at least one node; the first status information includes abnormal events and training parameter versions occurring in at least two working processes of at least one node; Analyze the first state information, and determine the faulty process where the fault occurs according to the working process where the abnormal event occurs; A globally available training parameter version is determined based on the faulty process, and the globally available training parameter version is used to restore the faulty process.

Images