Fault recovery method, computer device, and computer-readable storage medium

Abstract

The present application relates to the technical field of artificial intelligence, and particularly provides a fault recovery method, a computer device, and a computer-readable storage medium, aiming at ensuring the stability of a model training process and improving the training efficiency. To achieve this objective, the method provided by the present application comprises: acquiring and storing a temporary model generated by training in a model training process, and monitoring whether a fault occurs in model training; and when the occurrence of a fault is monitored, on the basis of the type of the fault, actively or passively eliminating the fault, and after the fault is eliminated, initiating a training recovery task; and using the temporary model to execute the training recovery task, so as to continue the model training. Based on the method, after a fault in model training is eliminated, a temporary model generated by training can be used to continue the model training without starting the model training from the beginning, so that the model training efficiency is greatly improved, and the waste of training resources is reduced.

Description

Fault recovery method, computer device, and computer-readable storage medium

[0001] This application claims priority to Chinese patent application No. 202311220238.0, filed on September 20, 2023, entitled “Method for fault recovery, computer device and computer-readable storage medium”. The entire contents of the above Chinese patent application are incorporated into this application by reference. Technical Field

[0002] The present application relates to the field of artificial intelligence technology, and in particular to a fault recovery method, a computer device, and a computer-readable storage medium. Background Art

[0003] Large AI (Artificial Intelligence) models (such as those for autonomous driving) are typically trained in a distributed environment with multiple machines and GPUs. During training, each machine and each GPU (Graphics Processing Unit) on the machine may fail, resulting in training interruptions. Furthermore, because the training process for large AI models is typically lengthy, the probability of machine and GPU failure increases, increasing the risk of training interruptions. Restarting training after an interruption inevitably wastes significant time and GPU resources, reducing the training efficiency of large AI models.

[0004] Accordingly, this field requires a new technical solution to solve the above problems.

[0005] Summary of the Invention

[0006] In order to overcome the above-mentioned defects, the present application is proposed to provide a fault recovery method, computer device and computer-readable storage medium that solve or at least partially solve the technical problem of how to ensure the stability of the model training process and improve the training efficiency.

[0007] In a first aspect, a method for fault recovery is provided, the method comprising:

[0008] Obtain and store the temporary model generated during model training, and monitor whether model training fails;

[0009] When a fault is detected, the system will proactively or passively resolve the fault based on the type of fault and initiate a training recovery task after the fault is resolved.

[0010] The temporary model is used to perform the training recovery task to continue model training.

[0011] In one technical solution of the above fault recovery method, the step of "initiating a training recovery task" specifically includes:

[0012] Obtaining storage information of the temporary model;

[0013] The training recovery task is initiated according to the stored information so that the corresponding temporary model can be called according to the stored information to perform the training recovery task.

[0014] In one technical solution of the above fault recovery method, the step of "obtaining and storing a temporary model generated during the model training process" specifically includes: regularly obtaining and storing a temporary model generated during the model training process;

[0015] The step of "obtaining storage information of the temporary model" specifically includes: determining whether designated information of the temporary model is received;

[0016] If yes, then obtaining storage information of a temporary model specified by the specified information, wherein the specified temporary model is one of all temporary models obtained regularly before the failure occurs;

[0017] If not, then obtain storage information of a preset temporary model among all temporary models obtained regularly before the failure occurs.

[0018] In one technical solution of the above-mentioned fault recovery method, the step of "actively or passively resolving the fault according to the type of fault and initiating a training recovery task after the fault is resolved" includes:

[0019] If the fault type is known, the system will proactively resolve the fault and initiate a training recovery task after the fault is resolved.

[0020] If the fault type is unknown, the fault is resolved passively and a training recovery task is initiated after the fault is resolved.

[0021] In one technical solution of the above fault recovery method, the method further includes monitoring whether an unknown type of fault occurs by the following means:

[0022] During the model training process, the effect indicators of the model training are obtained in real time or at regular intervals, and it is determined whether the effect indicators are not obtained within a preset time period and the model training is not stopped;

[0023] If so, it is determined that an unknown type of fault has occurred;

[0024] If not, it is determined that no unknown type of fault has occurred.

[0025] In one technical solution of the above-mentioned fault recovery method, before the step of "actively performing fault resolution and initiating a training recovery task after the fault resolution", the method includes:

[0026] Determining whether the known fault type is a fault type that requires manual intervention to resolve;

[0027] If so, the fault is resolved passively and a training recovery task is initiated after the fault is resolved, rather than proactively resolving the fault and initiating a training recovery task after the fault is resolved;

[0028] If not, the system will proactively troubleshoot the problem and initiate a training recovery task after the problem is solved.

[0029] In one technical solution of the above-mentioned fault recovery method, the method further includes actively resolving the fault and initiating a training recovery task after the fault is resolved in the following manner: obtaining and executing a preset fault resolution operation corresponding to the fault type according to the fault type to resolve the fault, and initiating the training recovery task after the fault resolution operation is completed;

[0030] The method also includes passively performing fault resolution and initiating a training recovery task after the fault resolution in the following manner: performing fault resolution in response to a user's fault resolution operation, and initiating a training recovery task in response to a user's training recovery operation after the fault resolution.

[0031] In one technical solution of the above fault recovery method, the step of "obtaining and storing a temporary model generated during the model training process" specifically includes:

[0032] When a distributed training system is used for model training, controlling each processor of each computer device in the distributed training system to respectively obtain model fragments of the temporary model;

[0033] The model fragments obtained by each processor are stored separately through the distributed storage system.

[0034] In one technical solution of the above fault recovery method, the distributed storage system is a distributed cache system, and the step of "storing the model fragments obtained by each processor separately through the distributed storage system" specifically includes:

[0035] For each model fragment, determine whether the storage space occupied by the model fragment is greater than the set value;

[0036] If so, the model fragment is persistently cached;

[0037] If not, the model fragment is cached in a non-persistent manner.

[0038] In a second aspect, a computer device is provided, which includes a processor and a storage device, wherein the storage device is suitable for storing multiple program codes, and the program codes are suitable for being loaded and run by the processor to execute the method described in any one of the technical solutions of the above-mentioned fault recovery method.

[0039] In a third aspect, a computer-readable storage medium is provided, wherein a plurality of program codes are stored in the computer-readable storage medium, wherein the program codes are suitable for being loaded and run by a processor to execute the method described in any one of the technical solutions of the above-mentioned fault recovery method.

[0040] The above one or more technical solutions of this application have at least one or more of the following beneficial effects:

[0041] In the technical solution of the fault recovery method provided in the present application, a temporary model generated during the model training process can be obtained and stored, and the model training can be monitored for faults; when a fault is detected, the fault can be actively or passively resolved according to the type of fault and a training recovery task can be initiated after the fault is resolved, and then the temporary model can be used to perform the training recovery task to continue model training. Based on the above implementation, the model training can be continued using the temporary model generated during training after the fault of model training is resolved, without the need to retrain the model from scratch, which greatly improves the efficiency of model training and reduces the waste of training resources (such as GPU resources). In addition, the above implementation can use active or passive methods to resolve faults and restore training according to different fault types. The automation of fault resolution and recovery training is achieved based on the active method, and the faults that cannot be automatically resolved can be effectively resolved based on the passive method, ensuring that model training can proceed normally. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The disclosure of this application will become more easily understood with reference to the accompanying drawings. Those skilled in the art will readily appreciate that these drawings are for illustrative purposes only and are not intended to limit the scope of protection of this application. Among them:

[0043] FIG1 is a flow chart showing the main steps of a method for fault recovery according to an embodiment of the present application;

[0044] FIG2 is a flow chart showing the main steps of a method for acquiring and storing a temporary model generated during a model training process according to an embodiment of the present application;

[0045] FIG3 is a flowchart illustrating the main steps of a method for performing fault resolution and initiating a training recovery task according to an embodiment of the present application;

[0046] FIG4 is a flowchart illustrating the main steps of a method for initiating a training recovery task according to an embodiment of the present application;

[0047] FIG5 is a schematic diagram of the main structure of a computer device according to an embodiment of the present application. DETAILED DESCRIPTION

[0048] Some embodiments of the present application are described below with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are only used to explain the technical principles of the present application and are not intended to limit the scope of protection of the present application.

[0049] In the description of this application, a "processor" may include hardware, software, or a combination of the two. A processor may be a central processing unit, a microprocessor, an image processor, a digital signal processor, or any other suitable processor. A processor has data and / or signal processing capabilities. A processor may be implemented in software, hardware, or a combination of the two. Computer-readable storage media include any suitable medium capable of storing program code, such as a magnetic disk, a hard disk, an optical disk, flash memory, read-only memory, random access memory, and the like.

[0050] The relevant user personal information that may be involved in the various embodiments of this application is strictly in accordance with the requirements of laws and regulations, following the principles of legality, legitimacy and necessity, and based on the reasonable purposes of business scenarios, to process the personal information that users actively provide during the use of products / services or generated due to the use of products / services, as well as the personal information obtained with the user's authorization.

[0051] The user personal information processed by this application will vary depending on the specific product / service scenario and must be based on the specific scenario in which the user uses the product / service. This may involve the user's account information, device information, driving information, vehicle information, or other related information. This application will treat the user's personal information and its processing with a high degree of diligence.

[0052] This application attaches great importance to the security of user personal information and has taken reasonable and feasible security protection measures that comply with industry standards to protect user information and prevent personal information from being accessed, disclosed, used, modified, damaged or lost without authorization.

[0053] The following describes an embodiment of the fault recovery method provided in this application.

[0054] Referring to FIG1 , FIG1 is a flow chart showing the main steps of a fault recovery method according to an embodiment of the present application. As shown in FIG1 , the fault recovery method in the embodiment of the present application mainly includes the following steps S101 to S103 .

[0055] Step S101: Acquire and store a temporary model generated during the model training process, and monitor whether the model training fails.

[0056] The model refers to an AI model. In some embodiments, the model can be a large AI model. For example, the model can be a large autonomous driving model, which includes but is not limited to an environment perception model, a planning and decision-making model, and the like.

[0057] When training a model, it is usually performed through multiple iterations until a preset model convergence condition is met, at which point training is stopped. The preset model convergence condition can be when a model training performance indicator (such as accuracy) meets a preset condition or when the number of iterations reaches a preset threshold.

[0058] In this embodiment, the model before training can be understood as the initial model, the model after training can be understood as the final model, and the model in the process between the start of training and the completion of training can be understood as a temporary model. In addition, in this process, after each iterative training of the model, the model that has completed the iterative training can be used as a temporary model, that is, each iterative training can generate a temporary model.

[0059] In this embodiment, a checkpoint (CKPT) algorithm can be used to obtain and store a temporary model generated during model training. Based on this, all model parameters in the temporary model can be saved to a file. When the temporary model is needed, the model parameters can be obtained from this file and then restored to the temporary model.

[0060] Step S102: When a fault is detected, the fault is resolved actively or passively according to the type of the fault and a training recovery task is initiated after the fault is resolved.

[0061] In this embodiment, model fault handling methods include active and passive methods. Active means proactively resolving the fault and initiating a training recovery task after the fault is resolved. Passive means passively resolving the fault and initiating a training recovery task after the fault is resolved. Different handling methods are used for different fault types. Specifically, if the fault type can be automatically resolved without human intervention, active handling can be used; otherwise, passive handling is required.

[0062] In some implementations, for fault types that require active handling, fault resolution operations can be pre-configured for each type. When a fault type that requires active handling occurs, the corresponding fault resolution operation can be retrieved and executed based on the fault type to resolve the fault. After the fault resolution operation is completed, the training recovery task can be initiated. For example, a fault type requiring active handling might be a network failure on a computer device performing model training, and the fault resolution operation might be connecting to a backup network.

[0063] For fault types that adopt a passive processing method, prompt information can be output so that the user can promptly understand that a fault has occurred in the model training according to the prompt information and take measures to resolve the fault. In addition, the fault can be resolved in response to the user's fault resolution operation, and the training recovery task can be initiated in response to the user's training recovery operation after the fault is resolved. The training recovery operation includes but is not limited to the training recovery instruction generated by the user trigger. For example, the fault type that adopts an active processing method may be that a disk failure occurs in the computer device for model training. After the user discovers the disk failure according to the prompt information, the user can manually replace a new disk to resolve the disk failure. This step is to resolve the fault in response to the user's fault resolution operation (manually replacing a new disk). After resolving the fault, the user can operate the above-mentioned computer device to generate a training recovery instruction, so that the training recovery task can be initiated according to this training recovery instruction.

[0064] Based on the above-mentioned active and passive processing methods, faults that can be automatically resolved can be automatically resolved, and faults that cannot be automatically resolved can be resolved in response to user operations, which significantly improves the ability to resolve faults and ensures that model training can proceed normally.

[0065] Step S103: Execute the training recovery task using the temporary model to continue model training. Specifically, for the remaining training iterations after the fault is resolved, the temporary model will be used as the basis for the remaining training iterations. If a fault is detected again during the remaining training iterations, the method described in steps S101 to S103 can still be used to handle it.

[0066] Based on the method described in steps S101 to S103 above, the model training can be continued using the temporary model generated by the training after the model training fault is resolved, without having to re-train the model from the beginning, which greatly improves the efficiency of the model training and reduces the waste of training resources (such as GPU resources).

[0067] The above steps S101 to S103 are further explained below.

[0068] 1. Explain step S101.

[0069] In some implementations of the above step S101, the temporary model generated by training can be obtained and stored through the following steps S1011 to S1012 shown in Figure 2.

[0070] Step S1011: When using a distributed training system for model training, control each processor of each computer device in the distributed training system to respectively obtain model fragments of a temporary model. In this embodiment, each iterative training can be completed by the processors working together, that is, in each iterative training, each processor jointly trains to generate a temporary model, and the models generated by each process can be understood as model fragments of this temporary model, and all model fragments can be combined into this temporary model. Similar to obtaining a temporary model in the aforementioned step S101, each processor can also use the CKPT algorithm to obtain and store the model fragments generated by its own training.

[0071] Step S1012: The model fragments acquired by each processor are stored separately in the distributed storage system. Compared with centralized storage, using the distributed storage system to store model fragments can flexibly expand the storage space according to actual needs and improve the storage reliability of the system.

[0072] In some embodiments, the distributed storage system can be a distributed cache system. In this case, for each model fragment, it can be determined whether the storage space occupied by the model fragment is greater than the set value; if so, it indicates that the data volume of the model fragment is relatively large. If the model fragment is lost due to system failure, etc., it will take a long time to cache it again. Therefore, in order to prevent the loss of the model fragment, the model fragment can be cached persistently; if not, it indicates that the data volume of the model fragment is relatively small. If the model fragment is lost due to system failure, etc., it will not take a long time to cache it again. Therefore, in order to improve the reading speed of the model fragment, the model fragment can be cached non-persistently.

[0073] Based on the method described in steps S1011 to S1012 above, a distributed training system can be used in combination with a distributed storage system to achieve distributed training of the model. In this embodiment, conventional distributed training methods in the field of artificial intelligence technology can be adopted, and model training can be performed based on conventional distributed training systems. This embodiment of the application does not specifically limit the distributed training method and distributed training system, as long as distributed training can be completed.

[0074] In some implementations of step S101 above, the fault type may include known types and unknown types. For known types, monitoring methods for each known type of fault may be pre-set, and each monitoring method may be executed during fault monitoring. For unknown types, monitoring may be performed through the following steps S1013 to S1015.

[0075] Step S1013: During the model training process, the effect indicators of the model training are obtained in real time or at regular intervals, and it is determined whether the effect indicators have not been obtained within the preset time period and the model training has not stopped. If so, it indicates that the model training has not stopped, but there is no output. At this time, it is very likely that some unknown fault has occurred in the model training, so go to step S1014; if not, it indicates that some unknown fault has not occurred at this time, so go to step S1015. Effect indicators include but are not limited to accuracy, precision, and recall rate. Step S1014: Determine whether an unknown type of fault has occurred; Step S1015: Determine whether an unknown type of fault has occurred.

[0076] Based on the method described in steps S1013 to S1015 above, it is possible to easily and accurately determine whether an unknown type of fault has occurred by monitoring the effect indicators.

[0077] 2. Explain step S102.

[0078] According to the description of the above-mentioned step S101, the types of faults may include known types and unknown types. In some implementations of the above-mentioned step S102, the fault can be resolved and the training recovery task can be initiated through the following steps S1021 to S1023 shown in Figure 3.

[0079] Step S1021: Get the type of fault, which includes known types and unknown types. Step S1022: If the type of fault is a known type, the operation to resolve the fault is usually also known, so the fault can be resolved actively and a training recovery task can be initiated after the fault is resolved. Step S1023: If the type of fault is an unknown type, the operation to resolve the fault is usually also unknown, and manual intervention is required to resolve it, so the fault needs to be resolved passively and a training recovery task needs to be initiated after the fault is resolved. Among them, the method of actively resolving the fault and initiating a training recovery task is similar to the method for the fault type that adopts an active processing method in the aforementioned step S102; the method of passively resolving the fault and initiating a training recovery task is similar to the method for the fault type that adopts a passive processing method in the aforementioned step S102, and will not be repeated here.

[0080] Based on the method described in steps S1021 to S1023 above, known types of faults are automatically resolved, and unknown types of faults can be resolved in response to user operations, which significantly improves the fault resolution capability and ensures that model training can proceed normally.

[0081] Furthermore, in some implementations of the above-mentioned step S1022, when the type of fault is a known type, it can be determined whether this known type is a fault type that requires manual intervention to resolve; if so, it indicates that the fault cannot be resolved automatically, so the fault is resolved passively and a training recovery task is initiated after the fault is resolved, rather than actively resolving the fault and initiating a training recovery task after the fault is resolved; if not, it indicates that the fault can be resolved automatically, so the fault is actively resolved and a training recovery task is initiated after the fault is resolved. Through the above-mentioned implementation, when it is determined that the fault is of a known type, it is possible to further analyze whether the current fault can be resolved automatically and whether manual intervention is required to ensure that the fault can be resolved reliably. For example, although the disk failure is of a known type, the user needs to manually replace the disk, so the fault cannot be resolved automatically and requires manual intervention. The fault must be resolved passively and a training recovery task must be initiated after the fault is resolved.

[0082] In some implementations of the above step S102, the training recovery task may be initiated through the following steps S1024 to S1025 shown in FIG4 .

[0083] Step S1024: Obtain storage information of the temporary model.

[0084] The storage information may be a storage address or a storage path of the temporary model. The temporary model can be accurately acquired from the storage space where the temporary model is located by using the storage information.

[0085] Step S1025: Initiate a training recovery task according to the stored information, so that the corresponding temporary model can be called according to the stored information to perform the training recovery task.

[0086] Specifically, the storage information may be used as task information of the training recovery task, so that the storage information may be obtained by parsing the training recovery task.

[0087] In some embodiments, a temporary model generated during model training can be periodically acquired and stored. During step S1025, it can be determined whether information specifying the temporary model has been received. If this information has been received, the stored information of the temporary model specified by this information is acquired. This designated temporary model is one of all temporary models acquired periodically before the fault occurs. If this information has not been received, the stored information of a preset temporary model among all temporary models acquired periodically before the fault occurs is acquired. In some preferred embodiments, the preset temporary model can be the stored information of the last temporary model acquired periodically before the fault occurs. Based on this, model training can be continued using the most recently stored temporary model, minimizing time lost due to interruptions in model training and improving model training efficiency. Furthermore, in some embodiments, the specified temporary model information can be manually entered or input by the user through a human-computer interaction device, allowing the user to select the temporary model to be used. Those skilled in the art can flexibly set the frequency of periodic acquisition and storage of temporary models based on actual needs, and this embodiment of the present application does not specifically limit this.

[0088] Based on the method described in steps S1024 to S1025 above, after initiating the training recovery task, the temporary model to be continued to be trained can be accurately obtained, ensuring that the model training can be carried out stably and reliably.

[0089] It should be pointed out that although the various steps in the above embodiments are described in a specific order, those skilled in the art will understand that in order to achieve the effect of the present application, different steps do not have to be performed in such an order. They can be performed simultaneously (in parallel) or in other orders. These adjusted solutions are equivalent to the technical solutions described in this application, and therefore will also fall within the scope of protection of this application.

[0090] It will be understood by those skilled in the art that all or part of the processes in the method for implementing the above embodiment of the present application can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps of each of the above method embodiments. The computer program includes computer program code, which can be in source code form, object code form, executable file or some intermediate form. The computer-readable storage medium can include: any entity or device, medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory, random access memory, electric carrier signal, telecommunication signal and software distribution medium that can carry the computer program code. It should be noted that the content contained in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable storage media do not include electric carrier signals and telecommunication signals.

[0091] Furthermore, the present application also provides a computer device.

[0092] Refer to Figure 5, which is a schematic diagram of the main structure of a computer device embodiment according to the present application. As shown in Figure 5, the computer device in the embodiment of the present application mainly includes a storage device and a processor. The storage device can be configured to store a program for executing the fault recovery method of the above-mentioned method embodiment, and the processor can be configured to execute the program in the storage device, which includes but is not limited to a program for executing the fault recovery method of the above-mentioned method embodiment. For ease of explanation, only the parts related to the embodiment of the present application are shown. For specific technical details not disclosed, please refer to the method section of the embodiment of the present application.

[0093] In the embodiment of the present application, the computer device may be a control device device formed by various electronic devices. In some possible implementations, the computer device may include multiple storage devices and multiple processors. The program for executing the fault recovery method of the above-mentioned method embodiment can be divided into multiple subroutines, and each subroutine can be loaded and run by the processor to execute different steps of the fault recovery method of the above-mentioned method embodiment. Specifically, each subroutine can be stored in different storage devices respectively, and each processor can be configured to execute the program in one or more storage devices to jointly implement the fault recovery method of the above-mentioned method embodiment, that is, each processor executes different steps of the fault recovery method of the above-mentioned method embodiment respectively to jointly implement the fault recovery method of the above-mentioned method embodiment.

[0094] The aforementioned multiple processors may be processors deployed on the same device. For example, the aforementioned computer device may be a high-performance device composed of multiple processors, and the aforementioned multiple processors may be processors configured on the high-performance device. Furthermore, the aforementioned multiple processors may also be processors deployed on different devices. For example, the aforementioned computer device may be a server cluster, and the aforementioned multiple processors may be processors on different servers in the server cluster.

[0095] Furthermore, the present application also provides a computer-readable storage medium.

[0096] In an embodiment of a computer-readable storage medium according to the present application, the computer-readable storage medium can be configured to store a program for executing the fault recovery method of the above-mentioned method embodiment, and the program can be loaded and run by the processor to implement the above-mentioned fault recovery method. For ease of explanation, only the parts related to the embodiment of the present application are shown. For specific technical details not disclosed, please refer to the method part of the embodiment of the present application. The computer-readable storage medium can be a storage device formed by various electronic devices. Optionally, the computer-readable storage medium in the embodiment of the present application is a non-transitory computer-readable storage medium.

[0097] Thus far, the technical solution of the present application has been described in conjunction with an embodiment shown in the accompanying drawings. However, it is readily understood by those skilled in the art that the scope of protection of the present application is obviously not limited to these specific embodiments. Without departing from the principles of the present application, those skilled in the art may make equivalent changes or substitutions to the relevant technical features, and the technical solutions after such changes or substitutions will fall within the scope of protection of the present application.

Claims

1. A method for fault recovery, characterized in that: The method comprises: Obtain and store temporary models generated during model training, and monitor whether model training fails; When a fault is detected, the fault is resolved actively or passively according to the type of fault and a training recovery task is initiated after the fault is resolved; The temporary model is used to perform the training recovery task to continue model training.

2. The method according to claim 1, characterized in that The steps of "initiating a training recovery task" include: Obtaining storage information of the temporary model; The training recovery task is initiated according to the storage information so that the corresponding temporary model can be called according to the storage information to perform the training recovery task.

3. The method according to claim 2, characterized in that The step of "obtaining and storing a temporary model generated during the model training process" specifically includes: regularly obtaining and storing a temporary model generated during the model training process; The step of "obtaining storage information of the temporary model" specifically includes: Determine whether the specified information of the temporary model is received; If yes, then obtaining storage information of a temporary model specified by the specified information, wherein the specified temporary model is one of all temporary models obtained regularly before the failure occurs; If not, then obtain storage information of a preset temporary model among all temporary models obtained regularly before the failure occurs.

4. The method according to claim 1, characterized in that The steps of "actively or passively performing fault elimination according to the type of fault and initiating a training recovery task after the fault elimination" include: If the fault type is known, the system will proactively resolve the fault and initiate a training recovery task after the fault is resolved. If the fault type is unknown, the fault is resolved passively and a training recovery task is initiated after the fault is resolved.

5. The method according to claim 4, characterized in that The method further includes monitoring whether a fault of an unknown type occurs by: Acquire the effect index of the model training in real time or periodically during the model training process, and determine whether the effect index is not acquired within a preset time and the model training is not stopped; If yes, it is determined that an unknown type of fault has occurred; If not, it is determined that no unknown type of fault has occurred.

6. The method according to claim 4, characterized in that Before the step of "actively performing fault resolution and initiating a training recovery task after the fault resolution", the method includes: Determine whether the known type is a fault type that requires manual intervention to resolve; If yes, the fault is resolved passively and a training recovery task is initiated after the fault is resolved, instead of actively resolving the fault and initiating a training recovery task after the fault is resolved; If not, the fault is corrected proactively and a training recovery task is initiated after the fault is corrected.

7. The method according to claim 1, characterized in that The method further includes actively performing fault removal and initiating a training recovery task after the fault removal in the following manner: acquiring and executing a preset fault removal operation corresponding to the type of the fault according to the type of the fault to remove the fault, and initiating a training recovery task after completing the execution of the fault removal operation; The method also includes passively performing fault relief and initiating a training recovery task after the fault relief in the following manner: performing fault relief in response to a user's fault relief operation, and initiating a training recovery task in response to a user's training recovery operation after the fault relief.

8. The method according to claim 1, characterized in that The steps of "obtaining and storing the temporary model generated during model training" specifically include: When a distributed training system is used for model training, each processor of each computer device in the distributed training system is controlled to respectively obtain model fragments of the temporary model; The model fragments obtained by each processor are stored separately through the distributed storage system.

9. The method according to claim 8, characterized in that The distributed storage system is a distributed cache system, and the step of "storing the model fragments obtained by each processor respectively through the distributed storage system" specifically includes: For each model fragment, determine whether the storage space occupied by the model fragment is greater than a set value; If yes, the model fragment is persistently cached; If not, the model fragment is cached in a non-persistent manner.

10. A computer device comprising a processor and a storage device, wherein the storage device is suitable for storing a plurality of program codes, characterized in that: The program code is suitable for being loaded and executed by the processor to execute the fault recovery method according to any one of claims 1 to 9.

11. A computer-readable storage medium storing a plurality of program codes, characterized in that: The program code is suitable for being loaded and run by a processor to execute the fault recovery method according to any one of claims 1 to 9.

Images