Error recovery method and device, equipment and storage medium

Abstract

The invention provides an error recovery method and device, equipment and a medium, and relates to the technical field of computers.The error recovery method comprises the steps that in response to a training task for a target large model, configuration information of the training task is obtained; according to the configuration information of the training task, the training task is divided into a plurality of training sub-tasks, each training sub-task is allocated to a corresponding training node, and each training node comprises a computing card and a target service; determining whether a calculation card used for executing the corresponding training subtask in the training node to which the target service belongs has an error or not through each target service; and when it is determined that any computing card has an error, error recovery is carried out on the computing card. According to the invention, the interruption duration of large model training can be reduced, and the continuity and efficiency of large model training can be improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular to an error recovery method, apparatus, device and medium. Background Art

[0002] With the continuous development of artificial intelligence (AI) technology, the application scenarios of AI technology are becoming more and more complex, and the corresponding models used by AI technology are becoming larger and larger. Therefore, relying on a single machine and acceleration equipment for large model training can no longer meet the demand, so distributed training has become the mainstream method for large model training.

[0003] In related technologies, the distributed training method is to use multiple training nodes in a cluster to jointly train a large model to improve the training speed and performance of the large model. Usually, a computing card is deployed on each training node in the cluster. Although the computing card has significant performance advantages over the central processing unit (CPU), the failure rate is relatively high. Since there are many training nodes in the cluster and the corresponding number of computing cards is also large, when a computing card on any training node fails due to a hardware error, the failure may spread rapidly, triggering a chain reaction, and thus seriously affecting the training of the entire large model.

[0004] At present, in order to effectively deal with the high failure rate of computing cards and ensure the stable operation of large model training, manual methods are generally used to determine the errors that occur in the computing cards and repair them, which takes a long time and causes long interruptions in large model training, affecting the continuity and efficiency of large model training. Summary of the invention

[0005] The embodiments of the present application provide an error recovery method, apparatus, device and medium, which can reduce the interruption duration of large model training and improve the continuity and efficiency of large model training.

[0006] In a first aspect, an embodiment of the present application provides an error recovery method, comprising: In response to a training task for a target large model, obtaining configuration information of the training task; According to the configuration information of the training task, the training task is divided into a plurality of training subtasks, and each of the training subtasks is assigned to a corresponding training node, wherein each of the training nodes includes a computing card and a target service; Determine, through each of the target services, whether an error occurs in a computing card in the training node for executing the corresponding training subtask; When it is determined that an error occurs in any computing card, error recovery is performed on the computing card.

[0007] In a second aspect, an embodiment of the present application provides an error recovery device, including: An information acquisition module, configured to obtain configuration information of a training task in response to a training task for a target large model; A task allocation module, used to divide the training task into multiple training subtasks according to the configuration information of the training task, and allocate each of the training subtasks to a corresponding training node, wherein each of the training nodes includes a computing card and a target service; An error detection module, used to determine whether an error occurs in a computing card in a training node for executing a corresponding training subtask through each of the target services; The error recovery module is used to perform error recovery on any computing card when it is determined that an error occurs in the computing card.

[0008] In a third aspect, an embodiment of the present application provides an electronic device, including: A processor and a memory, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute the error recovery method as described in the first aspect embodiment and various implementation methods mentioned above.

[0009] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium for storing a computer program, wherein the computer program enables a computer to execute the error recovery method as described in the embodiment of the first aspect and various implementation methods.

[0010] In a fifth aspect, an embodiment of the present application provides a computer program product comprising program instructions. When the program instructions are executed on an electronic device, the electronic device executes the error recovery method as described in the embodiment of the first aspect and various implementation methods.

[0011] The technical solution disclosed in the embodiment of the present application obtains the configuration information of the training task in response to the training task for the target large model, and divides the training task into multiple training subtasks according to the configuration information of the training task, and assigns each training subtask to the corresponding training node, and then determines through the target service whether an error occurs in the computing card used to execute the corresponding training subtask. When it is determined that an error occurs in any computing card, the computing method is error recovered. In this way, it is possible to automatically detect whether an error occurs in the computing card on the training node, and when it is determined that an error occurs in any computing card, the computing card with the error is automatically recovered, thereby reducing the interruption duration of the large model training and improving the continuity and efficiency of the large model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0013] Figure 1 A flowchart of an error recovery method provided in an embodiment of the present application; Figure 2 A schematic diagram of a cluster structure provided in an embodiment of the present application; Figure 3 A flowchart for determining whether an error occurs in a computing card is provided in an embodiment of the present application; Figure 4 A schematic block diagram of an error recovery device provided in an embodiment of the present application; Figure 5 A schematic block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0014] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. According to the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0015] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0016] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or scheme described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or schemes. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0017] In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more than two, that is, at least two. "At least one" means one or more than one, that is, one or more. "Any" means any one or any several.

[0018] As mentioned earlier, when using distributed training to train a large model, in order to effectively deal with the high failure rate of computing cards and ensure the stable operation of large model training, manual methods are generally used to determine the errors that occur in the computing cards and repair them, which takes a long time and causes long interruptions in large model training, affecting the continuity and efficiency of large model training.

[0019] In response to the above technical problems, the present application provides an error recovery method, device, equipment and medium to solve the problem that manually determining and repairing errors occurring in computing cards takes a long time, resulting in long interruptions in large model training, affecting the continuity and efficiency of large model training.

[0020] The technical solutions provided by the embodiments of the present application are described in detail below through some embodiments. The embodiments described below can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0021] Figure 1 A flowchart of an error recovery method provided for an embodiment of the present application. The error recovery method of the embodiment of the present application can be performed by an error recovery device, which can be composed of hardware and / or software and can be integrated into an electronic device. The electronic device can be selected as any hardware device with a data processing function, such as a server, a virtual machine, a computer, and other types of devices, etc., and the type of electronic device is not specifically limited here. In addition, a cluster for performing large model training tasks, such as a Kubernetes cluster (K8S cluster), etc., is deployed in the above-mentioned electronic device, so that the error recovery method provided in the present application can be applied to the above-mentioned cluster for performing large model training tasks, that is, the error recovery method is performed by the above-mentioned cluster for performing large model training tasks.

[0022] It should be understood that the cluster for executing the large model training task may include multiple nodes, and the multiple nodes include two types: the first type is the control node, and the second type is the working node, which can be Figure 2 As shown. The control node is used to manage and control the entire cluster, and may include an API server component, a scheduler component, a distributed key-value storage component, and a controller manager component. The working node is used to execute control instructions sent by the control node, such as executing training tasks sent by the control node. In other words, the working node in the cluster used to execute model training tasks can be called a training node.

[0023] like Figure 1 As shown, the method may include the following steps: S101, in response to a training task for a target large model, obtaining configuration information of the training task.

[0024] The target large model may be a large model that supports any processing function, such as image recognition, image classification, etc., and no limitation is imposed on it here.

[0025] It should be understood that a large model refers to a machine learning model with a large number of parameters and complex structure.

[0026] In this application, the configuration information of the training task can be understood as the training information of the training task, such as the training node identifier, distributed training parameters, model architecture and other parameter information, etc. Among them, the training node identifier refers to the unique information of the training node, that is, the training node used to execute the training task can be uniquely identified according to the training node identifier.

[0027] It should be understood that the above cluster may include two types of resource pools, the first type is the user's own private resource pool, and the second type is the cluster's public resource pool, wherein each type of resource pool may include multiple working nodes. Moreover, each working node in the public resource pool will not be located in any user's private resource pool.

[0028] That is, the training node corresponding to the training node identifier in the configuration information of the training task of the present application can be a working node in the user's private resource pool; or, it can also be a working node in the cluster's public resource pool; or, at least one can be a working node in the user's private resource pool and at least one can be a working node in the cluster's public resource pool.

[0029] Optionally, when the target large model needs to be trained, the user can send a training task for the target large model to the cluster through a visual interface, so that when the control node in the cluster receives the training task for the target large model sent by the user, it can obtain the configuration information of the training task by parsing the training task.

[0030] S102, dividing the training task into a plurality of training subtasks according to the configuration information of the training task, and assigning each training subtask to a corresponding training node, wherein each training node includes a computing card and a target service.

[0031] In the present application, the computing card is specifically a graphics processing unit (GPU).

[0032] In addition, the target service is a data collection service that is used to collect the operating information of the computing card on the training node during the execution of the corresponding training subtask, and to determine whether an error occurs in the computing card on the training node based on the operating information.

[0033] In some optional embodiments, the control node may determine the number of training nodes participating in the training of the target large model according to the training node identifier of the training task. Furthermore, the control node may divide the training task into a plurality of training subtasks corresponding to the number of training nodes according to the number of training nodes and the resource conditions or training capabilities of each training node. Then, each training subtask is assigned to the corresponding training node according to the corresponding relationship between the training node and the training subtask through the scheduler component in the control node, so that the training node can execute the corresponding training subtask in parallel.

[0034] Taking into account that the user sends the training task of the target large model, the training task can be manually split into multiple training subtasks according to the distributed training requirements, and the corresponding training node can be configured for each training subtask. Therefore, when it is determined that there are multiple split training subtasks and training nodes corresponding to each training subtask in the configuration information of the training task, the present application can allocate each training subtask to the corresponding training node through the scheduler in the control node according to the correspondence between the training node and the training subtask, so as to avoid the control node from performing the split operation on the training task and improve the training speed of the target large model.

[0035] S103, determining through each target service whether an error occurs in a computing card in the corresponding training node for executing the corresponding training subtask.

[0036] After receiving the training subtasks assigned by the scheduler component, the computing cards in each training node can execute the corresponding training subtasks in parallel. Considering that the probability of errors in the computing cards in the training nodes is relatively high, during the process of the computing cards in each training node executing the corresponding training subtasks, each target service can continuously monitor and collect the operation information of the computing cards on the training nodes to which it belongs during the execution of the corresponding training subtasks, and judge whether the computing cards on the training nodes to which it belongs have errors based on the operation information.

[0037] In some optional embodiments, each target service determines whether an error occurs in the computing card on the training node to which it belongs based on the collected operation information, and the operation information may be compared with each error information in the preset error set to determine whether the operation information hits the error information in the preset error set. If it is determined that the operation information hits any error information in the preset error set, it is determined that an error occurs in the computing card on the training node to which it belongs; if it is determined that the operation information does not hit all the error information in the preset error set, it is determined that no error occurs in the computing card on the training node to which it belongs.

[0038] In the present application, the preset error set is constructed in advance by collecting error information during the operation of the computing card and based on the collected error information.

[0039] S104: When it is determined that an error occurs in any computing card, error recovery is performed on the computing card.

[0040] In some optional embodiments, when an error occurs in any computing card, the present application may prohibit the continued allocation of the training subtask of the target large model to the training node to which the computing card where the error occurs belongs, and wait for the computing card where the error occurs to complete the task being executed. This can prevent the error from spreading and causing a wider impact on the training of the target large model.

[0041] Considering that the preset error set may include, in addition to the error information, an error repair file corresponding to the error information; or an error repair file corresponding to the error information may be stored in other storage modules. Therefore, when it is determined that an error has occurred in any computing card, the present application prohibits the continued allocation of the training subtask of the target large model to the training node to which the computing card with the error belongs, and at the same time, determines the error recovery file based on the error information of each computing card. Then, based on each error recovery file, the error that occurred in the corresponding computing card is recovered so that the computing card returns to normal, and the corresponding training subtask can continue to be executed.

[0042] Optionally, the error recovery file is determined based on the error information of each computing card, and the error recovery file can be determined from the mapping relationship between the error information and the error recovery file based on the error information generated by each computing card. The mapping relationship between the error information and the error recovery file can be stored in a preset error set or in other storage modules, and no limitation is imposed on it here.

[0043] In some optional embodiments, considering that when recovering the error occurring in the corresponding computing card according to the error recovery file, a situation that cannot be recovered may occur. Then, the present application can replace the training node including the computing card where the error occurred through the control node, so as to execute the corresponding training subtask through the computing card in the replaced training node. At the same time, the control node can also send a prompt message to the user to replace the training node, so that the user can grasp the training status of the target large model in real time. Alternatively, the control node can also send a prompt message to the user that an unrecoverable error has occurred in the computing card in any training node, please replace the training node, so that the user can replace the training node including the error according to the prompt message, thereby executing the corresponding training subtask through the computing card in the replaced training node.

[0044] In addition, after the training node including the erroneous computing card is replaced, the control node will also send a manual recovery prompt to the technician so that the technician can perform manual repairs on the training node including the erroneous computing card offline. This can reduce the interruption time of the target large model training, while also reducing manual intervention and saving a lot of time costs.

[0045] It should be noted that, considering that the training nodes participating in the training of the target large model may be nodes in the user's private resource pool, when an unrecoverable error occurs in the computing card in the private node, the present application can move the private node from the user's private resource pool to the cluster's public resource pool, and move the nodes of the same specification in the public resource pool to the user's private resource pool. In this way, technical personnel can manually repair the training nodes in the public resource pool where unrecoverable errors occur in the computing cards. At the same time, users can use the new nodes of the same specifications moved into the private resource pool to continue to participate in the training tasks of the target large model.

[0046] Considering that there is a situation in this application where no error occurs in each computing card, no processing is performed at this time, and the corresponding training subtask can continue to be executed through each computing card.

[0047] The technical solution disclosed in the embodiment of the present application obtains the configuration information of the training task in response to the training task for the target large model, and divides the training task into multiple training subtasks according to the configuration information of the training task, and assigns each training subtask to the corresponding training node, and then determines through the target service whether an error occurs in the computing card used to execute the corresponding training subtask. When it is determined that an error occurs in any computing card, the computing method is error recovered. In this way, it is possible to automatically detect whether an error occurs in the computing card on the training node, and when it is determined that an error occurs in any computing card, the computing card with the error is automatically recovered, thereby reducing the interruption duration of the large model training and improving the continuity and efficiency of the large model training.

[0048] Based on the above embodiments, considering that the target large model in this application is trained in a distributed manner, it is required that each training node can communicate normally. However, due to some hardware problems, calculation errors or communication failures may occur during the training of the target large model.

[0049] Therefore, after each training subtask is assigned to the corresponding training node through the scheduler component in the control node, the computing card in each training node can first be subjected to a hardware detection operation to determine whether the computing card in each training node is abnormal. If it is determined that there is no abnormality in the computing card in each training node, the corresponding training subtask is executed by the computing card in each training node. If it is determined that the computing card in any training node is abnormal, a reminder message is sent to the user to remind the computing card in which training node is abnormal, so that the user can replace the training node with the abnormal computing card according to the reminder message, thereby using the computing card in the new training node to execute the corresponding training subtask. In this way, it is possible to determine in advance the computing cards that are abnormal and participate in the training task of the target large model, thereby improving the stability and reliability of model training.

[0050] Optionally, after replacing the training node where the computing card has an abnormality, the present application may also perform hardware detection on the computing card in the new training node. When the computing card does not have an abnormality, the computing card in the new training node is used to execute the corresponding training subtask. Otherwise, the training node continues to be replaced and the computing cards in the replaced training nodes are detected until the computing card in the latest replaced training node does not have an abnormality.

[0051] The hardware detection operation on the computing card includes at least one of detecting whether the computing card in each training node operates normally and detecting whether the computing cards in each training node communicate normally.

[0052] In some optional embodiments, whether the computing card in each training node is operating normally and whether the computing cards in each training node are communicating normally can be detected by running a pre-created detection script file to detect whether the computing card in each training node is operating normally and whether the computing cards in each training node are communicating normally. The detection script file can be a targeted detection file created according to the target large model training task.

[0053] In some optional embodiments, the target service deployed in each training node of the present application may be a log collection service. That is, the present application may collect and monitor the system log information of the training node in real time through each log collection service to determine whether an error occurs in the computing card of the training node according to the system log information. Figure 3 , step S103 in the above embodiment is further explained. Figure 3 As shown, Figure 1 The S103 may include the following steps S103-1 and S103-2: S103-1, obtain the log information of the computing card in the corresponding training node through each target service.

[0054] S103-2, through each target service, according to the log information and error set of the computing card in the training node, determine whether an error occurs in the computing card in the training node.

[0055] In some optional embodiments, the present application can obtain the log information of the computing card used to execute the corresponding training subtask in the training node in real time through the log collection service in each training node during the process of the computing card in each training node executing the corresponding training subtask. Then, each log collection service compares the obtained log information with each error information in the preset error set to determine whether the log information hits the error information in the preset error set. If it is determined that the log information hits any error information in the preset error set, it is determined that an error has occurred in the computing card on the training node; if it is determined that the log information does not hit all the error information in the preset error set, it is determined that no error has occurred in the computing card on the training node.

[0056] That is to say, the present application determines whether an error occurs in the computing card in the training node to which it belongs based on the log information and error set of the computing card in the training node to which it belongs through each target service, specifically including: determining through each target service whether the log information of the computing card in the training node to which it belongs hits the error information in the error set; when determining through each target service that the log information of the computing card in the training node hits any error information in the error set, it is determined that an error occurs in the computing card in the training node; when determining through each target service that the log information of the computing card in the training node does not hit the error information in the error set, it is determined that no error occurs in the computing card in the training node.

[0057] In some embodiments, considering that the preset error set is pre-built based on the collected existing error information, it is possible that an error has indeed occurred on the computing card, but the error is not stored in the preset error set. In other words, a new error has occurred on the computing card. Then, based on the preset error set and the log information of the computing card, it may not be possible to determine that an error has occurred on the computing card, and thus it may be mistakenly believed that no error has occurred on the computing card. Then, when the computing card executes the corresponding training subtask, training failure or training interruption may occur.

[0058] To address this problem, the present application can output prompts of training failure or training interruption to the technicians, and enable the technicians to manually check what kind of error occurred in the computing card in the training node based on the prompt information, and provide a recovery file for the error, so as to recover new errors that occurred in the computing card based on the recovery file.

[0059] Furthermore, after resolving a new error that occurs on the computing card, the present application can also update the new error and the error repair file corresponding to the new error to a preset error set, so that when the same error occurs again in the future, the error can be automatically detected and automatically recovered, thereby effectively reducing manual involvement and improving the automation of error detection and error recovery of the computing card.

[0060] The technical solution disclosed in the embodiment of the present application obtains the configuration information of the training task in response to the training task for the target large model, and divides the training task into multiple training subtasks according to the configuration information of the training task, and assigns each training subtask to the corresponding training node, and then determines through the target service whether an error occurs in the computing card used to execute the corresponding training subtask. When it is determined that an error occurs in any computing card, the computing method is error recovered. In this way, it is possible to automatically detect whether an error occurs in the computing card on the training node, and when it is determined that an error occurs in any computing card, the computing card with the error is automatically recovered, thereby reducing the interruption duration of the large model training and improving the continuity and efficiency of the large model training.

[0061] Please refer to the attached Figure 4 , an error recovery device proposed in an embodiment of the present application is described. Figure 4 As shown, the error recovery device 300 includes: an information acquisition module 310 , a task allocation module 320 , an error detection module 330 and an error recovery module 340 .

[0062] The information acquisition module 310 is used to obtain configuration information of the training task in response to the training task for the target large model; A task allocation module 320, configured to divide the training task into a plurality of training subtasks according to the configuration information of the training task, and allocate each of the training subtasks to a corresponding training node, wherein each of the training nodes includes a computing card and a target service; An error detection module 330 is used to determine, through each of the target services, whether an error occurs in a computing card in a training node for executing a corresponding training subtask; The error recovery module 340 is used to perform error recovery on any computing card when it is determined that an error occurs in the computing card.

[0063] In an optional implementation of the embodiment of the present application, the error recovery module 340 is specifically configured to: Determining an error recovery file according to error information generated by each of the computing cards; According to each of the error recovery files, the error occurring in the corresponding computing card is recovered.

[0064] In an optional implementation of the embodiment of the present application, the error recovery device 300 further includes: The node replacement module is used to replace the training node including the computing card if the error repair of any computing card fails, so as to execute the corresponding training subtask through the computing card in the replaced training node.

[0065] In an optional implementation of the embodiment of the present application, the target service is a log collection service, and the error detection module 330 includes: A log acquisition module, used to acquire log information of a computing card in a training node through each target service; The determination module is used to determine whether an error occurs in the computing card in the training node according to the log information and error set of the computing card in the training node through each of the target services.

[0066] In an optional implementation of the embodiment of the present application, the determination module is specifically used to: Determine, through each of the target services, whether the log information of the computing card in the training node to which it belongs hits the error information in the error set; When it is determined through each of the target services that the log information of the computing card in the training node hits any error information in the error set, it is determined that an error occurs in the computing card in the training node; When it is determined through each of the target services that the log information of the computing card in the training node does not hit the error information in the error set, it is determined that no error occurs in the computing card in the training node.

[0067] In an optional implementation of the embodiment of the present application, the error recovery device 300 further includes: A detection module, used to perform hardware detection operations on the computing card in each of the training nodes; The hardware detection operation includes at least one of: detecting whether a computing card in each of the training nodes operates normally and detecting whether computing cards in each of the training nodes communicate normally.

[0068] It should be understood that the device embodiment and the aforementioned method embodiment may correspond to each other, and similar descriptions may refer to the method embodiment. To avoid repetition, they will not be described here. Specifically, Figure 4 The device 300 shown can perform Figure 1 The corresponding method embodiments, and the aforementioned and other operations and / or functions of each module in the device 300 are respectively to implement Figure 1 For the sake of brevity, the corresponding processes in each method are not repeated here.

[0069] In the above, the device 300 of the embodiment of the present application is described from the perspective of the functional module in conjunction with the accompanying drawings. It should be understood that the functional module can be implemented in hardware form, can be implemented by instructions in software form, and can also be implemented by a combination of hardware and software modules. Specifically, the steps of the first aspect method embodiment in the embodiment of the present application can be completed by the hardware integrated logic circuit and / or software form instructions in the processor, and the steps of the first aspect method disclosed in conjunction with the embodiment of the present application can be directly embodied as a hardware decoding processor to perform, or a combination of hardware and software modules in the decoding processor to perform. Optionally, the software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory, and completes the steps in the above-mentioned first aspect method embodiment in conjunction with its hardware.

[0070] Figure 5 A schematic block diagram of an electronic device provided in an embodiment of the present application. Figure 5 As shown, the electronic device 400 may include: a memory 410 and a processor 420, wherein the memory 410 is used to store a computer program and transmit the program code to the processor 420. In other words, the processor 420 may call and run the computer program from the memory 410 to implement the error recovery method described in the first aspect of the embodiment of the present application.

[0071] For example, the processor 420 may be configured to execute the above error recovery method according to instructions in the computer program.

[0072] In some embodiments of the present application, the processor 420 may include but is not limited to: General-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0073] In some embodiments of the present application, the memory 410 includes but is not limited to: Volatile memory and / or non-volatile memory. Among them, the non-volatile memory can be read-only memory (ROM), programmable ROM (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM) or flash memory. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct memory bus random access memory (Direct Rambus RAM, DR RAM).

[0074] In some embodiments of the present application, the computer program may be divided into one or more modules, which are stored in the memory 410 and executed by the processor 420 to complete the error recovery method provided by the present application. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program in the electronic device.

[0075] like Figure 5 As shown, the electronic device 400 may further include: The transceiver 430 may be connected to the processor 420 or the memory 410 .

[0076] The processor 420 may control the transceiver 430 to communicate with other devices, specifically, to send information or data to other devices, or to receive information or data sent by other devices. The transceiver 430 may include a transmitter and a receiver. The transceiver 430 may further include an antenna, and the number of antennas may be one or more.

[0077] It should be understood that the various components in the electronic device are connected via a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus and a status signal bus.

[0078] The present application also provides a computer storage medium on which a computer program is stored. When the computer program is executed by a computer, the computer is enabled to execute the error recovery method described in the first aspect above.

[0079] The present application also provides a computer program product including program instructions. When the program instructions are executed on an electronic device, the electronic device executes the error recovery method described in the first aspect above.

[0080] When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a server, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server, or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital video disc (DVD)), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0081] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0082] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the module is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0083] The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. For example, each functional module in each embodiment of the present application may be integrated into a processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0084] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories), or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0085] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An error recovery method, characterized in that: include: In response to a training task for a target large model, obtaining configuration information of the training task; According to the configuration information of the training task, the training task is divided into a plurality of training subtasks, and each of the training subtasks is assigned to a corresponding training node, wherein each of the training nodes includes a computing card and a target service; Determine, through each of the target services, whether an error occurs in a computing card in the training node for executing the corresponding training subtask; When it is determined that an error occurs in any computing card, error recovery is performed on the computing card.

2. The error recovery method according to claim 1, characterized in that: When it is determined that an error occurs in any computing card, error recovery is performed on the computing card, including: Determining an error recovery file according to error information generated by each of the computing cards; According to each of the error recovery files, the error occurring in the corresponding computing card is recovered.

3. The error recovery method according to claim 2, characterized in that: Also includes: If the error repair of any computing card fails, the training node including the computing card is replaced to execute the corresponding training subtask through the computing card in the replaced training node.

4. The error recovery method according to claim 1, characterized in that: The target service is a log collection service, and determining whether an error occurs in a computing card for executing a corresponding training subtask in a training node through each of the target services includes: Obtain log information of the computing card in the training node through each of the target services; Through each of the target services, it is determined whether an error occurs in the computing card in the training node to which it belongs according to the log information and error set of the computing card in the training node to which it belongs.

5. The error recovery method according to claim 4, characterized in that: The step of determining whether an error occurs in a computing card in a training node according to log information and an error set of the computing card in the training node by each of the target services includes: Determine, through each of the target services, whether the log information of the computing card in the training node to which it belongs hits the error information in the error set; When it is determined through each of the target services that the log information of the computing card in the training node hits any error information in the error set, it is determined that an error occurs in the computing card in the training node; When it is determined through each of the target services that the log information of the computing card in the training node does not hit the error information in the error set, it is determined that no error occurs in the computing card in the training node.

6. The error recovery method according to claim 1, characterized in that: Also includes: Performing hardware detection operations on the computing cards in each of the training nodes; The hardware detection operation includes at least one of: detecting whether a computing card in each of the training nodes operates normally and detecting whether computing cards in each of the training nodes communicate normally.

7. An error recovery device, characterized in that: include: An information acquisition module, configured to obtain configuration information of a training task in response to a training task for a target large model; A task allocation module, used to divide the training task into multiple training subtasks according to the configuration information of the training task, and allocate each of the training subtasks to a corresponding training node, wherein each of the training nodes includes a computing card and a target service; An error detection module, used to determine whether an error occurs in a computing card in a training node for executing a corresponding training subtask through each of the target services; The error recovery module is used to perform error recovery on any computing card when it is determined that an error occurs in the computing card.

8. An electronic device, characterized in that: include: A processor and a memory, the memory being used to store a computer program, and the processor being used to call and run the computer program stored in the memory to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: Used to store a computer program, wherein the computer program causes a computer to execute the method according to any one of claims 1 to 6.

10. A computer program product comprising program instructions, characterized in that When the program instructions are executed on an electronic device, the electronic device is caused to execute the method according to any one of claims 1 to 6.

Images