Fault detection method and device, equipment and storage medium

Abstract

The invention provides a fault detection method, device and equipment and a storage medium, and relates to the technical field of computers, in particular to the technical fields of deep learning, large model training, fault detection and the like. According to the specific implementation scheme, a plurality of computing devices are determined, wherein the computing devices are used for performing model training based on a pipeline parallel strategy; determining parameters and a scheduling strategy adopted by the pipeline parallel strategy; determining idle time of each computing device in the plurality of computing devices in a model training process based on the parameters and a scheduling strategy; and performing fault detection on each computing device in the idle time of each computing device in the model training process. According to the invention, online fault detection of the computing device can be realized.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to technical fields such as deep learning, large model training, and fault detection. Background Art

[0002] In recent years, with the continuous expansion of the scale of deep learning models, distributed cluster training has become a necessary technical means to train large deep learning models. In actual cluster training, computing devices often fail. Traditional fault detection methods usually require pausing the entire training task and performing offline fault detection. This method is not only time-consuming, but also reduces the efficiency of model training. Therefore, how to achieve real-time fault detection of computing devices without affecting training efficiency has become a problem that needs to be solved urgently. Summary of the invention

[0003] The present disclosure provides a fault detection method, apparatus, device and storage medium.

[0004] According to one aspect of the present disclosure, there is provided a fault detection method, comprising:

[0005] Determine a plurality of computing devices, where the plurality of computing devices are used for model training based on a pipeline parallel strategy;

[0006] Determine the parameters and scheduling strategy adopted by the pipeline parallel strategy;

[0007] Based on the parameter and the scheduling strategy, determine the idle time of each computing device in the plurality of computing devices during the model training process;

[0008] During the idle time of each computing device during the model training process, fault detection is performed on each computing device.

[0009] According to another aspect of the present disclosure, there is provided a fault detection device, comprising:

[0010] A first determination module is used to determine a plurality of computing devices, where the plurality of computing devices are used to perform model training based on a pipeline parallel strategy;

[0011] A second determination module is used to determine the parameters and scheduling strategy adopted by the pipeline parallel strategy;

[0012] A third determination module is used to determine the idle time of each computing device in the plurality of computing devices during the model training process based on the parameter and the scheduling strategy;

[0013] The fault detection module is used to perform fault detection on each computing device during the idle time of each computing device during the model training process.

[0014] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0015] at least one processor; and

[0016] a memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any method in the embodiments of the present disclosure.

[0018] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any method according to the embodiments of the present disclosure.

[0019] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements any method according to the embodiments of the present disclosure when executed by a processor.

[0020] The present disclosure determines the idle time of each computing device during the model training process by using the parameters and scheduling strategy based on the pipeline parallel scheduling strategy, and performs fault detection on each computing device during these spatial times. Since fault detection is performed during the idle time, there is no need to interrupt the model training process, and each computing device can be online for fault detection, which reduces the interruption time during the model training process and thus improves the efficiency of model training.

[0021] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0023] Figure 1 is a schematic diagram of an application scenario according to an embodiment of the present disclosure;

[0024] Figure 2 is a flow chart of an implementation of a fault detection method according to an embodiment of the present disclosure;

[0025] Figure 3 It is a schematic diagram of a non-interleaved pipeline scheduling strategy;

[0026] Figure 4 It is a schematic diagram of a staggered pipeline scheduling strategy;

[0027] Figure 5It is a schematic diagram of the idle time of each computing device in the non-interleaved pipeline scheduling strategy;

[0028] Figure 6 It is a schematic diagram of the idle time of each computing device in the staggered pipeline scheduling strategy;

[0029] Figure 7 is a structural schematic diagram of a fault detection device 700 according to an embodiment of the present disclosure;

[0030] Figure 8 A schematic block diagram of an example electronic device 800 that may be used to implement embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0031] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0032] The “and / or” in the embodiments of the present disclosure indicates that three relationships may exist. For example, A and / or B may indicate the following three situations: A exists alone, A and B exist at the same time, and B exists alone. The term “at least one” herein indicates any combination of at least two of any one or more of a plurality of types. For example, at least one of A, B, and C may indicate any one or more elements selected from the set consisting of A, B, and C. The terms “first” and “second” herein refer to and distinguish between multiple similar technical terms, and do not mean to limit the order or to limit the meaning to only two. For example, the first feature and the second feature refer to two types / two features. The first feature may be one or more, and the second feature may also be one or more.

[0033] In recent years, with the rapid development of deep learning technology, the scale of models has been expanding. This trend has made large-scale distributed training a necessary means to train large and complex deep learning models. The pipeline parallelism strategy is an efficient strategy in distributed training methods. This strategy distributes different layers of the model to different computing devices, allowing the model to perform forward and backward propagation on multiple computing devices simultaneously. Different computing devices can perform calculations at different stages of the model to achieve pipeline parallel computing. In pipeline parallel computing, data can be transmitted between adjacent computing devices through communication links. This parallel processing method can significantly reduce the computational load of a single computing device, but it also requires a more sophisticated and complex coordination mechanism to ensure the smooth progress of the training process.

[0034] However, in actual large-scale cluster training, although the pipeline parallel strategy has brought significant performance improvements, equipment failures still occur frequently. These failures may be caused by a variety of reasons, including: uncertainty in model calculation accuracy, errors caused by hardware aging, and unstable network connections. The occurrence of these failures will not only have a negative impact on the accuracy of training results and reduce model performance, but may also cause the entire training task to be interrupted, resulting in a waste of time and computing resources.

[0035] In the current fault detection methods, it is usually necessary to pause the entire training task and perform offline troubleshooting and repair on each device. This method is not only time-consuming, but also wastes computing resources due to long downtime. Therefore, how to achieve online fault detection of devices without affecting training efficiency, timely discover and handle potential problems, and ensure the continuity and stability of model training is an urgent problem to be solved in the field of deep learning.

[0036] In order to solve the above problem, an embodiment of the present disclosure proposes a fault detection method. Figure 1 is a schematic diagram of an application scenario according to an embodiment of the present disclosure, such as Figure 1As shown, the application scenario diagram of the embodiment of the present disclosure may include but is not limited to a fault detection device 110 and a computing device cluster 120, and the fault detection device 110 and the computing device cluster 120 may communicate through any type of wired or wireless network. Specifically, the computing device cluster 120 calls a fault detection program to the fault detection device 110 to perform fault detection on the computing devices in the computing device cluster 120 that are in an idle state. The embodiment of the present disclosure does not impose specific restrictions on the number of fault detection devices 110. For example, the application scenario diagram of the embodiment of the present disclosure may include one or more fault detection devices 110. In the embodiment of the present disclosure, the computing device may be a high-performance computer, a graphics processing unit (GPU), a tensor processing unit (TPU), or any other device that can support deep learning model training.

[0037] Figure 2 is a flowchart of a fault detection method according to an embodiment of the present disclosure, comprising:

[0038] S210, determining a plurality of computing devices, where the plurality of computing devices are used to perform model training based on a pipeline parallel strategy;

[0039] S220, determining the parameters and scheduling strategy adopted by the pipeline parallel strategy;

[0040] S230, based on the parameter and the scheduling strategy, determining the idle time of each computing device in the plurality of computing devices during the model training process;

[0041] S240. Perform fault detection on each computing device during the idle time of each computing device during the model training process.

[0042] By determining the idle time of each computing device when adopting the pipeline scheduling strategy for model training and performing fault detection on each computing device during the idle time, it is possible to avoid interruption of model training due to fault detection and improve the efficiency of model training.

[0043] In some embodiments, the parameters include pipeline dimensions.

[0044] In the disclosed embodiment, the parameters of the pipeline scheduling strategy include the pipeline dimension, also known as the pipeline depth, which refers to the number of computing devices participating in the model training in the pipeline scheduling strategy. According to the pipeline dimension combined with the specific scheduling strategy used, the idle time of each computing device can be determined.

[0045] In the embodiment of the present disclosure, the pipeline scheduling strategy includes non-interleaved pipeline scheduling and interleaved pipeline scheduling. A schematic diagram of a non-interleaved pipeline scheduling strategy is shown in FIG. Figure 3 As shown. The horizontal direction represents the number of calculation steps of a computing device. In this pipeline scheduling strategy, a computing device can calculate one forward calculation and one backward calculation of the model. The rectangles in the figure correspond to the number of steps of the forward calculation, and the squares correspond to the number of steps of the backward calculation. Assuming that 8 micro-batches need to be calculated, they can be divided into two blocks. First, use four computing devices to calculate the micro-batches of the first block (serial numbers 1 to 4) at a time, and then calculate the micro-batches of the second block (serial numbers 5 to 8) in sequence.

[0046] In the embodiment of the present disclosure, a schematic diagram of a staggered pipeline scheduling strategy is shown as follows: Figure 4 As shown. The interleaving times of this interleaving pipeline scheduling strategy is 2, and one computing device can perform multiple forward calculations and multiple backward calculations on the model. After the first computing device to the fourth computing device perform forward calculations on a micro-batch in turn, the fourth computing device needs to feed the calculation results back to the first computing device to perform forward calculations again. For example, after the first computing device performs the first forward calculation on micro-batch 1 in the first step, it performs the second forward calculation on micro-batch 1 in the fifth step. In this scheduling strategy, the backward calculation process is similar, and the first computing device needs to feed the backward calculation results back to the fourth computing device to perform backward calculations again.

[0047] In some implementations, determining the idle time of each computing device in the plurality of computing devices during the model training process based on the parameters and the scheduling strategy includes:

[0048] According to the pipeline dimension, determine the number of idle times of each computing device during the model training process;

[0049] Based on the number and the distribution pattern of idle time specified by the scheduling strategy, the idle time of each computing device during the model training process is determined.

[0050] In some embodiments, the model is divided into a plurality of mini-batches;

[0051] The distribution patterns include:

[0052] For each micro-batch, in the forward calculation process of the micro-batch, the plurality of computing devices sequentially perform the calculation process for the micro-batch according to the first order; and each computing device waits for other computing devices before it to complete calculation before performing calculation, and the waiting time is the idle time;

[0053] In the backward calculation process of the micro-batch, multiple computing devices execute the calculation process for the micro-batch in sequence in the reverse order of the first order; and each computing device waits for other computing devices behind it to complete the calculation before performing the calculation, and the waiting time is the idle time.

[0054] In the above two pipeline scheduling strategies, the number of idle times of each computing device is fixed. Assuming that the pipeline dimension is P, the number of idle times of each computing device is 2×(P-1). For example, Figure 3 or Figure 4 The pipeline dimension of the pipeline scheduling strategy is 4, and the number of idle times of each computing device is 6.

[0055] Based on the number of idle times of the computing devices that have been determined, and further combined with the distribution pattern of idle time clearly specified in the pipeline scheduling strategy, by analyzing the relationship between these patterns and the number of idle times generated by each computing device during the model training process, the idle time of the computing device in the entire model training cycle can be determined.

[0056] Among them, the idle time of each computing device in the pipeline scheduling strategy is mainly distributed in the two stages of forward calculation and backward calculation.

[0057] (1) In the process of forward computing of micro-batches, each computing device computes the micro-batches in sequence according to the first order. In the first order, the front computing device has not completed the calculation of the micro-batch. In this case, the subsequent computing device needs to wait for the front computing device to transmit data to it, and the waiting time is the idle time.

[0058] (2) In the process of backward calculation of micro-batches, each computing device calculates the micro-batches in sequence in the reverse order of the first order. In this order, the subsequent computing device has not completed the calculation of the micro-batch. In this case, the front computing device needs to wait for the subsequent computing device to transmit data to it, and the waiting time is the idle time.

[0059] Utilizing the idle time distribution pattern of each computing device in the pipeline scheduling strategy helps to determine the time that each computing device is in an idle state, and then use this idle time to realize online fault detection of the computing device.

[0060] In some implementations, performing fault detection on each computing device includes:

[0061] For each computing device, a fault detection program is called during the idle time of the computing device to implement fault detection of the computing device.

[0062] The fault detection program is called during the idle time of the computing device, aiming to use the idle time to perform fault detection on the computing device. This method ensures the continuity and stability of the computing device when training the model, and can perform online fault detection on the computing device without affecting the model training process.

[0063] In some implementations, the method further includes determining a fault detection program, where the fault detection program is used to detect at least one of a hardware state, a computing accuracy, and a memory and a storage medium of the computing device.

[0064] Figure 5 It is a schematic diagram of the idle time of each computing device in the non-interleaved pipeline scheduling strategy. Figure 6 It is a schematic diagram of the idle time of each computing device in the staggered pipeline scheduling strategy. Figure 5 and Figure 6 The graphically marked area in the figure is the idle time of each computing device. During each idle time, the fault detection program will be called to perform fault detection for the corresponding computing device. According to the fault detection result, the corresponding measures will be executed.

[0065] In some examples, the fault detection program is used to comprehensively check multiple core components and performance indicators of the computing device. The hardware status detection function focuses on the physical hardware components of the computing device, such as the central processing unit (CPU), GPU, motherboard, power supply, etc. The fault detection program runs a series of diagnostic tests to check whether the hardware is in normal working condition. For example, by checking the temperature, voltage, current and other parameters of the hardware, ensure that these parameters are within the safe range.

[0066] In some embodiments, the calculation accuracy is detected, including:

[0067] Control the computing device to execute the preset computing tasks and obtain the computing results;

[0068] comparing the calculation result with the expected result of the calculation task;

[0069] The detection result of the calculation accuracy is determined based on the comparison result.

[0070] Computational accuracy is one of the indicators for evaluating the performance of computing devices. The fault detection program will perform a series of preset mathematical operations or scientific computing tasks, and then compare the difference between the actual calculation results and the expected results. If the difference is found to be within an acceptable range, it means that the computing device has a problem with the model training accuracy and needs further maintenance. In this way, the computational accuracy of the computing device can be evaluated, and then the computing device with computational accuracy problems can be found based on the evaluation results.

[0071] In some embodiments, the memory and storage medium are tested, including:

[0072] At least one of a memory leak and a read / write anomaly is detected.

[0073] Memory and storage media (such as hard disks, solid-state drives, etc.) are key components for storing data and programs in computing devices. The fault detection program performs memory tests, including memory leaks and read and write anomalies. Detect whether there is a memory leak, that is, check whether there are memory blocks that are not released correctly or cannot be effectively recovered during the program running process. These unreleased memories may cause system resources to be gradually exhausted, thereby affecting the stability and performance of the system. At the same time, the fault detection program also includes detecting whether there are abnormal conditions in the memory and storage media during the reading and writing process, such as a significant decrease in reading and writing speed, frequent reading and writing errors, or damaged data integrity. These read and write anomalies may directly affect the accuracy and security of the data.

[0074] By testing the memory and storage media, it is possible to discover possible faults and potential problems in the memory management and storage media of the computing device.

[0075] By using the fault detection program to detect the hardware status, computing accuracy, memory and storage media of the computing device, potential anomalies and failures of the computing device can be discovered, and the computing device can be processed accordingly.

[0076] In some embodiments, it further comprises:

[0077] If any computing device fails, the model training process is stopped.

[0078] When the fault detection program detects a fault in a computing device, it will first trigger the fault recording mechanism, which ensures that key information about the fault can be accurately and completely recorded. The recorded information includes but is not limited to the device identification, fault type (such as hardware failure, abnormal calculation accuracy, memory error or storage medium damage, etc.), detection time, etc.

[0079] After the fault information is recorded, the monitoring system notifies the relevant operation and maintenance personnel or the automated processing module of the fault information. This step enables the fault to be known and corresponding measures to be taken. The notification mechanism may include email, SMS, instant message notification, or push notification through a dedicated operation and maintenance relationship platform. For the automated processing module, the notification may directly trigger the preset fault handling process, such as automatically restarting the device, interrupting the training task, etc.

[0080] In the event that any computing device is detected to have a fault, the system will trigger an emergency response process to stop the ongoing model training process. Specifically, the system will first confirm the authenticity and accuracy of the fault information. Once it is confirmed that the computing device has indeed failed, whether it is a hardware-level fault or a software-level abnormality, a global pause instruction will be triggered. This instruction will be conveyed to all computing devices involved in model training, requiring each computing device to stop the current task. When no computing device with a fault is detected, the model training task will continue to run.

[0081] The above measures can avoid the adverse effects of computing device failure on the model training results during the training process. At the same time, after stopping the model training, the computing device failure can be checked and repaired.

[0082] The present disclosure also provides a fault detection device. Figure 7 is a schematic diagram of the structure of a fault detection device 700 according to an embodiment of the present disclosure, comprising:

[0083] A first determination module 710 is used to determine a plurality of computing devices, where the plurality of computing devices are used to perform model training based on a pipeline parallel strategy;

[0084] A second determination module 720, used to determine the parameters and scheduling strategy adopted by the pipeline parallel strategy;

[0085] A third determination module 730, for determining the idle time of each computing device in the plurality of computing devices during the model training process based on the parameter and the scheduling strategy;

[0086] The fault detection module 740 is used to perform fault detection on each computing device during the idle time of each computing device during the model training process.

[0087] In some embodiments, the parameters include pipeline dimensions.

[0088] In some implementations, the third determination module 730 is configured to:

[0089] According to the pipeline dimension, determine the number of idle times of each computing device during the model training process;

[0090] Based on the number and the distribution pattern of idle time specified by the scheduling strategy, the idle time of each computing device during the model training process is determined.

[0091] In some embodiments, the model is divided into a plurality of mini-batches;

[0092] The distribution patterns include:

[0093] For each micro-batch, in the forward calculation process of the micro-batch, the plurality of computing devices sequentially perform the calculation process for the micro-batch according to the first order; and each computing device waits for other computing devices before it to complete calculation before performing calculation, and the waiting time is the idle time;

[0094] In the backward calculation process of a micro-batch, multiple computing devices execute the calculation process for the micro-batch in sequence in the reverse order of the first order; and each computing device waits for other computing devices behind it to complete the calculation before performing the calculation, and the waiting time is the idle time.

[0095] In some implementations, the fault detection module 740 is configured to:

[0096] For each computing device, a fault detection program is called during the idle time of the computing device to implement fault detection of the computing device.

[0097] In some implementations, the fault detection module 740 is further used to determine a fault detection program, which is used to detect at least one of the hardware status, computing accuracy, and memory and storage medium of the computing device.

[0098] In some implementations, the fault detection module 740 is configured to:

[0099] Control the computing device to execute the preset computing tasks and obtain the computing results;

[0100] Compare the computational results with the expected results of the computational task;

[0101] The detection result of the calculation accuracy is determined based on the comparison result.

[0102] In some implementations, the fault detection module 740 is configured to:

[0103] At least one of a memory leak and a read / write anomaly is detected.

[0104] In some embodiments, the fault detection module 740 is also used to stop the model training process when a fault is detected in any computing device.

[0105] For the description of specific functions and examples of each module and submodule of the device in the embodiment of the present disclosure, reference can be made to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0106] In the technical solution disclosed in the present invention, the acquisition, storage and application of personal information of users involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0107] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0108] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0109] like Figure 8 As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0110] A number of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0111] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as detection methods. For example, in some embodiments, the detection method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the detection method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the detection method in any other appropriate manner (e.g., by means of firmware).

[0112] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0113] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0114] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0115] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0116] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0117] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0118] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0119] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A fault detection method, comprising: Determine a plurality of computing devices, wherein the plurality of computing devices are used to perform model training based on a pipeline parallel strategy; Determining the parameters and scheduling strategy adopted by the pipeline parallel strategy; Based on the parameters and the scheduling strategy, determining the idle time of each computing device in the plurality of computing devices during the model training process; During the idle time of each computing device in the model training process, fault detection is performed on each computing device.

2. The method according to claim 1, wherein: The parameters include pipeline dimensions.

3. The method according to claim 2, wherein: The step of determining the idle time of each computing device in the plurality of computing devices during the model training process based on the parameters and the scheduling strategy includes: Determine, according to the pipeline dimension, the number of idle times of each computing device during the model training process; Based on the number and the distribution pattern of the idle time specified by the scheduling strategy, the idle time of each computing device during the model training process is determined.

4. The method according to claim 3, wherein: The model is divided into a plurality of mini-batches; The distribution rules include: For each of the micro-batches, in the forward calculation process of the micro-batch, the multiple computing devices sequentially perform the calculation process for the micro-batch in a first order; and each of the computing devices waits for other computing devices before it to complete calculation before performing calculation, and the waiting time is the idle time; In the backward calculation process of the micro-batch, the multiple computing devices sequentially execute the calculation process for the micro-batch in the reverse order of the first order; and each computing device waits for other computing devices behind it to complete the calculation before performing the calculation, and the waiting time is the idle time.

5. The method according to any one of claims 1 to 4, wherein: The performing fault detection on each of the computing devices comprises: For each of the computing devices, a fault detection program is called during the idle time of the computing device to implement fault detection of the computing device.

6. The method according to claim 5 further includes determining the fault detection program, wherein the fault detection program is used to detect at least one of the hardware status, calculation accuracy, and memory and storage medium of the computing device.

7. The method according to claim 6, wherein: Testing the calculation accuracy includes: Controlling the computing device to execute a preset computing task and obtain a computing result; comparing the calculation result with the expected result of the calculation task; The detection result of the calculation accuracy is determined according to the comparison result.

8. The method according to claim 7, wherein: Testing the memory and storage medium includes: At least one of a memory leak and a read / write anomaly is detected.

9. The method according to any one of claims 1 to 4, further comprising: When a failure is detected in any of the computing devices, the model training process is stopped.

10. A fault detection device, comprising: A first determination module is used to determine a plurality of computing devices, wherein the plurality of computing devices are used to perform model training based on a pipeline parallel strategy; A second determination module, used to determine the parameters and scheduling strategy adopted by the pipeline parallel strategy; A third determination module, configured to determine, based on the parameters and the scheduling strategy, the idle time of each computing device among the plurality of computing devices during the model training process; The fault detection module is used to perform fault detection on each computing device during the idle time of each computing device during the model training process.

11. The device according to claim 10, wherein: The parameters include pipeline dimensions.

12. The device according to claim 11, wherein: The third determining module is used to: Determine, according to the pipeline dimension, the number of idle times of each computing device during the model training process; Based on the number and the distribution pattern of the idle time specified by the scheduling strategy, the idle time of each computing device during the model training process is determined.

13. The device according to claim 12, wherein: The model is divided into a plurality of mini-batches; The distribution rules include: For each of the micro-batches, in the forward calculation process of the micro-batch, the multiple computing devices sequentially perform the calculation process for the micro-batch in a first order; and each of the computing devices waits for other computing devices before it to complete calculation before performing calculation, and the waiting time is the idle time; In the backward calculation process of the micro-batch, the multiple computing devices sequentially execute the calculation process for the micro-batch in the reverse order of the first order; and each computing device waits for other computing devices behind it to complete the calculation before performing the calculation, and the waiting time is the idle time.

14. The device according to any one of claims 10 to 13, wherein: The fault detection module is used to: For each of the computing devices, a fault detection program is called during the idle time of the computing device to implement fault detection of the computing device.

15. According to the device of claim 14, the fault detection module is also used to determine the fault detection program, and the fault detection program is used to detect at least one of the hardware status, calculation accuracy, and memory and storage medium of the computing device.

16. The device according to claim 15, wherein: The fault detection module is used to: Controlling the computing device to execute a preset computing task and obtain a computing result; comparing the calculation result with the expected result of the calculation task; The detection result of the calculation accuracy is determined according to the comparison result.

17. The device according to claim 16, wherein: The fault detection module is used to: At least one of a memory leak and a read / write anomaly is detected.

18. According to the apparatus according to any one of claims 10-13, the fault detection module is also used to stop the model training process when a fault is detected in any of the computing devices.

19. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 9.

20. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-9.

21. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 9.

Images