Fault handling method and electronic device

Abstract

The application provides a fault processing method and an electronic device, wherein the fault processing method comprises: obtaining an index sequence corresponding to at least one task index of a training task of a target large model, determining a mutation period sequence in the index sequence, determining a fault cause according to the mutation period sequence, and adjusting a training resource of the target large model according to the fault cause to execute the training task, wherein the type of the task index at least includes one of an error index, a process time consumption index, and a device usage index; the difference between the mutation period sequence and a reference period sequence corresponding to the index sequence is greater than a difference threshold, and the time period of the mutation period sequence is determined according to a time period corresponding to the reference period sequence.

Description

Technical Field

[0001] This application relates to computer technology, and more particularly to a fault handling method and an electronic device. Background Technology

[0002] Model training involves many training tasks that require collaboration. Different tasks are executed by different processes to update model parameters synchronously. If one of the processes fails while executing a training task, the entire training will be stuck or crash. Therefore, it is necessary to locate and handle the fault after it ends.

[0003] The fault handling methods in related technologies mainly rely on passive timeout detection mechanisms or post-event manual intervention analysis to detect and troubleshoot faults. The former depends on the timeout threshold built into the communication library, which only triggers an alarm after the blockage exceeds a preset time. The latter calls relevant data analysis tools, parses register information, or inserts monitoring logic into the self-developed communication library after the fault occurs to perform retrospective analysis on massive trajectory data or hardware status, thereby determining the root cause of the fault.

[0004] Therefore, all of the above-mentioned fault handling methods suffer from poor fault handling efficiency. Summary of the Invention

[0005] This application provides a fault handling method and an electronic device that at least partially solve the above-mentioned problems.

[0006] The technical solution of this application embodiment is implemented as follows: In a first aspect, embodiments of this application provide a fault handling method, the method comprising: For at least one training task of the target large model, obtain the indicator sequence corresponding to at least one task indicator of the training task. The type of task indicator includes at least one of error indicator, process time indicator and device usage indicator. The mutation cycle sequence in the indicator sequence is determined. The difference between the mutation cycle sequence and the benchmark cycle sequence corresponding to the indicator sequence is greater than the difference threshold. The time period of the mutation cycle sequence is determined based on the time period corresponding to the benchmark cycle sequence. Determine the cause of the failure based on the mutation cycle sequence; Adjust the training resources of the target large model according to the cause of the failure, and execute the training task.

[0007] Secondly, embodiments of this application provide a fault handling apparatus, comprising: The acquisition module is used to acquire the indicator sequence corresponding to at least one task indicator for at least one training task of the target large model. The type of task indicator includes at least one of error indicators, process time indicators and equipment usage indicators. The determination module is used to determine the mutation cycle sequence in the index sequence. The difference between the mutation cycle sequence and the benchmark cycle sequence corresponding to the index sequence is greater than the difference threshold. The time period of the mutation cycle sequence is determined based on the time period corresponding to the benchmark cycle sequence. The fault location module is used to determine the cause of the fault based on the mutation cycle sequence. The processing module is used to adjust the training resources of the target large model according to the cause of the fault and execute the training task.

[0008] Thirdly, embodiments of this application provide an electronic device, the device including a processor and an interface; The processor is used to acquire, for at least one training task of the target large model, an indicator sequence corresponding to at least one task indicator of the training task. The type of task indicator includes at least one of error indicators, process time indicators, and equipment usage indicators. It determines the mutation period sequence in the indicator sequence. The difference between the mutation period sequence and the baseline period sequence corresponding to the indicator sequence is greater than the difference threshold. The time period of the mutation period sequence is determined based on the time period corresponding to the baseline period sequence. It determines the cause of failure based on the mutation period sequence. It adjusts the training resources of the target large model based on the cause of failure and executes the training task.

[0009] This application provides a computer-readable storage medium storing a computer program or computer-executable instructions, which, when executed by a processor, implements the fault handling method provided in this application.

[0010] This application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, they implement the fault handling method provided in this application.

[0011] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of this application. Attached Figure Description

[0012] Figure 1 A schematic diagram of the implementation process of a fault handling method provided in this application embodiment. Figure 1 ; Figure 2 A schematic diagram of the implementation process of a fault handling method provided in this application embodiment. Figure 2 ; Figure 3 A schematic diagram of the implementation process of a fault handling method provided in this application embodiment. Figure 3 ; Figure 4 This application provides a schematic diagram of a root cause localization process. Figure 5 This application provides a schematic diagram of a fault handling process in accordance with an embodiment of the present application. Figure 6 A schematic diagram of a software architecture provided for an embodiment of this application; Figure 7 This is a schematic diagram of the composition structure of a fault handling device provided in an embodiment of this application; Figure 8 This is a schematic diagram of the hardware entity of an electronic device provided in an embodiment of this application.

[0013] It should be noted that the terms "first" and "second" mentioned above are only used to distinguish between different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application are further described in detail below with reference to the accompanying drawings and embodiments. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0015] In the following description, references to "some embodiments" refer to a subset of all possible embodiments. It is understood that "some embodiments" may be the same or different subsets of all possible embodiments and may be combined with each other without conflict. The terms "first / second / third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first / second / third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0016] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used herein is for descriptive purposes only and is not intended to limit the scope of this application.

[0017] Training large models typically requires many processing modules (such as image processors, central processing units, etc.) to work simultaneously. The processes on these processing modules cooperate to perform tasks and update model parameters synchronously. If one of the processes has a problem (such as hardware failure, network lag, or software freeze), the entire training will be stuck or crash. Such problems will lead to training interruption and waste a lot of time and computing power.

[0018] The following four solutions are used in related technologies to address this problem: (1) Timeout mechanism, but this mechanism takes time to detect problems and cannot determine which part is malfunctioning.

[0019] (2) Profiling performance analysis tools are designed for users to manually call the tool to troubleshoot after a failure occurs. However, this tool is usually used for fine-grained performance analysis. For large-scale model training with a large workload, small-scale troubleshooting requires a lot of time.

[0020] (3) The Graphics Processing Unit (GPU) has many registers that record the hardware status. By reading and parsing this register information, it is possible to determine whether the GPU is "frozen" or has encountered a hardware error. However, this method is mainly for hardware anomalies of the GPU itself. If the problem lies with the Central Processing Unit (CPU), memory, Peripheral Component Interconnect Express (PCIe) link, or network switch, this method is ineffective; moreover, parsing register information usually requires low-level operations and involves a large amount of data, which is also time-consuming.

[0021] (4) Develop your own communication library. This is a more proactive approach. By modifying the source code of the communication library, fine-grained indicator data collection logic is inserted into some core code, thereby collecting different data that can be used to detect whether there is a fault.

[0022] None of the above troubleshooting methods can accurately pinpoint the specific cause of the fault. Both fault location and fault detection take time, and subsequent fault handling is done manually, resulting in poor fault handling efficiency.

[0023] In view of this, embodiments of this application provide a fault handling method and an electronic device. The fault handling method includes: for at least one training task of a target large model, obtaining an indicator sequence corresponding to at least one task indicator of the training task, thereby determining a mutation period sequence in the indicator sequence, and then determining the cause of the fault based on the mutation period sequence. Thus, the training resources of the target large model can be adjusted according to the cause of the fault, and the training task can be executed. The type of task indicator includes at least one of error indicator, process time indicator, and equipment usage indicator. The difference between the mutation period sequence and the baseline period sequence corresponding to the indicator sequence is greater than a difference threshold, and the time period of the mutation period sequence is determined based on the time period corresponding to the baseline period sequence.

[0024] In this way, by continuously monitoring multi-dimensional indicators of large model training tasks (including error indicators, process time indicators, and equipment usage indicators), a time-series indicator sequence is constructed, and abruptly changing periodic sequences that differ significantly from the baseline periodic sequence are automatically identified. This allows for precise location of the fault cause, and then automatic adjustment of training resources and resumption of task execution based on the fault cause. On the one hand, compared to fault handling methods in related technologies that only begin troubleshooting after an anomaly has occurred and caused a long period of work stoppage, the technical solution provided in this application continuously monitors the multi-dimensional indicator sequence of large model training tasks. When a fault occurs, it can be quickly detected and precisely located to determine the cause, shortening the fault discovery time. On the other hand, after determining the fault cause, training resources are automatically adjusted to resume the training task. Compared to manual processing in related technologies, the technical solution provided in this application automates fault handling based on the detected fault cause, reducing the time and human resource waste caused by manual processing. Compared to fault handling methods in related technologies, the technical solution provided in this application improves fault handling efficiency.

[0025] This application provides a fault handling method, which can be executed by an electronic device. The electronic device may be, but is not limited to, a server, laptop, tablet, desktop computer, smart TV, set-top box, mobile device (such as mobile phone, portable video player, personal digital assistant, dedicated messaging device, portable gaming device), or other device with data processing capabilities.

[0026] Figure 1 A schematic diagram of the implementation process of a fault handling method provided in this application embodiment. Figure 1 ,like Figure 1 As shown, the method includes the following steps S101 and S102: Step S101: For at least one training task of the target large model, obtain the index sequence corresponding to at least one task index of the training task.

[0027] The types of task indicators include at least one of the following: error indicators, process time indicators, and equipment usage indicators.

[0028] Here, the target large model can refer to a deep learning model with a huge number of parameters and a large training scale. It can be, but is not limited to, large-scale language models, multimodal models, etc., and usually requires hundreds or thousands of processing modules (GPUs, CPUs) to train together.

[0029] Different training tasks will be executed in parallel on threads of different processing modules. Here, training tasks can refer to distributed training tasks designed for the target large model. Different training tasks are processed by different threads on each processing module, which may include, but are not limited to, data loading, forward propagation, back propagation, gradient synchronization, parameter updating, etc.

[0030] In some embodiments, different task metrics are set in order to observe the execution status of each training task. Here, task metrics refer to quantifiable feature data used to characterize the running status of training tasks, and the types include at least one of error metrics, process time metrics, and device usage metrics.

[0031] In some implementations, different types of task metrics reflect different situations during the training process. Process time metrics include the time required for different function calculation processes and the time required for data interaction processes; device usage metrics include the operating status of different hardware during the training process; and error metrics include the number of errors that occur when different hardware transmits data.

[0032] Here, by collecting process time metrics, we can observe the progress of each training task during execution and whether there is any stagnation. The functions to be observed in the process time metrics can include, but are not limited to, performing single-precision (float) ordinary matrix multiplication; performing batch single-precision matrix multiplication, general matrix multiplication, and batch version of cublasGemmEx (a very core and powerful matrix multiplication function in the CUDA math library); the data interaction process can include the process of different training task corresponding processes performing reduction operations on their input data, and the process corresponding to different training task performing reduction operations on the data and sending the results to the corresponding root process, etc.

[0033] Here, by collecting device usage metrics, we can observe whether there are any hardware problems during the training process. Device usage metrics may include, but are not limited to, GPU utilization, GPU memory usage, GPU PCIe input / output data volume, and Ethernet card input / output data volume.

[0034] Here, by collecting error metrics, we can intuitively observe abnormal events that occur during the training process, which may include, but are not limited to, the number of single-byte errors, the number of double-byte errors, and the number of errors generated when the network card receives and sends data.

[0035] In this way, by using different task indicators of the equipment, we can observe the training status of the target large model in different training tasks during the training process in a comprehensive and detailed manner, thus providing an effective data foundation for subsequent fault perception and fault location.

[0036] Understandably, in large-scale model training scenarios, faults often do not occur instantaneously, but rather manifest in advance through abnormal changes in certain task metrics. By collecting multi-dimensional and multi-type task metrics at different times and constructing time series, a rich data foundation can be provided for subsequent fault perception and localization, enabling timely monitoring of the training process and ensuring that any subtle abnormal fluctuations can be captured.

[0037] Therefore, in this embodiment of the application, it is necessary to obtain the index sequence corresponding to at least one task index for each training task. Here, the index sequence can refer to the set of observations of the same task index arranged in chronological order, which constitutes a time series data and reflects the dynamic trajectory of the index changing over time.

[0038] Regarding the timing of starting to acquire the indicator sequence, in one possible implementation, the indicator sequence corresponding to at least one task indicator for each training task can be acquired in real time or periodically.

[0039] In another possible implementation, when the execution time after the start of the training task reaches the waiting time length, the indicator sequence corresponding to at least one task indicator of the training task is obtained.

[0040] Here, the waiting time can refer to a preset threshold of time that needs to be waited after the task starts, and the collection of indicator sequences will only begin after the threshold is reached.

[0041] It is understandable that at the beginning of the training task, all data and devices are in the initial warm-up phase, such as data loader initialization, memory allocation, communication link establishment, framework compilation optimization, etc. These initial operations may cause large fluctuations in the indicators of each training task in the initial stage. These fluctuations are not real operational anomalies. If they are included in the indicator sequence without distinction, it will reduce the accuracy of subsequent fault detection.

[0042] Therefore, in some implementations, after each training task is started, the index sequence is collected only after its execution time reaches a preset waiting time.

[0043] Here, the waiting time can be set to a fixed value (e.g., 30 seconds after the task starts) or a dynamic value (e.g., waiting for the warm-up period after the first N iterations of training are completed). The purpose is to avoid the unstable state at the beginning of the training task and ensure that the collected index sequence can reflect the steady-state operation characteristics of the task.

[0044] Here, execution time can refer to the duration from the start of the training task to the current moment. It can be physical time (such as seconds or minutes) or the number of training iterations (such as the Nth iteration). The specific unit of measurement is determined according to the monitoring requirements.

[0045] By setting a waiting time, the collected indicator data is ensured to come from the steady-state operation phase of the task, thereby improving the accuracy and reliability of anomaly detection.

[0046] Regarding the method of obtaining indicator data, in one possible implementation, each indicator sequence corresponding to each training task is recorded in a default storage location, which can be located in memory. When obtaining the indicator sequence, it can be obtained directly from the default storage location.

[0047] Step S102: Determine the mutation cycle sequence in the index sequence.

[0048] Among them, the difference between the mutation cycle sequence and the benchmark cycle sequence corresponding to the index sequence is greater than the difference threshold, and the time period of the mutation cycle sequence is determined according to the time period corresponding to the benchmark cycle sequence.

[0049] Here, "period" can refer to the time span of recurring patterns or features in an indicator sequence. It's important to emphasize that for large-scale model training tasks, the "period" of an indicator sequence doesn't refer to a fixed, equal time interval, but rather to time segments with similar statistical characteristics or behavioral patterns. For example, the time sequence of each training iteration might exhibit the common characteristics of "stable forward propagation time and small fluctuations in gradient synchronization time," but the specific values ​​and the forms of these small fluctuations will differ.

[0050] The baseline periodic sequence refers to a reference sequence that reflects the changing pattern of indicators under normal conditions, representing the expected normal behavior of the indicators. In some embodiments, the baseline periodic sequence is not necessarily a continuous time segment, but a representative feature sequence obtained through statistical analysis (such as clustering, periodic graph analysis), reflecting the shape, fluctuation range, or statistical distribution that the indicators should exhibit during normal operation.

[0051] Regarding the process of determining the benchmark periodic sequence, in one possible implementation, the benchmark periodic sequence can be determined by extracting sequence data of a preset time length from the index sequence and performing data analysis.

[0052] In another possible implementation, when the index sequence collection begins, an initial benchmark periodic sequence can be set, and then a preset iterative averaging method can be used. Each time an index sequence is collected, the initial benchmark periodic sequence can be iteratively adjusted to obtain a more accurate benchmark periodic sequence.

[0053] It is understandable that when there are anomalies in the training task, this is reflected in the indicator sequence, that is, the indicator sequence contains a mutation periodic sequence. Here, the mutation periodic sequence can refer to the sequence segment in the indicator sequence whose feature difference from the benchmark periodic sequence exceeds a preset difference threshold, reflecting the abnormal fluctuation of the indicator.

[0054] The difference threshold can refer to the quantitative standard for judging whether an indicator sequence has undergone a sudden change. It can be the absolute difference, relative rate of change, statistical deviation (such as exceeding 3 times the standard deviation), slope deviation, peak difference, etc.

[0055] In some implementations, the time period of the mutation cycle sequence is determined based on the time period of the baseline cycle sequence—meaning that the mutation cycle sequence is truncated to a segment of the same length as the baseline cycle sequence for comparison, so as to measure differences on a uniform scale. However, it is important to emphasize that the cycles in the original index sequence may themselves vary in length; they are simply aligned to the length of the baseline cycle for comparative analysis when identifying mutations, for ease of comparison.

[0056] Regarding the process of determining the mutation periodic sequence, in one possible implementation, a sequence fragment of length L can be extracted from the latest arriving index sequence using a sliding window method, where L is the time period length of the reference periodic sequence. Then, the sequence fragment is compared with the reference periodic sequence, and the difference found in the comparison determines whether the sequence fragment is a mutation periodic sequence.

[0057] In another possible implementation, for the obtained index sequence fragment, the sequence fragment corresponding to the last training cycle is obtained. It is understood that the time length corresponding to the last training cycle is not necessarily the same as the time length of the benchmark cycle sequence. In this case, the time period of the sequence fragment can be aligned with the time period of the benchmark cycle sequence according to the preset time alignment method. Then, the aligned sequence fragment is compared with the benchmark cycle sequence, and the difference found in the comparison is used to determine whether the sequence fragment is a mutation cycle sequence.

[0058] Regarding the comparison process, in one possible embodiment, features of the sequence segment, such as peak value, curvature, and slope, can be identified. These features are then compared with the features corresponding to a baseline periodic sequence to obtain difference data. If the difference data is greater than a preset difference threshold, the sequence segment is determined to be a mutation periodic sequence. It is understood that this difference data can be determined based on differences between features; for example, it can be determined based on at least one of peak value difference, curvature difference, and slope difference, or it can be determined based on a combination of two or more differences. The combined determination method can be a statistical method such as a weighted average.

[0059] In another possible implementation, the Euclidean distance between the sequence fragment and the reference period can be calculated to quantify the difference between the sequence fragment and the reference period sequence. If the Euclidean distance representing the difference exceeds a preset size (i.e., a difference threshold), the sequence fragment is determined to be a mutated periodic sequence.

[0060] It is understandable that the above analysis targets the global indicator sequence and is a data analysis of the entire scope. In some cases, in order to reduce the waste of computing resources, the scope of analysis can be narrowed.

[0061] In some embodiments, before step S102 "determine the mutation cycle sequence in the indicator sequence", the method may further include: in response to a hardware positioning instruction, determining at least one hardware task indicator corresponding to the target hardware indicated by the hardware positioning instruction.

[0062] Thus, step S102, "determining the mutation cycle sequence in the indicator sequence", may specifically include: determining the mutation cycle sequence in the indicator sequence corresponding to the hardware task indicator.

[0063] Here, hardware location instructions can refer to instructions issued by users, operation and maintenance systems, or upper-level schedulers to specify specific hardware devices that need to be focused on or investigated.

[0064] In some embodiments, the hardware location instruction may include hardware identifiers, such as GPU index number (GPU 0-7), node IP address, PCIe device address, switch port number, etc., to focus the fault detection scope on specific hardware.

[0065] It is understandable that the hardware pointed to by the hardware location command, that is, the target hardware, is a specific hardware device or set of hardware devices that needs to be monitored or investigated. It can be a single piece of hardware (such as a GPU), a set of similar hardware on the same node (such as all GPUs on node A), or hardware corresponding to a specific type of hardware across nodes (such as network cards on all nodes).

[0066] Understandably, the task metrics include some hardware-related metrics, namely hardware task metrics.

[0067] Here, hardware task metrics can refer to task metrics that are directly related to the target hardware and can reflect the operating status of the hardware. Compared with general task metrics, hardware task metrics have hardware relevance. For example, they can be the utilization rate, temperature, and video memory usage of a specific GPU, the transmit / receive rate and retransmission rate of a specific network card, the bandwidth utilization of a PCIe link, etc.

[0068] Regarding the process of obtaining hardware location commands, in one possible implementation, the input can be pre-set by the user. That is, during the training task, the user judges based on experience which hardware is most likely to cause anomalies during the training of the target large model. In actual analysis, the indicator sequence analysis can be performed first on the target hardware pointed to by the hardware location commands input by the user.

[0069] In another possible implementation, a historical fault database can be maintained, which includes fault data from historical times when faults occurred. By analyzing the data in the historical fault database, it is possible to predict which devices the current fault may be related to and generate corresponding hardware location instructions.

[0070] In this embodiment of the application, after determining the hardware task indicators corresponding to the target hardware, subsequent analysis only requires obtaining the indicator sequence corresponding to the hardware task indicators. Similarly, determining the mutation cycle sequence only requires determining the mutation cycle sequence in the indicator sequence corresponding to the hardware task indicators. The method of determination can be referred to the above embodiment, and will not be repeated here.

[0071] Understandably, in large-scale training clusters, tens of thousands of hardware devices run simultaneously. If all hardware is subjected to indiscriminate full-scale indicator analysis, not only will the computational cost be enormous, but it will also generate a massive amount of alarm information when a fault occurs, obscuring the true root cause. In the above embodiment, the scope of indicator collection and analysis is compressed from "global indiscriminate monitoring" to "targeted detection of specific hardware". Through hardware location instructions, the system can be guided to prioritize or specifically focus on "suspect hardware" suspected by the user or upper-level system, thereby narrowing the scope of fault perception and improving detection efficiency and accuracy.

[0072] Step S103: Determine the cause of the failure based on the mutation cycle sequence.

[0073] Understandably, simply knowing that an indicator has changed abruptly is not enough to guide subsequent resource adjustments. Different causes of failures require different handling strategies. By analyzing the indicator types, change patterns, and causal relationships between multiple indicators in the mutation cycle sequence, the root cause of the failure can be accurately identified, thereby avoiding blind handling or misjudgment. Therefore, it is necessary to determine the cause of the failure based on the mutation cycle sequence.

[0074] Here, the cause of the malfunction can refer to the root cause that leads to the abnormality of the training task.

[0075] In some embodiments, the training tasks are interconnected. If a training task is abnormal, it may be due to another training task rather than its own internal cause.

[0076] Therefore, in some implementations, the cause of the failure could be a training task that caused the training task to malfunction.

[0077] In some implementations, the cause of the failure can be a specific hardware failure or a fine-grained cause such as a software failure. For example, hardware failures may include, but are not limited to, GPU freezing, memory corruption, PCIe link slowdown, etc.; network failures may include, but are not limited to, switch congestion, network card failure, link retransmission, etc.; software failures may include, but are not limited to, video memory overflow, driver abnormality, etc.; and storage failures may include, but are not limited to, I / O latency, file system read-only, etc.

[0078] Regarding the process of determining the cause of the fault, in one possible implementation, each mutation cycle sequence can be directly input into a pre-trained fault location model to obtain the fault cause output by the fault location model.

[0079] In another possible implementation, the difference between the mutated periodic sequence and the baseline periodic sequence can be quantified as difference data, and analysis can be performed based on all difference data to find the abnormal training tasks that cause anomalies in other training tasks from all training tasks.

[0080] Step S104: Adjust the training resources of the target large model according to the cause of the fault, and execute the training task.

[0081] Understandably, when an anomaly occurs in the training task, it is necessary to address the cause of the fault so that the training task can continue to be executed normally.

[0082] In some embodiments, the training resources of the target large model can be adjusted according to the cause of the failure so that the training task can continue to be performed subsequently.

[0083] Here, the training resources for the target large model can refer to the computing resources, network resources, and storage resources required to perform the training task, including GPU nodes, CPU nodes, network bandwidth, storage volumes, memory, etc.

[0084] For example, when a GPU hardware malfunctions, the training tasks running on that GPU can be rescheduled to a backup healthy node; or, when network communication fails, a backup network path can be switched to.

[0085] In some embodiments, different fault causes correspond to different training resource adjustment strategies. The corresponding adjustment strategy can be determined according to the fault cause, and the training resources can be adjusted according to the adjustment strategy.

[0086] In one possible implementation, each fault cause can be classified into different fault types, and different fault types correspond to different general training resource adjustment strategies. In practical applications, the corresponding training resource adjustment strategy can be determined first based on the fault type corresponding to the fault cause, and then the training resources can be adjusted according to the training resource adjustment strategy.

[0087] In another possible implementation, the cause of the failure can be input into a preset policy generation model to obtain the policy output by the policy generation model, and the training resources can be adjusted based on the policy.

[0088] In the above embodiments, by continuously monitoring multi-dimensional indicators (including error indicators, process time indicators, and equipment usage indicators) of the large model training task, a time-series indicator sequence is constructed, and abrupt change periodic sequences that differ significantly from the baseline periodic sequence are automatically identified. This allows for precise location of the fault cause, and then automatic adjustment of training resources and resumption of task execution based on the fault cause. On the one hand, compared to fault handling methods in related technologies that only perform fault investigation after an anomaly occurs and causes a long period of work stoppage, the technical solution provided in this application continuously monitors the multi-dimensional indicator sequence of the large model training task. When a fault occurs, it can be quickly detected and precisely located to determine the cause, shortening the fault detection time. On the other hand, after determining the fault cause, training resources are automatically adjusted to resume the training task. Compared to manual processing in related technologies, the technical solution provided in this application automates fault handling based on the detected fault cause, reducing the time and human resource waste caused by manual processing. Compared to fault handling methods in related technologies, the technical solution provided in this application improves fault handling efficiency.

[0089] Figure 2 A schematic diagram of the implementation process of a fault handling method provided in this application embodiment. Figure 2 In this embodiment of the application, the process of step S102, "determining the mutation periodic sequence in the index sequence", can be referred to Figure 2 The process specifically includes the following steps: Step S201: Obtain the benchmark period sequence corresponding to the indicator sequence.

[0090] Here, the baseline periodic sequence can refer to a reference sequence that reflects the variation pattern of the indicator under normal conditions, representing the expected normal behavior of the indicator. In some embodiments, the baseline periodic sequence may not be a continuous time segment, but rather a representative feature sequence obtained through statistical analysis (such as clustering, periodic graph analysis), reflecting the shape, fluctuation range, or statistical distribution that the indicator should exhibit during normal operation.

[0091] Regarding the method of obtaining the benchmark period sequence, in one possible implementation, the benchmark period sequence is obtained by fitting and analyzing the sequence segments of the index data for all training periods except the last training period.

[0092] In another possible implementation, the process of step S201, "obtaining the benchmark period sequence corresponding to the indicator sequence," may include the following steps: Step S2011: Determine the initial reference sequence based on the normal periodic sequences other than the mutation periodic sequences in the index sequence.

[0093] Step S2012: Based on the vector distance between the vector corresponding to the first normal periodic sequence and the vector corresponding to the initial reference sequence, update the initial reference sequence to determine the updated reference sequence.

[0094] Step S2013: If there is a second normal periodic sequence that is not used for the baseline sequence update, then the updated baseline sequence is updated according to the vector corresponding to the second normal periodic sequence until all normal periodic sequences are used for the baseline sequence update, and the updated baseline sequence corresponding to the last round of the baseline sequence update process is determined as the baseline periodic sequence.

[0095] Here, the mutated cycle sequence can refer to the sequence segment corresponding to the last training cycle in the index sequence, or in other words, a sequence segment that has been identified as abnormal, usually exhibiting significant differences from normal behavior patterns. Understandably, these abnormal sequence segments need to be excluded when determining the baseline cycle sequence to avoid contaminating the construction of the baseline sequence.

[0096] Here, a normal periodic sequence can refer to those sequence segments in the index sequence that are not marked as mutations and represent normal operating conditions. These segments are the "pure" samples used to construct the baseline sequence.

[0097] The initial baseline sequence can refer to a reference template representing a typical behavior pattern, which is first calculated from a normal cycle sequence. It can be the mean sequence, median sequence, or center sequence obtained by clustering of multiple normal segments, or the sequence segment corresponding to the first training cycle in time.

[0098] After determining the initial reference sequence, the reference sequence can be iteratively updated based on each normal periodic sequence, thereby achieving dynamic adjustment of the reference periodic sequence and making the accuracy of the current reference periodic sequence used for evaluation higher.

[0099] Regarding the iterative update process, in this embodiment of the application, the initial reference sequence will first be updated using the first normal periodic sequence.

[0100] Here, the first normal periodic sequence can refer to the sample sequence selected from the normal periodic sequence for the initial update of the benchmark. It can be a specific normal segment or a representative of a batch of normal segments.

[0101] After an update, it checks whether there is a second normal periodic sequence for the update. If there is, it continues; otherwise, the result of this update is used as the final baseline periodic sequence.

[0102] For a single benchmark sequence update process, taking the update process corresponding to the first normal periodic sequence as an example, we can first determine the vector corresponding to the first normal periodic sequence and the vector corresponding to the initial benchmark sequence to be updated in this round.

[0103] Here, a vector can refer to a vector in a multi-dimensional space when the periodic sequence is viewed. In this embodiment, the vector can be labeled as X. i ={x0,x1,…,x n}, where i represents the sequence identifier corresponding to the first normal periodic sequence, x n This represents the index value corresponding to the nth time in the first periodic sequence. The initial baseline sequence behaves similarly to the first normal periodic sequence, and can be denoted as H. norm This will not be elaborated upon here.

[0104] It is understandable that the time period of the normal periodic sequence used to update the reference sequence may be the same as or different from the time period of the reference sequence. Therefore, a preset averaging method can be used to align the two periods and update them in parallel. This preset averaging method can be the iterative DTW Barycenter Averaging (DBA).

[0105] In the averaging update process, the vector distance between the vector corresponding to the first normal periodic sequence and the vector corresponding to the initial reference sequence can be calculated. This vector distance can be obtained using Euclidean distance or other distance calculation methods. Then, the initial reference sequence is updated based on this vector distance. For example, if the vector distance is less than a preset threshold, the initial reference sequence is used as the updated initial reference sequence for the next round of updates. If it is greater than the preset threshold, each point corresponding to a time series in the sequence is mapped to each point corresponding to a time series in the initial reference sequence. The mapping result can be... The mapping can be one-to-one, one-to-many, or many-to-one. Then, based on the mapping result, the points corresponding to each time series in the initial reference sequence are updated. For example, if the mapping result is one-to-one, the point in the first normal periodic sequence replaces the mapped point in the initial reference sequence. If the mapping result is one-to-many, the point in the first normal periodic sequence replaces those mapped points in the initial reference sequence. If the mapping result is many-to-one, the mean of multiple points in the first normal periodic sequence is calculated, and the mean is used to replace the mapped point in the initial reference sequence. Finally, the updated reference sequence corresponding to this round of reference sequence update process is obtained.

[0106] After an update, it can be checked whether there is a second normal periodic sequence in the normal periodic sequence that can be used for updating. If there is, the second normal periodic sequence is used to update the updated reference sequence. If not, the result of the last round of updates is used as the final reference periodic sequence.

[0107] In this way, the purity of the initial benchmark is first ensured by removing abnormal segments, and then the benchmark gradually incorporates information from new samples through vector distance measurement and weighted update mechanism. Finally, all normal samples are fully utilized through iterative traversal or convergence judgment, so that the benchmark sequence can accurately reflect the normal behavior pattern of the current stage of the training task. A benchmark periodic sequence generation system that can dynamically evolve and improve itself is constructed, which lays a solid foundation for the accurate identification of subsequent mutation periodic sequences.

[0108] Step S202: Compare the benchmark periodic sequence with the sequences of different periods in the index sequence.

[0109] Step S203: The target sequence is determined to be a mutated periodic sequence if the difference between it and the baseline periodic sequence is greater than the difference threshold.

[0110] Here, sequences of different periods can refer to sequence segments corresponding to different training periods, or they can refer to sequence segments extracted from the index sequence that have the same length as the baseline period sequence.

[0111] It is understandable that the sequences of each period are continuous in time, and may even overlap. When each sequence is compared with the reference period sequence, the length of its time period will be aligned with the reference period sequence.

[0112] Therefore, the comparison can be made by comparing the Euclidean distance between the baseline periodic sequence and the corresponding sequences of each period, or by comparing the cosine similarity between the two.

[0113] The form of the difference threshold is related to the comparison method and can be set based on empirical values.

[0114] In some implementations, when there is a target sequence with a quantified difference value greater than a difference threshold, the target sequence is determined to be a mutation periodic sequence.

[0115] In the above embodiments, a baseline periodic sequence is constructed and accurately compared with sequence segments of different periods. Finally, based on the comparison of differences and difference thresholds, mutation determination is performed to ensure that only segments that truly deviate from the normal range are identified as mutated periodic sequences. This effectively avoids misjudgments caused by changes in the training phase or normal fluctuations, and provides high-quality, high-confidence input for subsequent fault cause determination.

[0116] Figure 3 A schematic diagram of the implementation process of a fault handling method provided in this application embodiment. Figure 3 , refer to Figure 3 In this embodiment of the application, the process of step S103 "determining the cause of failure based on the mutation cycle sequence" may include the following steps: Step S301: For each training task, determine the difference data corresponding to the indicator sequence based on the baseline periodic sequence and / or mutation periodic sequence.

[0117] Understandably, when determining the cause of the failure, it is necessary to check whether there are any mutations in the indicator sequence corresponding to each training task, and if so, whether the mutation is abnormal. Therefore, it is necessary to determine the difference data corresponding to the indicator sequence.

[0118] Here, differential data can refer to data that quantifies the degree of deviation between a mutated periodic sequence and a normal baseline periodic sequence. It can be a scalar (such as the average deviation of the entire segment), a vector (such as the deviation value at each time point), or structured data (such as the statistical characteristics of the deviation). Differential data reflects the degree, form, and characteristics of anomalies in task metrics under this training task.

[0119] In one possible implementation, the deviation between the baseline periodic sequence and the mutation periodic sequence can be directly used as the difference data corresponding to the index sequence.

[0120] In another possible implementation, the process of "determining the difference data corresponding to the index sequence based on the baseline periodic sequence and / or the mutation periodic sequence" can be achieved in the following two ways: Method 1: If the task indicator corresponding to the indicator sequence is an error indicator, then the cumulative value of the time series data corresponding to different times in the mutation cycle sequence is used as the difference data corresponding to the indicator sequence. The difference data corresponding to the first indicator represents the number of times the error corresponding to the first indicator occurs during the execution of the training task.

[0121] Method 2: If the task indicator corresponding to the indicator sequence is either a process time indicator or an equipment usage indicator, the first vector distance between the vector corresponding to the baseline periodic sequence and the vector corresponding to the mutation periodic sequence is determined as the difference data of the indicator sequence in the training task.

[0122] In some embodiments, different calculation methods can be used for different types of task metrics, which enables accurate calculation of differential data.

[0123] Regarding method one: Error metrics can refer to the number of errors that occur when different hardware transmits data. By collecting error metrics, abnormal events that occur during training can be observed intuitively. These may include, but are not limited to, the number of single-byte errors, the number of double-byte errors, and the number of errors generated when the network card receives and sends data.

[0124] In some embodiments, in the index sequence corresponding to the error index, the time series data corresponding to each time series can refer to the observation value at each time point, such as the error count recorded once per second.

[0125] Therefore, the time series data in the mutation cycle sequence corresponding to the error index can be summed directly, and the resulting cumulative value is the total number of error events that occurred during that time period. It can be understood that the cumulative value reflects the severity of error accumulation during the abnormal period.

[0126] Regarding method two: Here, the process time consumption index refers to the time required for different function calculation processes and the time required for data interaction processes. By collecting the process time consumption index, we can observe the progress of each training task in the execution process and whether there is a standstill. The device usage index can characterize the operating status of different hardware in the training process. By collecting the device usage index, we can observe whether there are any hardware-level problems in the training process.

[0127] In some embodiments, the normal operating mode of process time consumption indicators and equipment usage indicators manifests as a specific time series pattern, such as periodic peaks and troughs. Abnormal performance may be amplitude changes, phase shifts, or morphological distortions. Therefore, it is necessary to quantify the degree of deviation between the baseline periodic sequence and the abrupt periodic sequence to determine the corresponding difference data. That is, the difference data corresponding to the process time consumption indicators and equipment usage indicators characterize the difference between the abrupt periodic sequence and the baseline periodic sequence.

[0128] In this embodiment of the application, the first vector distance between the vector corresponding to the baseline periodic sequence and the vector corresponding to the mutation periodic sequence can be determined as the difference data of the index sequence in the training task. Here, the first vector distance can be obtained by the Euclidean distance calculation method.

[0129] In this way, through methods one and two described above, precise adaptation processing of different types of task indicators is achieved. For discrete count error indicators, the cumulative value is used as the difference data, transforming the frequency of error events into an intuitive measure of anomaly severity. For continuously changing process time and equipment usage indicators, vector distance is used as the difference data, compressing complex temporal deviations into a unified numerical measure. This differentiated processing respects the inherent characteristics of various indicators while providing unified, comparable, and information-rich input for subsequent steps, laying a solid data foundation for fault root cause localization in large-scale training clusters.

[0130] Step S302: Determine the first matrix based on the difference data corresponding to each task indicator in each training task.

[0131] The first matrix includes indicator vectors corresponding to each task indicator, and the elements in the indicator vectors are the difference data of the task indicators in different training tasks.

[0132] Here, the first matrix can refer to an N-row × M-column (or M-row × N-column) matrix, where the N rows correspond to N training tasks, the M columns correspond to M task metrics (or vice versa), and the matrix elements are the difference data of the i-th task on the j-th task metric.

[0133] Correspondingly, the index vectors included in the first matrix can refer to column vectors in the first matrix (or row vectors if the columns in the first matrix represent task indicators, depending on the design). Each index vector corresponds to a task indicator, and the elements therein are the difference data of that indicator in different training tasks.

[0134] In some embodiments, for example, the first matrix is ​​set to G. n×m Then its included index vector K j ={G i,j |0 <i<=n,0<j<=m},Gi,j This represents the difference data of the j-th task metric in the i-th training task.

[0135] Therefore, in this embodiment of the application, the first matrix can be determined by combining the various difference data according to a preset matrix arrangement.

[0136] Step S303: Determine the cause of the fault based on the first matrix.

[0137] In one possible implementation, the column vectors (indicator vectors) of the first matrix can be analyzed to identify which indicators exhibit spatial clustering anomalies, thereby pinpointing the fault type as the final cause of the fault.

[0138] In another possible implementation, when there are multiple task indicators, the process for step S303, "determining the cause of failure based on the first matrix," may include: Step S3031: Denoise the first matrix based on the standard scores corresponding to each index vector to determine at least one target index vector.

[0139] Here, the standard score, also known as the Z-score, can be used to measure the relative position of a value in the overall data distribution. For an indicator vector, based on the standard scores corresponding to each element included in it, it is possible to identify which indicators have significant anomalies across the entire cluster.

[0140] Denoising can be performed based on the standard scores corresponding to each element in the indicator vector. Here, denoising can refer to using statistical methods to screen out the target indicator vector that truly has abnormal elements, thereby eliminating those "noise indicators" caused by random fluctuations or irrelevant factors.

[0141] In some embodiments, the denoising process may be as follows: for an index vector, a standard score corresponding to each element can be calculated, and it can be determined whether the standard score is greater than a preset constant threshold, thereby determining whether there are elements with a very large change range in the index vector. If so, the index vector can be determined as the target index vector.

[0142] Understandably, the target indicator vector can refer to the indicator vector that has been filtered and retained, and that contains significant anomalous elements within the cluster. These indicators are more likely to be related to the actual fault.

[0143] Step S3032: Determine the second matrix based on each target index vector. The second matrix includes the task vector corresponding to each training task.

[0144] The elements included in the task vector are the difference data corresponding to different target indicators in the training task.

[0145] Here, each target metric is the task metric corresponding to each target vector.

[0146] Here, the second matrix can refer to a new matrix composed of target metric vectors. For example, the dimension can be N×P (P is the number corresponding to the target metric). However, in this step, the emphasis is on the conversion from the metric perspective to the task perspective. Therefore, each row in the second matrix corresponds to the task vector of a training task and contains the difference data of this task on all target metrics. That is, the second matrix is the column screening result of the original first matrix, but is used for subsequent analysis in the form of row vectors.

[0147] In some embodiments, for the representation form of the task vector, a task vector can be denoted as C i , where 0 < i < N, and each element in the task vector is the difference data of the corresponding training task on all target metrics.

[0148] It can be understood that the target metric vectors can be processed according to a preset vector composition format to obtain the second matrix.

[0149] Step S3033: Classify each task vector according to a preset classification algorithm to determine the target task vectors of at least one target type; Here, the preset classification algorithm can refer to a machine learning or statistical algorithm preselected for classifying task vectors into different categories, including but not limited to unsupervised algorithms such as K-means clustering algorithm (K-means), Density-Based Spatial Clustering of Applications with Noise (DBSCAN), hierarchical clustering, Gaussian mixture model, self-organizing map, etc., or supervised classification algorithms (such as random forest) in the case of labeled data.

[0150] In the embodiments of the present application, each task vector can be classified according to a preset classification algorithm, so that tasks within the same class have similar abnormal patterns, and tasks between different types have obvious differences.

[0151] It can be understood that each type includes a target type, and the target type can refer to the abnormal category in the classification result. That is, the training tasks corresponding to those task vectors determined to be of this type are abnormal tasks.

[0152] The target task vectors can refer to those task vectors belonging to the target type.

[0153] In some implementations, to facilitate processing, each task vector can be normalized when classifying it. This transforms the differences in data of different dimensions and orders of magnitude in the task vectors to the same scale, eliminates the influence of dimensions between indicators, and makes the task vectors comparable in subsequent cluster analysis.

[0154] Regarding the normalization process, in one possible implementation, a standard score can be calculated for each element in the task vector, and the standard score can be used to replace each element, thereby achieving the normalization of the task vector.

[0155] Step S3034: Determine the cause of the fault based on the reliability of the classification results and the training tasks corresponding to each target task vector.

[0156] Here, the credibility of the classification results can refer to the quantitative assessment of the reliability of the classification results.

[0157] Based on the above embodiments, it can be determined that the classification result includes task vectors of the target type and task vectors of non-target type, that is, binary classification. For this classification result, in one possible implementation, the silhouette coefficient can be used to describe the credibility of the classification result.

[0158] For the process of calculating confidence level using the profile coefficient, please refer to the following: For each target task vector, the average distance between the target task vector and all other task vectors is determined, and the inter-cluster separation S is obtained based on this. out Understandably, S out The larger the value, the more significant the difference between the abnormal target task vector and other normal task vectors in the feature space.

[0159] Then, the average distance between the target task vector and other target task vectors is calculated, and the intra-cluster separation S is obtained based on this. in Understandably, S in The smaller the value, the more similar the features are between the target task vectors.

[0160] Based on the above S out and S in The confidence level of the classification result can be obtained, namely the silhouette coefficient P, where the silhouette coefficient P is the confidence level of the classification result.

[0161] In this embodiment of the application, the credibility of the current classification result can be determined based on the credibility level of the classification result. If the credibility level is greater than the preset credibility threshold, the credibility of the classification result is determined to be high. It can be determined that the training task corresponding to the target task vector in the classification result is an abnormal training task. Based on these abnormal training tasks, the cause of the failure can be determined.

[0162] In one possible implementation, the cause of failure can refer to the abnormal training task that caused the anomaly. Therefore, the training task corresponding to each target task vector can be directly identified as the cause of failure.

[0163] In another possible implementation, after determining that the training task corresponding to each target task vector is an abnormal training task, the elements in each target task vector can be used to determine which task indicators in the abnormal training task are more abnormal, thereby further narrowing down the fault location range, and taking the abnormal training task and the abnormal task indicators in the abnormal training task as the cause of the fault.

[0164] In this way, through a series of processing steps on the first matrix, including denoising, classification, and confidence diagnosis, the abnormal training tasks affected by the fault are accurately identified from all training tasks. Finally, the cause of the fault is inferred, and a complete diagnostic link from "multi-index matrix" to "precise root cause" is constructed, providing a high-confidence decision basis for automated fault perception.

[0165] The following describes the application of the embodiments of this application in a real-world scenario.

[0166] Large-scale model training jobs are massive in scale, and the training processes within a job are interdependent. Software or hardware failures in a single process can cause the entire job to stall or terminate, resulting in the interruption of the training process. This solution mainly addresses the identification and root cause localization of implicit anomalies such as job stalling in large-scale training scenarios.

[0167] The existing solution has the following problems: (1) It relies on the timeout mechanism of the NVIDIA Collective Communications Library (NCCL), which results in a long fault detection time and a lack of root cause localization mechanism.

[0168] (2) After the fault occurs, profiling tools are called to check one by one, but the scope of location is limited and it takes a long time.

[0169] (3) Parsing GPU register information takes a long time and has a limited location range.

[0170] (4) The self-developed NCCL library inserts fine-grained indicator collection logic in some core code locations, but the positioning range is limited.

[0171] All of the above methods suffer from low fault handling efficiency.

[0172] Based on this, this application adopts a training task stagnation detection and root cause localization method based on pattern analysis.

[0173] Figure 4 This is a schematic diagram of a root cause localization process provided in an embodiment of this application, referring to... Figure 4 As can be seen, for each task Rank, time-series data of the metric can be obtained, i.e., the curve in the figure, where the vertical axis v is the value of the metric and the horizontal axis t is time.

[0174] Understandably, when anomalies occur in the training task, each task can be analyzed to quantify the difference between the abnormal sequence interval and the normal later sequence. The difference values ​​can be combined into a multi-dimensional quantization matrix and binary classified to find the root cause abnormal task that caused the job to stall, i.e., the root cause localization result.

[0175] Compared to existing fault detection processes, the solution provided in this application has the following advantages: 1. Highly responsive and time-efficient. Real-time analysis eliminates the need to call various profiling or debugging tools after a job has stalled, enabling root cause localization as soon as a fault occurs.

[0176] 2. Loosely coupled and non-intrusive. Bypass analysis can be performed without modifying existing training software stacks or user jobs.

[0177] 3. High accuracy. It breaks through the traditional diagnostic model that relies on expert experience and manual rules for case-by-case analysis, avoiding misjudgments caused by the discrepancy between expert knowledge and fault modes.

[0178] The following are the specific technical solutions provided in this application: Figure 5 This is a schematic diagram of a fault handling process provided in an embodiment of this application, referring to... Figure 5 It can be seen that the overall processing is divided into four stages: Step S501, Waiting phase after each training task in the training job starts: When the training job starts, the indicators fluctuate greatly. During this phase, it is necessary to wait for a preset time to ensure that a stable initial normal window benchmark is collected.

[0179] Step S502, Preparation Phase after the Waiting Phase: The execution process of each training task in the training job exhibits obvious periodicity. When the job is running stably, the fluctuations in indicators such as the number of function calls, execution time, and resource utilization during different periods show obvious similarities. In this phase, a time series analysis algorithm is first used to divide the period. Then, the core indicators determined by feature engineering are used to model the training process within a single period window. By continuously analyzing the characteristics of each period window during the training process, the expected characteristics of the training job in the current single period are obtained, and the data of the current window is continuously updated as the benchmark characteristics for the stable operation of the job.

[0180] Step S503, Real-time monitoring stage (Discovery): Continuous stall detection will introduce excessive system overhead. This stage is a pre-stage of stall detection. If the execution of core computation and communication functions is not detected within the time set by the preset timeout mechanism, it will be determined that the current training task has stalled (i.e., the kernel event has not been updated). When all training tasks stall, it means that the job has stalled.

[0181] Step S504, Detection Stage: This stage aims to identify the root cause task leading to the job stagnation. The detection process is divided into two sub-stages: local and global. The local detection stage primarily involves a comparative analysis of each training task. This is achieved by collecting data on the duration of the time period preceding the stagnation (the aforementioned mutation cycle sequence) and comparing it with the expected features modeled in the preparation stage (current window data, the aforementioned baseline cycle sequence). The difference value (the aforementioned difference data) is then quantified. The global detection stage aggregates the difference values ​​from each training task. Since these difference values ​​are described using multi-dimensional data, this stage employs multi-dimensional clustering methods to identify significant outliers, thereby pinpointing the root cause task (the aforementioned cause of the failure).

[0182] Figure 6 This is a schematic diagram of a software architecture provided in an embodiment of this application, with reference to... Figure 6 The above four stages correspond to the software architecture level, which is mainly the collaborative work of three core modules: MetricProcess 601, LocalAnalysis 602, and GlobalAnalysis 603. The MetricProcess module is mainly responsible for the collection and preprocessing of indicator data, while the LocalAnalysis module performs local analysis on each training task and sends the analysis results to GlobalAnalysis for global analysis. The analysis content mainly includes the functions described in the Discovery and Detection stages above.

[0183] For the operations performed during the Discovery phase, the MetricProcess module collects the computation and communication-related kernel function time consumption metrics of each training process in real time and writes them to shared memory. LocalAnalysis monitors whether any metrics are written to this file in real time (i.e., whether the kernel events have been updated as mentioned above). If no metrics are written within a certain period of time, the process is marked as stalled and the status is synchronized to the GlobalAnalysis module. When the GlobalAnalysis module receives stall information from all training tasks, it determines that the entire training job has stalled, thereby triggering the root cause localization process for job stalling.

[0184] The specific applications of the three modules in the process are as follows: 1. MetricProcess module. This module is mainly responsible for collecting and processing raw metrics, which mainly include three categories of metrics: process time, equipment usage, and error information. These three categories of metrics are represented by the T, D, and E sets, respectively.

[0185] For details regarding the metrics for T-type tasks—specifically, the process time consumption metrics—please refer to Table 1: Table 1

[0186] For details regarding Category D task indicators—equipment usage task indicators—please refer to Table 2: Table 2

[0187] For details regarding the E-category task metrics—error task metrics—please refer to Table 3: Table 3

[0188] 2. LocalAnalysis Module: The core function of this module is to quantify the degree of abrupt change in the working state of the training process when a fault occurs. Experiments verify that the root cause process, being directly affected by the fault, exhibits a significantly stronger degree of abrupt change than other indirectly affected processes. This paper proposes a set of abrupt change quantification indicators, which describe the degree of change of the indicator at the moment the anomaly occurs compared to the indicator within a normal iteration cycle. Therefore, to construct the final abrupt change quantification indicator, we first need to generate a baseline training cycle indicator. The specific construction process of this indicator is as follows.

[0189] (1) Benchmark training cycle index (i.e., the aforementioned benchmark cycle sequence). In this stage, time series indexes corresponding to N training iterations within the stable training phase are collected. The Iterative DTW Barycenter Averaging (DBA) algorithm is used to solve for the actual alignment sequence Hnorm when the minimum distance between these N time series data is reached. This sequence is the benchmark cycle during normal training. The length of the cycle is denoted as period. For any index type X∈(T∪D∪E), its time series index within a training iteration cycle i is denoted as X. i ={x0,x1,…,x n The baseline training period metric is denoted as Norm, and the baseline training period can be written as: .

[0190] (2) Mutation index.

[0191] For both process time and equipment usage metrics, the DTW algorithm is used, with Euclidean distance of vectors selected as the metric to quantify time-series metrics after training stagnation. The difference from the benchmark training cycle metric is calculated from the start time of the last iteration after the stagnation occurred. Durational time-series data. Let V be the duration. X Let X represent the mutation value corresponding to index X. Then: .

[0192] For the error category metric, since it can directly characterize anomalies in the training process, the observation periods are directly accumulated here. The total number of errors within the range is: .

[0193] So far, we have only obtained the quantitative result of a single indicator. This case uses multiple indicators collected by the MetricProcess module. The final mutation index generated by this module is a vector V containing the mutation values ​​of various time series indicators, denoted as: .

[0194] 3. GlobalAnalyzer module.

[0195] This module receives mutation value vectors V from the LocalAnalysis module distributed across various training tasks, analyzes them from a global perspective, and identifies the root cause process (i.e., the abnormal task) that caused the stagnation. The mutation value vectors V from all training tasks form a vector matrix G (i.e., the first matrix mentioned above), with the number of rows being m, the number of training tasks being m, and the number of columns being n, the number of metrics collected by the MetricProcess module.

[0196] There are many types of faults that can cause job stalls, and the metrics that characterize the stalls differ for each situation. For example, in training stalls caused by GPU faults, the abrupt changes in GPU metrics reported by the root cause task will differ from those reported by other tasks. However, for training stalls caused by Remote Direct Memory Access (RDMA) network card faults, the abrupt changes in RDMA communication metrics are more meaningful. For simplicity, we define key metrics as those whose abrupt changes show significant differences accompanying the fault, and other metrics as non-key metrics.

[0197] First, it is necessary to identify the key indicator matrix G from all indicators that is causing the current work to stall. key For the input vector matrix G, each column represents the mutation value of each task under the current metric dimension, which is extracted and denoted as a vector. (That is, the aforementioned task indicator vectors i and j represent the row and column indices, respectively, and m represents the number of task indicators.) Calculate the Z-score (standard score) of all values ​​in the vector to determine whether there are any abrupt changes.

[0198] When there is a standard score with a certain difference At that time, Z thresh If K is a pre-set constant threshold, then it means that K j If an indicator exhibits a highly variable range, then that dimension of the indicator is considered a key indicator. By traversing all columns of matrix G, we can obtain the key vector matrix G representing the current stagnation of the operation. key (i.e., the second matrix mentioned above), the number of rows in the matrix is ​​m, the number of training tasks, and the number of columns is l, where l <n。

[0199] Secondly, according to the key indicator matrix G key The row vectors are divided into two clusters. Many algorithms exist for binary classification; this case uses the K-Means algorithm. To avoid clustering anomalies caused by differences in the dimensions of the vectors, the vectors in G are sorted column-wise before clustering. key Normalization is performed, and Z-score is used for quantization. The calculation formula is similar to the previous step and will not be repeated here. Let G be the normalization process. norm This represents the key indicator matrix after normalization, and then the K-Means algorithm is used to analyze G. norm We perform binary clustering on each row vector (i.e., task vector). Based on our insights, for large-scale training tasks, the simultaneous occurrence of widespread failures is a low-probability event. Correspondingly, in the clustering results, the cluster with fewer constituent elements is identified as the root cause cluster, denoted as C. hang (Target task vector), denoted as C for non-root cause clusters. health .

[0200] Finally, the reliability of the current detection method is calculated. We use the silhouette coefficient to indirectly describe the accuracy of the detection results.

[0201] The final output of the solution is C. hang The training task corresponding to the row vector is the root cause of the entire job stalling. The confidence level of this detection result is P (this value is greater than 0 and less than or equal to 1; generally, a confidence level greater than 0.5 is considered relatively reliable). Finally, a complete detection chain is completed, from original metrics -> time-series metrics -> local severity metrics -> denoised metrics -> binary classification -> (diagnosis) confidence level.

[0202] Next, several practical implementation examples of the solution provided in this application will be given: This section uses the training process involving four training processes to introduce the specific handling procedures for three typical faults that cause training to stall: rdma exception, kernel function (cublas) exception, and Python garbage collection (py gc).

[0203] (1) rdma abnormality.

[0204] When this type of failure occurs, the amount of RDMA data transmitted on the failed node will fluctuate significantly compared to other indirectly affected nodes, and RDMA-related error indicators such as packet loss rate and error count will also be generated. This section will use real data from a real-world environment as the raw input to introduce the specific implementation of this case.

[0205] A. This process occurs in the MetricProcess module. First, it collects the metric sequences corresponding to the four task metrics: mlx_port_rcv_byte (number of bytes received by the network card port), mlx_port_xmit_byte (number of bytes sent by the network card port), port_rcv_constraint_errors (port receive constraint errors), and port_xmit_constraint_errors (port send constraint errors) from the four tasks from rank0 to rank3.

[0206] B. Severity Measurement. The LocalAnalysis module processes the collected indicator sequences using the DBA algorithm described above to determine the baseline periodic sequences for different task indicators (rank 0-rank 3) and the actual indicator sequences during failures. Next, the DTW algorithm is used to analyze the differences between the baseline periodic sequences and the failure periodic sequences, which serve as input to the GlobalAnalysis module.

[0207] C. Fault Determination. This process occurs in the GlobalAnalysis module. The input to this module is the difference values ​​of different indicators corresponding to each training task sent by the LocalAnalysis module. The difference values ​​form a key indicator matrix. This matrix is ​​denoised and then normalized. The row vectors included in the normalized matrix are then subjected to binary classification using the k-means algorithm.

[0208] D. Detection reliability assessment. The reliability of fault determination is assessed using the profile coefficient.

[0209] (2) Cublas kernel delivery anomaly: When this type of failure occurs, the most obvious feature is a significant decrease in GPU utilization on the root cause node. Other indirectly affected processes will occupy the GPU for a long time because they will be waiting at the nccl set communication operation. Therefore, the GPU utilization of these processes will remain high. Thus, the key indicator for this type of failure is GPU utilization (GRACT).

[0210] A. Original metrics: Collect the GRACT metric sequence corresponding to the rank0-rank3 training tasks.

[0211] B. Severity Measurement: LocalAnalysis uses the DBA algorithm to analyze the indicator sequence and determine the baseline periodic sequence and the fault mutation periodic sequence corresponding to the GRACT indicator in different training tasks. The difference between the two is used as input to the GlobalAnalysis module. C. Fault Determination: The GlobalAnalysis module receives the difference values ​​from LocalAnalysis to form a key indicator matrix. Since only the indicator sequence corresponding to one task indicator is collected here, no noise reduction is needed. The key indicator matrix is ​​directly normalized, and the k-means algorithm is used to perform binary classification on the row vectors included in the normalized matrix.

[0212] D. Detection Reliability Assessment. The profile coefficient is used to assess the reliability of the fault determination.

[0213] (3) Py gc exception: This exception describes how repeated garbage collection at the Python level can block the training process due to unreasonable code logic or memory resource settings. When this problem occurs, the Python garbage collection count (py_gc_num) of the root cause process will continue to rise, and the GPU utilization GRACT metric will also show the same characteristics as when the cublas kernel issues an exception.

[0214] A. Original metrics: The metric sequences corresponding to the py_gc_num metric and the GRACT metric in the rank0-rank3 tasks.

[0215] B. Severe Quantification: LocalAnalysis uses the DBA algorithm to analyze the indicator sequences corresponding to the py_gc_num and GRACT indicators, determine the baseline and mutation cycle sequences of the two indicators under different tasks, and calculate the difference data for input to the GlobalAnalysis module.

[0216] C. Fault Determination: The GlobalAnalysis module processes the received differential data to obtain a key indicator matrix. The key indicator matrix is ​​then denoised and normalized. The row vectors included in the normalized matrix are then classified using the k-means algorithm.

[0217] D. Detection Reliability Assessment. The profile coefficient is used to assess the reliability of the fault determination.

[0218] Based on the foregoing embodiments, this application provides a fault handling device, which includes various units and modules included in each unit, and can be implemented by a processor in an electronic device; of course, it can also be implemented by specific logic circuits; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor unit (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA), etc.

[0219] Figure 7 This is a schematic diagram of the composition structure of a fault handling device provided in an embodiment of this application, as shown below. Figure 7 As shown, the fault handling device 700 includes: an acquisition module 701, a determination module 702, a fault location module 703, and a processing module 704, wherein: The acquisition module 701 is used to acquire the indicator sequence corresponding to at least one task indicator for at least one training task of the target large model. The type of task indicator includes at least one of error indicator, process time indicator and equipment usage indicator. The determination module 702 is used to determine the mutation cycle sequence in the index sequence. The difference between the mutation cycle sequence and the benchmark cycle sequence corresponding to the index sequence is greater than the difference threshold. The time period of the mutation cycle sequence is determined based on the time period corresponding to the benchmark cycle sequence. The fault location module 703 is used to determine the cause of the fault based on the mutation cycle sequence. The processing module 704 is used to adjust the training resources of the target large model according to the cause of the fault and to execute the training task.

[0220] In some embodiments, the determining module 702 includes: The benchmark acquisition unit is used to acquire the benchmark period sequence corresponding to the indicator sequence. The comparison unit is used to compare the benchmark periodic sequence with sequences of different periods in the index sequence; The mutation determination unit is used to determine the target sequence as a mutated periodic sequence if the difference between the target sequence and the reference periodic sequence is greater than the difference threshold.

[0221] In some embodiments, the reference acquisition unit is specifically used for: The initial baseline sequence is determined based on the normal periodic sequence (excluding the mutation periodic sequence) in the index sequence. Based on the vector distance between the vector corresponding to the first normal periodic sequence and the vector corresponding to the initial reference sequence, the initial reference sequence is updated to determine the updated reference sequence. If there is a second normal periodic sequence that is not used for updating the reference sequence, then the updated reference sequence is updated according to the vector corresponding to the second normal periodic sequence until all normal periodic sequences are used for updating the reference sequence. The updated reference sequence corresponding to the last round of reference sequence update process is then determined as the reference periodic sequence.

[0222] In some embodiments, the fault location module 703 includes: The difference determination unit is used to determine the difference data corresponding to the indicator sequence for each training task, based on the baseline period sequence and / or mutation period sequence. The matrix determination unit is used to determine the first matrix based on the difference data corresponding to each task indicator in each training task. The first matrix includes the indicator vector corresponding to each task indicator, and the elements included in the indicator vector are the difference data corresponding to the task indicator in different training tasks. The positioning unit is used to determine the cause of the fault based on the first matrix.

[0223] In some embodiments, the difference determination unit is specifically used to perform: If the task indicator corresponding to the indicator sequence is an error indicator, then the cumulative value of the time series data corresponding to different times in the mutation cycle sequence is used as the difference data corresponding to the indicator sequence. The difference data corresponding to the first indicator represents the number of times the error corresponding to the first indicator occurs during the execution of the training task. If the task indicator corresponding to the indicator sequence is either a process time indicator or an equipment usage indicator, the first vector distance between the vector corresponding to the baseline periodic sequence and the vector corresponding to the mutation periodic sequence is determined as the difference data of the indicator sequence in the training task.

[0224] In some embodiments, there are multiple training tasks and multiple task metrics, and the localization unit specifically performs the following: The first matrix is ​​denoised based on the standard scores corresponding to each index vector to determine at least one target index vector. Based on each target indicator vector, a second matrix is ​​determined. The second matrix includes the task vector corresponding to each training task. The elements in the task vector are the difference data corresponding to different target indicators in the training task. Each target indicator is the task indicator corresponding to each target pointer. Based on the preset classification algorithm, each task vector is classified to determine at least one target task vector of a target type; the cause of the fault is determined based on the reliability of the classification results and the training task corresponding to each target task vector.

[0225] In some embodiments, process time metrics include the time required for different function calculation processes and the time required for data interaction processes; device usage metrics include the operating status of different hardware during the training process; and error metrics include the number of errors that occur when different hardware transmits data.

[0226] In some embodiments, the acquisition module 701 includes: The waiting unit is used to obtain the indicator sequence corresponding to at least one task indicator of the training task when the execution time after the start of the training task reaches the waiting time length.

[0227] In some embodiments, the device 700 further includes: The instruction response module is used to respond to a hardware positioning instruction and determine at least one hardware task indicator corresponding to the target hardware indicated by the hardware positioning instruction. The determination module 702 includes: a designation unit, used to determine the mutation cycle sequence in the indicator sequence corresponding to the hardware task indicator.

[0228] The descriptions of the apparatus embodiments above are similar to those of the method embodiments above, and have similar beneficial effects. In some embodiments, the functions or modules included in the apparatus provided in this application can be used to perform the methods described in the method embodiments above. For technical details not disclosed in the apparatus embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0229] It should be noted that, in the embodiments of this application, if the above-mentioned fault handling method is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the related technology, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, mobile hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware, software, or firmware, or any combination of hardware, software, and firmware.

[0230] This application provides an electronic device, which includes a processor and an interface. The processor is configured to, for at least one training task of a target large model, acquire an indicator sequence corresponding to at least one task indicator for the training task. The type of task indicator includes at least one of error indicators, process time indicators, and device usage indicators. The processor determines a mutation period sequence in the indicator sequence, wherein the difference between the mutation period sequence and the baseline period sequence corresponding to the indicator sequence is greater than a difference threshold, and the time period of the mutation period sequence is determined based on the time period corresponding to the baseline period sequence. The processor determines the cause of the failure based on the mutation period sequence. The processor adjusts the training resources of the target large model based on the cause of the failure and executes the training task.

[0231] This application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements some or all of the steps in the above-described method. The computer-readable storage medium can be transient or non-transient.

[0232] This application provides a computer program including computer-readable code, wherein when the computer-readable code is executed in an electronic device, a processor in the electronic device performs some or all of the steps in the above-described method.

[0233] This application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program. When the computer program is read and executed by a computer, it implements some or all of the steps in the above-described method. This computer program product can be implemented specifically through hardware, software, or a combination thereof. In some embodiments, the computer program product is specifically embodied as a computer storage medium; in other embodiments, the computer program product is specifically embodied as a software product, such as a software development kit (SDK), etc.

[0234] It should be noted that the descriptions of the various embodiments above tend to emphasize the differences between them, while their similarities or commonalities can be referred to interchangeably. The descriptions of the above embodiments of the device, storage medium, computer program, and computer program product are similar to the descriptions of the above method embodiments and have similar beneficial effects. For technical details not disclosed in the embodiments of the device, storage medium, computer program, and computer program product of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0235] Figure 8 This is a hardware entity diagram of an electronic device provided in an embodiment of this application, such as... Figure 8 As shown, the hardware entity of the electronic device 800 includes: a processor 801, a communication interface 802, and a memory 803, wherein: When processor 801 executes a program, it implements the steps of any of the above-mentioned fault handling methods. Processor 801 typically controls the overall operation of electronic device 800.

[0236] Communication interface 802 enables electronic devices to communicate with other terminals or servers via a network.

[0237] The memory 803 is configured to store instructions and applications executable by the processor 801, and can also cache data to be processed or already processed (e.g., image data, audio data, voice communication data, and video communication data) in the processor 801 and various modules in the electronic device 800. It can be implemented using flash memory or random access memory (RAM). Data transfer between the processor 801, the communication interface 802, and the memory 803 can be performed via bus 804.

[0238] This application provides a computer storage medium that stores one or more programs, which can be executed by one or more processors to implement the steps of the fault handling method as described in any of the above embodiments.

[0239] It should be noted that the descriptions of the storage medium and device embodiments above are similar to those of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0240] The aforementioned processor can be at least one of the following: Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), Central Processing Unit (CPU), Controller, Microcontroller, and Microprocessor. It is understood that other electronic devices can also implement the functions of the aforementioned processor, and this application does not specifically limit the specific implementation.

[0241] The aforementioned computer storage media / memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM), etc.; or it can be various terminals that include one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0242] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A fault handling method, the method comprising: For at least one training task of the target large model, obtain an indicator sequence corresponding to at least one task indicator of the training task, wherein the type of the task indicator includes at least one of error indicator, process time indicator and equipment usage indicator. A mutation cycle sequence is determined in the index sequence, wherein the difference between the mutation cycle sequence and the benchmark cycle sequence corresponding to the index sequence is greater than a difference threshold, and the time period of the mutation cycle sequence is determined based on the time period corresponding to the benchmark cycle sequence. The cause of the failure was determined based on the mutation cycle sequence; Adjust the training resources of the target large model according to the cause of the fault, and execute the training task.

2. The method according to claim 1, wherein determining the mutation periodic sequence in the index sequence comprises: Obtain the benchmark period sequence corresponding to the indicator sequence; The benchmark periodic sequence is compared with sequences of different periods in the index sequence; The target sequence is identified as the mutation periodic sequence if the difference between it and the baseline periodic sequence is greater than the difference threshold.

3. The method according to claim 2, wherein obtaining the benchmark period sequence corresponding to the indicator sequence comprises: The initial baseline sequence is determined based on the normal periodic sequence in the index sequence, excluding the mutation periodic sequence. Based on the vector distance between the vector corresponding to the first normal periodic sequence and the vector corresponding to the initial reference sequence, the initial reference sequence is updated to determine the updated reference sequence. If there is a second normal periodic sequence that is not used for updating the reference sequence, then the updated reference sequence is updated according to the vector corresponding to the second normal periodic sequence until all the normal periodic sequences are used for updating the reference sequence, and the updated reference sequence corresponding to the last round of reference sequence update process is determined as the reference periodic sequence.

4. The method according to any one of claims 1 to 3, wherein determining the cause of failure based on the mutation cycle sequence comprises: For each training task, the differential data corresponding to the index sequence is determined based on the baseline period sequence and / or the mutation period sequence; A first matrix is ​​determined based on the difference data corresponding to each task indicator in each training task. The first matrix includes an indicator vector corresponding to each task indicator, and the elements included in the indicator vector are the difference data corresponding to the task indicator in different training tasks. The cause of the fault is determined based on the first matrix.

5. The method according to claim 4, wherein determining the difference data corresponding to the index sequence based on the baseline periodic sequence and / or the mutation periodic sequence comprises: If the task indicator corresponding to the indicator sequence is the error indicator, then the cumulative value of the time series data corresponding to different times in the mutation cycle sequence is used as the difference data corresponding to the indicator sequence. The difference data corresponding to the first indicator represents the number of times the error corresponding to the first indicator occurs during the execution of the training task. If the task indicator corresponding to the indicator sequence is one of the process time indicator or the equipment usage indicator, the first vector distance between the vector corresponding to the baseline periodic sequence and the vector corresponding to the mutation periodic sequence is determined as the difference data of the indicator sequence in the training task.

6. The method according to claim 4, wherein the number of training tasks is multiple, the number of task indicators is multiple, and determining the cause of the fault based on the first matrix includes: The first matrix is ​​denoised based on the standard scores corresponding to each of the indicator vectors to determine at least one target indicator vector. Based on each of the target indicator vectors, a second matrix is ​​determined. The second matrix includes the task vectors corresponding to each of the training tasks. The elements included in the task vectors are the difference data corresponding to different target indicators in the training tasks. Each target indicator is the task indicator corresponding to each target pointer. According to a preset classification algorithm, each task vector is classified to determine at least one target task vector of a target type; the cause of the fault is determined based on the reliability of the classification results and the training task corresponding to each target task vector.

7. The method according to any one of claims 1 to 3, wherein the process time consumption index includes the time required for different function calculation processes and the time required for data interaction processes; the device usage index includes the operating status of different hardware during the training process; and the error index includes the number of errors that occur when different hardware transmits data.

8. The method according to any one of claims 1 to 3, wherein obtaining the index sequence corresponding to at least one task index corresponding to the training task comprises: When the execution time of the training task reaches the waiting time length after it is started, obtain the indicator sequence corresponding to at least one task indicator of the training task.

9. The method according to any one of claims 1 to 3, wherein before determining the mutation periodic sequence in the index sequence, the method further comprises: In response to a hardware positioning command, at least one hardware task indicator corresponding to the target hardware indicated by the hardware positioning command is determined. Determining the mutation cycle sequence in the index sequence includes: Determine the mutation cycle sequence in the indicator sequence corresponding to the hardware task indicator.

10. An electronic device, the device comprising a processor and an interface; The processor is configured to, for at least one training task of a target large model, acquire an indicator sequence corresponding to at least one task indicator for the training task, wherein the type of the task indicator includes at least one of error indicators, process time indicators, and equipment usage indicators; determine a mutation period sequence in the indicator sequence, wherein the difference between the mutation period sequence and the baseline period sequence corresponding to the indicator sequence is greater than a difference threshold, and the time period of the mutation period sequence is determined based on the time period corresponding to the baseline period sequence; determine the cause of failure based on the mutation period sequence; adjust the training resources of the target large model based on the cause of failure, and execute the training task.

Images