A fault diagnosis method and device for distributed model training

Abstract

The application discloses a kind of distributed model training's fault diagnosis method and device, it is related to artificial intelligence computing technical field, comprising: based on global unique identification with software data and hardware data are associated to generate fusion data stream;With the abnormal data of continuous characteristics monitored is dumped into disk;Abnormal data is injected into the target chip determined based on minimum connected subgraph to carry out fault reproduction and obtain fault reproduction result;According to the abnormal data dumped into disk and fault reproduction result carries out fault diagnosis and obtains abnormal reason.Synchronous acquisition is carried out to hardware data to ensure the comprehensiveness of the acquisition data;With double sliding window, the abnormal data with continuity is dumped, and the log storage pressure is reduced;Dimensionality reduction playback is carried out based on minimum connected subgraph to reduce the cost of fault reproduction;According to the abnormal data dumped and fault reproduction result, automatic root cause positioning is carried out, to improve the efficiency and accuracy of fault diagnosis.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence computing technology, and in particular to a method and apparatus for fault diagnosis of distributed model training. Background Technology

[0002] With the continuous expansion of large models and the widespread application of distributed computing, a large amount of tensor data is usually involved in the collective communication in a multi-machine, multi-card environment. Sometimes, problems such as task blockage, abnormal results, and performance fluctuations may occur during model training. In such cases, the cause of the above problems is diagnosed by reproducing the problem to ensure the normal execution of model training.

[0003] However, current monitoring of model training involves fully calculating the massive amounts of logs generated during long-term training, severely impacting input / output performance and storage resources, leading to increased log storage pressure. Although the log volume is large, it only contains software-level call parameters, losing hardware data from the chip's internal communication engine, thus failing to support in-depth performance tuning. When a fault is identified through monitoring and the detected abnormal data is used to reproduce the fault, a complex multi-machine, multi-card environment needs to be rebuilt, which is not only time-consuming but also significantly increases the cost of fault reproduction. Furthermore, the fault diagnosis process is highly dependent on human experience and lacks automated communication reproduction and verification methods, resulting in reduced efficiency and accuracy in locating the root cause from massive logs. Summary of the Invention

[0004] This invention provides a method and apparatus for fault diagnosis in distributed model training, so as to achieve accurate fault diagnosis in distributed model training.

[0005] According to a first aspect of the present invention, a fault diagnosis method for distributed model training is provided, the method comprising: During distributed training of the model, software data and hardware data are collected, and the software data and hardware data are associated with each other based on a globally unique identifier to generate a fused data stream; The fused data stream is classified and aggregated according to the primitive type to obtain sub-data streams. Anomaly monitoring is performed on each sub-data stream using a dual sliding window, and the detected abnormal data with continuous characteristics is dumped to the disk. Extract the minimum connected subgraph related to the anomaly from the communication topology graph constructed based on the anomaly data, and inject the anomaly data into the target chip determined based on the minimum connected subgraph to reproduce the fault and obtain the fault reproduction result; Based on the abnormal data dumped to the disk and the results of the fault reproduction, fault diagnosis is performed to obtain the cause of the abnormality.

[0006] According to another aspect of the present invention, a fault diagnosis apparatus for distributed model training is provided, the apparatus comprising: The fusion data stream generation module is used to collect software data and hardware data when the model is being trained in a distributed manner, and to associate the software data and hardware data based on a globally unique identifier to generate a fusion data stream. An abnormal data dumping module is used to classify and aggregate the fused data stream according to the primitive type to obtain sub-data streams, use a dual sliding window to monitor abnormalities for each sub-data stream, and dump the monitored abnormal data with continuous characteristics to the disk. The fault reproduction module is used to extract the minimum connected subgraph related to the anomaly from the communication topology graph constructed based on the anomaly data, and inject the anomaly data into the target chip determined based on the minimum connected subgraph to reproduce the fault and obtain the fault reproduction result. The anomaly location module is used to diagnose the anomaly by performing fault diagnosis based on the anomaly data dumped to the disk and the fault reproduction results to obtain the cause of the anomaly.

[0007] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: one or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any embodiment of the present invention.

[0008] According to another aspect of the present invention, a storage medium for computer-executable instructions is provided, on which a computer program is stored, which, when executed by a processor, implements the method as described in any of the embodiments of the present invention.

[0009] The technical solution of this invention simultaneously collects hardware data while collecting software data, thus ensuring the comprehensiveness of the collected data; through a dual sliding window anomaly detection mechanism, only continuous anomaly data is dumped, significantly reducing log storage pressure; based on the dumped anomaly data, the lowest connected subgraph of fault association is identified, and dimensionality reduction and replay are performed based on the lowest connected subgraph to reduce the cost of fault reproduction; and root cause localization is automatically performed based on the dumped anomaly data and fault reproduction results, thereby improving the efficiency and accuracy of fault diagnosis.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a flowchart of a fault diagnosis method for distributed model training according to Embodiment 1 of the present invention; Figure 2 This is a flowchart of another fault diagnosis method for distributed model training provided in Embodiment 2 of the present invention; Figure 3 This is a schematic diagram of the structure of a fault diagnosis device for distributed model training according to Embodiment 3 of the present invention; Figure 4 This is a structural block diagram of an electronic device provided in Embodiment 4 of the present invention. Detailed Implementation

[0013] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0014] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, apparatus, product, or terminal device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or terminal devices.

[0015] Example 1 Figure 1This is a flowchart of a fault diagnosis method for distributed model training provided in Embodiment 1 of the present invention. This embodiment is applicable to fault diagnosis of distributed model training. The method can be executed by a fault diagnosis device for distributed model training. This device can be implemented in hardware and / or software, and can be integrated into an electronic device with data processing capabilities. Figure 1 As shown, the method includes: S101 collects software and hardware data during distributed training of the model, and associates the software and hardware data based on a globally unique identifier to generate a fused data stream.

[0016] Optionally, during distributed model training, software and hardware data are collected, and the software and hardware data are correlated based on a globally unique identifier to generate a fused data stream. This includes: collecting software parameters through a collection communication interface during distributed model training, where the software parameters include communicator identifier, compute node identifier, operation type, data scale, and topology information; obtaining the globally unique identifier, communicator hash value, and the timestamp corresponding to the collection, and binding the globally unique identifier, communicator hash value, and timestamp to the software parameters to generate software data; and using non-intrusive hardware deployed on a specified communication component during distributed model training. The probe collects hardware data and uploads it to the diagnostic management unit via a dedicated acquisition channel. The hardware data includes root cause events and observed symptoms. Root cause events include event types and globally unique identifiers associated with those event types. Event types include cyclic redundancy check errors, recovery triggers, and credit exhaustion. Observed symptoms include symptom types and globally unique identifiers associated with those symptom types. Observed symptom types include physical transmission delays and buffer levels. The diagnostic management unit performs lightweight lossless compression on the hardware data and associates the compressed hardware data with the software data based on the globally unique identifiers to generate a fused data stream.

[0017] Specifically, during the distributed training of large models, both software data from the software layer and hardware data from the hardware layer are captured simultaneously. A unique globally recognized identifier precisely maps these two types of data together, forming a complete fused data stream. This allows for rapid identification of whether a problem arises from a software code logic error or a hardware configuration misconfiguration. During software data capture, software parameters are intercepted through a set communication interface. This interface serves as the interaction interface between the upper-layer framework and the lower-layer communication library. Software parameters include communicator identifier, compute node identifier, operation type, data size, and topology information. The operation type refers to the specific communication action, such as full reduction, broadcast, or full collection. The compute node identifier identifies which GPU / NPU (e.g., Rank 0 or Rank 7) initiated the operation, distinguishing the roles of different devices. The data size estimate directly affects communication time and is a crucial reference for determining whether congestion has occurred. The topology information details the specific network structure involved in the operation, specifying which chips are involved in the communication interaction. In addition, for each operation, a globally unique identifier, a communicator hash value, and a timestamp corresponding to the time of collection are obtained. The globally unique identifier is a key index connecting software logs and hardware probe data; the timestamp records the precise moment the operation occurred, used for analyzing timing relationships; in distributed training, hundreds or thousands of processes may belong to the same communication group. To prevent topology parsing errors due to process number confusion, the system calculates a hash value for the communicator's configuration information. If the hash values ​​are the same, it means that these processes belong to the same logical group, and their topology views are consistent. This ensures that the constructed communication topology map is accurate when reproducing a fault. In this embodiment, the obtained globally unique identifier, communicator hash value, and timestamp are bound to the software parameters collected through interception to generate software data. Of course, this embodiment is only an example and does not limit the specific content of the software data.

[0018] In addition, this embodiment deploys non-invasive hardware probes on designated communication components within the chip. These designated communication components can be communication engines, transmission engines, or memory controllers, etc. The non-invasive probes collect hardware data involved in the distributed modeling process. The non-invasive hardware probes deployed in this embodiment do not interfere with or slow down normal training tasks during monitoring, and they upload the collected hardware data to the diagnostic management unit through a dedicated acquisition channel, thus not consuming service bandwidth. Specifically, when software tasks are issued, a globally unique identifier is written to a shadow register. The probe can obtain the globally unique identifier by reading the register and append it to the collected data. Therefore, the hardware data includes root cause events and observed symptoms. Root cause events include event types and globally unique identifiers bound to those event types. Event types include cyclic redundancy check errors, recovery triggers, and credit exhaustion. Observed symptoms include symptom types and globally unique identifiers bound to those symptom types. Observed symptom types include physical transmission delay and buffer water level. Additionally, root cause events are used to indicate the cause of a fault and can be injected during fault reproduction. Cyclic Redundancy Check (CRC) errors refer to packet verification failures triggering retransmissions; recovery triggers refer to the link entering a retraining state and transmission interruption; and credit depletion refers to the sender having no available credit, resulting in back pressure. The hardware recording format for root cause events involves recording a timestamp and event type each time an event occurs, forming an event sequence, such as T1: CRC error, T2: Recovery enters, T3: Recovery exits, etc. Observed symptoms are used to indicate the consequences of a fault and can be verified through fault playback results. Buffer level reflects the degree of back pressure, and physical transmission delay reflects transmission efficiency. The hardware recording format for observed symptoms involves periodic sampling (e.g., recording the buffer level every 10µs), recording the transmission and reception timestamps for each data packet, and calculating the delay to form a time sequence.

[0019] It should be noted that, after receiving hardware data, the diagnostic management unit in this embodiment can perform lightweight lossless compression on the hardware data and associate the software data with the compressed hardware data. The software data can be used to provide task boundaries (timestamp, globally unique identifier, communication primitive parameters, etc.), the root cause events in the hardware data are used to record hardware anomalies (timestamp, globally unique identifier, event type), and the observed symptoms in the hardware data are used to record physical states (timestamp, globally unique identifier, buffer bits, and transmission delay). Therefore, in this embodiment, the software data and hardware data can be associated and aligned using globally unique identifiers to form a fused data stream containing a complete causal chain of "task → root cause → symptom". Of course, this embodiment is merely illustrative and does not limit the specific content included in the fused data stream.

[0020] S102, classify and aggregate the fused data stream according to the primitive type to obtain sub-data streams, use a double sliding window to monitor anomalies for each sub-data stream, and dump the monitored abnormal data with continuous characteristics to the disk.

[0021] Optionally, anomaly monitoring is performed using a dual sliding window for each sub-data stream, including: using a first window to perform sliding calculation to obtain the mean and standard deviation for each sub-data stream, and using a second window to calculate the trend slope, wherein the length of the second window is greater than that of the first window; determining the anomaly type of the corresponding sub-data stream based on the mean, standard deviation, and trend slope, wherein the anomaly type includes transient characteristics and continuous characteristics.

[0022] Specifically, after acquiring the fused data stream generated during model training through monitoring, in order to save disk space and capture truly fault-related anomaly data, the system does not store all detected data. Instead, it analyzes the data in real time using a dual sliding window. Only when a serious anomaly with continuous characteristics (i.e., a true fault trend) is detected will the saving mechanism be triggered, writing the key data to the disk. The fused data stream containing software and hardware information includes different primitive types. Primitive types refer to the specific types or modes of aggregated communication operations, such as AllReduce or Broadcast. Because different primitive types of communication operations have different normal data distributions and fluctuation ranges, this implementation can classify and aggregate the fused data stream according to primitive types to obtain sub-data streams, thereby processing data streams of different primitive types separately, which can significantly improve the detection accuracy.

[0023] In this embodiment, a dual sliding window approach is used for anomaly monitoring of each sub-data stream. This involves simultaneously scanning data using two windows with different functions, each capturing different anomaly characteristics. Specifically, a first window is used for sliding calculations to obtain the mean and standard deviation for each sub-data stream, focusing on the data's dispersion and local fluctuations. This reflects whether the current value deviates from the recent average level; therefore, the first window excels at capturing instantaneous anomalies, such as sudden spikes, momentary network jitter, or single checksum errors. A second window is used for sliding calculations to obtain the trend slope, focusing on the direction of data change and long-term trends. Due to its longer window size, it smooths out short-term noise and reveals the overall trend; therefore, the second window excels at capturing continuous anomalies, such as gradually decreasing bandwidth, continuously increasing latency, and slowly rising buffer levels (precursor to congestion). In this embodiment, the second window is longer than the first window. The first window is sensitive but easily affected by interference, while the second window is less sensitive but robust. The combination of both can detect both sudden changes and gradual shifts. Therefore, this embodiment can determine the anomaly type of the corresponding sub-data stream based on the mean, standard deviation, and trend slope, and the anomaly type includes both instantaneous and continuous characteristics.

[0024] It should be noted that in this embodiment, a hierarchical response strategy is executed based on the anomaly detection results. When the data is determined to be transient, it indicates that it may be environmental noise. In this case, a first-level response is executed, which simply marks the anomaly without writing it to the disk. When the data is determined to be continuous, it indicates that a substantial failure has occurred in the model training process. In this case, a second-level response is executed, which triggers a dump operation to dump the detected continuous anomaly data to the disk for subsequent root cause analysis. Of course, this embodiment is only an example and does not limit the specific form of the hierarchical response strategy.

[0025] S103: Extract the minimum connected subgraph related to the anomaly from the communication topology graph constructed based on the anomaly data, and inject the anomaly data into the target chip determined based on the minimum connected subgraph to reproduce the fault and obtain the fault reproduction result.

[0026] Optionally, extracting the minimum connected subgraph related to the anomaly from the communication topology graph constructed based on the anomaly data includes: extracting the communicator hash values ​​contained in the anomaly data, and aggregating anomaly data with the same communicator hash values ​​to obtain an anomaly data subset; extracting the topology information contained in each software data for each anomaly data subset, and concatenating the topology information to generate a communication topology graph corresponding to the anomaly data subset; marking the anomaly edges for each communication topology graph according to the corresponding anomaly data subset, and extracting the minimum connected subgraph containing all anomaly edges from the communication topology graph.

[0027] Specifically, after acquiring abnormal data through dual sliding windows, fault reproduction is performed based on this data. If the reproduction still uses the entire large-scale hardware environment involved in the actual training, it requires rebuilding the entire computing environment, preparing data, and executing a lengthy process, resulting in cumbersome steps and high time costs. In this implementation, to reduce reproduction costs, only the smallest connected subgraph where the problem actually occurred is segmented from the massive communication topology graph constructed based on the abnormal data. Since different tasks or model layers may use different communication groups during large-scale training, the system first uses the communicator hash value as a group label to select and aggregate abnormal data belonging to the same communication group to obtain a subset of abnormal data. This ensures that subsequent analysis focuses on faults within the same logical network, avoiding mixing data from different tasks and preventing subsequent root cause localization. Each subset of abnormal data includes hardware and software data corresponding to each timestamp, and the software data includes topology information, such as chip connection relationships and master-slave relationships. By concatenating all the topology information contained in each subset of abnormal data, the complete network connection relationship under the corresponding communication group can be reconstructed, i.e., the communication topology graph. Therefore, a corresponding communication topology map can be constructed for each subset of abnormal data. Of course, this embodiment is only an example and does not limit the specific number of communication topology maps based on abnormal data. As long as the number of communication topology maps is the same as the number of abnormal data subsets, they are all within the protection scope of this application.

[0028] In this implementation, for each communication topology graph, the erroneous links (i.e., abnormal edges) are marked based on the corresponding subset of abnormal data. Then, the marked communication topology graph is traversed to extract a minimum connected subgraph. This minimum connected subgraph must satisfy the following two conditions: 1. It contains all fault points: all marked abnormal edges are included; 2. It maintains connectivity: the connections between these fault points are complete and cannot be broken. For example, when the model is being trained in a distributed manner, an 8000-card cluster is used, but only cards 101 to 108 experience deadlock during communication. In this case, the system will not reproduce the entire 8000-card network, but will only extract the subgraph of these 8 cards and their interconnections as the minimum connected subgraph. Of course, the above example is only used to illustrate the construction of a minimum connected subgraph containing a communication group. In actual applications, the specific number of minimum connected subgraphs is not limited.

[0029] Optionally, the abnormal data is injected into the target chip determined based on the minimum connected subgraph to reproduce the fault and obtain the fault reproduction result. This includes: determining the minimum number of chips required for playback based on the minimum connected subgraph, and configuring the target chip that matches the minimum number of chips; injecting the software data and root cause events in the abnormal data and the hardware data into the target chip to reproduce the distributed training process of the model and obtain the fault reproduction result, wherein the fault reproduction result includes the reproduction of hardware data and software error conditions; verifying the fidelity of the fault reproduction result, and outputting the fault reproduction result when the verification is successful.

[0030] Specifically, in this implementation, the system utilizes the minimum connected subgraph extracted in the previous step (i.e., the core local network of the fault) to perform low-cost, high-fidelity fault reproduction on a small number of physical chips. It goes beyond simply analyzing logs; it actually injects software instructions and hardware faults into the target chip to see if the original problem can be reproduced and to verify the authenticity and reliability of the reproduced results. Specifically, the minimum number of chips required for playback can be determined based on the minimum connected subgraph. For example, when the minimum connected subgraph involves only a ring connection of 8 nodes, the system automatically calculates the minimum hardware resources required to reproduce this specific fault, i.e., 8 chips, and automatically configures these 8 target chips, which greatly reduces the reproduction cost. After determining the target chip, the system injects software data from the abnormal data and root cause events from the hardware data into the target chip to reproduce the distributed training process of the model to obtain fault reproduction results. The system then observes the fault reproduction results obtained under this software and hardware data injection, where the fault reproduction results include hardware data and software error information. In addition, this embodiment will also verify the fidelity of the fault reproduction results, that is, verify the fault reproduction results from three aspects: data correctness, state timing matching degree, and water level distribution consistency, to ensure that the fault can be stably reproduced.

[0031] S104. Based on the abnormal data dumped to the disk and the results of fault reproduction, perform fault diagnosis to obtain the cause of the abnormality.

[0032] Optionally, based on the abnormal data dumped to the disk and the fault reproduction results, fault diagnosis is performed to obtain the cause of the anomaly, including: determining whether the software error has disappeared abnormally. If so, the cause of the anomaly is directly determined to be a topology fault. Otherwise, the fault is reproduced again based on the abnormal data to obtain the latest fault reproduction result. The latest fault reproduction result is compared with the fault reproduction result obtained from the first reproduction to determine whether the two results are different. If so, the cause of the anomaly is determined to be uncertain and node isolation is recommended. Otherwise, the abnormal hardware data contained in the abnormal data is extracted, and fault diagnosis is performed based on the abnormal hardware data to obtain the cause of the anomaly.

[0033] Specifically, this implementation first checks whether the software error in the fault reproduction results has disappeared abnormally. If it has, it indicates that the fault is not caused by a specific faulty component, but rather depends on a specific network structure. When the system attempts to replay using the minimum connected subgraph, the interference disappears because it is no longer in the original complex network environment, and the fault disappears as well. Therefore, the cause of the anomaly can be determined to be a topology fault, i.e., a problem with the original network connection method or path (such as routing deadlock or cross-node congestion), rather than a single chip failure. Furthermore, if the anomaly has not disappeared, i.e., the fault still exists, the system will perform another reproduction, comparing the latest fault reproduction result from the second run with the first. If the two runs are different (e.g., the first run reports a timeout, the second reports a verification error, or one succeeds and the other fails), this indicates that the fault behavior is extremely unstable, usually due to poor signal integrity, voltage fluctuations, or a critical state. In this case, the cause of the anomaly is determined to be uncertain, i.e., an intermittent fault. Since it is difficult to pinpoint the specific component causing the problem, the safest strategy is node isolation (removing the unstable device from the cluster) to prevent it from slowing down the overall training. If the fault does not disappear and the structure is highly consistent each time it is reproduced, then intermittent problems can be ruled out. The system will then conduct in-depth analysis of the specific abnormal hardware data and perform fault diagnosis based on the abnormal hardware data to obtain the cause of the abnormality. At this time, the cause of the abnormality is collectively referred to as a deterministic hardware fault.

[0034] Optionally, extract the abnormal hardware data contained in the abnormal data, and perform fault diagnosis based on the abnormal hardware data to obtain the cause of the abnormality, including: extracting the root cause events and observed symptoms contained in the abnormal data; when the event type of the root cause event is cyclic redundancy check error and the observed symptom is physical transmission delay, the cause of the abnormality is determined to be physical layer signal integrity failure; when the event type of the root cause event is credit depletion and the observed symptom is physical transmission delay, the cause is refined based on the local transmission delay and the peer reception delay in the physical transmission delay; when the event type of the root cause event is credit depletion and the observed symptom is buffer water level, the cause of the abnormality is determined to be flow control congestion or peer processing bottleneck.

[0035] Optionally, the attribution can be refined based on the local transmission delay and the peer reception delay in the physical transmission delay, including: when the local transmission delay exceeds the corresponding threshold, the cause of the anomaly is determined to be a local transmission path failure; when the peer reception delay exceeds the corresponding threshold, the cause of the anomaly is determined to be a peer reception path failure.

[0036] When the cause of the anomaly is determined to be a deterministic hardware failure, the underlying physical fault type is further precisely determined based on the specific details of the root cause event and observed symptoms. For example, when the root cause event type is Cyclic Redundancy Check (CRC) error and the observed symptom is physical transmission delay, the occurrence of a CRC error indicates that data has been flipped or corrupted during transmission. When accompanied by physical transmission delay, this usually points to physical layer problems such as optical module aging, excessive bending of fiber optic cables, poor interface contact, or electromagnetic interference. Therefore, the cause of the anomaly can be determined to be a physical layer signal integrity failure, i.e., poor current physical link quality. Furthermore, when the root cause event type is credit depletion and the observed symptom is physical transmission delay, if the physical transmission delay is a local transmission delay exceeding a corresponding threshold, the cause of the anomaly is determined to be a local transmission path failure; if the physical transmission delay is a peer reception delay exceeding a corresponding threshold, the cause of the anomaly is determined to be a peer reception path failure. Additionally, when the root cause event is credit exhaustion and the observed symptom is buffer level, it indicates that the receiving end is processing data too slowly (e.g., GPU computing cannot keep up with the network card's receiving speed), resulting in buffer backlog and inability to release new credit to the sender. In this case, the cause of the anomaly is determined to be flow control congestion or a bottleneck in the peer's processing.

[0037] The technical solution of this invention simultaneously collects hardware data while collecting software data, thus ensuring the comprehensiveness of the collected data; through a dual sliding window anomaly detection mechanism, only continuous anomaly data is dumped, significantly reducing log storage pressure; based on the dumped anomaly data, the lowest connected subgraph of fault association is identified, and dimensionality reduction and replay are performed based on the lowest connected subgraph to reduce the cost of fault reproduction; and root cause localization is automatically performed based on the dumped anomaly data and fault reproduction results, thereby improving the efficiency and accuracy of fault diagnosis.

[0038] Example 2 Figure 2 This is a flowchart of another fault diagnosis method for distributed model training provided by an embodiment of the present invention. Based on the above embodiment, after obtaining the cause of the anomaly through fault diagnosis based on the abnormal data dumped to disk and the fault reproduction results, this embodiment further includes: determining an optimization strategy matching the cause of the anomaly; and executing the optimization strategy to adjust the distributed training process of the model, such as... Figure 2 As shown, the method includes: S201 collects software and hardware data during distributed training of the model, and associates the software and hardware data based on a globally unique identifier to generate a fused data stream.

[0039] Optionally, during distributed model training, software and hardware data are collected, and the software and hardware data are correlated based on a globally unique identifier to generate a fused data stream. This includes: collecting software parameters through a collection communication interface during distributed model training, where the software parameters include communicator identifier, compute node identifier, operation type, data scale, and topology information; obtaining the globally unique identifier, communicator hash value, and the timestamp corresponding to the collection, and binding the globally unique identifier, communicator hash value, and timestamp to the software parameters to generate software data; and using non-intrusive hardware deployed on a specified communication component during distributed model training. The probe collects hardware data and uploads it to the diagnostic management unit via a dedicated acquisition channel. The hardware data includes root cause events and observed symptoms. Root cause events include event types and globally unique identifiers associated with those event types. Event types include cyclic redundancy check errors, recovery triggers, and credit exhaustion. Observed symptoms include symptom types and globally unique identifiers associated with those symptom types. Observed symptom types include physical transmission delays and buffer levels. The diagnostic management unit performs lightweight lossless compression on the hardware data and associates the compressed hardware data with the software data based on the globally unique identifiers to generate a fused data stream.

[0040] S202, classify and aggregate the fused data stream according to the primitive type to obtain sub-data streams, use a dual sliding window to monitor anomalies for each sub-data stream, and dump the monitored abnormal data with continuous characteristics to the disk.

[0041] Optionally, anomaly monitoring is performed using a dual sliding window for each sub-data stream, including: using a first window to perform sliding calculation to obtain the mean and standard deviation for each sub-data stream, and using a second window to calculate the trend slope, wherein the length of the second window is greater than that of the first window; determining the anomaly type of the corresponding sub-data stream based on the mean, standard deviation, and trend slope, wherein the anomaly type includes transient characteristics and continuous characteristics.

[0042] S203: Extract the minimum connected subgraph related to the anomaly from the communication topology graph constructed based on the anomaly data, and inject the anomaly data into the target chip determined based on the minimum connected subgraph to reproduce the fault and obtain the fault reproduction result.

[0043] Optionally, extracting the minimum connected subgraph related to the anomaly from the communication topology graph constructed based on the anomaly data includes: extracting the communicator hash values ​​contained in the anomaly data, and aggregating anomaly data with the same communicator hash values ​​to obtain an anomaly data subset; extracting the topology information contained in each software data for each anomaly data subset, and concatenating the topology information to generate a communication topology graph corresponding to the anomaly data subset; marking the anomaly edges for each communication topology graph according to the corresponding anomaly data subset, and extracting the minimum connected subgraph containing all anomaly edges from the communication topology graph.

[0044] Optionally, the abnormal data is injected into the target chip determined based on the minimum connected subgraph to reproduce the fault and obtain the fault reproduction result. This includes: determining the minimum number of chips required for playback based on the minimum connected subgraph, and configuring the target chip that matches the minimum number of chips; injecting the software data and root cause events in the abnormal data and the hardware data into the target chip to reproduce the distributed training process of the model and obtain the fault reproduction result, wherein the fault reproduction result includes the reproduction of hardware data and software error conditions; verifying the fidelity of the fault reproduction result, and outputting the fault reproduction result when the verification is successful.

[0045] S204. Based on the abnormal data dumped to the disk and the results of fault reproduction, perform fault diagnosis to obtain the cause of the abnormality.

[0046] Optionally, based on the abnormal data dumped to the disk and the fault reproduction results, fault diagnosis is performed to obtain the cause of the anomaly, including: determining whether the software error has disappeared abnormally. If so, the cause of the anomaly is directly determined to be a topology fault. Otherwise, the fault is reproduced again based on the abnormal data to obtain the latest fault reproduction result. The latest fault reproduction result is compared with the fault reproduction result obtained from the first reproduction to determine whether the two results are different. If so, the cause of the anomaly is determined to be uncertain and node isolation is recommended. Otherwise, the abnormal hardware data contained in the abnormal data is extracted, and fault diagnosis is performed based on the abnormal hardware data to obtain the cause of the anomaly.

[0047] Optionally, extract the abnormal hardware data contained in the abnormal data, and perform fault diagnosis based on the abnormal hardware data to obtain the cause of the abnormality, including: extracting the root cause events and observed symptoms contained in the abnormal data; when the event type of the root cause event is cyclic redundancy check error and the observed symptom is physical transmission delay, the cause of the abnormality is determined to be physical layer signal integrity failure; when the event type of the root cause event is credit depletion and the observed symptom is physical transmission delay, the cause is refined based on the local transmission delay and the peer reception delay in the physical transmission delay; when the event type of the root cause event is credit depletion and the observed symptom is buffer water level, the cause of the abnormality is determined to be flow control congestion or peer processing bottleneck.

[0048] Optionally, the attribution can be refined based on the local transmission delay and the peer reception delay in the physical transmission delay, including: when the local transmission delay exceeds the corresponding threshold, the cause of the anomaly is determined to be a local transmission path failure; when the peer reception delay exceeds the corresponding threshold, the cause of the anomaly is determined to be a peer reception path failure.

[0049] S205, determine the optimization strategy that matches the cause of the anomaly, and execute the optimization strategy to adjust the distributed training process of the model.

[0050] Specifically, in this implementation, the system searches for corresponding optimization strategies in a pre-built expert knowledge base based on the determined cause of the anomaly. For example, if the hardware is faulty, the strategy is to isolate the node; if the network is congested, the strategy is to adjust the routing or limit the rate. This implementation does not limit the specific content of the optimization strategy. The system will translate the selected strategy into specific configuration instructions or operations and apply them to the current training task. This may involve modifying configuration files, restarting specific processes, or adjusting communication parameters online.

[0051] For example, when the cause of the anomaly is determined to be a topology fault, the corresponding optimization strategy is route reconfiguration. The specific action is to modify the routing algorithm of the aggregated communication (such as changing from adaptive routing to fixed routing) to avoid network paths that could cause deadlocks. When the cause of the anomaly is determined to be uncertain, the corresponding optimization strategy is fault-tolerant migration. The specific action is to save the current checkpoint and migrate the task to a backup healthy node for restart. When the cause of the anomaly is determined to be flow control congestion or a bottleneck in the peer's processing, the corresponding optimization strategy is flow control and rate reduction. The specific action is to dynamically reduce the data injection rate of the sender or increase the buffer size of the receiver to prevent overflow. When the cause of the anomaly is determined to be a physical layer signal integrity failure, the corresponding optimization strategy is node / link isolation. Specifically, this involves marking the faulty network interface card (NIC) or node as unavailable, replanning the communication topology, and bypassing the faulty component to continue training. Of course, this embodiment is merely an example and does not limit the optimization strategy or specific actions.

[0052] The technical solution of this invention simultaneously collects hardware data while collecting software data, thus ensuring the comprehensiveness of the collected data; through a dual sliding window anomaly detection mechanism, only continuous anomaly data is dumped, significantly reducing log storage pressure; based on the dumped anomaly data, the lowest connected subgraph of fault association is identified, and dimensionality reduction and replay are performed based on the lowest connected subgraph to reduce the cost of fault reproduction; and root cause localization is automatically performed based on the dumped anomaly data and fault reproduction results, thereby improving the efficiency and accuracy of fault diagnosis.

[0053] Example 3 Figure 3This is a schematic diagram of a fault diagnosis device for distributed model training provided in an embodiment of the present invention. Figure 3 As shown, the device includes: a fused data stream generation module 310, an abnormal data dumping module 320, a fault reproduction module 330, and an abnormal location module 340.

[0054] Among them, the fusion data stream generation module 310 is used to collect software data and hardware data when the model is being trained in a distributed manner, and to associate the software data and hardware data based on a globally unique identifier to generate a fusion data stream. The abnormal data dumping module 320 is used to classify and aggregate the fused data stream according to the primitive type to obtain sub-data streams, use a dual sliding window to monitor abnormalities for each sub-data stream, and dump the monitored abnormal data with continuous characteristics to the disk. The fault reproduction module 330 is used to extract the minimum connected subgraph related to the anomaly from the communication topology graph constructed based on the anomaly data, and inject the anomaly data into the target chip determined based on the minimum connected subgraph to reproduce the fault and obtain the fault reproduction result. The anomaly location module 340 is used to diagnose and obtain the cause of the anomaly based on the abnormal data dumped to the disk and the results of the fault reproduction.

[0055] Optionally, a data stream generation module is used to collect software parameters through a collection communication interface when the model is being trained in a distributed manner. The software parameters include communicator identifier, computing node identifier, operation type, data size, and topology information. Obtain the globally unique identifier, communicator hash value, and the timestamp corresponding to the data collection time. Bind the globally unique identifier, communicator hash value, and timestamp to the software parameters to generate software data. During distributed training of the model, hardware data is collected by non-intrusive hardware probes deployed on designated communication components and uploaded to the diagnostic management unit through a dedicated acquisition channel. The hardware data includes root cause events and observed symptoms. Root cause events include event types and globally unique identifiers bound to the event types. Event types include cyclic redundancy check errors, recovery triggers, and credit depletion. Observed symptoms include symptom types and globally unique identifiers bound to the symptom types. Observed symptom types include physical transmission delay and buffer level. The diagnostic management unit performs lightweight lossless compression on hardware data and associates the compressed hardware data and software data based on a globally unique identifier to generate a fused data stream.

[0056] Optionally, an abnormal data dumping module is used to obtain the mean and standard deviation by using a first window to perform sliding calculation for each sub-data stream, and to calculate the trend slope by using a second window, wherein the length of the second window is greater than that of the first window; For the mean, standard deviation, and trend slope, determine the anomaly type of the corresponding sub-data stream, where the anomaly type includes transient characteristics and continuous characteristics.

[0057] Optionally, the fault reproduction module includes a minimum connected subgraph extraction unit, which is used to extract the communicator hash value contained in the abnormal data and aggregate the abnormal data with the same communicator hash value to obtain a subset of abnormal data; For each subset of abnormal data, extract the topology information contained in each software data, and then stitch the topology information together to generate a communication topology diagram corresponding to the subset of abnormal data. For each communication topology graph, mark the abnormal edges according to the corresponding abnormal data subset, and extract the minimum connected subgraph containing all abnormal edges from the communication topology graph.

[0058] Optionally, the fault reproduction module includes a fault reproduction unit for determining the minimum number of chips required for playback based on the minimum connected subgraph, and configuring a target chip that matches the minimum number of chips. The distributed training process of the model is reproduced by injecting software data from abnormal data and root cause events from hardware data into the target chip to obtain fault reproduction results. The fault reproduction results include the reproduction of hardware data and software error conditions. Verify the fidelity of the fault reproduction results, and output the fault reproduction results when the verification is successful.

[0059] Optional, an anomaly location module is used to determine whether the software error message indicates an abnormal disappearance. If so, the cause of the anomaly is directly determined to be a topology fault. Otherwise, reproduce the fault again based on the abnormal data to obtain the latest fault reproduction result, and compare the latest fault reproduction result with the fault reproduction result obtained from the first reproduction to determine whether the two results are different. If they are different, determine that the cause of the abnormality is uncertain and recommend node isolation. Otherwise, extract the abnormal hardware data contained in the abnormal data, and perform fault diagnosis based on the abnormal hardware data to obtain the cause of the abnormality.

[0060] Optionally, an anomaly localization module is used to extract the root cause events and observed symptoms contained in the anomaly data. When the event type of the root cause event is a cyclic redundancy check error and the observed symptom is physical transmission delay, the cause of the anomaly is determined to be a physical layer signal integrity failure. When the root cause event is credit depletion and the observed symptom is physical transmission delay, the attribution is refined based on the local transmission delay and the peer reception delay within the physical transmission delay. When the root cause event type is credit depletion and the observed symptom is buffer level, the cause of the anomaly is determined to be flow control congestion or peer processing bottleneck.

[0061] Optionally, the anomaly location module is also used to determine that the cause of the anomaly is a local transmission path failure when the local transmission delay exceeds the corresponding threshold. When the receiving delay at the peer exceeds the corresponding threshold, the cause of the anomaly is determined to be a failure in the peer's receiving path.

[0062] Optionally, the device also includes an adjustment module for determining an optimization strategy that matches the cause of the anomaly; Execute optimization strategies to adjust the distributed training process of the model.

[0063] The fault diagnosis device for distributed model training provided in this embodiment of the invention can execute the fault diagnosis method for distributed model training provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0064] Example 4 Figure 4 A schematic diagram of an electronic device 10, which can be used to implement embodiments of the present invention, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0065] The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the invention described and / or claimed herein.

[0066] like Figure 4 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0067] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other electronic devices through computer networks such as the Internet and / or various telecommunications networks.

[0068] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as fault diagnosis methods for distributed model training.

[0069] That is, when the model is being trained in a distributed manner, software data and hardware data are collected, and the software data and hardware data are associated with each other based on a globally unique identifier to generate a fused data stream; The fused data stream is classified and aggregated according to the primitive type to obtain sub-data streams. Anomaly monitoring is performed on each sub-data stream using a dual sliding window, and the detected anomaly data with continuous characteristics is dumped to disk. Extract the minimum connected subgraph related to the anomaly from the communication topology graph constructed based on the anomaly data, and inject the anomaly data into the target chip determined based on the minimum connected subgraph to reproduce the fault and obtain the fault reproduction result; Based on the abnormal data dumped to the disk and the results of fault reproduction, fault diagnosis is performed to obtain the cause of the abnormality.

[0070] In some embodiments, the fault diagnosis method for distributed model training can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the fault diagnosis method for distributed model training described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to perform the fault diagnosis method for distributed model training by any other suitable means (e.g., by means of firmware).

[0071] Various embodiments of the apparatuses and techniques described above herein can be implemented in digital electronic circuit devices, integrated circuit devices, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), device-on-a-chip (SoC) devices, complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable device including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage device, at least one input device, and at least one output device, and transmitting data and instructions to the storage device, the at least one input device, and the at least one output device.

[0072] Computer programs used to implement the fault diagnosis method for distributed model training of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to the processor of a general-purpose computer or a special-purpose computer, such that when executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer programs can be executed entirely on the machine, partially on the machine, or as a standalone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0073] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution apparatus, device, or electronic device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage electronics, magnetic storage electronics, or any suitable combination thereof.

[0074] To provide interaction with a user, the devices and techniques described herein can be implemented on an electronic device having: a display device (e.g., a touchscreen) for displaying information to the user; and buttons through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0075] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0076] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A fault diagnosis method for distributed model training, characterized in that, The method includes: During distributed training of the model, software data and hardware data are collected, and the software data and hardware data are associated with each other based on a globally unique identifier to generate a fused data stream; The fused data stream is classified and aggregated according to the primitive type to obtain sub-data streams. Anomaly monitoring is performed on each sub-data stream using a dual sliding window, and the detected abnormal data with continuous characteristics is dumped to the disk. Extract the minimum connected subgraph related to the anomaly from the communication topology graph constructed based on the anomaly data, and inject the anomaly data into the target chip determined based on the minimum connected subgraph to reproduce the fault and obtain the fault reproduction result; Based on the abnormal data dumped to the disk and the results of the fault reproduction, fault diagnosis is performed to obtain the cause of the abnormality.

2. The method according to claim 1, characterized in that, The process of collecting software and hardware data during distributed training of the model, and associating the software and hardware data based on a globally unique identifier to generate a fused data stream, includes: During distributed training of the model, software parameters are collected through a collective communication interface, wherein the software parameters include communicator identifier, computing node identifier, operation type, data scale, and topology information; Obtain a globally unique identifier, a communicator hash value, and a timestamp corresponding to the time of data collection; bind the globally unique identifier, the communicator hash value, and the timestamp to the software parameters to generate the software data. During distributed training of the model, hardware data is collected by non-invasive hardware probes deployed on designated communication components and uploaded to the diagnostic management unit through a dedicated acquisition channel. The hardware data includes root cause events and observed symptoms. The root cause events include event types and globally unique identifiers bound to the event types. The event types include cyclic redundancy check errors, recovery triggers, and credit exhaustion. The observed symptoms include symptom types and globally unique identifiers bound to the symptom types. The observed symptom types include physical transmission delay and buffer water level. The diagnostic management unit performs lightweight lossless compression on the hardware data and associates the compressed hardware data and software data based on the globally unique identifier to generate the fused data stream.

3. The method according to claim 1, characterized in that, The method of using a dual sliding window for anomaly monitoring of each of the sub-data streams includes: For each of the sub-data streams, a first window is used to perform sliding calculation to obtain the mean and standard deviation, and a second window is used to calculate the trend slope, wherein the length of the second window is greater than that of the first window; For the mean, standard deviation and trend slope, the anomaly type of the corresponding sub-data stream is determined, wherein the anomaly type includes transient characteristics and continuous characteristics.

4. The method according to claim 2, characterized in that, Extracting the minimum connected subgraph related to the anomaly from the communication topology graph constructed based on the anomaly data includes: Extract the communicator hash value contained in the abnormal data, and aggregate the abnormal data with the same communicator hash value to obtain a subset of abnormal data; For each subset of abnormal data, extract the topology information contained in each software data, and concatenate the topology information to generate a communication topology map corresponding to the subset of abnormal data. For each of the communication topologies, abnormal edges are marked according to the corresponding abnormal data subset, and the minimum connected subgraph containing all abnormal edges is extracted from the communication topologies.

5. The method according to claim 4, characterized in that, The step of injecting the abnormal data into the target chip determined based on the minimum connected subgraph to reproduce the fault and obtain the fault reproduction result includes: The minimum number of chips required for playback is determined based on the minimum connected subgraph, and a target chip matching the minimum number of chips is configured. The software data in the abnormal data and the root cause events in the hardware data are injected into the target chip to reproduce the distributed training process of the model to obtain the fault reproduction results, wherein the fault reproduction results include the reproduction of hardware data and software error conditions. The fidelity of the fault reproduction result is verified, and the fault reproduction result is output when the verification is successful.

6. The method according to claim 5, characterized in that, The step of diagnosing the fault and obtaining the cause of the fault based on the abnormal data dumped to the disk and the fault reproduction results includes: Determine whether the software error message indicates an abnormal disappearance. If so, directly determine the cause of the abnormality as a topology fault. Otherwise, based on the abnormal data, the fault is reproduced again to obtain the latest fault reproduction result, and the latest fault reproduction result is compared with the fault reproduction result obtained in the first reproduction to determine whether the two results are different. If they are different, the cause of the abnormality is determined to be uncertain and node isolation is recommended. Otherwise, the abnormal hardware data contained in the abnormal data is extracted, and the fault diagnosis is performed based on the abnormal hardware data to obtain the cause of the abnormality.

7. The method according to claim 6, characterized in that, The step of extracting abnormal hardware data contained in the abnormal data and performing fault diagnosis based on the abnormal hardware data to obtain the cause of the abnormality includes: Extract the root cause events and observed symptoms contained in the abnormal data. When the event type of the root cause event is cyclic redundancy check error and the observed symptom is physical transmission delay, the cause of the abnormality is determined to be a physical layer signal integrity failure. When the root cause event is of credit depletion and the observed symptom is physical transmission delay, the attribution is refined based on the local transmission delay and the peer reception delay in the physical transmission delay. When the root cause event is of credit depletion and the observed symptom is buffer level, the cause of the anomaly is determined to be flow control congestion or peer processing bottleneck.

8. The method according to claim 7, characterized in that, The detailed attribution based on the local transmission delay and the peer reception delay in the physical transmission delay includes: When the local transmission delay exceeds the corresponding threshold, the cause of the anomaly is determined to be a local transmission path failure. When the peer's reception delay exceeds the corresponding threshold, the cause of the anomaly is determined to be a peer's reception path failure.

9. The method according to claim 1, characterized in that, After obtaining the cause of the anomaly by performing fault diagnosis based on the abnormal data dumped to the disk and the fault reproduction results, the method further includes: Determine the optimization strategy that matches the cause of the anomaly; The optimization strategy is implemented to adjust the distributed training process of the model.

10. A fault diagnosis device for distributed model training, characterized in that, The device includes: The fusion data stream generation module is used to collect software data and hardware data when the model is being trained in a distributed manner, and to associate the software data and hardware data based on a globally unique identifier to generate a fusion data stream. An abnormal data dumping module is used to classify and aggregate the fused data stream according to the primitive type to obtain sub-data streams, use a dual sliding window to monitor abnormalities for each sub-data stream, and dump the monitored abnormal data with continuous characteristics to the disk. The fault reproduction module is used to extract the minimum connected subgraph related to the anomaly from the communication topology graph constructed based on the anomaly data, and inject the anomaly data into the target chip determined based on the minimum connected subgraph to reproduce the fault and obtain the fault reproduction result. The anomaly location module is used to diagnose the anomaly by performing fault diagnosis based on the anomaly data dumped to the disk and the fault reproduction results to obtain the cause of the anomaly.

11. An electronic device, characterized in that, The electronic device includes: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-9.

12. A storage medium for computer-executable instructions, wherein a computer program is stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-9.

Images