Method and apparatus for processing gpu cluster failure, device, medium, program product

Abstract

The present disclosure provides a method and apparatus, device, medium, program product for processing GPU cluster failure. By performing a desensitization operation on the failure features associated with the GPU cluster failure event at the client side, and sending the desensitized failure features to the server side to match the historical failure fingerprints of other clients, the risk of information leakage of the client identification information can be reduced while retaining the information required for failure diagnosis, and the historical failure processing experience of different clients can be used to assist the diagnosis and processing of the current GPU cluster failure event, thereby realizing the reuse of historical failure experience and improving the processing efficiency of GPU cluster failure diagnosis.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to a method, apparatus, computer device, computer readable storage medium and computer program product for processing GPU cluster failure. BACKGROUND

[0002] With the development of technologies such as artificial intelligence, graphics processing units (GPUs) are widely used in computationally intensive scenarios such as large model training due to their strong parallel computing capabilities. To meet the growing demand for computing resources, multiple GPUs or computing nodes carrying GPUs are usually interconnected through high-speed communication networks to form a GPU cluster, and large-scale computing tasks are executed cooperatively through the GPU cluster.

[0003] Due to the complex structure of the GPU cluster, various types of failures that occur during the execution of the GPU cluster are complex, making it difficult to quickly locate the cause of the failure and quickly recover. For example, due to the complex association between different components in the GPU cluster, the same failure cause can cause different failure phenomena, and similar failure phenomena can be caused by different failure causes. Therefore, failure diagnosis and processing of the GPU cluster is still an important problem that needs to be solved in the process of running and maintaining the GPU cluster. SUMMARY

[0004] In the face of the above problem that the GPU cluster failure is difficult to diagnose, the present disclosure forms a current failure fingerprint by desensitizing the related information of the GPU cluster failure, and matches a historical failure fingerprint based on the current failure fingerprint, to assist the diagnosis and recovery of the current GPU cluster failure with historical experience, thereby reducing the risk of client identification information leakage while improving the efficiency of GPU cluster failure diagnosis and processing.

[0005] To this end, the present disclosure provides a method, apparatus, computer device, computer readable storage medium and computer program product for processing GPU cluster failure.

[0006] According to an aspect of the present disclosure, a method for handling GPU cluster failure performed by a client is provided, which comprises: obtaining one or more failure features associated with a failure event occurred in a GPU cluster associated with the client; performing a de-identification operation on the one or more failure features to suppress identification information capable of identifying the client in the one or more failure features, thereby obtaining one or more de-identified failure features; mapping the one or more de-identified failure features to a corresponding dimension in a set of GPU cluster failure dimensions to generate a current failure fingerprint; sending the current failure fingerprint to a server side; and performing a recovery action for the failure event based on a recommended diagnosis result corresponding to the current failure fingerprint received from the server side.

[0007] According to an aspect of the present disclosure, obtaining one or more failure features associated with a failure event occurred in the GPU cluster comprises: obtaining original diagnosis evidence through diagnosis data interfaces respectively corresponding to at least two running levels of the GPU cluster; and obtaining one or more failure features associated with the failure event in the original diagnosis evidence based on the original diagnosis evidence.

[0008] According to an aspect of the present disclosure, obtaining one or more failure features associated with the failure event in the diagnosis evidence based on the original diagnosis evidence comprises: parsing the original diagnosis evidence to determine a data representation type corresponding to the original diagnosis evidence; and performing a standardization processing on the original diagnosis evidence according to the determined data representation type to convert the original diagnosis evidence into the one or more failure features.

[0009] According to an aspect of the present disclosure, performing a standardization processing on the original diagnosis evidence according to the determined data representation type comprises: determining a target standardization processing rule from a plurality of standardization processing rules respectively corresponding to different data representation types according to the determined data representation type; and converting the original diagnosis evidence into a standardized feature value according to the target standardization processing rule to obtain the one or more failure features.

[0010] According to an aspect of the present disclosure, the at least two running levels comprise at least two of a device level, a communication level, a system and container level, and a training and recovery level of the GPU cluster.

[0011] According to an aspect of the present disclosure, the desensitization operation on the one or more fault features comprises: determining at least one of a first type of feature, a second type of feature and a third type of feature from the one or more fault features, wherein the first type of feature contains identification information of the client and does not participate in diagnosis of the fault event, the second type of feature contains identification information of the client and participates in diagnosis of the fault event, and the third type of feature does not contain identification information of the client; in response to determining that the first type of feature exists, deleting the first type of feature; in response to determining that the second type of feature exists, converting the second type of feature into a generalized feature that retains its diagnostic semantics and does not contain the identification information; and in response to determining that the third type of feature exists, retaining the third type of feature.

[0012] According to an aspect of the present disclosure, the identification information comprises one or more of computing resource identification information, communication entity identification information, task subject identification information and running environment information of the GPU cluster.

[0013] According to an aspect of the present disclosure, mapping the one or more desensitized fault features to respective dimensions in a set of GPU cluster fault dimensions to generate a current fault fingerprint comprises: determining diagnostic semantics respectively represented by the one or more desensitized fault features; and mapping the one or more desensitized fault features to corresponding dimensions according to a correspondence between the diagnostic semantics and each dimension in the set of GPU cluster fault dimensions.

[0014] According to an aspect of the present disclosure, the set of GPU cluster fault dimensions comprises at least two of an environment dimension, a symptom dimension, an indicator and relationship dimension, a recovery dimension and a privacy and trustworthiness dimension.

[0015] According to an aspect of the present disclosure, performing a recovery action for the fault event based on a recommended diagnosis result corresponding to the current fault fingerprint received from the server side comprises: determining a target recovery object in the GPU cluster and a target recovery action for the target recovery object according to the recommended diagnosis result; and performing the target recovery action corresponding to the fault event on the target recovery object.

[0016] According to an aspect of the present disclosure, the recommended diagnosis result comprises fault matching information matching the current fault fingerprint, wherein the fault matching information indicates one or more similar fault fingerprints determined from a set of fault fingerprints stored in the server side.

[0017] According to an aspect of the present disclosure, the recommended diagnosis result comprises fault inference information and verification information, wherein the fault inference information indicates a fault inference cause, and the verification information indicates evidence to be obtained for verifying the fault inference information.

[0018] According to one aspect of this disclosure, the recommended diagnostic results include recovery decision information, wherein the recovery decision information includes recommended recovery actions, and wherein the recovery decision information further includes at least one of the historical execution success rate of the recommended recovery actions and other historical recovery actions different from the recommended recovery actions.

[0019] According to one aspect of this disclosure, the method further includes: after performing the recovery action, sending the actual fault handling result of the fault event to the server, wherein the actual fault handling result includes one or more of the following: actual fault cause, the actual fault cause indicating the fault cause of the fault event as verified; actual recovery action, the actual recovery action being a recovery action performed based on the recommended diagnostic result; execution result, the execution result indicating whether the actual recovery action was successfully executed; and verification result, the verification result indicating whether the fault event was eliminated after performing the actual recovery action.

[0020] According to another aspect of this disclosure, a method for handling graphics processing unit (GPU) cluster failures, executed by a server, is provided. The method includes: receiving a current failure fingerprint from a client, wherein the current failure fingerprint includes one or more de-identified failure features mapped to a corresponding dimension in a GPU cluster failure dimension set, and indicating a failure event occurring in the GPU cluster associated with the client; determining one or more similar failure fingerprints from the failure fingerprint set based on the degree of matching between the current failure fingerprint and historical failure fingerprints in a server-maintained failure fingerprint set, wherein the failure fingerprint set includes one or more historical failure fingerprints from other clients; obtaining a recommended diagnostic result corresponding to the current failure fingerprint based on the one or more similar failure fingerprints; and sending the recommended diagnostic result to the client.

[0021] According to another aspect of this disclosure, determining one or more similar fault fingerprints from the fault fingerprint set based on the matching degree between the current fault fingerprint and each historical fault fingerprint in the fault fingerprint set maintained by the server includes: for each historical fault fingerprint, determining the dimensional similarity between the current fault fingerprint and the historical fault fingerprint in each dimension of the GPU cluster fault dimension set; determining the composite similarity between the current fault fingerprint and the historical fault fingerprint based on the dimensional similarity between the current fault fingerprint and the historical fault fingerprint in each dimension of the GPU cluster fault dimension set; and determining the one or more similar fault fingerprints from the fault fingerprint set based on the multiple composite similarities between the current fault fingerprint and each historical fault fingerprint.

[0022] According to another aspect of this disclosure, the method further includes: determining a recommended confidence level for the one or more similar fault fingerprints; and ranking the one or more similar fault fingerprints based on the recommended confidence level, wherein a recommended diagnostic result corresponding to the current fault fingerprint is also based on the ranking.

[0023] According to another aspect of this disclosure, the recommended confidence level is determined based on at least one of the composite similarity and fingerprint confidence level corresponding to the similar fault fingerprint, wherein the fingerprint confidence level indicates the degree of confidence of the fault features of the similar fault fingerprint, and the recommended confidence level indicates the priority of the similar fault fingerprint among the one or more similar fault fingerprints.

[0024] According to another aspect of this disclosure, obtaining a recommended diagnostic result corresponding to the current fault fingerprint based on the one or more similar fault fingerprints includes: for each of the one or more similar fault fingerprints, determining one or more candidate diagnostic items indicated by the similar fault fingerprint based on the one or more desensitized fault features mapped by the similar fault fingerprint on the GPU cluster fault dimension set, wherein the one or more candidate diagnostic items include at least one of candidate fault causes, candidate verification evidence, and candidate recovery actions; aggregating the candidate diagnostic items indicated by the one or more similar fault fingerprints respectively to determine a recommended diagnostic item for the fault event, thereby obtaining the recommended diagnostic result.

[0025] According to another aspect of this disclosure, the recommended diagnostic result includes fault matching information, wherein the fault matching information indicates one or more similar fault fingerprints that match the current fault fingerprint and a composite similarity corresponding to each similar fault fingerprint.

[0026] According to another aspect of this disclosure, the recommended diagnostic result includes fault inference information and verification information, wherein the fault inference information indicates the inferred cause of the fault event, and the verification information indicates evidence to be obtained to verify the fault inference information.

[0027] According to another aspect of this disclosure, the recommended diagnostic results include recovery decision information, which includes at least one of recommended recovery actions, the historical success rate of the recommended recovery actions, and other historical recovery actions different from the recommended recovery actions.

[0028] According to another aspect of this disclosure, the method further includes: receiving an actual fault handling result corresponding to the fault event from the client; updating the current fault fingerprint based on the actual fault handling result; and storing the updated current fault fingerprint in the fault fingerprint set; wherein the actual fault handling result includes one or more of the following: actual fault cause, the actual fault cause indicating a verified fault cause of the fault event; actual recovery action, the actual recovery action being a recovery action actually performed by the client for the fault event; execution result, the execution result indicating whether the actual recovery action was successfully executed; and verification result, the verification result indicating whether the fault event was eliminated after the actual recovery action was executed.

[0029] According to another aspect of this disclosure, an apparatus is provided for a method of handling graphics processing unit (GPU) cluster failures. The apparatus includes: a failure feature acquisition module configured to acquire one or more failure features associated with a failure event occurring in the GPU cluster; a desensitization module configured to perform a desensitization operation on the one or more failure features to suppress identification information in the one or more failure features that could identify the client, thereby obtaining one or more desensitized failure features; a fingerprint generation module configured to map the one or more desensitized failure features to corresponding dimensions in a GPU cluster failure dimension set to generate a current failure fingerprint; a communication module configured to send the current failure fingerprint to a server; and a recovery module configured to perform a recovery action for the failure event based on a recommended diagnostic result received from the server corresponding to the current failure fingerprint.

[0030] According to another aspect of this disclosure, an apparatus is provided for a method of handling GPU cluster failures, the apparatus comprising: a receiving module configured to receive a current failure fingerprint from a client, wherein the current failure fingerprint includes one or more desensitized failure features mapped to a corresponding dimension in a GPU cluster failure dimension set, and indicating a failure event associated with the client; a matching module configured to determine one or more similar failure fingerprints from the failure fingerprint set based on the degree of matching between the current failure fingerprint and each historical failure fingerprint in the failure fingerprint set maintained by the server, wherein the failure fingerprint set includes one or more historical failure fingerprints from other clients; a diagnostic result obtaining module configured to obtain a recommended diagnostic result corresponding to the current failure fingerprint based on the one or more similar failure fingerprints; and a sending module configured to send the recommended diagnostic result to the client.

[0031] According to another aspect of this disclosure, a computer device is provided, the computer device comprising: at least one processor; and a memory having a computer program stored thereon, wherein, when executed by the at least one processor, the computer program causes the at least one processor to perform the aforementioned method for handling graphics processor (GPU) cluster failures executed by a client.

[0032] According to another aspect of this disclosure, a computer device is provided, the computer device comprising: at least one processor; and a memory having a computer program stored thereon, wherein, when executed by the at least one processor, the computer program causes the at least one processor to perform the aforementioned method for handling graphics processing unit (GPU) cluster failures executed by a server.

[0033] According to another aspect of this disclosure, a computer-readable storage medium is provided that stores a computer program, which, when executed by a processor, causes the processor to perform the aforementioned method for handling graphics processor (GPU) cluster failures executed by a client.

[0034] According to another aspect of this disclosure, a computer-readable storage medium is provided, wherein a computer program is stored on the computer-readable storage medium, and when executed by a processor, the computer program causes the processor to perform the aforementioned method for handling graphics processing unit (GPU) cluster failures executed by a server.

[0035] According to another aspect of this disclosure, a computer program product is provided, wherein the computer program product includes a computer program that, when executed by a processor, causes the processor to perform the aforementioned method for handling graphics processor (GPU) cluster failures executed by a client.

[0036] According to another aspect of this disclosure, a computer program product is provided, wherein the computer program product includes a computer program that, when executed by a processor, causes the processor to perform a method for handling graphics processing unit (GPU) cluster failures executed by a server.

[0037] By desensitizing the fault features associated with GPU cluster failure events on the client side and sending the desensitized fault features to the server side to match the historical fault fingerprints of other clients, the risk of client identification information leakage can be reduced while retaining the information required for fault diagnosis. Furthermore, the historical fault handling experience of different clients can be used to assist in the diagnosis and handling of current GPU cluster failure events, thereby realizing the reuse of historical fault experience and improving the processing efficiency of GPU cluster fault diagnosis. Attached Figure Description

[0038] The accompanying drawings exemplify embodiments and form part of the specification, serving together with the textual description to explain exemplary implementations of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of this disclosure. Throughout the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0039] Figure 1 A schematic flowchart illustrating a method for handling graphics processor (GPU) cluster failures executed by a client according to an embodiment of the present disclosure is shown.

[0040] Figure 2 A schematic flowchart of a method for handling GPU cluster failures, executed by a server according to an embodiment of the present disclosure, is shown.

[0041] Figures 3A-3B An embodiment of this disclosure is shown.

[0042] Figures 4A-4B Another embodiment of this disclosure is shown.

[0043] Figure 5 A lifecycle diagram of a current fault fingerprint is shown according to an embodiment of the present disclosure.

[0044] Figure 6 A schematic diagram of an apparatus for a method of handling graphics processor (GPU) cluster failures according to an embodiment of the present disclosure is shown.

[0045] Figure 7 A schematic diagram of another apparatus for a method of handling graphics processor (GPU) cluster failures according to an embodiment of the present disclosure is shown.

[0046] Figure 8 A schematic diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0047] Figure 9 A schematic diagram of a computer-readable storage medium according to an embodiment of the present disclosure is shown.

[0048] Figure 10 A schematic diagram of a computer device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0049] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0050] In this disclosure, unless otherwise stated, the use of terms such as "first," "second," etc., to describe various elements is not intended to limit the positional, temporal, or importance relationships of these elements; such terms are merely used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of that element, while in other cases, based on the context, they may refer to different instances.

[0051] The terminology used in the description of the various examples described in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context explicitly indicates otherwise, an element may be one or more unless the number of elements is specifically limited. As used herein, the term "multiple" means two or more, and the term "based on" should be interpreted as "at least partially based on". Furthermore, the terms "and / or" and "at least one of..." cover any one of the listed items and all possible combinations thereof.

[0052] In related technologies, different clients have relatively abundant cases of GPU cluster failure handling in their local environments. However, the reference value of these original cases needs to be considered. For example, in a scenario where communication timeouts occur in a multi-node GPU training task, the communication logs of the GPU cluster at client A may record communication timeouts, while network monitoring information shows a decrease in link bandwidth and an increase in lossless network packet loss; while the GPU cluster at client B may also show communication timeouts in its training logs, but its device logs further record GPU device abnormalities. Although both failure events may ultimately result in the training task stalling or exiting, in the former failure event, the communication timeout may be caused by network problems, and applicable handling measures may include replacing the abnormal link; in the latter failure event, the communication timeout may be caused by GPU device abnormalities, and handling measures may include restarting the node or migrating the training task. Therefore, even if different GPU clusters have similar failure behaviors, their corresponding failure causes and applicable handling measures may not be the same.

[0053] Therefore, even if different clients have accumulated a large number of historical failure cases, it is difficult to directly determine the degree of correlation between the historical failure cases of one client and the current failure events of another client, or whether they have reference value for the current failure events, thus limiting the effective reuse of historical failure cases in GPU cluster fault diagnosis.

[0054] Furthermore, fault information from different clients may contain identifying information that can pinpoint a specific client. For example, training logs may contain sensitive information such as usernames and specific model names. This type of information is usually related to the infrastructure or business environment of a specific client. This makes it difficult to directly share and utilize related fault cases between different clients.

[0055] Therefore, how to reduce the risk of client identification information leakage while enabling historical failure cases from different clients to form diagnostic criteria suitable for comparison and reuse, and from which to determine diagnostic suggestions that are of reference value for current failure events, is a technical problem that needs to be solved in the process of GPU cluster failure diagnosis.

[0056] To address the aforementioned issues, this disclosure provides a method for handling GPU cluster failures. In this method, GPU cluster failure events on clients can be anonymized, and the anonymized failure features can be converted into failure fingerprints suitable for comparison and reuse across different clients. Based on this, recommended diagnostic results related to the failure event can be determined according to the degree of matching between the current failure fingerprint and historical failure fingerprints. Thus, on the one hand, the risk of client identification information leakage can be reduced while retaining the diagnostic information needed for comparison between different historical failure cases; on the other hand, historical failure cases from different clients can be compared based on a unified failure fingerprint format, improving the reusability of historical failure cases across different clients, thereby assisting in the diagnosis of current GPU cluster failure events.

[0057] Figure 1 A schematic flowchart of a method 100 for handling graphics processor (GPU) cluster failures, executed by a client according to an embodiment of the present disclosure, is shown.

[0058] At S101, obtain one or more fault characteristics associated with a fault event occurring in the GPU cluster associated with the client.

[0059] At S102, a desensitization operation is performed on one or more fault features to suppress identification information in one or more fault features that can identify the client, thereby obtaining one or more desensitized fault features.

[0060] At S103, one or more desensitized fault features are mapped to the corresponding dimensions in the GPU cluster fault dimension set to generate the current fault fingerprint.

[0061] At S104, send the current fault fingerprint to the server.

[0062] At S105, based on the recommended diagnostic results received from the server corresponding to the current fault fingerprint, a recovery action is performed for the fault event.

[0063] By implementing the aforementioned method, the risk of leakage of sensitive client-side identification information can be reduced, and recommended diagnostic results can be obtained based on server-side knowledge, thereby improving the processing efficiency of GPU cluster fault event diagnosis. It is understood that the de-identification operation can suppress or reduce only the identification information that can be directed to the client, without deleting semantic information that can be used for diagnosis. Through this distinction, the risk of leakage of sensitive client-side identification information can be reduced, and recommended results can be effectively obtained. For example, the recommended diagnostic result received from the server corresponding to the current fault fingerprint can be based on historical cases on the server, such as the similarity between the server and other historical faults, which will be further explained below in conjunction with other exemplary embodiments.

[0064] The following describes in detail one or more exemplary embodiments or variations of the method 100 according to the present disclosure.

[0065] Return to reference Figure 1 As mentioned earlier, at S101, one or more fault characteristics associated with a fault event occurring in the GPU cluster associated with the client can be obtained. In such an example, a fault event can be an anomaly detected during the operation of the GPU cluster that requires fault diagnosis. For example, if the aggregation communication between GPUs fails to complete normally during the execution of a computing task by the GPU cluster, this communication anomaly can be identified as a fault event. As another example, when a GPU device cannot be accessed normally through the bus, a corresponding error code may appear in the kernel log, recording that the GPU device has lost connection from the bus; this bus disconnection anomaly can be identified as a fault event. As yet another example, when the communication connection between different computing nodes in the GPU cluster is abnormally interrupted, connection anomaly information such as "Connection closed by remote peer" or "Connection reset by peer" may appear in the relevant communication log; this connection interruption anomaly can be identified as a fault event. A fault event can be manifested by one or more abnormal parameters or phenomena such as error codes, log messages, and runtime metrics.

[0066] For example, fault features can be feature values ​​that characterize the abnormal behavior of a fault event in a predetermined standardized form. Compared to the fault event itself, fault features are used to describe the specific anomaly exhibited by the fault event in a form suitable for subsequent processing. Therefore, the same abnormal behavior can be characterized using the same or corresponding fault features even if it is recorded in different forms at different operating levels of the GPU cluster. For example, if communication between GPUs fails to complete within the expected time, the communication log may record a "timeout" message indicating a communication timeout, and the user-operated software may also generate a corresponding communication timeout alarm. Although the original representations used by the two are different, they both reflect that a timeout anomaly occurred in the communication process, thus obtaining fault features characterizing the communication timeout.

[0067] For example, a fault event can correspond to one or more fault features. These fault features can each characterize different anomalies exhibited by the fault event. For instance, after an anomaly occurs in the communication state between GPUs, communication performance may first degrade, followed by communication timeouts, and further causing the executing task to stall. For this fault event, fault features characterizing the aforementioned abnormal behaviors can be formed separately, thereby describing the fault event from different perspectives.

[0068] It should be understood that the standardized form of fault characteristics can give consistent or comparable feature values ​​to information that reflects the same abnormal behavior but uses different original representations. Exemplary embodiments regarding how clients can obtain such standardized fault characteristics will be further described later.

[0069] In some embodiments, obtaining one or more fault features associated with a fault event occurring in the GPU cluster may include: obtaining raw diagnostic evidence through diagnostic data interfaces corresponding to at least two operating levels of the GPU cluster; and obtaining one or more fault features associated with the fault event in the raw diagnostic evidence based on the raw diagnostic evidence.

[0070] When a GPU cluster experiences a failure event, the corresponding anomaly can typically manifest differently at different runtime levels. For example, if communication between GPUs fails, training tasks executed at higher levels may further stall. Therefore, by obtaining raw diagnostic evidence from different runtime levels, it is possible to determine, for example, whether the anomaly is a direct result of the failure or a consequence of other anomalies. This, combined with the manifestation of the failure event at different runtime levels, allows us to obtain the failure characteristics associated with the failure event.

[0071] The following describes specific details of the operational hierarchy according to some exemplary embodiments of this disclosure.

[0072] In some embodiments, at least two operational tiers may include at least two of the following: a device tier, a communication tier, a system and container tier, and a training and recovery tier for the GPU cluster. Exemplarily, raw diagnostic evidence obtained from at least two operational tiers may be complementary.

[0073] For example, different operational levels can have corresponding diagnostic data interfaces. A diagnostic data interface can be an interface used by a client to obtain raw diagnostic evidence for the corresponding operational level. The raw diagnostic evidence obtained from the diagnostic data interfaces corresponding to different operational levels can reflect different aspects of the fault event.

[0074] For example, raw diagnostic evidence can be data obtained directly through a diagnostic data interface that has not yet been converted into fault characteristics according to a predetermined standardized form. For instance, raw diagnostic evidence could be error codes recorded in logs or text messages describing abnormal states. For diagnostic data interfaces that reflect the state of a GPU cluster using operational metrics, raw diagnostic evidence can also be measured values ​​of the corresponding metrics. For example, raw diagnostic evidence obtained at the communication layer could be communication bandwidth or communication latency measured over a certain period, while raw diagnostic evidence obtained at the device layer could be GPU temperature or device error counts. These values ​​can change as the GPU cluster operates and can reflect abnormal states related to fault events through their current values ​​or changes.

[0075] For example, the client can obtain fault features associated with the current fault event based on the original diagnostic evidence, without having to directly use all the raw data output by the diagnostic data interface as the component of the subsequent fault fingerprint. Thus, on the one hand, fault features for diagnosis can be obtained by combining abnormal behaviors related to the fault event at different operational levels; on the other hand, information related to fault diagnosis can be retained from the original diagnostic evidence, providing a foundation for subsequent standardization, desensitization, and dimensional mapping of fault features.

[0076] For example, the device hierarchy can be a hierarchy reflecting the GPU's own operating state, and the diagnostic data interface corresponding to the device hierarchy can include, for example, a device management interface for reading the GPU device status, or a log interface for obtaining kernel records related to the GPU device. In some embodiments, the device management interface can obtain values ​​such as GPU temperature, device utilization, or memory usage that change with GPU operation; in the event of a device malfunction, it can also obtain the device error status or error count. In some embodiments, the log interface can obtain error codes or corresponding log messages generated by GPU device malfunctions.

[0077] For example, the communication layer can reflect the hierarchy of data transmission status between GPUs, and the diagnostic data interface corresponding to the communication layer can include, for example, a status query interface provided by the communication library and a connection status interface provided by the network protocol stack. In some embodiments, the execution status of a collection of communication operations can be obtained through the status query interface of the communication library. In some embodiments, the current status or retransmission status of the communication connection can be obtained through the connection status interface of the network protocol stack. The data obtained from these interfaces can serve as raw diagnostic evidence for the communication layer.

[0078] For example, the system and container hierarchy can reflect the hierarchy of the system and container environments in which the GPU cluster runs, and the diagnostic data interface corresponding to the system and container hierarchy can, for example, include a container status interface or a container device access permission query interface provided by the container runtime. In some embodiments, the container runtime interface can be used to determine whether the allocated GPUs are correctly mapped to the containers running the tasks. In some embodiments, the container device access permission query interface can be used to determine the original diagnostic evidence of whether the container has GPU access permissions.

[0079] For example, the training and recovery hierarchy can reflect the hierarchical status of the tasks executed by the GPU cluster before and after a failure event, and the diagnostic data interface corresponding to the training and recovery hierarchy can, for example, include a runtime record interface generated by the training task. In some embodiments, the runtime record interface can be used to obtain data such as whether the training iteration continues to execute and whether the time required for one iteration has changed significantly.

[0080] It is understood that the diagnostic data interfaces for each operational level described here are merely examples. In actual GPU cluster operation, each operational level may have one or more of the above diagnostic data interfaces, or fewer or more diagnostic data interfaces. It is also understood that clients do not require raw diagnostic evidence from all of the above operational levels, and the operational level used can be determined based on the GPU cluster deployment method and the data available at the time of the failure event.

[0081] Table 1 provides examples of operational levels and corresponding original diagnostic evidence according to exemplary embodiments of this disclosure.

[0082] Table 1: Example Original Diagnostic Evidence

[0083] As previously mentioned, the fault features acquired by the client may include standardized feature values. In some embodiments, raw diagnostic evidence from different diagnostic data interfaces may retain the original data representation format used by the respective interfaces. In such an embodiment, obtaining one or more fault features associated with a fault event in the diagnostic evidence based on the raw diagnostic evidence may include: parsing the raw diagnostic evidence to determine the data representation type corresponding to the raw diagnostic evidence; and standardizing the raw diagnostic evidence according to the determined data representation type to convert the raw diagnostic evidence into one or more fault features. According to such an embodiment, the data representation type of the raw diagnostic evidence can be identified before conversion, and standardization processing adapted to the corresponding representation format can be applied accordingly.

[0084] In such examples, the data representation type of the original diagnostic evidence can be determined based on the parsing results. For instance, for original diagnostic evidence obtained through a log interface, the error code or text message used to represent the anomaly can be identified by parsing the log record format, and the corresponding error code or text format can be used to represent the original diagnostic evidence. Similarly, for original diagnostic evidence obtained through a status query interface, the predefined status value used to represent the GPU cluster's operating status can be determined based on the interface's return value definition. Furthermore, for original diagnostic evidence obtained through an interface related to operating metrics, the corresponding operating status can be determined to be represented by a measured value. It is understandable that determining the data representation type can help determine how to perform subsequent transformations on the original diagnostic evidence. For example, the parsed log message "timeout" can be determined to represent a communication anomaly in text format.

[0085] After determining the data representation type, in such an example, the original diagnostic evidence can be standardized according to the determined data representation type through standardization. It is understood that throughout this disclosure, standardization can be a process of converting the original diagnostic evidence into a predetermined standardized form in a manner appropriate to the data representation type of the original diagnostic evidence, so that original diagnostic evidence using different original representation forms but having the same or comparable diagnostic meaning can form consistent or comparable fault characteristics.

[0086] It is understood that the standardization process according to one or more embodiments of this disclosure involves only the standardization of data representation or data format to eliminate representational differences caused by different diagnostic data interfaces or different data representation types. Therefore, according to one or more embodiments of this disclosure, the data after standardization can still retain the original diagnostic semantics.

[0087] In some embodiments, standardizing the original diagnostic evidence according to the determined data representation type may include: determining a target standardization rule from a plurality of standardization rules corresponding to different data representation types; and converting the original diagnostic evidence into standardized feature values ​​according to the target standardization rule to obtain one or more fault features.

[0088] In some embodiments, associations between different data representation types and corresponding standardization rules can be pre-established. Different standardization rules can specify the transformation method used for original diagnostic evidence of a corresponding data representation type. After determining the data representation type of the original diagnostic evidence through parsing, the standardization rule corresponding to that data representation type can be selected as the target standardization rule based on the pre-defined associations between that data representation type and multiple standardization rules. This process will be described in the embodiments below.

[0089] For example, when the original diagnostic evidence is determined to represent an anomaly using error codes or status codes, the standardized processing rules used to map the error codes or status codes can be identified as the target standardized processing rules. These standardized processing rules can pre-record the mapping relationships between different error codes or status codes and standardized feature values. For instance, in the event of a GPU device reset, the kernel interface can record the device reset event using the corresponding error code, while the device management interface can return a status code indicating that the GPU device has been reset. Exemplarily, the client can map these different codes to standardized feature values ​​representing device reset, such as the standardized feature values ​​"RESET" or "device reset," which characterize fault features of a device reset anomaly, according to the target standardized processing rules.

[0090] For example, when the original diagnostic evidence is determined to be abnormal using text messages, the standardization processing rules used for matching and standardizing these text messages can be defined as the target standardization processing rules. These rules can specify the text content representing a predetermined abnormality and the corresponding standardized feature values. For instance, in the event of a communication timeout, different communication components can record text messages such as "operation timeout" and "communication timeout" in their logs. Exemplarily, the client can convert different text messages representing communication timeouts into the same standardized feature value, such as the standardized feature value "TIMEOUT" or "communication timeout," representing the fault characteristics of a communication timeout anomaly, according to the target standardization processing rules.

[0091] For example, when the original diagnostic evidence is determined to represent the operating status of a GPU cluster using measured values, the standardization rule used to determine the degree of change relative to the normal operating state based on the measured values ​​can be defined as the target standardization rule. This standardization rule can specify the baseline value for the normal operating state and the conversion method for obtaining standardized feature values ​​based on the current measured value and the baseline value. For example, the original diagnostic evidence may include the currently measured high-speed network bandwidth. Assuming the bandwidth of the corresponding communication link is 200 Gbps under normal operating conditions, and the bandwidth measured at the time of the fault event is 80 Gbps, then according to the target standardization rule, the bandwidth attenuation ratio can be determined to be 60% based on the degree of decrease in the current bandwidth relative to the normal bandwidth, and this bandwidth attenuation ratio can be used as the standardized feature value to obtain a fault feature such as "bandwidth attenuation ratio exceeds a threshold".

[0092] As can be seen from the above embodiments, the standardized feature values ​​here are fault characteristics that represent fault events in a predetermined standardized form, obtained after converting the original diagnostic evidence according to the corresponding standardization processing rules. For original diagnostic evidence using different data representation types, the standardized feature values ​​can adopt standardized state representation, text representation, or numerical representation, etc. Through the above standardization processing, original diagnostic evidence with different original representation forms but reflecting the same abnormal manifestation can form consistent fault characteristics, and original diagnostic evidence reflecting abnormal manifestation using different measurement scales can form fault characteristics with comparable representation.

[0093] It is understood that the standardized feature values ​​mentioned in the above embodiments are merely examples. Standardized feature values ​​for different data representation types can be flexibly set according to actual conditions, or different data representation types can be mapped using standardized feature values ​​in a unified format. For example, fault features include, but are not limited to, the following: device category, device series, driver major version family, communication library family, network type, containerization status, scheduling environment category, and training framework category; device hangup, device loss, device reset, uncorrectable memory errors, collective communication timeout, training task stuck, abnormal training task exit, high-speed network bandwidth decrease, lossless network packet loss increase, GPU invisible within the container, and runtime initialization failure; temperature range, bandwidth attenuation ratio, ECC error increment, device utilization change, memory usage range, packet loss trend, and latency trend; the relative order of error code occurrence, abnormal indicators, communication timeout, training task failure, and recovery actions, as well as topology roles such as single device anomaly, single node multi-device anomaly, and multi-node anomaly on the same switch; recovery action type, action execution result, recovery verification result, recurrence status, and case confirmation status.

[0094] Return to reference Figure 1As described above, at S102, a desensitization operation can be performed on one or more fault features to suppress client-identifying information in the fault features, thereby obtaining one or more desensitized fault features. It is understood that fault features generated during a fault may reflect sensitive client-identifying information, which may be undesirable in some cases. For example, a fault feature formed by communication status may simultaneously contain the specific communication address where the communication anomaly occurred, and a fault feature formed by device status may also contain the specific computing resource identifier where the anomaly occurred. If this data is directly included in the current fault fingerprint sent to the server, the server may determine that the fault feature originates from a specific client. Therefore, a further desensitization operation can be performed on the fault features before generating the current fault fingerprint. It is understood that the one or more fault features to be desensitized here can be the original one or more fault features, or fault features obtained by standardizing the original diagnostic evidence as described above, and will not be elaborated further here.

[0095] It is understood that throughout this disclosure, identification information can be information within a fault feature that enables the fault feature to establish a correspondence with a specific client. This can include direct identification information that directly identifies the client itself, or indirect identification information that identifies a specific resource or running object of the client, enabling the receiving party to identify the source of the fault feature. In some embodiments, identification information can also be embodied by multiple fault features. That is, a single fault feature may not be sufficient to identify a specific client, but when combined with other fault features, if the resulting feature combination has high specificity, it may allow the receiving party to infer the source of the fault feature. Therefore, when determining whether identification information exists in a fault feature, the identifiability of a single fault feature or a combination of multiple fault features can also be considered. For example, as will be further explained below, in addition to the fault feature content itself, the occurrence time and frequency of the fault feature may also provide identification information that can identify the client.

[0096] In some embodiments, the identification information may include one or more of the following: GPU cluster computing resource identification information, communication entity identification information, task subject identification information, and runtime environment information.

[0097] For example, computing resource identification information can be used to uniquely or specifically identify computing resources in a GPU cluster. For instance, a fault characteristic might include the name of the cluster where the anomaly occurred. Such a characteristic might expose the resource organization of the GPU cluster within the client and could therefore be considered as computing resource identification information.

[0098] For example, communication entity identification information can be used to identify the specific entity participating in GPU cluster communication. For instance, when a connection failure occurs between two compute nodes, the corresponding fault characteristics might include the specific network addresses of the source and destination nodes. In other examples, the fault characteristics might retain a unique identifier for a specific network device or communication port. In such examples, such characteristics can be used to identify the client's network node or internal communication entity, and therefore can be considered part of the communication entity identification information.

[0099] For example, task subject identification information can be used to associate a fault event with a specific task executed in the GPU cluster or a subject related to that task. For instance, when a training task exits abnormally, the corresponding fault characteristics may include the job identifier of that training task. Such characteristics may establish a correspondence between the fault event and the specific task executed by the client, thereby exposing the client's task information, and thus can be regarded as belonging to the task subject identification information.

[0100] For example, runtime environment information can be information that characterizes the deployment conditions on which the GPU cluster depends when performing tasks. For instance, the specific container namespace name or internal resource address used within the client may reflect the client-specific deployment environment. Such characteristics may allow a correspondence between failure events and client-specific deployment conditions, and therefore can be considered runtime environment information.

[0101] In some embodiments, performing an anonymization operation on one or more fault features may include: determining at least one of a first type of feature, a second type of feature, and a third type of feature from the one or more fault features. In such an example, the first type of feature may contain client identification information and not participate in the diagnosis of fault events; the second type of feature may contain client identification information and participate in the diagnosis of fault events; and the third type of feature may not contain client identification information. In such an example, the method may include deleting the first type of feature in response to determining the existence of the first type of feature; converting the second type of feature into a generalized feature that retains its diagnostic semantics and does not contain identification information in response to determining the existence of the second type of feature; and retaining the third type of feature in response to determining the existence of the third type of feature.

[0102] Based on this example, diagnostic semantics related to fault diagnosis, fault relationship analysis, and the reuse of recovery experience can be preserved while suppressing client-identifiable information, thus balancing the privacy protection of fault fingerprints with the integrity of diagnostic information. For example, identification information that does not participate in fault diagnosis can be suppressed by deletion. For identification information that still has diagnostic significance, its identifiability can be reduced while preserving diagnostic semantics through generalization methods such as entity replacement, reducing numerical granularity, temporal relativization, or topological generalization. Fault features that do not contain identification information can be retained.

[0103] For example, whether a fault feature participates in the diagnosis of a fault event can be determined based on whether the content represented by the fault feature is used to distinguish fault states, determine fault causes, or judge recovery methods. In other words, if removing a fault feature does not affect the subsequent diagnosis of the fault event, then the fault feature can be considered not to participate in the corresponding diagnosis; if the abnormal state or relationship represented by the fault feature is still needed for subsequent diagnosis, then the fault feature can be considered to participate in the corresponding diagnosis.

[0104] It is understandable that, generally speaking, determining at least one of the first, second, and third categories of features from one or more fault features can be done by the client based on current fault information, or by the client referring to other knowledge. For example, the client can identify the content contained in the fault features based on the categories of identification information described above in conjunction with the embodiments and pre-set de-identification rules, such as extracting specific resource numbers, node names, network addresses, job identifiers, precise measurement values, and absolute time information, and determine that it belongs to the first and / or second categories of features that need to be processed.

[0105] In other embodiments, the identifiability of certain fault features does not lie in the content itself, but requires the combination of other knowledge for identification and determination. For example, a fault feature may not contain direct identifying information such as client name, computing resource identifier, or network address, but if the fault feature only appears in the historical faults of a very small number of clients, it may still have strong specificity or identifiability. Similarly, if a feature combination formed by several fault features corresponds to only a very small number of fault cases, this feature combination may also allow the receiver to narrow down the scope of the fault feature sources. In such cases, the identification of features that need to be deleted or generalized may require the aid of other knowledge, such as fault frequency or fault feature statistics stored in the client or obtained from the server.

[0106] In some embodiments, desensitization rules can be pre-set to identify information in different fault features and determine whether the corresponding fault features are involved in diagnosis. The client can classify the acquired fault features according to the desensitization rules to determine which desensitization method to use for the corresponding fault features.

[0107] For example, the first type of feature can be deleted. The first type of feature contains identification information that can identify the client, but it does not participate in the diagnosis of fault events. Therefore, deleting the first type of feature can suppress the identification information and does not affect the subsequent diagnosis of fault events. For instance, if one of the fault features is only used to record the unique number of the computing node that experienced the anomaly, and this number is not used to subsequently determine the anomaly type or the cause of the fault, then the fault feature containing this unique number can be identified as a first type of feature and deleted, thereby achieving the separation of sensitive entities.

[0108] For example, for the second type of feature, the corresponding diagnostic semantics can be retained while suppressing the identification information therein. In such an example, the second type of feature may contain the client's identification information and simultaneously carry semantic information about the fault event. Therefore, by suppressing the identification information while retaining its diagnostic semantics, the fault feature can still be used to diagnose the fault event after desensitization. It is understood that throughout this disclosure, diagnostic semantics can be semantic information that can describe the actual fault nature of the fault feature and / or can be used for subsequent fault diagnosis. Accordingly, retaining the diagnostic semantics of the second type of feature can mean retaining the meaning of the fault feature that, apart from being used to identify a specific client or an entity within the client, can still be used for subsequent fault diagnosis. In other words, after removing or generalizing the specific identifier in the fault feature, as long as the retained information can still reflect the content related to fault diagnosis, it can be considered that the diagnostic semantics of the fault feature has been retained.

[0109] In some embodiments, the second type of features can be converted into generalized features that retain their diagnostic semantics but do not contain identification information. Generalization can convert features that can be mapped to a specific client into features that cannot identify a specific client but can still reflect the information required for fault diagnosis.

[0110] For example, in the event of an abnormal interruption of communication between computing nodes, the fault characteristic obtained by S101 might indicate that the communication connection between a specific source IP address and a specific destination IP address has been reset. While the source and destination IP addresses can identify the specific communication entities within the client, the anomaly of "communication connection between nodes being reset" still participates in the diagnosis of the fault event. Therefore, the specific source and destination IP addresses can be converted to node roles that do not correspond to specific client communication entities, while retaining the anomaly of the communication connection being reset. After this generalization, a generalized characteristic representing, for example, "communication connection between the source and destination nodes has been reset" can be obtained, no longer including the original IP address.

[0111] In some embodiments, generalizing the second type of feature may also include reducing the granularity of the precise values, making the values ​​range-bound. While some precise values ​​are useful for fault diagnosis, excessively high numerical precision may establish a correspondence with specific client-side device states or operational records. Therefore, precise values ​​can be converted into numerical ranges or variations that retain the degree of anomaly. For example, a fault feature might represent a GPU temperature of "87 degrees Celsius." During diagnosis, the focus may be on the GPU temperature being within a higher temperature range, rather than the precise value of that temperature. Therefore, "87 degrees Celsius" can be generalized to "80 to 90 degrees Celsius," ensuring that the generalized feature still reflects the GPU's temperature state while reducing the identifiability of precise measurements.

[0112] In some embodiments, generalizing the second type of feature may further include converting absolute time information into relative time relationships, thus relativizing time. For example, the order of occurrence of multiple anomalies can be used for fault diagnosis rather than the absolute time itself. Therefore, the specific time of occurrence of each anomaly can be suppressed, while their relative chronological relationships are preserved. For example, when the specific occurrence times of index anomalies, communication timeouts, and training task stagnation are recorded separately in the fault features, the corresponding absolute timestamps can be removed, forming generalized features such as "index anomalies occur before communication timeouts" or "training tasks stagnate shortly after communication timeouts." This preserves the temporal correlation between multiple anomalies while avoiding the association of the client's specific operational records with absolute time information.

[0113] In some embodiments, generalizing the second type of feature may further include converting the real topology identifiers in the GPU cluster into structural roles that describe topology relationships, thereby achieving topology generalization. For example, fault features may include the specific node name, switch name, or port number where the anomaly occurred. While this information reflects the real topology within the client, the topology relationships between multiple anomalous entities themselves may be useful for fault diagnosis. Therefore, the real names that identify specific nodes or network devices can be removed, while retaining the structural relationships between the corresponding entities. For example, when multiple anomalous compute nodes are connected to the same switch, the fault feature described based on the specific node name and switch name can be generalized to "multiple nodes anomalous on the same switch." Similarly, when only a single GPU device exhibits anomalies while other GPU devices within the same node are functioning normally, a generalized feature of "isolated single-device anomaly" can be formed. Or, when multiple anomalous communication processes correspond to the same network interface card (NIC), a generalized feature of "multiple communication processes anomalous on a single NIC" can be formed.

[0114] In some embodiments, generalizing the second type of feature may also include merging the second type of feature with other fault features having similar diagnostic meanings into a single fault feature, thereby merging the features. Some fault features represent specific abnormal states and occur less frequently in historical fault features. Therefore, their specificity may allow for reverse identification of clients or specific operating scenarios, while the information needed for fault diagnosis sometimes primarily lies in the category to which the abnormality belongs. In this case, a fault feature category with similar diagnostic meaning to which the fault feature belongs can be determined, and the original fault feature can be replaced with a fault feature representing that category. For example, different GPU devices may output different specific error codes for the same type of device abnormality, and each specific error code may occur less frequently in historical fault features. In this case, multiple specific error codes with different values ​​but all representing the same type of device abnormality can be merged into a single error code family, and this error code family can be used as a generalization feature. As another example, for multiple fault features representing "abnormal exit of training task" and "termination of training task by scheduling system" respectively, if the information needed in subsequent diagnosis is that the training task failed to continue execution, the above fault features can be merged into a generalized feature representing "abnormal termination of training task". After the above merging, the generalized features can still reflect the anomaly categories required for fault diagnosis.

[0115] In such examples, the client can, for instance, determine whether the current fault feature is specific or identifiable based on historical fault features available to the client itself. In other embodiments, the client can obtain corresponding fault cases or fault statistics results issued by the server or actively request them from the server. Thus, the client can determine the number of times or frequency of occurrence of a single fault feature or combination of fault features in historical fault features, and identify the corresponding features (e.g., identifiable low-frequency / extremely high-frequency fault features) as the corresponding features to be generalized, and perform desensitization processing through fuzzing, merging with other features, etc.

[0116] For example, third-category features can be retained. Third-category features do not contain identifying information that can identify the client; therefore, there is no need to change these third-category features for desensitization. For instance, a fault feature obtained from high-speed network bandwidth measurements could be "bandwidth attenuation rate is 60%". If this fault feature does not contain information that can identify the client, it can be identified as a third-category feature and retained.

[0117] Return to reference Figure 1 As mentioned earlier, at S103, one or more desensitized fault features can be mapped to the corresponding dimension in the GPU cluster fault dimension set to generate the current fault fingerprint.

[0118] For example, the GPU cluster fault dimension can correspond to different categories of objective states or attributes exhibited by the GPU cluster before and after a fault event. For instance, through the diagnostic data interface at the communication layer, raw diagnostic evidence representing communication timeouts and raw diagnostic evidence representing high-speed network bandwidth can be obtained. After processing in S101 and S102, the former can form a de-identified fault feature characterizing "communication timeout," and the latter can form a de-identified fault feature such as "bandwidth attenuation ratio of 60%." Although both originate from the communication layer, "communication timeout" reflects the communication symptoms exhibited by the fault event, while "bandwidth attenuation ratio of 60%" reflects an abnormal change in communication performance indicators, and therefore can be mapped to different dimensions.

[0119] The current fault fingerprint can be a structured representation formed by organizing one or more desensitized fault features associated with the current fault event according to the corresponding dimensions in the GPU cluster fault dimension set. Therefore, by mapping and converging discrete fault features from different clients and with varying manifestations into a unified GPU cluster fault diagnosis dimension, a consistent semantic comparison benchmark can be constructed, effectively eliminating feature heterogeneity between different cluster environments. This facilitates subsequent multi-dimensional comparison and similarity matching of different fault fingerprints on the server side.

[0120] In some embodiments, mapping one or more desensitized fault features to corresponding dimensions in the GPU cluster fault dimension set may include: determining the diagnostic semantics represented by one or more desensitized fault features respectively; and mapping one or more desensitized fault features to corresponding dimensions according to the correspondence between the diagnostic semantics and the dimensions in the GPU cluster fault dimension set.

[0121] Based on this example, the dimensional assignment of fault features within a fault fingerprint can be determined according to the diagnostic semantics expressed by the fault features, enabling the fault features to be organized according to the fault dimensions of the GPU cluster. This reduces the likelihood of similar fault features being assigned to different dimensions in different fault fingerprints, improves the structural consistency of fault fingerprints, and allows different fault fingerprints to be compared according to their corresponding dimensions, thus providing a basis for subsequently determining the degree of matching between historical and current fault fingerprints.

[0122] In some embodiments, a correspondence between different diagnostic semantics and dimensions in the GPU cluster fault dimension set can be pre-established. This correspondence can specify the dimension to which a fault feature with a specific diagnostic meaning should be assigned. After determining the diagnostic semantics represented by the desensitized fault feature, the dimension corresponding to the diagnostic semantics can be determined according to the correspondence, and the desensitized fault feature can be mapped to the determined dimension.

[0123] For example, for the anonymized fault feature "communication timeout," its diagnostic semantics can be determined as the abnormal symptom that the communication operation failed to complete within the expected time, reflecting the abnormal symptoms exhibited by the fault event and mapping it to the symptom dimension. For the anonymized fault feature "bandwidth attenuation ratio of 60%," its diagnostic semantics can be determined as the communication performance index decreasing relative to the normal operating state, reflecting the abnormal change in the operating index and mapping it to the index and relationship dimension.

[0124] In some embodiments, the GPU cluster fault dimension set includes at least two of the following: environment dimension, symptom dimension, indicator and relationship dimension, recovery dimension, and privacy and trust dimension.

[0125] For example, the environment dimension can be used to organize de-identified fault features that characterize the operating conditions of the GPU cluster at the time of the fault event. For instance, the de-identified fault feature "data center GPU" characterizes the hardware category of the GPU to which the fault event occurred, and its diagnostic semantics describe the hardware conditions at the time of the fault event; therefore, this fault feature can be mapped to the environment dimension. As another example, the de-identified fault feature "high-speed lossless network" characterizes the network type used by the GPU cluster for communication, and its diagnostic semantics describe the network conditions at the time of the fault event; therefore, it can also be mapped to the environment dimension.

[0126] In some embodiments, the environment dimension may further include corresponding fields for carrying the aforementioned fault characteristics. For example, “data center GPU” may be mapped to the hardware category field in the environment dimension, and “high-speed lossless network” may be mapped to the network type field in the environment dimension.

[0127] For example, the symptom dimension can be used to organize desensitized fault features that characterize the fault manifestations of a fault event. For instance, the desensitized fault feature "collective communication timeout" characterizes the failure of a collective communication operation to complete within the expected time; its diagnostic semantics describe the communication anomaly exhibited by the fault event, thus "collective communication timeout" can be mapped to the symptom dimension. As another example, if the communication anomaly further prevents training from continuing, the desensitized fault feature "training task stall" characterizes the abnormal running state of the training task; its diagnostic semantics also belong to the symptoms presented by the fault event, and therefore can be mapped to the symptom dimension.

[0128] In some embodiments, the symptom dimension may include a symptom set field to carry fault features of one or more symptom dimensions obtained for the same fault event. For example, for a fault event in which both ensemble communication timeout and training task stall occur simultaneously, both "ensemble communication timeout" and "training task stall" can be mapped to the symptom set field in the symptom dimension.

[0129] For example, metrics and relationship dimensions can be used to organize desensitized fault features that characterize abnormal changes in operational metrics or relationships between multiple fault-related events.

[0130] For example, the de-identified fault feature "bandwidth below 40% of baseline" characterizes the degree of decline in high-speed network bandwidth relative to the normal operating baseline. Its diagnostic semantics describe the abnormal direction and degree of communication performance indicators, so this fault feature can be mapped to the indicator and relationship dimension. In some embodiments, it can be further mapped to the indicator trend field in the indicator and relationship dimension.

[0131] For example, anonymized fault characteristics could represent "increased packet loss → communication timeout → training stagnation". In such an example, the fault characteristic can be mapped to an indicator and relationship dimension based on the event relationship it represents. In some embodiments, it can be further mapped to an event sequence field within that dimension.

[0132] For example, after generalizing the real topology of the GPU cluster in S102, the anonymized fault feature "multiple nodes abnormal on the same switch" can be obtained. This fault feature does not contain specific node names or switch names, but still represents the relative positional relationship of multiple abnormal nodes in the GPU cluster topology. Since its diagnostic semantics lie in describing the topological relationship between entities involved in the fault, it can be mapped to the indicator and relationship dimension. In some embodiments, it can be further mapped to the topology role field in this dimension.

[0133] Therefore, even if there are obvious differences in content between “bandwidth below 40% of baseline”, “packet loss increase → communication timeout → training stagnation” and “multi-node anomaly on the same switch”, as long as their diagnostic semantics involve changes in indicators or relationships between fault-related objects, they can be organized in the indicator and relationship dimension.

[0134] For example, the recovery dimension can be used to organize de-identified fault characteristics that characterize the recovery processes implemented for a fault event and the effects of those processes. For instance, for a fault event for which recovery processing has already been performed, the de-identified fault characteristic "Network configuration repair successful" can be obtained. This fault characteristic not only indicates that network configuration repair was performed, but also that the fault was eliminated after the recovery action was performed. Its diagnostic semantics reflect the handling methods and their effects on the corresponding fault; therefore, this fault characteristic can be mapped to the recovery dimension.

[0135] In some embodiments, the recovery dimension may include a recovery action field, which can map fault features such as "network configuration repair successful" to the field. If recovery processing has not yet been performed for the current fault event, the recovery dimension in the current fault fingerprint may temporarily exclude the corresponding fault features; after obtaining the actual fault processing result, the corresponding content can be supplemented or updated based on the actual fault processing result. For example, the method may include the client (e.g., before generating and sending the current fault fingerprint) performing fault processing a predetermined number of times on the fault, and generating or updating the current fault fingerprint based on the fault processing results.

[0136] For example, the privacy and trustworthiness dimension can be used for de-identified fault features related to the sharing status of fault fingerprints or the trustworthiness of information therein. For instance, after de-identification, if the corresponding fault feature no longer contains identification information that can identify the client and meets the conditions for sharing between different clients, a fault feature representing "cross-client shareability" can be formed. This fault feature can indicate the shareability status of the corresponding fault fingerprint and can therefore be mapped to the privacy and trustworthiness dimension. In some embodiments, it can be further mapped to the privacy level field in that dimension.

[0137] For example, for historical faults for which actual fault handling results have been obtained, if the actual cause of the fault has been confirmed, and the fault event has been further verified to have been eliminated after the actual recovery action is performed, a fault feature representing "confirmed and recovery verified" can be formed. Since this feature is used to represent the degree of confirmation of the information contained in the corresponding fault case, it can be mapped to the privacy and trustworthiness dimension, such as mapping it to the case trustworthiness field.

[0138] Table 2 exemplarily illustrates examples of fields and semantics for different dimensions of an exemplary fault fingerprint according to one or more embodiments of this disclosure.

[0139] Table 2: Example Fault Fingerprints

[0140] In some embodiments, fault fingerprints can be represented using mathematical formulas. For example, a fault fingerprint can be represented as follows: F_q = [V_e, V_s, V_m, V_r, V_p] .in, V_e Indicates the environmental dimension. V_s Indicates symptom dimensions, V_m Indicates the dimensions of indicators and relationships. V_r Indicates the dimension to be recovered. V_p This indicates the dimensions of privacy and trustworthiness.

[0141] Return to reference Figure 1 As mentioned earlier, at S104, the current fault fingerprint can be sent to the server. In some embodiments, after generating the current fault fingerprint, the client can send the current fault fingerprint to the server so that the server can obtain recommended diagnostic results for the fault event based on the current fault fingerprint. The specific processing by the server after receiving the current fault fingerprint will be further described below in conjunction with the method executed by the server.

[0142] Since the fault features in the current fault fingerprint have been anonymized as described above, the client can provide the current fault fingerprint related to fault event diagnosis to the server without sending identification information that can identify the client to the server.

[0143] Return to reference Figure 1As mentioned earlier, at S105, recovery actions for the fault event can be performed based on the recommended diagnostic results received from the server corresponding to the current fault fingerprint. In some embodiments, after receiving the current fault fingerprint, the server can obtain the recommended diagnostic results corresponding to the fault event based on the current fault fingerprint and send the recommended diagnostic results to the client. The recommended diagnostic results can provide the client with fault matching information matching the current fault event, fault inference information inferring the cause of the fault, verification information verifying the fault inference information, and recovery decision information for the recovery processes that can be adopted, thereby assisting the client in determining the handling method for the current fault event. The specific method by which the server obtains the recommended diagnostic results will be described below in conjunction with the method executed by the server.

[0144] In some embodiments, performing a recovery action for a fault event based on a recommended diagnostic result received from the server corresponding to the current fault fingerprint may include: determining a target recovery object in the GPU cluster and a target recovery action for the target recovery object based on the recommended diagnostic result; and performing the target recovery action corresponding to the fault event on the target recovery object.

[0145] Based on this example, the server can provide the client with recommended diagnostic results for recovery processing without needing to know the actual identifiers of the task objects in the GPU cluster. The client can then further map these recommended diagnostic results to specific target recovery objects and actions within the current GPU cluster. This allows the recommended diagnostic results to be applied to the specific faulty object corresponding to the current fault event without exposing the client's internal resource identifiers to the server, thus balancing the reuse of cross-client diagnostic experience with the executability of recovery processing.

[0146] For example, the target recovery object can be a specific object in the GPU cluster that requires recovery processing. The client can combine the recommended diagnostic results with locally stored information corresponding to the currently occurring fault event to determine the actual object on which the recommended recovery action should be applied in the current GPU cluster.

[0147] For example, recommended diagnostic results can indicate the need for network configuration repair on network entities experiencing communication anomalies. Clients can identify the actual network entity experiencing the communication anomaly based on the fault information corresponding to the current fault event locally, designate it as the target recovery object, and specify network configuration repair as the target recovery action. Subsequently, the corresponding target recovery action can be performed on the target recovery object.

[0148] In some embodiments, the recommended diagnostic results can also provide the client with different information to determine the target recovery action. The following will further explain the fault matching information, fault inference information, verification information, and recovery decision information that the recommended diagnostic results may include.

[0149] In some embodiments, the recommended diagnostic result may include fault matching information that matches the current fault fingerprint, wherein the fault matching information indicates one or more similar fault fingerprints determined from a set of fault fingerprints stored on the server. Thus, the client can know whether similar fault cases have occurred previously and can use these similar fault fingerprints as a reference for handling the current fault event.

[0150] For example, the current fault fingerprint may include anonymized fault features such as "aggregate communication timeout," "bandwidth below 40% of baseline," and "packet loss increase precedes communication timeout." If the server determines that similar fault fingerprints exist in the fault fingerprint set, it can indicate one or more corresponding similar fault fingerprints to the client through fault matching information. The client can then determine that the current fault event has a certain similarity to past fault cases without needing to know which specific client the corresponding historical fault fingerprint originated from.

[0151] In some embodiments, the recommended diagnostic results may include fault inference information and verification information.

[0152] For example, fault inference information can indicate the cause of the fault. For instance, for a fault event that manifests as a collective communication timeout accompanied by a significant decrease in high-speed network bandwidth, fault inference information can indicate that the fault event may be related to an abnormal network configuration.

[0153] For example, the verification information indicates the evidence to be obtained to verify the fault inference information. The client can obtain the corresponding evidence locally based on the verification information and use the obtained evidence to determine whether the fault inference cause matches the current fault event. For instance, if the fault inference cause is an abnormal network configuration, the verification information can indicate that further checks are needed to ensure the corresponding network configuration meets predetermined requirements. Thus, the fault inference given by the server can be further verified before performing recovery actions, ensuring that the performed recovery actions correspond to the verified fault inference cause.

[0154] In some embodiments, the recommended diagnostic results may include recovery decision information. In such an example, the recovery decision information may include recommended recovery actions. For instance, if the fault is inferred to be caused by an abnormal network configuration, the recommended recovery action may be to repair the corresponding network configuration.

[0155] In such an example, the recovery decision information may also include the historical success rate of the recommended recovery action and at least one of other historical recovery actions different from the recommended one. The historical success rate reflects the number of times the recommended recovery action successfully eliminated the corresponding fault in historical cases reflected by similar fault fingerprints. For example, a recommended recovery action may have a high historical success rate if it eliminates the corresponding fault in most similar fault fingerprints. The client can use this historical success rate as a reference when choosing whether to execute the recommended recovery action.

[0156] For example, other historical recovery actions, different from the recommended recovery action, can be recovery actions that have been used in historical cases reflected by similar failure fingerprints. By providing these historical recovery actions to the client, the client can understand other processing methods that have been used for similar failure fingerprints. For example, for recovery actions that have been historically verified as invalid or potentially have a significant impact, the client can avoid prioritizing such recovery actions based on the corresponding historical processing.

[0157] The information mentioned above in the recommended diagnostic results is not required to be present in every fault diagnosis process. For example, when similar fault fingerprints can be identified but clear fault inference information cannot yet be formed, the recommended diagnostic results can mainly include fault matching information. As another example, when sufficient historical fault fingerprints and their processing information are available, the recommended diagnostic results may further include fault inference information, verification information, or recovery decision information.

[0158] In some embodiments, the recommended diagnostic results can further provide information to assist the client in understanding and handling the currently occurring fault event, based on one or more similar fault fingerprints. For example, the recommended diagnostic results may indicate the similar historical fault clusters corresponding to the current fault event, the recommended confidence levels of similar fault fingerprints, common and differing fault features between the current fault fingerprint and similar historical fault fingerprints, one or more candidate fault causes arranged according to recommended confidence levels or degrees of trust, and risk warnings. Thus, the recommended diagnostic results can provide the client with diagnostic references for the currently occurring fault event from multiple perspectives.

[0159] For example, for a current failure event manifesting as multi-node aggregated communication timeout, high-speed network bandwidth degradation, and training task stagnation, the recommended diagnostic results could be as follows: The current failure event has a recommendation confidence of 0.86 for similar failure fingerprints to the lossless network communication timeout category. Common failure features include increased lossless network packet loss, decreased high-speed network bandwidth, aggregated communication timeout, and training task stagnation. Among similar failure fingerprints, lossless network configuration anomalies can have a high proportion of failure causes; therefore, they can be prioritized for verification as a possible cause. Further evidence related to switch QoS configuration, flow control counts, physical layer packet loss counts, or high-speed network port rates is recommended. Regarding recovery, simply restarting the training task has a low historical success rate, while repairing the network configuration has a high historical success rate; therefore, network configuration repair is the preferred recommended recovery action.

[0160] In some embodiments, after performing a recovery action, the client can send the actual fault handling result of the fault event to the server. For example, the actual fault handling result can be used to reflect the final confirmed handling status of this fault event.

[0161] In some embodiments, the actual fault handling result may include one or more of the following: the actual fault cause, the actual recovery action, the execution result, and the verification result.

[0162] In some embodiments, the actual fault handling result may include the actual fault cause. The actual fault cause may indicate the cause of the fault event that has been verified. For example, if the initial fault is inferred to be due to an abnormal network configuration, and the client confirms through further verification that the network configuration is indeed abnormal, then the verified abnormal network configuration can be taken as the actual fault cause.

[0163] In some embodiments, the actual fault handling result may include actual recovery actions. Actual recovery actions may be recovery actions performed by the client based on recommended diagnostic results. For example, the client may determine the specific handling method by combining local verification results, and the actual recovery actions may be used to reflect the final, actually implemented recovery process.

[0164] In some embodiments, the actual fault handling result may also include the execution result. The execution result can indicate whether the actual recovery action was successfully executed. For example, for a network configuration repair recovery action, the execution result of the recovery action can be determined based on whether the corresponding configuration was successfully modified and took effect.

[0165] In some embodiments, the actual fault handling result may also include a verification result. The verification result may indicate whether the fault event has been eliminated after the actual recovery action was performed. For example, after performing network configuration repair, if the previously occurring aggregate communication timeout no longer occurs and the training task can continue to run, the corresponding verification result indicates that the fault event has been eliminated.

[0166] For example, successful execution of a recovery action does not necessarily mean that the recovery action has eliminated the fault event. For instance, a network configuration modification operation may execute successfully, but a communication timeout may still occur. In this case, the execution result may indicate that the recovery action was successful, while the verification result may indicate that the fault event has not been eliminated. By recording the above information separately, the actual effect of the recovery action on the current fault event can be more accurately reflected.

[0167] The client can send one or more of the actual fault handling results mentioned above to the server. The server can further refine the fault characteristics or related information corresponding to the current fault fingerprint based on the actual fault handling results. The specific processing method will be further explained below in conjunction with the method executed by the server.

[0168] Next, a method executed by the server side according to one or more embodiments of this disclosure will be described. It should be understood that, unless the context otherwise requires, the embodiments, specific implementations, and variations described above in conjunction with the client side are also applicable to the corresponding embodiments described below in conjunction with the server side; conversely, the corresponding content described in conjunction with the server side can also be applied to the embodiments on the client side. For example, content regarding fault characteristics, fault fingerprints, dimensions, matching methods between fault fingerprints, and recommended diagnostic results, which has been described on one side, can be correspondingly applied to embodiments involving the same or corresponding technical content on the other side, and therefore will not be repeated here.

[0169] Figure 2 A schematic flowchart of a method 200 for handling graphics processor (GPU) cluster failures, executed by a server according to an embodiment of the present disclosure, is shown.

[0170] At S201, the current fault fingerprint is received from the client. In such an example, the current fault fingerprint includes one or more de-identified fault features mapped to the corresponding dimension in the GPU cluster fault dimension set, and indicates a fault event that occurred in the GPU cluster associated with the client.

[0171] At S202, based on the degree of matching between the current fault fingerprint and each historical fault fingerprint in the fault fingerprint set maintained by the server, one or more similar fault fingerprints are identified from the fault fingerprint set. In such an example, the fault fingerprint set includes one or more historical fault fingerprints from other clients.

[0172] At S203, based on one or more similar fault fingerprints, a recommended diagnostic result corresponding to the current fault fingerprint is obtained.

[0173] At S204, the recommended diagnostic results are sent to the client.

[0174] By executing method 200, the server can utilize the fault characteristics carried by historical fault fingerprints from different clients to obtain recommended diagnostic results for the current fault fingerprint. This enables the reuse of GPU cluster fault diagnosis experience among different clients.

[0175] The following describes in detail one or more exemplary embodiments or variations of the method 200 according to the present disclosure.

[0176] Return to reference Figure 2 As mentioned above, at S201, the current fault fingerprint can be received from the client. In some embodiments, the current fault fingerprint received by the server can be the one obtained by the client according to the preceding description. Figure 1 The method 100 generates a fault fingerprint for a currently occurring fault event. One or more fault features in the current fault fingerprint have been de-identified, and each de-identified fault feature is mapped to a corresponding GPU cluster fault dimension. The current fault fingerprint can indicate the fault event that occurred in the GPU cluster associated with the client that sent the current fault fingerprint.

[0177] Return to reference Figure 2 As previously described, at S202, one or more similar fault fingerprints can be determined from the fault fingerprint set based on the degree of matching between the current fault fingerprint and each historical fault fingerprint in the fault fingerprint set maintained by the server. In some embodiments, the server may maintain a fault fingerprint set. Historical fault fingerprints in the fault fingerprint set may be fault fingerprints from other clients for fault events prior to the currently occurring fault event. The fault fingerprint set may include at least one or more historical fault fingerprints from other clients different from the current client, thereby enabling the server to match the current fault fingerprint using historical fault fingerprints previously formed by other clients.

[0178] The degree of matching between the current fault fingerprint and each historical fault fingerprint can be used to represent the similarity of the fault features contained in both on the fault dimension set of the GPU cluster. Therefore, the server can compare the current fault fingerprint with each historical fault fingerprint, and based on the comparison results, determine one or more historical fault fingerprints similar to the current fault fingerprint from the fault fingerprint set as one or more similar fault fingerprints.

[0179] In some embodiments, determining one or more similar fault fingerprints from the fault fingerprint set based on the matching degree between the current fault fingerprint and each historical fault fingerprint in the fault fingerprint set maintained by the server may include: determining the dimensional similarity between the current fault fingerprint and each historical fault fingerprint in each dimension of the GPU cluster fault dimension set for each historical fault fingerprint; determining the composite similarity between the current fault fingerprint and the historical fault fingerprint based on the dimensional similarity between the current fault fingerprint and the historical fault fingerprint in each dimension of the GPU cluster fault dimension set; and determining one or more similar fault fingerprints from the fault fingerprint set based on the multiple composite similarities between the current fault fingerprint and each historical fault fingerprint.

[0180] Based on this example, the matching degree between the current fault fingerprint and historical fault fingerprints can be examined separately along different dimensions, and then the overall matching degree can be determined by combining the matching results of each dimension. This allows the contribution of different dimensions to the matching results to be reflected separately, and historical fault fingerprints that are close to the current fault fingerprint across the entire GPU cluster fault dimension set are identified as similar fault fingerprints. This improves the reliability of selecting similar fault fingerprints from the fault fingerprint set and provides more targeted historical references for generating recommended diagnostic results based on similar fault fingerprints.

[0181] For example, dimensional similarity can be used to characterize the degree of similarity between the current fault fingerprint and a certain historical fault fingerprint in the same dimension.

[0182] In some embodiments, each dimension may include one or more fields, and each field may carry fault features mapped to that dimension. The server may compare the fault features contained in the corresponding fields of the current fault fingerprint and historical fault fingerprints in the same dimension to determine the degree of similarity between the fields, and further determine the dimensional similarity of the dimension based on the similarity of one or more fields. This disclosure does not limit the specific calculation method of the similarity between fields or the dimensional similarity.

[0183] For example, the symptom dimension can include a symptom set field. The server can compare the symptom tags contained in the symptom set field of the current fault fingerprint and historical fault fingerprints, and determine the similarity of the symptom set field based on the overlap of identical or similar symptom tags. For example, if the symptom set field of the current fault fingerprint includes "aggregate communication timeout" and "training task stall," and the symptom set field of historical fault fingerprints also includes one or more of the above symptoms, then the corresponding symptom similarity can be determined based on the number of overlapping symptom tags (e.g., using...). S_s (Indicates symptom similarity).

[0184] In some embodiments, the symptom dimension may further include an error code family field. The error code family field can carry error code categories formed by standardizing different original error codes. The server can compare the error code families contained in the error code family field of the current fault fingerprint and historical fault fingerprints, and determine the error code family similarity based on the overlap of identical error code families (e.g., using...). S_c (Indicates the similarity of error code families).

[0185] For example, the metrics and relationship dimensions can include a metric trend field. The server can compare the fault characteristics of two fault fingerprints for the same operational metric in this field. For instance, if both the current fault fingerprint and the historical fault fingerprint indicate a decrease in high-speed network bandwidth, and their decrease rates are in the same or similar ranges, then they can be considered to have a high degree of similarity in this metric trend field. Based on the direction and degree of change of one or more operational metrics, metric trend similarity can be determined (e.g., denoted by S_m).

[0186] For example, metrics and relationship dimensions can also include an event sequence field. The server can compare the relative order of abnormal events recorded in two fault fingerprints. For instance, if the event sequence field of the current fault fingerprint includes "increased network card packet loss → timeout of aggregate communication → interruption of training task", and historical fault fingerprints also include the same or similar event sequences, then the two can be considered to have a high degree of similarity in the event sequence field. In some embodiments, the event sequence can be converted into a sequence representation or a vector representation, and the similarity can be determined based on the closeness between the representation results.

[0187] For example, metrics and relationship dimensions can also include a topology role field. The server can compare the generalized topology relationships between two fault fingerprints. For instance, if the topology role field of the current fault fingerprint includes "multiple GPU anomalies on the same node," while the topology role field of a certain historical fault fingerprint includes "single GPU anomalies on the same node," then because the anomaly distribution ranges reflected by the two are different, a relatively low topology role similarity can be determined.

[0188] In some embodiments, the comparison results of the event order field and the topology role field can be used together to determine the similarity between event order and topology role (e.g., using...). S_t (This indicates the similarity between the order of events and topological roles).

[0189] For example, the recovery dimension may include a recovery result field. The server can compare the recovery processing results reflected in this field between the current fault fingerprint and historical fault fingerprints. In some embodiments, the recovery result can be represented by the number, proportion, or proportion range of results such as successful recovery, failed recovery, or fault recurrence. For example, the recovery result field of the current fault fingerprint can indicate that in multiple processing steps using a certain recovery action for the corresponding fault, the success rate of recovery is between 70% and 80%; if the success rate of recovery after using the same or similar recovery action in a historical fault fingerprint is also between 70% and 80%, then the two can be considered to have a high degree of similarity in the recovery result. If the success rate of recovery corresponding to a historical fault fingerprint is only 20% to 30%, then the similarity between the two in the recovery result can be low. For cases where successful recovery, failed recovery, and fault recurrence are recorded simultaneously, the proportions corresponding to each type of result can be compared separately, and the similarity of the recovery results can be determined by combining these comparison results (e.g., using...). S_r (This indicates the similarity of the recovery results). However, if the current fault fingerprint does not yet contain recovery results, the similarity of the corresponding dimension can be determined without relying on the recovery result field. The above comparison can be performed when the current fault fingerprint contains information about recovery processes previously performed for that fault event.

[0190] For example, the environment dimension can include hardware category, network type, or other fields used to describe the operating environment. The server can compare the environment-related fault features of the current fault fingerprint and historical fault fingerprints in these corresponding fields, and determine the similarity of the environment dimension based on the consistency between the fields.

[0191] When a dimension includes multiple fields, the server can determine the dimensional similarity of that dimension based on the comparison results between the current fault fingerprint and historical fault fingerprints in their respective fields. Specifically, a weighted combination, average, or other combination of field similarity can be used, and this disclosure does not impose any restrictions on this.

[0192] After obtaining the dimensional similarity between the current fault fingerprint and a historical fault fingerprint on the GPU cluster fault dimension set, the composite similarity between the current fault fingerprint and the historical fault fingerprint can be determined based on these dimensional similarities. The composite similarity can be used to comprehensively characterize the overall matching degree of the two fault fingerprints on the GPU cluster fault dimension set.

[0193] In some embodiments, the current fault fingerprint can be represented as F_q Let the i-th historical fault fingerprint in the fault fingerprint set be represented as F_i Therefore, it is possible to base decisions on symptom similarity. S_s Error code family similarity S_c Similarity of indicator trends S_m Event sequence and topological role similarity S_t Similarity of recovery results S_r and environmental compatibility similarity S_e Determine the current fault fingerprint F_q fingerprints of historical faults F_i The composite similarity between them. For example, composite similarity can be expressed as: S(F_q,F_i)=w_s S_s+w_c S_c+w_m S_m+w_t S_t+w_r S_r+w_e S_e .in, w_s,w_ c,w_m,w_t,w_r,w_e These represent the weights corresponding to the similarity scores mentioned above. Each weight can be a non-negative value, and their sum can be 1. By setting different weights, the influence of different fault features on the overall matching degree of two fault fingerprints can be adjusted.

[0194] For example, when performing fault fingerprint matching for GPU device reset anomalies, symptom similarity, error code family similarity, indicator trend similarity, and event sequence and topology role similarity can be given relatively high weights. If both the current fault fingerprint and a certain historical fault fingerprint include fault features such as "GPU device reset" and "GPU device lost," and their corresponding device error codes belong to the same or similar error code families, and both show a continuous increase in bus link error count and a similar event sequence of "bus link anomaly → GPU device reset → task abnormal exit," then the above-mentioned multiple similarities can have high values, thus giving the current fault fingerprint and the historical fault fingerprint a high composite similarity.

[0195] In some embodiments, the current fault fingerprint or the historical fault fingerprint to be compared may not contain fault features in a certain dimension or field. For example, if no recovery action has been performed on the current fault event, the current fault fingerprint may not contain a recovery result. In this case, the composite similarity can be determined without using the similarity of the corresponding fields, and the corresponding weights can be adjusted according to the actual similarity involved in the comparison.

[0196] It is understood that the weighted summation of similarities across various dimensions described above is merely one exemplary method for determining composite similarity. In other embodiments, other methods capable of comprehensively considering the similarity across the entire set of GPU cluster fault dimensions may also be used to determine composite similarity, and this disclosure does not impose any limitations on this.

[0197] In some embodiments, the method may further include: determining a recommended confidence level for one or more similar fault fingerprints; and ranking the one or more similar fault fingerprints based on the recommended confidence level. In such an example, the recommended diagnostic result corresponding to the current fault fingerprint is also based on the ranking.

[0198] Based on this example, after identifying multiple similar fault fingerprints, their reference priorities can be further distinguished according to the recommended confidence level of each similar fault fingerprint. This allows historical fault fingerprints that are more relevant to the current fault to be prioritized in the generation of recommended diagnostic results, thereby improving the fit between the recommended diagnostic results and the current fault event.

[0199] For example, the aforementioned composite similarity can be used to determine which historical fault fingerprints have a high degree of matching with the current fault fingerprint, while the recommendation confidence can be further used to determine the recommendation order among one or more identified similar fault fingerprints. That is, in such an example, composite similarity can be used to filter similar fault fingerprints from the set of fault fingerprints, while recommendation confidence can be used to determine the reference priority of these similar fault fingerprints when obtaining recommended diagnostic results in the future.

[0200] In some embodiments, one or more similar fault fingerprints can be sorted from high to low recommended confidence levels. Similar fault fingerprints with higher recommended confidence levels can have higher ranking positions and be given priority reference when obtaining recommended diagnostic results subsequently.

[0201] For example, the server can determine the first, second, and third similar fault fingerprints from the fault fingerprint set based on the current fault fingerprint, and determine the corresponding recommended confidence level for each. If the second similar fault fingerprint has the highest recommended confidence level, followed by the first, and then the third, the fingerprints can be sorted in the order of second, first, and third. When obtaining recommended diagnostic results subsequently, the information contained in the higher-ranked similar fault fingerprints can be used preferentially.

[0202] In some embodiments, the recommended diagnostic results can also be obtained based on the above ranking. For example, when multiple similar fault fingerprints contain different recovery actions, the similar fault fingerprints ranked higher can be given priority. As another example, when aggregating multiple similar fault fingerprints, the similar fault fingerprints ranked higher can be given higher reference priority. How to obtain recommended diagnostic results based on multiple similar fault fingerprints will be further explained below.

[0203] In some embodiments, the recommendation confidence level is determined based on at least one of the composite similarity and fingerprint confidence level corresponding to the similar fault fingerprint. In such an example, the fingerprint confidence level indicates the degree of confidence of the fault features of the similar fault fingerprint, and the recommendation confidence level indicates the priority of the similar fault fingerprint among one or more similar fault fingerprints.

[0204] Based on such examples, the recommendation priority can be determined according to the degree of matching between similar fault fingerprints and the current fault fingerprint, the credibility of the similar fault fingerprints themselves, or a combination of the two. This allows the recommendation confidence to reflect the reference value of similar fault fingerprints for the current fault diagnosis from different perspectives and helps to form a more reasonable recommendation order among multiple similar fault fingerprints.

[0205] For example, the recommended confidence level can be determined directly based on the composite similarity between similar faulty fingerprints and the current faulty fingerprint, so that similar faulty fingerprints with higher composite similarity have higher recommended confidence levels. Alternatively, the recommended confidence level can be determined based on the fingerprint credibility of the similar faulty fingerprint itself. Yet another example is the combination of composite similarity and fingerprint credibility to determine the recommended confidence level.

[0206] In some embodiments, fingerprint confidence can be determined based on the degree of confirmation of historical fault cases corresponding to similar fault fingerprints. For example, historical fault cases that have been manually confirmed can correspond to higher fingerprint confidence, while historical fault cases that have not yet been confirmed can correspond to relatively lower fingerprint confidence. For example, the fingerprint confidence of the former can be set to 0.9, and the latter can be set to 0.5. The above values ​​are for illustrative purposes only and do not limit the specific way in which fingerprint confidence is determined. Assume the current fault fingerprint is... F_q The i-th similar fault fingerprint is F_i The recommended confidence level for this similar fault fingerprint is C_i The recommendation confidence level can be expressed as follows: C_i=S(F_q,F_i)· Q_i As mentioned above, S(F_q,F_i) This represents the composite similarity between the current faulty fingerprint and similar faulty fingerprints. Q_i This can indicate the credibility of the fingerprint.

[0207] It is understood that the above formula is only one specific way to determine the recommended confidence level. In other embodiments, the recommended confidence level may be determined based solely on composite similarity or fingerprint confidence, or it may be determined by combining both of these and other information that can reflect the applicability of similar fault fingerprints. This disclosure does not impose any limitations on this.

[0208] Return to reference Figure 2 As mentioned earlier, at S203, a recommended diagnostic result corresponding to the current fault fingerprint can be obtained based on one or more similar fault fingerprints.

[0209] After the server identifies one or more similar fault fingerprints, it can utilize the fault features contained in these similar fault fingerprints to obtain recommended diagnostic results for diagnosing the current fault event. Since similar fault fingerprints and the current fault fingerprint have a high degree of matching in the corresponding dimensions of the GPU cluster fault dimension set, the past fault handling information reflected by the similar fault fingerprints can serve as a reference for diagnosing the current fault event.

[0210] In some embodiments, obtaining a recommended diagnostic result corresponding to the current fault fingerprint based on one or more similar fault fingerprints may include: for each of the one or more similar fault fingerprints, determining one or more candidate diagnostic items indicated by the similar fault fingerprint based on one or more desensitized fault features mapped by the similar fault fingerprint onto the GPU cluster fault dimension set. In such an example, the one or more candidate diagnostic items include at least one of candidate fault causes, candidate verification evidence, and candidate recovery actions; aggregating the candidate diagnostic items indicated by the one or more similar fault fingerprints respectively to determine recommended diagnostic items for the fault event, thereby obtaining a recommended diagnostic result.

[0211] Based on this example, candidate diagnostic items related to the current fault event can be extracted from the fault features contained in each similar fault fingerprint. Then, the candidate diagnostic items from multiple similar fault fingerprints can be integrated. This allows the recurring or mutually corroborating fault causes, verification evidence, and recovery actions from multiple similar historical faults to be reflected in the aggregation process, thereby improving the reference value of the recommended diagnostic results for the current fault event.

[0212] Understandably, candidate diagnostic items can be candidate content determined based on the fault characteristics of similar fault fingerprints, which can be used to diagnose the fault event indicated by the current fault fingerprint. Correspondingly, recommended diagnostic items can be diagnostic content determined for the fault event indicated by the current fault fingerprint after synthesizing candidate diagnostic items indicated by one or more similar fault fingerprints.

[0213] For a similar fault fingerprint, one or more candidate diagnostic items can be determined based on the fault features contained in the fault dimension set of the GPU cluster. As mentioned earlier, the fault features of similar fault fingerprints in the environmental dimension, symptom dimension, indicator and relationship dimension, recovery dimension, and privacy and trustworthiness dimension can respectively reflect the operating conditions, fault performance, indicator changes and event relationships, recovery processing and its effect, and case trustworthiness of the historical faults corresponding to the similar fault fingerprints. Therefore, at least one of the candidate fault causes, candidate verification evidence, and candidate recovery actions can be determined by combining these fault features.

[0214] For example, the symptom dimension of a similar fault fingerprint may include "device hardware error" and "abnormal training task exit," while the indicator and relationship dimensions may include fault features such as "continuous growth of uncorrectable memory errors" and "isolated single device anomaly." If the combination of the above fault features corresponds to a GPU memory hardware anomaly in the similar fault fingerprint, then "GPU memory hardware anomaly" can be identified as a candidate fault cause indicated by the similar fault fingerprint based on the device anomaly performance and anomaly distribution reflected by these fault features.

[0215] In some embodiments, candidate verification evidence can be determined based on fault features in similar fault fingerprints that can be used to verify the cause of the fault. For example, a similar fault fingerprint may include fault features such as "continuous growth of uncorrectable memory errors," "appearance of critical device error codes," and "anomaly occurring only in a single GPU." These fault features can be used to confirm or distinguish between GPU hardware anomalies and anomalies caused by the training task itself or the software operating environment. Therefore, memory error correction counts, device error states, or the operating states of other GPUs in the same node can be determined as candidate verification evidence for verifying the candidate cause of "GPU memory hardware anomaly."

[0216] In some embodiments, candidate recovery actions can be determined based on fault features in the recovery dimension of similar fault fingerprints. For example, the recovery dimension of similar fault fingerprints may include fault features such as "fault persists after node reboot" and "successful recovery after replacing the faulty GPU". These fault features indicate that node reboot failed to eliminate the corresponding fault, while replacing the faulty GPU eliminated the fault. Therefore, "replacing the faulty GPU" can be identified as a candidate recovery action based on these fault features.

[0217] For example, when the server identifies multiple similar fault fingerprints, different similar fault fingerprints may indicate the same candidate diagnostic item or different candidate diagnostic items. Therefore, these candidate diagnostic items can be aggregated to determine the recommended diagnostic item for the currently occurring fault event.

[0218] In some embodiments, aggregation can consider the occurrence of candidate diagnostic items among multiple similar fault fingerprints. For example, if the current fault fingerprint includes fault features such as "runtime initialization failed" and "GPU not visible in container," and multiple similar fault fingerprints indicate that the driver is incompatible with the runtime environment, while only a few similar fault fingerprints indicate a device hardware malfunction, then "driver incompatibility with runtime environment" can be given a higher recommendation priority. Similarly, if multiple similar fault fingerprints indicate that reconfiguring the runtime environment can eliminate the corresponding fault, then the corresponding recovery action can be prioritized as the recommended recovery action.

[0219] In some embodiments, when aggregating candidate diagnostic items, the recommendation confidence level determined in the aforementioned steps can also be utilized. For example, candidate diagnostic items indicated by similar fault fingerprints with higher recommendation confidence levels can have higher reference priority. Thus, even if multiple candidate fault causes appear in similar fault fingerprints with similar occurrence counts, the recommendation order can be further determined by combining the matching degree and confidence level of the corresponding historical fault fingerprints. This disclosure does not limit the specific aggregation method of candidate diagnostic items.

[0220] Return to reference Figure 2 As mentioned earlier, at S204, recommended diagnostic results can be sent to the client.

[0221] In some embodiments, the server can send recommended diagnostic results to the client, enabling the client to further diagnose the current fault event and perform corresponding recovery actions based on the recommended diagnostic results. The recommended diagnostic results may include fault matching information, fault inference information, verification information, or recovery decision information. The specific processing by the client after receiving the recommended diagnostic results can be referred to the content described above in conjunction with method 100, and will not be repeated here.

[0222] In some embodiments, after the client processes the current fault event based on the recommended diagnostic results, the server can also receive the actual fault handling result corresponding to the fault event from the client; update the current fault fingerprint based on the actual fault handling result; and store the updated current fault fingerprint in the fault fingerprint set. In such an example, the actual fault handling result may include one or more of the following: actual fault cause, indicating the verified cause of the fault event; actual recovery action, which is the recovery action actually performed by the client for the fault event; execution result, indicating whether the actual recovery action was successfully executed; and verification result, indicating whether the fault event was eliminated after the actual recovery action was performed.

[0223] Based on this example, the current fault fingerprint generated for the current fault event can be further supplemented after the actual fault handling is completed, and added to the fault fingerprint set as a new historical fault fingerprint. As the actual fault handling results corresponding to different fault events are continuously fed back, the fault fingerprint set can accumulate fault handling experience verified by actual diagnosis and recovery, which can be used for subsequent matching and diagnostic recommendations for other current fault fingerprints.

[0224] In some embodiments, the server can supplement or update the fault features related to the fault handling result in the current fault fingerprint based on the actual fault handling result. For example, if the actual fault cause has been verified, the verified actual fault cause can be associated with the current fault fingerprint, so that the current fault fingerprint not only reflects the fault features obtained when the fault event occurred, but also further reflects the verified fault cause. If a recovery action has been actually performed, the content related to recovery processing in the current fault fingerprint can also be updated based on the actual recovery action, the execution result, and the verification result.

[0225] For example, the current fault fingerprint may include fault features such as "GPU temperature continuously rising," "device operating frequency decreasing," and "training task performance deteriorating." After the client further verifies and confirms that the actual fault is caused by abnormal GPU cooling, and restores the GPU temperature to normal by restoring the corresponding cooling conditions, the server can further associate the current fault fingerprint with the confirmed "GPU cooling anomaly" based on the received actual fault handling results, and supplement the recovery dimension with the content indicating "heating anomaly successfully repaired." If the verification results show that the GPU temperature has returned to the normal range after the recovery action, and the training task's running status has returned to normal, the content reflecting the degree of case confirmation in the current fault fingerprint can also be updated accordingly.

[0226] In some embodiments, the actual fault handling result can also be used to correct content in the current fault fingerprint that has not been previously confirmed or is inconsistent with the actual handling result. For example, if the fault inference initially indicated by the recommended diagnostic result is the first fault cause, but the client determines after further verification that the actual fault cause is a different second fault cause, the server can record the verified second fault cause based on the actual fault handling result so that the actually confirmed fault handling result can be used as a reference when using the fault fingerprint subsequently.

[0227] After the update is completed, the server can store the updated current fault fingerprint in the fault fingerprint set. The updated current fault fingerprint can then be used as a historical fault fingerprint and matched with other currently received fault fingerprints. Since the updated current fault fingerprint can further include verified fault causes and the results of actual recovery processing, when the historical fault fingerprint is subsequently identified as a similar fault fingerprint, candidate fault causes, candidate verification evidence, or candidate recovery actions can be obtained based on the confirmed fault processing information contained therein.

[0228] To further illustrate the aforementioned GPU cluster fault diagnosis process executed collaboratively by the client and server, two specific fault scenarios are provided below as examples. It should be understood that the following embodiments are only for helping to understand the technical solutions of this disclosure and do not limit the specific content of the fault type, fault characteristics, or the recommended specific form of the diagnostic results.

[0229] In one embodiment, a client's multi-node GPU training task experiences a ensemble communication timeout. The client obtains raw diagnostic evidence related to decreased high-speed network bandwidth, increased lossless network packet loss, ensemble communication timeout, and training task stagnation through multiple diagnostic data interfaces. This evidence also includes information that can identify internal client resources, such as node network addresses, switch ports, job names, and container identifiers. After standardizing and anonymizing the raw diagnostic evidence, the client can generate a corresponding current fault fingerprint, as shown below.

[0230] Table 3: Another example of a current fault fingerprint

[0231] In such an example, after receiving the current fault fingerprint, the server can identify one or more similar fault fingerprints from the fault fingerprint set. If multiple similar fault fingerprints indicate that the similar fault is related to lossless network configuration anomalies, and the corresponding historical faults have been recovered through network configuration repair, the server can consider lossless network configuration anomalies as a candidate fault cause and recommend further verification by obtaining evidence such as network flow control, link packet loss, or port rate, while also considering network configuration repair as a candidate recovery action. After the client completes the actual fault handling and reports the result, the server can also update the current fault fingerprint based on the actual fault cause and recovery result, further including fault features such as "lossless network configuration anomaly confirmed" and "network configuration repair successful," and store it as a historical fault fingerprint in the fault fingerprint set.

[0232] In another embodiment, a single GPU on a client experiences a device hardware malfunction, causing the training task to terminate. The client can obtain raw diagnostic evidence such as critical device error codes, uncorrectable memory errors, and the abnormal termination of the training task, and after standardization and de-identification, form a current fault fingerprint.

[0233] Table 4: Another example of a current fault fingerprint

[0234] In such an example, the server can match the current fault fingerprint with historical fault fingerprints from the fault fingerprint set. If similar fault fingerprints indicate that similar faults are mostly related to single-device hardware anomalies, and historically, node restarts failed to eliminate the fault while hardware replacement resolved the issue, the server can prioritize hardware anomalies as a candidate cause of failure and recommend performing hardware checks instead of repeatedly restarting the node. After the client confirms the actual cause of the fault and completes the recovery process, the server can also update the current fault fingerprint using the feedback results and use the updated fault fingerprint as historical experience that can be reused for subsequent fault diagnosis.

[0235] Figure 3A and Figure 3B A schematic flow 300 for handling GPU cluster failures is illustrated according to a specific embodiment of this disclosure, which is not limiting. In such an example, Figure 3A and Figure 3B Connected through point A. It is understandable that, unless explicitly stated otherwise, the combination... Figure 3A and Figure 3B The details of the various implementation methods described can be applied to other embodiments of this disclosure (e.g., method 100 and method 200), and vice versa. Furthermore, it is understood that since specific examples of fault feature acquisition, desensitization, fault fingerprint generation, fault fingerprint matching, diagnostic recommendation, and processing result feedback have been described in detail above in conjunction with method 100 and method 200, similar or identical details will not be repeated here for the sake of brevity.

[0236] At point S301, obtain the raw diagnostic evidence for the GPU cluster. For example, the raw diagnostic evidence can be obtained through the diagnostic data interfaces corresponding to the different runtime levels mentioned above.

[0237] At S302, the original diagnostic evidence is mapped to fault features. For example, different representations of original diagnostic evidence can be converted into corresponding fault features through parsing and standardization.

[0238] At S303, sensitive entity stripping is performed on the fault characteristics to remove or replace sensitive information that can identify the client or entities within the client.

[0239] At S304, numerical range conversion and time relativization are performed on fault features that need to retain diagnostic semantics. For example, specific measured values ​​can be converted into corresponding numerical ranges, and absolute time information can be converted into relative time relationships between fault events.

[0240] At S305, topology generalization is performed on the topology information in the fault characteristics to replace specific topology entity information with topology roles or topology relationships that can reflect diagnostic significance.

[0241] At step S306, it is determined whether the low-frequency fault characteristics meet a predetermined privacy threshold condition. This privacy threshold condition can be used to determine whether the corresponding low-frequency fault characteristics may increase the risk of the client being reverse-identified due to their high specificity. If step S306 is yes, proceed to step S307; otherwise, proceed to step S308.

[0242] At S307, if the fault feature meets the predetermined privacy threshold condition, that is, for fault features that do not require low-frequency suppression, the corresponding structured fault features can be retained.

[0243] At S308, for low-frequency fault features that do not meet the privacy threshold conditions, the low-frequency fault feature can be deleted or generalized.

[0244] At step S309, multi-dimensional fault features are generated based on the fault features processed as described above. For example, the fault features can be organized into the corresponding dimensions according to the correspondence between the aforementioned diagnostic semantics and the dimensions in the GPU cluster fault dimension set.

[0245] At S310, it is determined whether the corresponding fault characteristics are authorized for sharing. This authorization for sharing can be used to determine whether the corresponding de-identified fault characteristics can be used for cross-client fault diagnosis on the server side.

[0246] At S311, if the corresponding fault characteristics are authorized to be shared, the corresponding fault fingerprint can be written into the fault fingerprint set maintained on the server side.

[0247] At step S312, a fault fingerprint cluster can be formed based on the fault fingerprints written into the fault fingerprint set, or an existing fault fingerprint cluster can be updated using the fault fingerprint. The fault fingerprint cluster can be used to aggregate historical fault fingerprints with the same or similar fault characteristics.

[0248] At S313, if the corresponding fault characteristics are not authorized to be shared, the corresponding fault fingerprint can be saved locally on the client or used for local fault diagnosis without sharing it to the server.

[0249] Figure 3B Continued Figure 3A Connection point A is shown.

[0250] At S314, when a fault event to be diagnosed occurs, the current fault fingerprint corresponding to the fault event can be generated in the manner described above.

[0251] At S315, the composite similarity between the current fault fingerprint and historical fault fingerprints in the fault fingerprint set is calculated to identify historical fault fingerprints similar to the current fault fingerprint.

[0252] At step S316, candidate fault causes and recommended recovery actions are generated based on the identified similar fault fingerprints. For example, the corresponding recommendations can be determined from the candidate diagnostic items indicated by the similar fault fingerprints in the manner described above.

[0253] At point S317, common symptoms, differential symptoms, and risk warnings can be output. Common symptoms and differential symptoms can reflect the same and different fault characteristics between the current fault fingerprint and similar fault fingerprints, while risk warnings can be used to indicate the potential risks of corresponding diagnosis or recovery procedures.

[0254] At S318, confirmation and recovery results for the currently occurring fault event are received. For example, the actual fault cause, actual recovery action, execution result, or verification result reported by the client can be received.

[0255] At S319, the credibility of the fault fingerprint cluster and the validity weight of the recovery action can be updated based on the received confirmation and recovery results, so that subsequent fault diagnosis can further utilize historical fault experience verified by actual processing results.

[0256] pass Figure 3A and Figure 3B The processing flow shown can integrate fault evidence processing, privacy protection, fault experience sharing, fault fingerprint matching, diagnostic recommendations, and actual processing result feedback into a continuous process. The specific implementation of each step can be referred to the corresponding embodiments described above in conjunction with methods 100 and 200, and will not be repeated here.

[0257] Figure 4A and Figure 4B A schematic diagram illustrating module interactions during a GPU cluster fault diagnosis process according to an embodiment of this disclosure is shown. In such an example, Figure 4A and Figure 4B Together, they illustrate exemplary interaction processes between the client, local authorization and policy module, fault fingerprint set, composite similarity matching module, diagnostic recommendation generation module, and operation and maintenance or automatic recovery system. It is understood that, unless explicitly stated otherwise, the combination of... Figure 4A andFigure 4B The details of the various implementation methods described can be applied to other embodiments of this disclosure (e.g., method 100 and method 200), and vice versa. Furthermore, it is understood that since specific examples of fault feature acquisition, desensitization, fault fingerprint generation, fault fingerprint matching, diagnostic recommendation, and processing result feedback have been described in detail above in conjunction with method 100 and method 200, similar or identical details will not be repeated here for the sake of brevity.

[0258] In some embodiments, the local authorization and policy module can be set on the client side to determine whether the corresponding fault information is allowed to be shared. The fault fingerprint set, composite similarity matching module, and diagnostic recommendation generation module can be set on the server side to perform fault matching and diagnostic recommendations using historical fault fingerprints from different clients. The operation and maintenance or automatic recovery system can be set on the server side, or it can be set on the client side or in the local operating environment associated with the client, depending on the actual deployment method. This disclosure does not limit the specific deployment location of the above functional modules, as long as they can be implemented. Figure 4A and Figure 4B The corresponding functions shown are sufficient.

[0259] At S401, the client can obtain raw diagnostic evidence associated with failure events occurring in the GPU cluster.

[0260] At S402, the client can obtain the corresponding fault characteristics based on the original diagnostic evidence, for example, by converting the original diagnostic evidence into the aforementioned standardized fault characteristics.

[0261] At S403, the client can perform a desensitization operation on fault characteristics to suppress identification information that could identify the client.

[0262] At S404, the client can request a sharing policy judgment from the local authorization and policy module to determine whether the de-identified fault information is allowed to be shared.

[0263] At S405, the local authorization and policy module can return the corresponding authorization result to the client.

[0264] At S406, if the authorization result indicates that sharing is allowed, the client can write the corresponding fault fingerprint into the fault fingerprint set.

[0265] At S407, the fault fingerprint set can be updated with the corresponding fault fingerprint cluster based on the newly written fault fingerprint. For example, multiple historical fault fingerprints with similar fault characteristics can be organized into corresponding fault fingerprint clusters. It should be understood that using fault fingerprint clusters is only one optional implementation method; in other embodiments, subsequent matching can also be performed directly based on each historical fault fingerprint.

[0266] At S408, if the authorization result indicates that sharing is not allowed, the client can save the corresponding fault fingerprint locally or use the fault fingerprint for local diagnosis without sharing it to the server.

[0267] At S409, when it is necessary to diagnose the current fault event, the client can submit the current fault fingerprint to the composite similarity matching module.

[0268] At S410, the composite similarity matching module can retrieve historical fault fingerprints or fault fingerprint clusters from the fault fingerprint set for matching with the current fault fingerprint.

[0269] At S411, the fault fingerprint set can return to the composite similarity matching module one or more similar fault fingerprints or corresponding fault fingerprint clusters that match the current fault fingerprint.

[0270] At S412, the composite similarity matching module can calculate the composite similarity between the current fault fingerprint and the historical fault fingerprint, and further determine the corresponding recommendation confidence to sort the historical fault experience obtained by matching.

[0271] At S413, the composite similarity matching module can submit sorted similar fault fingerprints or clusters of similar fault fingerprints to the diagnostic recommendation generation module.

[0272] At S414, the diagnostic recommendation generation module can obtain recommended diagnostic results based on sorted similar fault fingerprints and return one or more of the following to the client: candidate fault causes, candidate verification evidence, and candidate recovery actions.

[0273] At S415, the client can display corresponding recovery suggestions or execute recovery actions for the current fault event based on the recommended diagnostic results through the operation and maintenance or automatic recovery system.

[0274] At S416, the maintenance or automatic recovery system can return actual fault handling information to the client, such as the confirmed actual fault cause and the execution result of the recovery action.

[0275] At S417, the client can perform corresponding desensitization processing on the actual fault handling information and feed back the desensitized confirmation result to the fault fingerprint set for supplementing or updating the corresponding historical fault fingerprints and their fault handling information.

[0276] pass Figure 4A and Figure 4BThe interactive process shown allows the client to retain local control over fault fingerprint sharing. When sharing is permitted, it leverages historical fault experience accumulated on the server side to perform fault fingerprint matching and diagnostic recommendations, while simultaneously feeding back the actual fault handling results to the fault fingerprint set. The specific processing methods for each of the above steps can be referred to the corresponding embodiments described above in conjunction with methods 100 and 200, and will not be repeated here.

[0277] Figure 5 A schematic flowchart illustrating the lifecycle of a fault fingerprint according to a specific embodiment of this disclosure is shown. The processes of fault fingerprint generation, desensitization, sharing, matching, diagnostic recommendation, and result feedback have already been described in conjunction with methods 100 and 200; therefore, the following only describes... Figure 5 The fault fingerprint lifecycle shown is briefly explained. It is understood that, unless explicitly stated otherwise, the fault fingerprint lifecycle will be interpreted in conjunction with... Figure 5 The details of the various implementation methods described can be applied to method 100 and method 200, and vice versa.

[0278] At S501, the current fault fingerprint is generated. For example, the current fault fingerprint can be generated based on the fault characteristics associated with the fault event occurring in the GPU cluster, according to the corresponding dimension in the aforementioned GPU cluster fault dimension set.

[0279] At S502, a desensitization operation is performed on the fault features corresponding to the current fault fingerprint to suppress the identification information that can identify the client.

[0280] In some embodiments, the subsequent processing method for the de-identified current fault fingerprint can be determined based on the client's sharing authorization.

[0281] At S503, if the current fault fingerprint is authorized to be shared, the current fault fingerprint can be written to the fault fingerprint set maintained by the server for subsequent fault diagnosis.

[0282] At step S504, historical fault fingerprints with the same or similar fault characteristics in the fault fingerprint set can be aggregated into fault fingerprint clusters. It should be understood that aggregating historical fault fingerprints into fault fingerprint clusters is only one optional organization method. In other embodiments, subsequent matching can also be performed directly based on each historical fault fingerprint in the fault fingerprint set.

[0283] At S505, when another current fault fingerprint is subsequently received, the historical fault fingerprint or fault fingerprint cluster that the current fault fingerprint matches can be determined through fault fingerprint matching.

[0284] At S506, recommended diagnostic results corresponding to the current fault fingerprint can be generated based on the hit historical fault fingerprints or fault fingerprint clusters.

[0285] At S507, the actual cause of the fault and the execution result of the recovery action can be confirmed based on the actual handling of the fault event by the client, and the corresponding actual fault handling results can be fed back to update historical fault experience.

[0286] At S508, the credibility of the corresponding fault fingerprint cluster and the weights related to recovery processing can be updated based on the confirmed actual fault cause and execution result, so that subsequent fault diagnosis can further utilize historical fault information verified by actual processing results.

[0287] At S509, if the current fault fingerprint is not authorized to be shared, the current fault fingerprint can be saved locally on the client, or local fault diagnosis can be performed based on the current fault fingerprint, without writing it to the fault fingerprint set maintained on the server.

[0288] thus, Figure 5 The fault fingerprint shown can undergo processes such as generation, desensitization, authorized sharing, historical experience accumulation, fault matching, diagnostic recommendation, and feedback and updating of actual processing results, so that the fault experience verified by actual diagnosis and recovery can be continuously used for subsequent fault diagnosis. The specific implementation of each of the above steps can be referred to the corresponding embodiments described above, and will not be repeated here.

[0289] Figure 6 A schematic diagram of an apparatus 600 for a method of handling graphics processing unit (GPU) cluster failures according to an embodiment of the present disclosure is shown. It can be utilized... Figure 6 The device shown is used to perform Figure 1 Method 100 is described.

[0290] like Figure 6 As shown, the device 600 includes a fault feature acquisition module 601, a desensitization module 602, a fingerprint generation module 603, a communication module 604, and a recovery module 605. In addition to these modules, the device 600 may also include other components, the details of which are omitted here.

[0291] The fault feature acquisition module 601 is configured to acquire one or more fault features associated with a fault event occurring in the GPU cluster.

[0292] The desensitization module 602 is configured to perform a desensitization operation on one or more fault features to suppress identification information in one or more fault features that can identify the client, thereby obtaining one or more desensitized fault features.

[0293] The fingerprint generation module 603 is configured to map one or more desensitized fault features to the corresponding dimension in the GPU cluster fault dimension set to generate the current fault fingerprint.

[0294] The communication module 604 is configured to send the current fault fingerprint to the server.

[0295] The recovery module 605 is configured to perform recovery actions for the fault event based on the recommended diagnostic results received from the server corresponding to the current fault fingerprint.

[0296] It should be understood that Figure 6 The various modules of the device 600 shown can be connected to the reference. Figure 1 The various steps in the described method 100 correspond to each other. Therefore, the above description pertains to method 100 and its various variations (including but not limited to those surrounding...). Figure 2 , Figures 3A-3B , Figures 4A-4B , Figure 5 The operations, features, and advantages described in the embodiments (or variations thereof) also apply to device 600 and its included modules. For the sake of brevity, some operations, features, and advantages will not be repeated here.

[0297] Figure 7 A schematic diagram of another apparatus 700 for a method of handling graphics processor (GPU) cluster failures according to an embodiment of the present disclosure is shown.

[0298] like Figure 7 As shown, another device 700 includes a receiving module 701, a matching module 702, a diagnostic result acquisition module 703, and a sending module 704. In addition to these modules, the other device 700 may also include other components, the details of which are omitted here.

[0299] The receiving module 701 is configured to receive the current fault fingerprint from the client. In such an example, the current fault fingerprint includes one or more de-identified fault features mapped to the corresponding dimension in the GPU cluster fault dimension set, and indicates the fault event associated with the client.

[0300] The matching module 702 is configured to determine one or more similar fault fingerprints from the fault fingerprint set based on the degree of matching between the current fault fingerprint and each historical fault fingerprint in the fault fingerprint set maintained by the server. In such an example, the fault fingerprint set includes one or more historical fault fingerprints from other clients.

[0301] The diagnostic result acquisition module 703 is configured to obtain recommended diagnostic results corresponding to the current fault fingerprint based on one or more similar fault fingerprints.

[0302] The sending module 704 is configured to send recommended diagnostic results to the client.

[0303] It should be understood that Figure 7The various modules of the other device 700 shown can be connected to the reference. Figure 2 The various steps in the described method 200 correspond to each other. Therefore, the above description pertains to method 200 and its various variations (including but not limited to those surrounding...). Figure 1 , Figures 3A-3B , Figures 4A-4B , Figure 5 The operations, features, and advantages described in the embodiments (or variations thereof) also apply to another device 700 and its included modules. For the sake of brevity, some operations, features, and advantages will not be repeated here.

[0304] Although specific functions have been discussed with reference to specific modules above, it should be noted that the functions of the various modules discussed in this article can be divided into multiple modules, and / or at least some functions of multiple modules can be combined into a single module.

[0305] Figure 8 A schematic diagram of an electronic device 800 according to an embodiment of the present disclosure is shown.

[0306] like Figure 8 As shown, the electronic device 800 may include a memory 801 and a processor 802.

[0307] The memory 801 may store a computer program non-transitoryly. The processor 802 may be configured to run the computer program. When the computer program is run by the processor 802, it may implement one or more methods described according to embodiments of this disclosure, including but not limited to those described with reference to... Figure 1 Method 100 described, reference Figure 2 Methods described include 200, etc.

[0308] Figure 8 The structure shown is for illustrative purposes only. Electronic device 800 may also include buses, input / output interfaces, communication interfaces, power supplies, or other components not shown, and some components may be omitted or combined as needed.

[0309] Figure 9 A schematic diagram of a computer-readable storage medium 900 according to an embodiment of the present disclosure is shown.

[0310] like Figure 9 As shown, a computer-readable storage medium 900 stores a computer program 901, which, when executed by a processor, implements the steps of any of the method embodiments described above.

[0311] In some embodiments, the computer-readable storage medium 900 may include any combination of one or more computer program products. For example, without limitation, the non-transitory computer-readable storage medium may include volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB memory, flash memory, etc.

[0312] According to one aspect of this disclosure, a computer program product is also provided. In such an example, the computer program product includes a computer program that, when executed by a processor, causes the processor to perform steps implementing any of the method embodiments described above.

[0313] Figure 10 A schematic diagram of a computer device 1000 according to an embodiment of the present disclosure is shown.

[0314] The computer device 1000 may specifically correspond to the electronic device or processing system in the foregoing embodiments. It should be noted that... Figure 10 The components of the computer device 1000 shown are merely exemplary and not restrictive. Depending on the actual application requirements, the computer device 1000 may have other components or omit some components.

[0315] like Figure 10 As shown, computer device 1000 includes processing device 1001. Processing device 1001 can perform various appropriate actions and processes based on computer-executable instructions stored in read-only memory (ROM) 1002 or loaded from storage device 1008 into random access memory (RAM) 1003. For example, processing device 1001 can be configured to perform one or more methods described according to embodiments of this disclosure, including but not limited to those described with reference to… Figure 1 Method 100 described, reference Figure 2 Method 200 is described above. Various programs and data required for the operation of the computer device 1000 can also be temporarily stored in RAM 1003. The processing unit 1001, ROM 1002, and RAM 1003 are interconnected and exchange data via bus 1004. Input / output (I / O) interface 1005 is also connected to bus 1004.

[0316] In some embodiments, the processing device 1001 may include a central processing unit (CPU), a graphics processing unit (GPU), a tensor processor (TPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. In implementing embodiments of this disclosure, the processing device 1001 may have built-in or coupled logic as described above for handling GPU cluster failures to execute the aforementioned method 100 or method 200.

[0317] Typically, the following devices can be connected to the I / O interface 1005: input devices 1006 including, for example, touch screens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 1007 including, for example, liquid crystal displays (LCDs), light-emitting diode displays (LEDs), speakers, vibrators, etc.; storage devices 1008 including, for example, magnetic tapes, hard disks, solid-state drives (SSDs), flash memory cards, etc.; and communication devices 1009.

[0318] The storage device 1008 can serve as a high-capacity, non-volatile storage medium for persistently storing the operating system, applications, and fault fingerprints related to the embodiments of this disclosure. The communication device 1009 allows the computer device 1000 to exchange data with other devices (e.g., cloud servers, other computing nodes in a distributed computing cluster, etc.) via a network, either wirelessly or via wired communication. The communication device 1009 may include a local area network (LAN) card, a wireless local area network (WLAN) card, a Bluetooth module, a near field communication (NFC) module, a cellular communication module, etc.

[0319] Although Figure 10 A computer device 1000 with various devices is shown, but it should be understood that it is not required to implement or possess all of the devices shown, and the computer device 1000 may alternatively implement or possess more or fewer devices. For example, when the computer device 1000 is used as an AI computing accelerator card or server node in a data center, it may not include input devices 1006 and output devices 1007 for human-computer interaction such as touch screens and keyboards, but may mainly consist of a processing unit 1001, high-bandwidth memory, and high-speed communication interfaces.

[0320] The functions of the systems, devices, modules, or units described above herein can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field-Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application-Specific Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on. Furthermore, the methods described in this disclosure can also be implemented by software or microcode containing computer-executable instructions that, when executed by the processing device 1001, cause the machine to perform the operations defined in this disclosure.

[0321] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0322] While embodiments or examples of this disclosure have been described with reference to the accompanying drawings, it should be understood that the methods, systems, and devices described above are merely exemplary embodiments or examples, and the scope of this disclosure is not limited by these embodiments or examples, but only by the claims and their equivalents. Various elements in the embodiments or examples may be omitted or replaced by their equivalents. Furthermore, the steps may be performed in a different order than that described in this disclosure. Further, various elements in the embodiments or examples may be combined in various ways. Importantly, as technology evolves, many elements described herein can be replaced by equivalents that appear after this disclosure.

Claims

1. A method for handling GPU cluster failures, characterized in that, The method is executed by the client, and the method includes: Obtain one or more fault characteristics associated with a fault event occurring in the GPU cluster associated with the client; Perform a desensitization operation on the one or more fault features to suppress the identification information in the one or more fault features that can identify the client, thereby obtaining one or more desensitized fault features; The one or more desensitized fault features are mapped to the corresponding dimensions in the GPU cluster fault dimension set to generate the current fault fingerprint; Send the current fault fingerprint to the server; and Based on the recommended diagnostic results received from the server corresponding to the current fault fingerprint, a recovery action is performed for the fault event.

2. The method according to claim 1, characterized in that, Obtain one or more fault characteristics associated with a fault event occurring in the GPU cluster, including: Original diagnostic evidence is obtained through diagnostic data interfaces corresponding to at least two operational levels of the GPU cluster; and Based on the original diagnostic evidence, obtain one or more fault features in the original diagnostic evidence that are associated with the fault event.

3. The method according to claim 2, characterized in that, Obtaining one or more fault features associated with the fault event from the original diagnostic evidence includes: The original diagnostic evidence is parsed to determine the data representation type corresponding to the original diagnostic evidence; Based on the determined data representation type, the original diagnostic evidence is standardized to convert it into one or more fault characteristics.

4. The method according to claim 3, characterized in that, Based on the determined data representation type, the original diagnostic evidence is standardized, including: Based on the determined data representation type, a target standardization rule is determined from multiple standardization rules corresponding to different data representation types; and According to the target standardization processing rules, the original diagnostic evidence is converted into standardized feature values ​​to obtain the one or more fault features.

5. The method according to claim 2, characterized in that, The at least two operating levels include at least two of the following: the device level, the communication level, the system and container level, and the training and recovery level of the GPU cluster.

6. The method according to any one of claims 1-5, characterized in that, Performing a desensitization operation on the one or more fault features includes: At least one of a first type of feature, a second type of feature, and a third type of feature is determined from the one or more fault features, wherein the first type of feature contains the client's identification information and does not participate in the diagnosis of the fault event, the second type of feature contains the client's identification information and participates in the diagnosis of the fault event, and the third type of feature does not contain the client's identification information. In response to the determination that the first type of feature exists, delete the first type of feature; In response to determining the existence of the second type of feature, the second type of feature is converted into a generalized feature that retains its diagnostic semantics but does not contain the identification information; and In response to the determination that the third type of feature exists, the third type of feature is retained.

7. The method according to any one of claims 1-5, characterized in that, The identification information includes one or more of the following: computing resource identification information of the GPU cluster, communication entity identification information, task subject identification information, and operating environment information.

8. The method according to any one of claims 1-5, characterized in that, Mapping the one or more desensitized fault features to the corresponding dimensions in the GPU cluster fault dimension set includes: Determine the diagnostic semantics represented by the one or more desensitized fault features; and Based on the correspondence between the diagnostic semantics and each dimension in the GPU cluster fault dimension set, the one or more desensitized fault features are mapped to the corresponding dimensions.

9. The method according to any one of claims 1-5, characterized in that, The GPU cluster fault dimension set includes at least two of the following: environment dimension, symptom dimension, indicator and relationship dimension, recovery dimension, and privacy and trustworthiness dimension.

10. The method according to any one of claims 1-5, characterized in that, Based on the recommended diagnostic results received from the server corresponding to the current fault fingerprint, a recovery action is performed for the fault event, including: Based on the recommended diagnostic results, the target recovery objects in the GPU cluster and the target recovery actions for the target recovery objects are determined; and Perform the target recovery action corresponding to the fault event on the target recovery object.

11. The method according to any one of claims 1-5, characterized in that, The recommended diagnostic results include fault matching information that matches the current fault fingerprint, wherein the fault matching information indicates one or more similar fault fingerprints determined from the set of fault fingerprints stored on the server.

12. The method according to any one of claims 1-5, characterized in that, The recommended diagnostic results include fault inference information and verification information, wherein the fault inference information indicates the cause of the fault inference, and the verification information indicates the evidence to be obtained to verify the fault inference information.

13. The method according to any one of claims 1-5, characterized in that, The recommended diagnostic results include recovery decision information, wherein the recovery decision information includes recommended recovery actions, and wherein the recovery decision information also includes the historical execution success rate of the recommended recovery actions and at least one of other historical recovery actions different from the recommended recovery actions.

14. The method according to any one of claims 1-5, characterized in that, The method further includes: After performing the recovery action, the actual fault handling result of the fault event is sent to the server, wherein the actual fault handling result includes one or more of the following: Actual cause of failure, wherein the actual cause of failure indicates the verified cause of the failure event; Actual recovery actions, which are recovery actions performed based on the recommended diagnostic results; The execution result indicates whether the actual recovery action was successfully executed; and The verification result indicates whether the fault event has been eliminated after the actual recovery action is performed.

15. A method for handling GPU cluster failures, characterized in that, The method is executed by the server and includes: Receive a current fault fingerprint from the client, wherein the current fault fingerprint includes one or more desensitized fault features mapped to a corresponding dimension in the GPU cluster fault dimension set, and indicates a fault event that occurred in the GPU cluster associated with the client; Based on the degree of matching between the current fault fingerprint and each historical fault fingerprint in the fault fingerprint set maintained by the server, one or more similar fault fingerprints are determined from the fault fingerprint set, wherein the fault fingerprint set includes one or more historical fault fingerprints from other clients. Based on the one or more similar fault fingerprints, a recommended diagnostic result corresponding to the current fault fingerprint is obtained; and The recommended diagnostic results are sent to the client.

16. The method according to claim 15, characterized in that, Based on the matching degree between the current fault fingerprint and each historical fault fingerprint in the fault fingerprint set maintained by the server, one or more similar fault fingerprints are determined from the fault fingerprint set, including: For each historical fault fingerprint, determine the dimensional similarity between the current fault fingerprint and the historical fault fingerprint on each dimension of the fault dimension set of the GPU cluster; Based on the dimensional similarity between the current fault fingerprint and the historical fault fingerprint in each dimension of the GPU cluster fault dimension set, a composite similarity between the current fault fingerprint and the historical fault fingerprint is determined; and Based on multiple composite similarities between the current fault fingerprint and each historical fault fingerprint, one or more similar fault fingerprints are determined from the fault fingerprint set.

17. The method according to claim 15, characterized in that, The method further includes: Determine the recommended confidence level for the one or more similar fault fingerprints; and Based on the recommended confidence level, the one or more similar fault fingerprints are ranked. The recommended diagnostic results corresponding to the current fault fingerprint are also based on the sorting.

18. The method according to claim 17, characterized in that, The recommended confidence level is determined based on at least one of the composite similarity and fingerprint confidence level corresponding to the similar fault fingerprint, wherein the fingerprint confidence level indicates the degree of confidence of the fault features of the similar fault fingerprint, and the recommended confidence level indicates the priority of the similar fault fingerprint among the one or more similar fault fingerprints.

19. The method according to claim 15, characterized in that, Based on the one or more similar fault fingerprints, a recommended diagnostic result corresponding to the current fault fingerprint is obtained, including: For each of the one or more similar fault fingerprints, based on one or more desensitized fault features mapped by the similar fault fingerprint on the fault dimension set of the GPU cluster, one or more candidate diagnostic items indicated by the similar fault fingerprint are determined, wherein the one or more candidate diagnostic items include at least one of candidate fault causes, candidate verification evidence, and candidate recovery actions; The candidate diagnostic items indicated by the one or more similar fault fingerprints are aggregated to determine the recommended diagnostic items for the fault event, thereby obtaining the recommended diagnostic results.

20. The method according to any one of claims 15 to 19, characterized in that, The recommended diagnostic results include fault matching information, wherein the fault matching information indicates one or more similar fault fingerprints that match the current fault fingerprint and the composite similarity corresponding to each similar fault fingerprint.

21. The method according to any one of claims 15 to 19, characterized in that, The recommended diagnostic results include fault inference information and verification information. The fault inference information indicates the inferred cause of the fault event, and the verification information indicates the evidence to be obtained to verify the fault inference information.

22. The method according to any one of claims 15 to 19, characterized in that, The recommended diagnostic results include recovery decision information, which includes at least one of the following: recommended recovery action, the historical success rate of the recommended recovery action, and other historical recovery actions different from the recommended recovery action.

23. The method according to any one of claims 15 to 19, characterized in that, The method further includes: Receive the actual fault handling result corresponding to the fault event from the client; Based on the actual fault handling results, update the current fault fingerprint; and Store the updated current fault fingerprint into the fault fingerprint set; The actual fault handling results include one or more of the following: Actual cause of failure, wherein the actual cause of failure indicates the verified cause of the failure event; Actual recovery action, which is the recovery action actually performed by the client in response to the failure event; The execution result indicates whether the actual recovery action was successfully executed; and The verification result indicates whether the fault event has been eliminated after the actual recovery action is performed.

24. An apparatus for handling GPU cluster failures, characterized in that, The device is used for a client, and the device includes: The fault feature acquisition module is configured to acquire one or more fault features associated with a fault event occurring in the GPU cluster; The desensitization module is configured to perform a desensitization operation on the one or more fault features to suppress the identification information in the one or more fault features that can identify the client, thereby obtaining one or more desensitized fault features; The fingerprint generation module is configured to map the one or more desensitized fault features to the corresponding dimensions in the GPU cluster fault dimension set to generate the current fault fingerprint. The communication module is configured to send the current fault fingerprint to the server; and The recovery module is configured to perform recovery actions for the fault event based on the recommended diagnostic results received from the server corresponding to the current fault fingerprint.

25. An apparatus for handling GPU cluster failures, characterized in that, The device is used on a server side, and the device includes: The receiving module is configured to receive a current fault fingerprint from a client, wherein the current fault fingerprint includes one or more de-identified fault features mapped to a corresponding dimension in a set of fault dimensions of the GPU cluster, and indicates a fault event associated with the client. The matching module is configured to determine one or more similar fault fingerprints from the fault fingerprint set based on the degree of matching between the current fault fingerprint and each historical fault fingerprint in the fault fingerprint set maintained by the server, wherein the fault fingerprint set includes one or more historical fault fingerprints from other clients. The diagnostic result acquisition module is configured to obtain a recommended diagnostic result corresponding to the current fault fingerprint based on the one or more similar fault fingerprints; and The sending module is configured to send the recommended diagnostic results to the client.

26. A computer device, characterized in that, The computer device includes: At least one processor; A memory having a computer program stored thereon, wherein, when executed by the at least one processor, the computer program causes the at least one processor to perform the method of any one of claims 1 to 14.

27. A computer device, characterized in that, The computer device includes: At least one processor; A memory having a computer program stored thereon, wherein, when executed by the at least one processor, the computer program causes the at least one processor to perform the method of any one of claims 15 to 23.

28. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to perform the method of any one of claims 1 to 14.

29. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to perform the method of any one of claims 15 to 23.

30. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, causes the processor to perform the method according to any one of claims 1 to 14.

31. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, causes the processor to perform the method of any one of claims 15 to 23.

Images