GPU server fault pre-diagnosis system and self-healing method

Abstract

The invention provides a GPU server fault pre-diagnosis system and a self-healing method. The method comprises the following steps: periodically collecting operation data of a GPU, calculating a window average value for SM utilization rate and power consumption, calculating a window accumulated value for ECC error count, and calculating a power consumption change slope; constructing a state vector based on the operation data, and performing normalization processing to obtain an operation state vector; inputting the running state vector into a pre-trained auto-encoder model, and calculating a mean square error of the state vector and a reconstruction vector as a reconstruction error; adding the reconstruction error and the regular term to obtain a risk score; calculating a benefit score of each repair action based on the risk score, the operation data and the environment context information; and executing a selected repair action, the repair action including holding, underclocking, migration or resetting, and updating the node state flag according to an execution result.

Description

Technical Field

[0001] This invention belongs to the field of computer equipment technology, and in particular relates to a GPU server fault diagnosis system and self-healing method. Background Technology

[0002] With the rapid development of artificial intelligence, big data, and high-performance computing, GPU servers have become the core of computing power in scientific research, cloud computing, and enterprise data centers. Under prolonged high-load operation, GPU servers often face hidden risks such as power consumption fluctuations, abnormal temperature control, hardware aging, and memory errors. These risks often do not immediately trigger system errors but accumulate gradually, eventually leading to serious failures. Once a GPU node malfunctions, it is often characterized by its sudden onset, rapid propagation, and high difficulty in localization, easily causing training task interruptions or business system paralysis. Current GPU server fault handling primarily relies on manual troubleshooting by operations and maintenance personnel, supplemented by log analysis, temperature monitoring, or hardware self-testing tools. This method is not only slow to react but also prone to misjudgment and missed detection in multi-GPU parallel and cross-platform deployment scenarios due to the complex coupling between fault characteristics. While some platforms have attempted to introduce threshold-based automated alarm mechanisms, the thresholds are difficult to adapt to different tasks and hardware environments, often failing to identify GPUs in an early, degraded "sub-healthy" state, and lacking a complete fault handling loop. Therefore, current technology cannot provide a complete solution for GPU servers that can take into account early diagnosis, refined repair and system self-healing, and it is difficult to meet the stability and continuity requirements of high-performance computing environments. Summary of the Invention

[0003] The purpose of this invention is to design a GPU server fault diagnosis system and self-healing method that can identify the sub-health state of the GPU, generate appropriate repair paths in response to different deployment environments and task types, and improve the stability of operation and the overall recovery capability of the system through a status marking mechanism, thereby realizing intelligent management of GPU servers from diagnosis to self-healing in engineering.

[0004] To achieve the above objectives, a method for pre-diagnosis and self-healing of GPU server faults is provided in a first aspect of the present invention, the method comprising: The GPU's operating data is periodically collected, including SM utilization, ECC error count, power consumption readings, and cumulative running time data; the average value of the SM utilization and power consumption is calculated within a window, the cumulative value of the ECC error count is calculated within a window, and the slope of power consumption change is calculated; a state vector is constructed based on the operating data and normalized to obtain the operating state vector; The running state vector is input into the pre-trained autoencoder model, and the mean square error between the state vector and the reconstruction vector is calculated as the reconstruction error. The reconstruction error is added to a regularization term to obtain a risk score, where the regularization term is the weighted sum of squares of the cumulative ECC error value and the slope of the power consumption change. Based on risk scores, operational data, and environmental context information, the benefit score of each remediation action is calculated; the benefit score is a weighted combination of risk score, task importance, resource sufficiency, and action cost; the action with the highest benefit score is selected as the output. Perform the selected repair action, which includes hold, downclocking, migration, or reset, and update the node status flag based on the execution result.

[0005] Furthermore, the specific steps for periodic data acquisition are as follows: set a primary data acquisition period, and within the primary data acquisition period, sample at time intervals shorter than the primary data acquisition period to form multiple sets of sampled data sequences within a time window.

[0006] Furthermore, the autoencoder model has a symmetric neural network structure, in which the encoder part progressively compresses the input state vector into a low-dimensional latent space, while the decoder part progressively reconstructs the original vector from the latent representation; both the encoder and decoder are implemented using multi-layer fully connected neural networks.

[0007] Furthermore, the environmental context information specifically includes: a task importance score provided by the cluster task scheduling system, which identifies the criticality of the task currently being run by the GPU, and a load margin assessment value provided by the resource management platform, which reflects the current remaining schedulable computing resources of the corresponding GPU node.

[0008] Furthermore, in the calculation of the benefit score, the weights of each factor are configured according to the following rules: risk score occupies the dominant weight, task importance occupies the secondary weight, and resource adequacy occupies the auxiliary weight, thereby reflecting the decision-making logic of prioritizing risk while taking into account the impact on business.

[0009] Furthermore, the action cost is set based on the potential impact of different operations on system continuity and task stability. The numerical relationships from high to low are: reset operation, migration operation, frequency reduction operation, and hold operation; among them, the hold operation has no cost.

[0010] Furthermore, the repair actions are performed by calling the corresponding system interfaces: frequency reduction is performed by calling the device management command-line tool provided by the GPU manufacturer to adjust the core frequency; migration is performed by calling the application programming interface of the container orchestration platform or job scheduling system to evict and reschedule tasks; and reset is performed by calling the device control interface provided by the GPU driver layer to perform a soft reset function.

[0011] Furthermore, the update rules for node status markers are as follows: if the selected action is to keep, it is marked as idle; if the action is successfully executed, it is marked as executed; if the action fails during execution due to system reasons, it is marked as failed; if the action is not executed due to external conditions, it is marked as skipped. This marker is used in subsequent cycles to determine whether a retry or upgrade operation is needed.

[0012] Furthermore, the application of risk scoring includes setting grading thresholds: when the score is below the first threshold, the GPU is considered to be in a healthy state; when the score is between the first and second thresholds, the GPU is determined to be in a sub-healthy state and requires attention; when the score is above the second threshold, the GPU is determined to be in a high-risk state and the repair mechanism should be triggered first.

[0013] A second aspect of the present invention provides a GPU server fault prediction system, the system comprising: The data acquisition module is used to periodically acquire GPU operating data, including SM utilization, ECC error count, power consumption readings, and cumulative running time data; calculate the window average value for SM utilization and power consumption, calculate the window cumulative value for ECC error count, and calculate the power consumption change slope; construct a state vector based on the operating data and perform normalization processing to obtain the operating state vector; The risk assessment module is used to input the running state vector into the pre-trained autoencoder model, calculate the mean square error between the state vector and the reconstruction vector as the reconstruction error, and add the reconstruction error to a regularization term to obtain a risk score, wherein the regularization term is a weighted sum of squares of the cumulative ECC error value and the power consumption change slope. The action generation module is used to calculate the benefit score of each repair action based on risk score, operational data and environmental context information; the benefit score is a weighted combination of risk score, task importance, resource sufficiency and action cost; the action with the highest benefit score is selected as the output; The action execution module is used to execute selected repair actions, including hold, reduce frequency, migrate or reset, and update the node status flags based on the execution results.

[0014] The beneficial technical effects of the present invention are at least as follows: To address the aforementioned issues, this invention provides a GPU server fault prediction system and self-healing method. It constructs a state vector by collecting key underlying GPU operational data to represent the GPU's operational characteristics using a unified structure. Unsupervised modeling is then used to score the risk of this state vector, enabling early identification of potential degradation trends. Based on this, repair actions are generated by combining risk scores and operational environment information. The optimal choice is made based on the cost and task attributes of each action, ensuring appropriate and reasonable control strategies are adopted at different risk levels. Finally, the system translates the selected repair actions into specific control commands for execution and writes state tags to the GPU nodes to prevent repeated interventions in subsequent cycles, thus achieving a complete closed loop of perception, judgment, decision-making, and execution. Compared to traditional threshold- or rule-based solutions, this invention not only identifies the sub-health state of the GPU but also adaptively generates suitable repair paths for different deployment environments and task types. Furthermore, the state tagging mechanism improves operational stability and overall system recovery capabilities, thereby achieving intelligent management of GPU servers from prediction to self-healing in engineering applications. Attached Figure Description

[0015] The present invention will be further described with reference to the accompanying drawings, but the embodiments in the drawings do not constitute any limitation on the present invention. For those skilled in the art, other drawings can be obtained based on the following drawings without creative effort.

[0016] Figure 1 This is a flowchart of the GPU server fault pre-diagnosis and self-healing method of the present invention.

[0017] Figure 2 This is a framework diagram of the GPU server fault prediction system of the present invention. Detailed Implementation

[0018] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0019] In one or more embodiments, such as Figure 1 As shown, a method for pre-diagnosis and self-healing of GPU server faults is disclosed, the method comprising the following: S1: Periodically collect GPU operating data, including SM utilization, ECC error count, power consumption readings, and cumulative operating time data; calculate the window average value for SM utilization and power consumption, calculate the window cumulative value for ECC error count, and calculate the power consumption change slope; construct a state vector based on the operating data and perform normalization processing to obtain the operating state vector; Specifically, the goal of this step is to construct a structured input for the subsequent risk scoring model, namely the running state vector of each GPU in the current time period. This state vector needs to possess two key characteristics: first, it should reflect, as far as possible, the potential risks of performance degradation and instability during GPU operation; second, it should have a simple structure and low computational cost, facilitating high-frequency acquisition and rapid processing in practical systems. The perceived data of GPU operating status is mainly obtained by calling the monitoring interface of the GPU driver layer. In an NVIDIA hardware-based deployment environment, DCGM (DataCenterGPUManager) is used as the primary acquisition channel. In the actual acquisition logic, the system runs the data acquisition process as a periodic task (a period of 60 seconds is recommended), sampling once every 1 second within each period, forming an observation window with a length of [missing information]. The time series of the indicators. The key indicators involved in the data collection are as follows: SM utilization sequence The `dcgmFieldGroupGetLatestValues` or `dcgmGetValuesSince` command retrieves a sequence of metrics with the field ID `DCGM_FI_PROF_SM_ACTIVE`. These metrics represent the activity ratio of each computing unit on the GPU per unit of time, typically ranging from 0 to 100. For example, during a peak training task, the A100 card's... The sequence may be: 85,88,92,95,91,90,...,87.

[0020] ECC Error Count Use dcgmGetLatestValuesForFields with field ID DCGM_FI_DEV_ECC_SBE_AGG_TOTAL (configurable to aggregation methods such as SBE and DBE) to collect the cumulative number of ECC single-bit error events per second. This value is 0 most of the time, but may suddenly increase when the GPU enters a high-temperature or high-bandwidth conflict state, for example: [0,0,1,0,2,0,3,...].

[0021] Power consumption readings The current GPU power (in milliwatts) is collected using nvmlDeviceGetPowerUsage(), and recorded as a time series by the number of calls per second. Under high load, power consumption is usually stable between 80% and 100% of the set power limit. Some hardware problems (such as fan failure) can cause abnormal power consumption drift.

[0022] GPU cumulative runtime This value is calculated by the difference between the GPU device's power-on time recorded by the system (e.g., via the uptime command or the bootTimestamp field in the driver) and the current time. This value does not change with task fluctuations, but for devices that have been running for extended periods, it can serve as an important reference for measuring the degree of hardware aging.

[0023] After data collection, the system performs in-window averaging on the three time series sets (for each time series set). , ) and cumulative summation (for ), and calculate the power consumption sequence. The first-order difference slope is used to construct the enhanced state vector. The definition is as follows: ; The variables are explained below: This is the window average of SM utilization, derived from the field DCGM_FI_PROF_SM_ACTIVE; This is the cumulative value for ECC errors, derived from the field DCGM_FI_DEV_ECC_SBE_AGG_TOTAL; The power consumption is the average value of the series, derived from the interface nvmlDeviceGetPowerUsage(). The cumulative GPU runtime is calculated from the system time and the device power-on timestamp. The linear slope of the power consumption sequence within the time window is calculated as follows: ; in: This represents the power consumption sample value at the end of the time window; The power consumption sample value at the start of the time window; This is the number of sampling points within the sampling window, which is fixed at 60.

[0024] Each variable comes from the standard driver interface and does not depend on external devices or middleware. For example, A value of 2.5 indicates that the average power consumption increased by 2.5 milliwatts per second in the past minute. If it is much higher than the historical operating value, it can be regarded as an early sign of abnormal temperature rise.

[0025] The aforementioned indicators differ in physical dimensions. After generating the state vector, the system performs uniform normalization on each dimension, ensuring comparability in numerical scale between different indicators and avoiding bias in subsequent model learning. All processing operations are completed in the configuration file set before deployment, eliminating the need for online automatic adjustments and ensuring stable system operation.

[0026] Finally, the state vector This data will serve as direct input to the risk scoring model in subsequent steps, requiring no further data transformation or aggregation. This architecture offers excellent deployment adaptability, suitable for real-time sensing at scales ranging from single-GPU servers and dual-GPU nodes to entire rack-wide GPU clusters.

[0027] S2: Input the running state vector into the pre-trained autoencoder model, calculate the mean square error between the state vector and the reconstruction vector as the reconstruction error; add the reconstruction error to the regularization term to obtain the risk score, wherein the regularization term is the weighted sum of squares of the cumulative ECC error value and the power consumption change slope; Specifically, this step aims to utilize the GPU state vector constructed in the previous step. The system assigns a sub-health risk score to each GPU in the current cycle. The score measures the deviation between the current state and historical healthy behavior, thereby identifying potential degenerative states that have not yet triggered system errors. This score serves as direct input to subsequent policy modules, determining whether to execute self-healing actions, and therefore holds a central position in the entire system.

[0028] In actual operation and maintenance scenarios of GPU servers, traditional monitoring systems often rely on static thresholds to issue abnormal alarms. However, the early degradation state of the GPU may not directly cause "hard failure" indicators such as overheating or ECC outbreak. Therefore, the innovation of this step lies in building a scoring mechanism that can quantify slight abnormal trends and has adaptive learning capabilities, without the need for fault labels to participate in training.

[0029] The input is a state vector. The meanings and sources of the five variables were fully defined and uniformly named in step one. In the model design, this step employs an improved symmetric autoencoder, whose encoder and decoder both consist of two fully connected neural networks (dimensional structure 5→3→2→3→5). During the model training phase, only healthy GPU states from historical sampling are used as input, with the objective function being minimizing the reconstruction error; after deployment, only inference is performed.

[0030] To enhance the "directional recognition" capability of GPU state deviations, this step introduces a risk-sensitive regularization term based on the reconstruction error. This term improves the model's ability to perceive the weights of certain key indicators (such as ECC growth and power consumption fluctuations) by penalizing the degree of deviation in a specific dimension, thus reflecting the engineering perception of GPU fault characteristics.

[0031] Risk Score The final calculation method is as follows: ; ; in: and The pre-trained encoder and decoder networks each consist of two fully connected layers, stacked non-linearly without activation functions. For the model to state vector The reconstructed output; For the first Reconstruction error of each indicator; The regularization strength coefficient is determined by optimization on the training set before deployment. As an explicit penalty for the increase in ECC errors, it means that the contribution of ECC volatility to the risk score should be much higher than other dimensions; This is a penalty term for the slope of power consumption fluctuations, used to amplify trend anomalies caused by temperature control issues; All variables have been normalized in step one, so no unit conversion is needed and the model runs in a uniform scale space.

[0032] This structure enables the model to maintain its general reconstruction error judgment capability while possessing scene-customized risk enhancement perception capabilities. For example, ECC micro-jitter will be prioritized for detection and its impact on risk scoring will be amplified. The introduction of this regularization term is an engineering enhancement to address the typical characteristic of GPU servers often exhibiting "latent error accumulation leading to sudden failures" during the training of high-concurrency AI tasks or video rendering.

[0033] In actual inference, the deployment system performs state vector acquisition (step one) and score calculation on all GPU nodes once per minute. Scoring results. It is a floating-point number, usually located in Interval (the specific range varies with the normalization scale and) (Settings vary). Based on historical deployment statistics and experience, settings can be configured as follows: Determined to be in a sub-healthy state. Nodes identified as high-risk are used as input criteria for the strategy module.

[0034] Specifically, a certain GPU state vector Model output ,set up Then the calculation yields: ; Output rating If the value is significantly lower than the set threshold, the GPU node is determined to be in a healthy state.

[0035] S3: Calculate the benefit score of each remediation action based on risk score, operational data, and environmental context information; the benefit score is a weighted combination of risk score, task importance, resource sufficiency, and action cost; select the action with the highest benefit score as the output; Specifically, this step will use the GPU risk score output from step two. By jointly modeling with the original state variables partially retained in step one, a structured and executable action decision is formed. This will guide whether subsequent operations such as migration, downclocking, or reset of the GPU node are necessary.

[0036] This step is specifically designed for the "high task density, high computing pressure, and limited resources" operating characteristics of GPU servers. Its goal is to adopt the most appropriate and economical remediation method for different GPUs, given the known risks. To this end, the solution introduces several policy decision factors that are valuable only in GPU scenarios, such as: task execution importance. GPU resource load margin Operation cost function wait.

[0037] Additional input for this step: Some metrics from the state vector, including cumulative running time. ECC error accumulation Power consumption change rate ; Contextual information: such as the importance of the GPU's current task (Job tags from the scheduling system), GPU load margin (Estimated amount of schedulable resources from the scheduling platform).

[0038] All input values ​​have been normalized, supporting policy scoring modeling on a uniform scale.

[0039] To achieve strategy structuring, risk sensitivity, and cost perception, this step introduces an innovative action benefit scoring function. It is used to measure the actions performed in the current state. Is it worthwhile? The function is designed as follows: ; in: Indicated in GPU Execute the action Relative return score; A risk score is assigned, with higher scores indicating a more dangerous GPU. Rate the importance of the current task; the higher the rating, the more critical the task. This represents the current resource redundancy of the GPU. A higher value indicates that the GPU still has redundant resources and there is no need to operate on it immediately. The cost function for this action is, for example, the cost of resetting is much higher than the cost of downclocking; Weighting coefficients are set before deployment, such as... .

[0040] This formula models risk, task value, and resource status together, innovatively incorporating an "action cost function." By directly incorporating it into the scoring system, the system can identify risks, consider task value and system burden when making action selections, and balance safety and stability.

[0041] In actual operation, the system assigns each GPU node a set of optional actions. Scores are calculated separately above. Select the action with the highest score as the output action. : ; This formula provides a deployable optimal standard for action selection, moving away from a "hard segmentation based on risk thresholds" and instead seeking the most suitable operational path from the perspective of "maximizing returns." Specifically: like high, Low (task is not important). If resources are plentiful, the system tends to choose Reset. like high, high, When the threshold is low, the system tends to choose Throttle or Migrate rather than a direct Reset; like If it is low, then regardless of other values, Normally, maximize the repair speed to avoid frequent repairs; Action Cost The following settings can be preset: ; ; ; ; The aforementioned cost design is derived from the execution costs of actual GPU operations. For example, Reset involves driver calls and context loss, Migrate involves scheduling platform coordination, Throttle only involves changes to driver call parameters, and Hold has no system calls. This structure can be uniformly configured before any deployment, requiring no training and having no feedback dependencies.

[0042] Specifically, the state of a GPU node in the current cycle is: , , Then the four actions They are respectively: ; ; ; ; Final action selection This indicates that although the GPU poses a moderate risk, the task's importance and resource availability suggest that immediate action is not necessary.

[0043] S4: Execute the selected repair action, which includes hold, down-frequency, migration or reset, and update the node status flag based on the execution result.

[0044] Specifically, the task of this step is to perform the GPU repair actions selected in step three of the previous step. Execute the system-level control command and record the status flag of the operation on the GPU node. This information is used for subsequent periodic assessments. This step does not involve any scoring, modeling, or policy reasoning; it only performs actions and state updates.

[0045] The input is the repair action output from the previous step. Its type is an enumeration, and its value set is The definition is as follows: No action is taken. : Perform GPU downclocking operation; Migrate the current task to another GPU node; Perform a soft reset on the GPU.

[0046] The system receives Then, execution logic is distributed based on the action type. Execution utilizes the control interface provided by the system scheduling platform, such as nvidia-smi, Kubernetes, Slurm, DCGM plugins, etc., specifically implemented as follows: Hold: Skip this GPU node; no call is made. Throttle: Invokes nvidia-smi-lgc<low_clock> The instruction will reduce the GPU core frequency; Example: nvidia-smi-i2-lgc800,800 locks the frequency of GPU number 2 to 800MHz; Migrate: Calls the scheduling system API, such as kubectldrainnode-x in Kubernetes, to release node tasks; Example: kubectlcordonnode-gpu3+kubectldrainnode-gpu3--ignore-daemonsets; Reset: Invokes nvidia-smi--gpu-reset-iX to perform a soft reset of GPU X; Example: nvidia-smi--gpu-reset-i2 performs a reset on GPU #2, ensuring that its tasks have been migrated out.

[0047] After each action is completed, the system determines whether it was successful based on the action's return value, system log status, or interface feedback, and writes this information to the GPU's node status flag. Its set of values ​​is: No action was taken. The operation has been successfully executed. Operation failed; : The execution was skipped by the scheduling system (e.g., task locking, driver not supported); Status records are stored in the GPU node management table, with the field structure {gpu_id, action, status, timestamp}. They can be written to the platform database or local persistent storage and are supported for platform calls and queries.

[0048] The node state flag update logic is defined as follows: ; in: This refers to the GPU operation action output by the strategy module in the previous step; The value represents the result of the operation, with 1 indicating success and 0 indicating failure. The state update function has the following structure: ; Specifically, if a GPU node receives If executing nvidia-smi-lgc800 returns a code of 0 (indicating success), then... If the execution of gpu-reset is blocked by the operating system, it will return a non-zero value. In the next cycle, a retry mechanism or policy downgrade will be triggered.

[0049] The output consists of two fields: Control command execution result A boolean value used for logging and fault auditing; GPU node status marking This information is used by the next cycle's strategy module to determine whether to skip, retry, or upgrade the process.

[0050] In one or more embodiments, such as Figure 2 As shown, a GPU server fault diagnosis and self-healing system is disclosed, the system comprising: The data acquisition module is used to periodically acquire GPU operating data, including SM utilization, ECC error count, power consumption readings, and cumulative running time data; calculate the window average value for SM utilization and power consumption, calculate the window cumulative value for ECC error count, and calculate the power consumption change slope; construct a state vector based on the operating data and perform normalization processing to obtain the operating state vector; The risk assessment module is used to input the running state vector into the pre-trained autoencoder model, calculate the mean square error between the state vector and the reconstruction vector as the reconstruction error, and add the reconstruction error to a regularization term to obtain a risk score, wherein the regularization term is a weighted sum of squares of the cumulative ECC error value and the power consumption change slope. The action generation module is used to calculate the benefit score of each repair action based on risk score, operational data and environmental context information; the benefit score is a weighted combination of risk score, task importance, resource sufficiency and action cost; the action with the highest benefit score is selected as the output; The action execution module is used to execute selected repair actions, including hold, reduce frequency, migrate or reset, and update the node status flags based on the execution results.

[0051] It is worth noting that the specific workflow of the GPU server fault diagnosis and self-healing system provided in this embodiment of the invention is the same as that of the GPU server fault diagnosis and self-healing method described in the above embodiments, and will not be repeated here.

[0052] This invention also provides a GPU server fault prediction and self-healing device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the steps described in the above-described GPU server fault prediction and self-healing method embodiments, for example... Figure 1 The steps S1 to S4 described above; or, when the processor executes the computer program, it implements the functions of each module in the above system embodiments.

[0053] For example, the computer program may be divided into one or more modules, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules may be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the GPU server fault diagnosis and self-healing device.

[0054] The GPU server fault diagnosis and self-healing device can be a computing device such as a desktop computer, laptop, handheld computer, or cloud server. The GPU server fault diagnosis and self-healing device may include, but is not limited to, processors and memory. Those skilled in the art will understand that the GPU server fault diagnosis and self-healing device may also include input / output devices, network access devices, buses, etc.

[0055] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASACs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. This processor is the control center of the GPU server fault diagnosis and self-healing device, connecting all parts of the device via various interfaces and lines.

[0056] The memory can be used to store the computer programs and / or modules. The processor, by running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory, realizes various functions of the GPU server fault diagnosis and self-healing device. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function, etc.; the data storage area may store data created based on the operation of the air conditioner controller, etc. In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0057] If the modules integrated into the GPU server fault diagnosis and self-healing device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0058] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0059] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for fault diagnosis and self-healing of GPU servers, characterized in that, The method includes: The GPU's operating data is periodically collected, including SM utilization, ECC error count, power consumption readings, and cumulative running time data; the average value of the SM utilization and power consumption is calculated within a window, the cumulative value of the ECC error count is calculated within a window, and the slope of power consumption change is calculated; a state vector is constructed based on the operating data and normalized to obtain the operating state vector; The running state vector is input into the pre-trained autoencoder model, and the mean square error between the state vector and the reconstruction vector is calculated as the reconstruction error. The reconstruction error is added to a regularization term to obtain a risk score, where the regularization term is the weighted sum of squares of the cumulative ECC error value and the slope of the power consumption change. Based on risk scores, operational data, and environmental context information, the benefit score of each remediation action is calculated; the benefit score is a weighted combination of risk score, task importance, resource sufficiency, and action cost; the action with the highest benefit score is selected as the output. Perform the selected repair action, which includes hold, downclocking, migration, or reset, and update the node status flag based on the execution result.

2. The GPU server fault diagnosis and self-healing method according to claim 1, characterized in that, The specific steps of the periodic acquisition are as follows: set a main data acquisition period, and within the main data acquisition period, sample at time intervals shorter than the main data acquisition period to form multiple sets of sampled data sequences within a time window.

3. The GPU server fault diagnosis and self-healing method according to claim 1, characterized in that, The autoencoder model has a symmetrical neural network structure. Its encoder part gradually compresses the input state vector into a low-dimensional latent space, while the decoder part gradually reconstructs the original vector from the latent representation. Both the encoder and decoder are implemented using multi-layer fully connected neural networks.

4. The GPU server fault diagnosis and self-healing method according to claim 1, characterized in that, The environmental context information specifically includes: a task importance score provided by the cluster task scheduling system, which identifies the criticality of the task currently being run by the GPU, and a load margin assessment value provided by the resource management platform, which reflects the current remaining schedulable computing resources of the corresponding GPU node.

5. The GPU server fault diagnosis and self-healing method according to claim 1, characterized in that, In the calculation of the benefit score, the weights of each factor are configured according to the following rules: risk score has a dominant weight, task importance has a secondary weight, and resource adequacy has a secondary weight, thereby reflecting the decision-making logic of prioritizing risk while taking into account the impact on business.

6. The GPU server fault diagnosis and self-healing method according to claim 1, characterized in that, The cost of the operation is set based on the potential impact of different operations on the continuity of the system and the stability of the task. The numerical relationship from high to low is as follows: reset operation, migration operation, frequency reduction operation, and hold operation; among them, the hold operation has no cost.

7. The GPU server fault diagnosis and self-healing method according to claim 1, characterized in that, The repair actions are performed by calling the corresponding system interfaces: frequency reduction is performed by calling the device management command-line tool provided by the GPU manufacturer to adjust the core frequency; migration is performed by calling the application programming interface of the container orchestration platform or job scheduling system to evict and reschedule the task; and reset is performed by calling the device control interface provided by the GPU driver layer to perform a soft reset function.

8. The GPU server fault diagnosis and self-healing method according to claim 1, characterized in that, The update rule for the node status marker is as follows: if the selected action is to hold, it is marked as idle; if the action is successfully executed, it is marked as executed; if the action fails during execution due to system reasons, it is marked as failed; if the action is not executed due to external conditions, it is marked as skipped. This marker is used in subsequent cycles to determine whether a retry or upgrade operation is needed.

9. The GPU server fault diagnosis and self-healing method according to claim 1, characterized in that, The application of the risk score includes setting grading thresholds: when the score is below the first threshold, the GPU is considered to be in a healthy state; when the score is between the first and second thresholds, the GPU is determined to be in a sub-healthy state and requires attention; when the score is above the second threshold, the GPU is determined to be in a high-risk state and the repair mechanism should be triggered first.

10. A GPU server fault diagnosis and self-healing system, characterized in that, The system includes: The data acquisition module is used to periodically acquire GPU operating data, including SM utilization, ECC error count, power consumption readings, and cumulative running time data; calculate the window average value for SM utilization and power consumption, calculate the window cumulative value for ECC error count, and calculate the power consumption change slope; construct a state vector based on the operating data and perform normalization processing to obtain the operating state vector; The risk assessment module is used to input the running state vector into the pre-trained autoencoder model, calculate the mean square error between the state vector and the reconstruction vector as the reconstruction error, and add the reconstruction error to a regularization term to obtain a risk score, wherein the regularization term is a weighted sum of squares of the cumulative ECC error value and the power consumption change slope. The action generation module is used to calculate the benefit score of each repair action based on risk score, operational data and environmental context information; the benefit score is a weighted combination of risk score, task importance, resource sufficiency and action cost; the action with the highest benefit score is selected as the output; The action execution module is used to execute selected repair actions, including hold, reduce frequency, migrate or reset, and update the node status flags based on the execution results.

Images