A self-healing GPU firmware method
By using a self-healing GPU firmware method, real-time monitoring and intelligent diagnosis of GPU hardware status are achieved, and hardware faults are automatically repaired. This solves the problems of detection lag and manual reliance in existing technologies, and improves the reliability and operational efficiency of the system.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SUZHOU SUXIAN MICROELECTRONICS TECH CO LTD
- Filing Date
- 2026-05-13
- Publication Date
- 2026-07-31
AI Technical Summary
Existing GPU firmware technology lags behind in hardware anomaly detection, lacks real-time monitoring and intelligent diagnostic capabilities, relies on manual repair, resulting in high maintenance costs and long system downtime, and lacks remote operation and maintenance support, making it difficult to meet the needs of highly reliable and highly available computing systems.
It employs a combination of hardware status real-time monitoring module, anomaly detection and diagnosis module, dynamic self-repair execution module, operation log and analysis module, and remote maintenance and update module to achieve real-time monitoring, intelligent diagnosis and automated repair of GPU hardware status, and supports remote operation and maintenance.
It enables early detection and rapid repair of GPU hardware failures, reduces system downtime and maintenance costs, improves operational efficiency and system reliability, and adapts to complex and ever-changing failure scenarios.
Smart Images
Figure CN122489334A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of firmware management and maintenance of graphics processing units (GPUs), specifically a self-healing GPU firmware method. Background Technology
[0002] Graphics Processing Units (GPUs), as core computing units, play a crucial role in high-performance computing scenarios such as deep learning training and inference, high-performance graphics rendering, and large-scale scientific computing. With the increasing complexity and duration of computational tasks, GPU hardware operates under high load and high power consumption for extended periods, significantly increasing the risk of hardware failure. Common hardware anomalies include localized failures of computing cores due to electromigration or thermal fatigue, bit errors or physical damage to memory units caused by high-frequency access, and transmission anomalies in high-speed communication links due to signal integrity degradation.
[0003] However, existing GPU firmware technology has significant shortcomings in addressing these hardware anomalies. First, its fault detection capabilities are lagging. Traditional firmware typically lacks fine-grained, continuous hardware status monitoring capabilities, making it difficult to promptly capture transient anomalies or precursors to performance degradation. Often, these anomalies are only detected when they accumulate to the point of causing explicit functional failure or system crashes, missing the crucial opportunity for early intervention. Second, the repair process is highly dependent on manual intervention. Once a hardware fault is detected, its diagnosis, localization, and repair heavily rely on on-site intervention by professional technicians. Whether through physical replacement of hardware modules or complex low-level debugging, both result in high system maintenance costs and prolonged system service interruptions due to manual response and operation time. Third, remote operation and maintenance support is weak. In modern distributed computing clusters or edge computing node deployment environments, devices are widely distributed, making on-site maintenance extremely difficult. Existing firmware architectures generally lack stable and secure remote access and control interfaces, hindering efficient remote diagnosis, fault recovery, or policy adjustment, thus limiting overall operation and maintenance efficiency. Finally, intelligent diagnostic and predictive capabilities are lacking. Existing solutions mostly rely on static rules or simple thresholds for anomaly detection, failing to fully utilize the large amount of state data generated during GPU runtime. They also lack the ability to deeply mine fault modes and perform predictive maintenance based on advanced algorithms such as machine learning, resulting in a single repair strategy that is difficult to adapt to complex and ever-changing real-world fault scenarios.
[0004] In summary, existing technologies are insufficient to meet the operational and maintenance requirements of highly reliable and available GPU computing systems. Therefore, there is an urgent need in the field for a self-healing GPU firmware method that enables real-time hardware status monitoring, possesses intelligent diagnostic and predictive capabilities, supports automated repair, and integrates remote operation and maintenance functions, in order to significantly improve the reliability, availability, and maintainability of GPU clusters. Summary of the Invention
[0005] The purpose of this invention is to solve the aforementioned technical problems and provide a self-healing GPU firmware method. The technical solution adopted by this invention is as follows: A self-healing GPU firmware method, characterized in that the method is implemented collaboratively by the following modules:
[0006] The hardware status real-time monitoring module is used to periodically collect the operating status data of the internal hardware units of the GPU through the firmware layer interface.
[0007] An anomaly detection and diagnosis module is connected to the real-time hardware status monitoring module and is used to perform online analysis of the operating status data and identify and locate hardware faults using an integrated anomaly detection algorithm.
[0008] A dynamic self-repair execution module is connected to the anomaly detection and diagnosis module and is used to automatically perform repair operations based on the diagnosis results.
[0009] The operation log and analysis module is connected to the hardware status real-time monitoring module, the anomaly detection and diagnosis module, and the dynamic self-repair execution module. It is used to record hardware status, abnormal events and repair operations, and perform trend analysis based on historical data.
[0010] The remote maintenance and update module is connected to the operation log and analysis module and the dynamic self-healing execution module, and is used to realize remote firmware updates, repair strategy adjustments and status reports.
[0011] Preferably, in the real-time hardware status monitoring module, the status data acquisition adopts a combination of polling and event triggering. The acquired data includes the task queue status of the streaming multiprocessor of the computing core, the health status of the execution unit and the error count; the ECC correction count, memory access latency and bad page table information of the video memory unit; and the signal integrity, data packet retransmission rate and bandwidth utilization of the high-speed bus or high-speed interconnect link of the peripheral components of the communication interface.
[0012] Preferably, in the anomaly detection and diagnosis module, the integrated anomaly detection algorithm includes a threshold detection method based on statistical process control, an outlier detection method based on isolated forests, and a time-series prediction and anomaly detection method based on long short-term memory networks; fault location is achieved by analyzing the correlation between error patterns and hardware topology, specifically: when a sustained high error rate of a specific SM is detected, it is determined that the kernel unit has a local failure; when a concentrated outbreak of ECC errors in a specific address range of video memory is detected, it is determined that the physical memory area is unstable; when the bit error rate of the communication link exceeds a threshold, it is determined that the link quality has deteriorated.
[0013] Preferably, kernel task remapping: dynamically migrates the queued computing tasks on the failed streaming multiprocessor to other healthy streaming multiprocessors within the same graphics processor to continue execution;
[0014] Memory Failure Area Masking: Physical memory pages that frequently experience ECC errors are marked as unavailable and isolated from the graphics processor's memory address space, while their storage responsibilities are taken over by a firmware-managed spare memory area.
[0015] Communication link soft reset: Performs protocol layer reinitialization on the high-speed bus or high-speed interconnect link of peripheral components that has experienced persistent bit errors, and restores its communication state.
[0016] Preferably, in the operation log and analysis module, the log records adopt a structured format, including timestamps, hardware unit identifiers, status indicators, anomaly levels, repair actions, and results; the trend analysis identifies the precursor patterns of failures by analyzing historical logs, dynamically adjusts the detection threshold and sensitivity of the anomaly detection and diagnosis module, and provides predictive repair suggestions for the dynamic self-repair execution module.
[0017] Preferably, the remote maintenance and update module supports secure network communication protocols, can receive firmware image updates and repair strategy parameter adjustment instructions from the remote management server, and can proactively report system status summaries, key alarms, and log analysis results to the remote server.
[0018] Preferably, before performing kernel task remapping, memory fault area masking, or communication link soft reset operations, the dynamic self-repair execution module will assess the risk of the repair operation and the current system load. For high-risk operations, a second confirmation is required based on historical success rates and the criticality of the current task, or the operation may be performed during task intervals.
[0019] Preferably, the anomaly detection and diagnosis module and the operation log and analysis module constitute a closed-loop learning system. By continuously analyzing the success rate of repair operations and subsequent changes in hardware status, the system continuously optimizes the model parameters and repair strategies in the integrated anomaly detection algorithm using the results of the trend analysis.
[0020] Preferably, the sampling frequency of the real-time hardware status monitoring module can be dynamically adjusted according to the system load and the historical fault frequency obtained from the operation log and analysis module, and the sampling frequency can be automatically increased during high load or high risk periods.
[0021] Preferably, a self-healing GPU firmware method is characterized by comprising the following steps:
[0022] S1: The hardware status real-time monitoring module periodically collects the operating status data of the GPU computing core, video memory and communication interface;
[0023] S2: The anomaly detection and diagnosis module performs real-time analysis of the collected status data, uses an integrated anomaly detection algorithm to determine whether there is a hardware anomaly, and locates the faulty hardware unit.
[0024] S3: If an anomaly is detected, the dynamic self-repair execution module automatically performs the corresponding repair operation according to the fault type and location. The repair operation includes kernel task remapping, video memory fault area masking, or communication link soft reset.
[0025] S4: The operation log and analysis module records the entire process from status acquisition and anomaly detection to repair execution, and performs in-depth analysis based on historical log data to optimize future detection thresholds and repair strategies;
[0026] S5: The remote maintenance and update module provides a remote interface to support firmware updates, parameter configuration, and status monitoring, enabling remote operation and maintenance management of the system.
[0027] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:
[0028] 1. This invention continuously collects deep internal state data of the graphics processor through a real-time hardware status monitoring module, enabling early detection of potential hardware faults and overcoming the problem of delayed detection in traditional firmware.
[0029] 2. Through the integration of multiple intelligent algorithms in the anomaly detection and diagnosis module, it can accurately identify and locate hardware anomalies in complex environments, with high diagnostic accuracy and reduced false alarms and missed alarms.
[0030] 3. Through the various automated repair methods provided by the dynamic self-repair execution module, such as kernel remapping and video memory masking, hardware functions can be quickly restored without manual intervention, significantly reducing system downtime and maintenance costs.
[0031] 4. Through the recording and analysis functions of the operation log and analysis module, a complete data closed loop is formed from fault occurrence, diagnosis, repair to strategy optimization, enabling the system to have the ability to continuously learn and self-optimize.
[0032] 5. The remote maintenance and update module enables centralized and remote management of distributed graphics processor devices, greatly improving the operation and maintenance efficiency in large-scale deployment scenarios. Attached Figure Description
[0033] Figure 1 This is a flowchart of the method steps in this invention;
[0034] Figure 2 This is a system module connection diagram of the present invention;
[0035] Figure 3This is a schematic diagram of the repair operation of the dynamic self-repair execution module in this invention. Detailed Implementation
[0036] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.
[0037] Please see Figure 1-3 This invention relates to a self-healing GPU firmware method, which aims to achieve real-time detection, intelligent diagnosis, and automated repair of GPU hardware anomalies. The specific implementation methods of this invention are described in detail below.
[0038] Real-time Hardware Status Monitoring Module: This module, as the foundation of the system, is responsible for collecting status data from various hardware units of the GPU. Specifically, it obtains the utilization rate of the computing core, instruction execution error count, and thread beam stall cycles by accessing the GPU's internal function registers, performance counters, and error reporting registers; it obtains the ECC error count of the memory controller, memory access latency distribution, and bad page information marked by the driver or hardware; and it obtains the bandwidth utilization, packet retransmission count, and link training status of communication interfaces such as high-speed buses or high-speed interconnect links. Data collection is performed at fixed intervals, and a data collection and reporting is immediately triggered when specific events such as a sudden increase in the error count are detected.
[0039] Anomaly Detection and Diagnosis Module: This module receives raw data from the monitoring module and performs real-time analysis. Multiple detection algorithms run in parallel within the module.
[0040] Threshold detection based on statistical process control: set dynamic thresholds for various indicators, and trigger an alarm when the threshold is exceeded.
[0041] Outlier detection based on isolated forests: used to discover anomalous states that are significantly different from normal operating modes, such as a sudden change in the power consumption or temperature pattern of a certain SM.
[0042] Temporal prediction and anomaly detection based on long short-term memory networks: Recurrent neural networks are used to learn the normal temporal patterns of hardware status indicators. When real-time data deviates significantly from the prediction range, it is judged as an anomaly.
[0043] The diagnostic process integrates the outputs of these algorithms and combines them with the hardware topology to make a comprehensive judgment, ultimately outputting the fault unit identifier and fault level.
[0044] Dynamic self-healing execution module: This module executes specific repair actions based on the diagnostic results. The repair strategy strictly corresponds to the fault type.
[0045] In the event of a kernel failure, kernel task remapping is performed: the GPU scheduler stops assigning new computing tasks to the SM marked as faulty and reschedules any unfinished tasks on that SM to other healthy SMs. This process is transparent to upper-layer applications.
[0046] For memory errors, a faulty memory region masking process is implemented: the firmware accesses the GPU's memory management unit, marks frequently failing physical memory pages or rows as bad blocks, and removes them from the system's available address space. Subsequent memory allocations will avoid these regions. Some high-end GPUs offer redundant memory units that can be enabled by firmware to replace faulty units.
[0047] In case of communication link anomalies, a soft reset of the communication link is performed: the firmware will trigger the retraining of the high-speed bus link of the peripheral component interconnect or the reinitialization of the high-speed interconnect link in an attempt to clear the temporary signal integrity problem and restore link stability.
[0048] Before performing a high-risk operation, the module will assess the current load on the graphics processor. If it is not a critical task period or if remote confirmation is received, it will be executed immediately; otherwise, it may be delayed until the system is idle.
[0049] Operation Log and Analysis Module: This module acts like a "black box," recording all critical events. Each log entry contains a precise timestamp, the hardware ID involved, raw status data, triggered anomaly alarms, executed repair operations, and their results. These structured logs are not only used for post-event auditing but, more importantly, for trend analysis. By analyzing historical data, the system can identify common characteristics preceding certain failures, thereby adjusting the collection frequency of the monitoring module or the thresholds of the anomaly detection module to achieve predictive maintenance.
[0050] Remote Maintenance and Update Module: This module provides the system with an external window. It runs a secure communication daemon that listens for commands from the remote management platform. Administrators can remotely push new firmware images to upgrade functionality, adjust the sensitivity parameters of anomaly detection, and request real-time GPU health status reports and detailed log files. The module can also proactively send alerts to higher levels in the event of a serious fault.
[0051] Example 1: Data Center GPU Server Scenario
[0052] In a large AI computing data center, a server equipped with eight high-performance computing GPUs was running a distributed training task. The real-time hardware status monitoring module detected a sharp increase in the instruction error count of the 14th streaming multiprocessor core unit of GPU 3 within a short period. The anomaly detection and diagnosis module determined that the 14th streaming multiprocessor had a potential hardware failure. The dynamic self-healing execution module immediately intervened, safely migrating the training subtask running on the 14th streaming multiprocessor to other idle streaming multiprocessors within GPU 3 to continue computation. Throughout the process, the training task experienced only a brief, millisecond-level pause and was not interrupted. The operation log and analysis module recorded the event and marked the 14th streaming multiprocessor as degraded. The remote operations and maintenance center received notification of this self-healing event, eliminating the need to dispatch engineers to the site.
[0053] Example 2: Edge Computing Device Scenario
[0054] In an edge computing device deployed in the field, its built-in GPU is running continuously. The memory controller begins reporting frequent ECC correction events within a specific address range. The anomaly detection and diagnosis module determines, based on rules, that the reliability of this memory region has deteriorated. The dynamic self-healing execution module then masks the physical memory page and replaces it with a spare memory region. The operation log and analysis module records the masked memory address range and generates a diagnostic report. Through the remote maintenance and update module, maintenance personnel can remotely download the report, assess the long-term health of the GPU, and decide whether hardware replacement is necessary.
[0055] Example 3: Autonomous Driving Computing Unit Scenario
[0056] In the domain controller of an autonomous vehicle, an integrated GPU handles the inference of the perception model. The monitoring module detected a sudden increase in the bit error rate (BER) of the high-speed bus link connecting the peripheral components to the central processing unit (CPU), approaching a threshold that could affect data transmission integrity. The anomaly detection and diagnosis module quickly located the link problem. The dynamic self-healing execution module immediately initiated a soft reset of the high-speed bus link. After the reset, the BER returned to normal, the perception inference task was unaffected, and driving safety was ensured. This incident was recorded in detail and provided crucial data on the reliability of the vehicle's hardware system to the fleet management cloud platform.
[0057] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.
[0058] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.
[0059] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A self-healing GPU firmware method, characterized in that, The method is implemented collaboratively through the following modules integrated into the GPU firmware: The hardware status real-time monitoring module is used to periodically collect the operating status data of the internal hardware units of the GPU through the firmware layer interface. An anomaly detection and diagnosis module is connected to the real-time hardware status monitoring module and is used to perform online analysis of the operating status data and identify and locate hardware faults using an integrated anomaly detection algorithm. A dynamic self-repair execution module is connected to the anomaly detection and diagnosis module and is used to automatically perform repair operations based on the diagnosis results. The operation log and analysis module is connected to the hardware status real-time monitoring module, the anomaly detection and diagnosis module, and the dynamic self-repair execution module. It is used to record hardware status, abnormal events and repair operations, and perform trend analysis based on historical data. The remote maintenance and update module is connected to the operation log and analysis module and the dynamic self-healing execution module, and is used to realize remote firmware updates, repair strategy adjustments and status reports.
2. The self-healing GPU firmware method according to claim 1, characterized in that, In the real-time hardware status monitoring module, status data acquisition adopts a combination of polling and event triggering. The acquired data includes the task queue status of the streaming multiprocessor of the computing core, the health of the execution unit and the error count; the ECC correction count, memory access latency and bad page table information of the video memory unit; and the signal integrity, data packet retransmission rate and bandwidth utilization of the high-speed bus or high-speed interconnect link of the peripheral components of the communication interface.
3. The self-healing GPU firmware method according to claim 1, characterized in that, In the anomaly detection and diagnosis module, the integrated anomaly detection algorithm includes a threshold detection method based on statistical process control, an outlier detection method based on isolated forests, and a time-series prediction and anomaly detection method based on long short-term memory networks. Fault localization is achieved by analyzing the correlation between error patterns and hardware topology. Specifically, when a sustained high error rate of a specific SM is detected, it is determined that the kernel unit has a local failure; when a concentrated outbreak of ECC errors in a specific address range of video memory is detected, it is determined that the physical memory area is unstable; when the bit error rate of the communication link exceeds a threshold, it is determined that the link quality has deteriorated.
4. The self-healing GPU firmware method according to claim 3, characterized in that, The repair operation in the dynamic self-repair execution module includes: Kernel task remapping: Dynamically migrates queued computation tasks on a failed streaming multiprocessor to other healthy streaming multiprocessors within the same GPU to continue execution; Memory Failure Area Masking: Physical memory pages that frequently experience ECC errors are marked as unavailable and isolated from the graphics processor's memory address space, while their storage responsibilities are taken over by a firmware-managed spare memory area. Communication link soft reset: Performs protocol layer reinitialization on the high-speed bus or high-speed interconnect link of peripheral components that has experienced persistent bit errors, and restores its communication state.
5. The self-healing GPU firmware method according to claim 4, characterized in that, In the operation log and analysis module, the log records adopt a structured format, including timestamps, hardware unit identifiers, status indicators, anomaly levels, repair actions and results; the trend analysis identifies the precursor patterns of failures by analyzing historical logs, dynamically adjusts the detection threshold and sensitivity of the anomaly detection and diagnosis module, and provides predictive repair suggestions for the dynamic self-repair execution module.
6. The self-healing GPU firmware method according to claim 1, characterized in that, The remote maintenance and update module supports secure network communication protocols, can receive firmware image updates and repair strategy parameter adjustment instructions from the remote management server, and can proactively report system status summaries, key alarms, and log analysis results to the remote server.
7. The self-healing GPU firmware method according to claim 4, characterized in that, Before performing kernel task remapping, memory fault area masking, or communication link soft reset operations, the dynamic self-repair execution module will assess the risk of the repair operation and the current system load. For high-risk operations, a second confirmation is required based on historical success rates and the criticality of the current task, or the operation may be performed during task intervals.
8. The self-healing GPU firmware method according to claim 5, characterized in that, The anomaly detection and diagnosis module and the operation log and analysis module constitute a closed-loop learning system. By continuously analyzing the success rate of repair operations and subsequent changes in hardware status, the system continuously optimizes the model parameters and repair strategies in the integrated anomaly detection algorithm using the results of the trend analysis.
9. The self-healing GPU firmware method according to claim 2, characterized in that, The data collection frequency of the real-time hardware status monitoring module can be dynamically adjusted according to the system load and the historical fault frequency obtained from the operation log and analysis module, and the data collection frequency can be automatically increased during high load or high risk periods.
10. A self-healing GPU firmware method, characterized in that, Includes the following steps: S1: The real-time hardware status monitoring module periodically collects the operating status data of the GPU computing core, video memory and communication interface; S2: The anomaly detection and diagnosis module performs real-time analysis of the collected status data, uses an integrated anomaly detection algorithm to determine whether there is a hardware anomaly, and locates the faulty hardware unit. S3: If an anomaly is detected, the dynamic self-repair execution module automatically performs the corresponding repair operation according to the fault type and location. The repair operation includes kernel task remapping, video memory fault area masking, or communication link soft reset. S4: The operation log and analysis module records the entire process from status acquisition and anomaly detection to repair execution, and performs in-depth analysis based on historical log data to optimize future detection thresholds and repair strategies; S5: The remote maintenance and update module provides a remote interface to support firmware updates, parameter configuration, and status monitoring, enabling remote operation and maintenance management of the system.