Fault prediction system and electronic equipment

By combining a central control module with hardware redundancy design, and using a fault prediction model to analyze GPU operating data, the problem of not being able to predict faults in advance in existing technologies is solved. This enables intelligent management and fault prevention of GPUs, improving the stability and reliability of the server.

CN121349822APending Publication Date: 2026-01-16INSPUR SUZHOU INTELLIGENT TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202511430785.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-09-30
Publication Date
2026-01-16

AI Technical Summary

Technical Problem

Existing GPU monitoring methods cannot predict failures in advance and take measures, resulting in a decline in server graphics rendering capabilities and economic losses, and affecting user experience.

Method used

The central control module analyzes the historical and current operating data of the graphics processor using a pre-trained fault prediction model to predict future faults and execute corresponding preventive operations. Combined with hardware redundancy design and intelligent collaborative operation and maintenance mode, it realizes intelligent management of GPU.

Benefits of technology

It enables early prediction and prevention of GPU failures, improves server stability and reliability, reduces service interruption time, and enhances the overall operation and maintenance level of the data center.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN121349822A_ABST
    Figure CN121349822A_ABST
Patent Text Reader

Abstract

The invention discloses a fault prediction system and electronic equipment, and relates to the technical field of fault prediction.The fault prediction system is characterized in that in the process of monitoring a graphic processor in a server, a node server reports operation data of the detected graphic processor to a central control module, and the central control module carries out fault prediction on the graphic processor based on a pre-trained fault prediction model; and performing data prediction on historical operation data and current operation data included in the operation data, determining predicted operation data of the detected graphics processor in a preset duration, then determining a fault prediction result of the detected graphics processor based on the predicted operation data and a preset threshold value, and if the fault prediction result is that a fault occurs, determining that the detected graphics processor fails. Therefore, potential faults are predicted and prevented based on the operation data of the graphics processor, the problem that prediction cannot be carried out in advance and measures cannot be taken to avoid the faults is solved, and the requirements of a user for the stability and reliability of the GPU are met.
Need to check novelty before this filing date? Find Prior Art

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of fault prediction, and in particular to a fault prediction system and an electronic device. BACKGROUND

[0002] With the rapid development of information technology, a graphics processing unit (GPU) is configured on a server of multiple nodes to perform graphics rendering and the like. Normal operation of the GPU is a key to ensure that the server performs graphics rendering operations, and therefore, the GPU needs to be continuously monitored to ensure the graphics rendering capability of the server.

[0003] Generally, the GPU is continuously monitored based on an external monitoring method of a baseboard management controller (BMC) and an internal monitoring method of the GPU software itself. However, this monitoring method can only detect and report after the GPU predicts the occurrence of a fault, and cannot predict the fault of the GPU in advance, cannot take measures to avoid the occurrence of the fault in advance, and cannot meet the needs of users for stability and reliability of the GPU. SUMMARY

[0004] The present application provides a fault prediction system and an electronic device to at least solve the problems that the fault of the graphics processing unit cannot be predicted in advance and measures cannot be taken to avoid the occurrence of the fault in the related art.

[0005] The present application provides a fault prediction system, comprising: a center control module and a node server, the node server comprising a detected graphics processing unit;

[0006] The node server sends running data of the detected graphics processing unit to the center control module, wherein the running data comprises historical running data of the detected graphics processing unit before a current time and current running data of the current time;

[0007] The center control module uses a pre-trained fault prediction model to perform data prediction on the historical running data and the current running data of the detected graphics processing unit, and determines predicted running data of the detected graphics processing unit within a preset time length;

[0008] The center control module determines a fault prediction result of the detected graphics processing unit based on the predicted running data and a preset threshold;

[0009] If the fault prediction result is that a fault will occur, the center control module performs a fault prevention operation on the detected graphics processing unit.

[0010] The application further provides a fault prediction method applied to a center control module of a fault prediction system, the fault prediction system further comprising a node server, the node server comprising a detected graphic processor, comprising:

[0011] obtaining running data of the detected graphic processor, wherein the running data comprises historical running data of the detected graphic processor before a current time and current running data of the current time;

[0012] performing data prediction on the historical running data and the current running data of the detected graphic processor by using a pre-trained fault prediction model to determine predicted running data of the detected graphic processor within a preset time length;

[0013] determining a fault prediction result of the detected graphic processor based on the predicted running data and a preset threshold value;

[0014] if the fault prediction result is that a fault will occur, performing a fault prevention operation on the detected graphic processor.

[0015] The application further provides a fault prediction device configured in a center control module of a fault prediction system, the fault prediction system further comprising a node server, the node server comprising a detected graphic processor, comprising:

[0016] a first obtaining module configured to obtain running data of the detected graphic processor, wherein the running data comprises historical running data of the detected graphic processor before a current time and current running data of the current time;

[0017] a first determining module configured to perform data prediction on the historical running data and the current running data of the detected graphic processor by using a pre-trained fault prediction model to determine predicted running data of the detected graphic processor within a preset time length;

[0018] a second determining module configured to determine a fault prediction result of the detected graphic processor based on the predicted running data and a preset threshold value;

[0019] a first executing module configured to perform a fault prevention operation on the detected graphic processor if the fault prediction result is that a fault will occur.

[0020] The application further provides an electronic device comprising the fault prediction system.

[0021] The application further provides a computer readable storage medium, and the computer readable storage medium stores a computer program.

[0022] The application further provides a computer program product, comprising a computer program, and the computer program is executed by a processor to implement the steps of any of the above failure prediction methods.

[0023] Through the application, in the process of monitoring the graphic processor in the server, the node server reports the running data of the detected graphic processor to the central control module, and then the central control module, based on the pre-trained failure prediction model, performs data prediction on the historical running data and the current running data contained in the running data, determines the predicted running data of the detected graphic processor within a preset time length, and then based on the predicted running data and the preset threshold, determines the failure prediction result of the detected graphic processor, and when the failure prediction result is that a failure will occur, performs the failure prevention operation on the detected graphic processor. Therefore, potential failures can be predicted based on the running data of the graphic processor, and necessary measures can be taken to prevent the failure of the graphic processor, solving the problems of being unable to predict the failure of the graphic processor in advance and being unable to take measures to avoid the failure in advance, thereby meeting the user's demand for GPU stability and reliability. BRIEF DESCRIPTION OF DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the application, the following will briefly introduce the drawings needed in the embodiments. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0025] Figure 1 A system architecture diagram of a failure prediction system provided for the embodiments of the application;

[0026] Figure 2 A flowchart of a failure prediction method provided for the embodiments of the application;

[0027] Figure 3 A structure diagram of an MLP model provided for the embodiments of the application;

[0028] Figure 4 A structure diagram of a failure prediction device provided for the embodiments of the application. DETAILED DESCRIPTION

[0029] With reference to the drawings of the embodiments of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all embodiments of the present application. Based on the embodiments of the present application, any other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0030] It should be noted that, in the description of the present application, the terms "comprise", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or equipment comprising a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or equipment. The terms "first", "second" and the like in the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0031] In order for those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0032] In today's information technology era, GPU plays a crucial role in artificial intelligence, big data computing, and graphics rendering and many other fields. With the continuous expansion of GPU application scenarios and the continuous increase of usage scale, its stability and reliability have become a key problem to be solved.

[0033] The current monitoring method of GPU is an external monitoring method based on BMC and an internal monitoring method of GPU software itself. In addition to predicting the failure of GPU in advance and taking measures to avoid the occurrence of failure, this monitoring method also lacks a perfect and effective recovery mechanism, often leading to a significant decline in the performance of related service terminals or systems, causing great inconvenience to users, and may also cause huge economic losses.

[0034] For example, in a large-scale artificial intelligence training cluster, if a GPU suddenly fails due to potential failure, it is likely to cause the ongoing training task to be forced to interrupt, and restarting the training will not only waste a lot of computing resources and time, but also may affect the accuracy and consistency of the training results. For another example, in a real-time graphics rendering scenario, GPU failure may cause problems such as screen freezing and frame dropping, resulting in the screen being unable to display normally, thereby affecting the user's experience.

[0035] It can be seen that the existing GPU monitoring method cannot meet the user's demand for the stability and reliability of GPU.

[0036] Embodiments of the present application provide a fault prediction system, and the system is described in detail in combination with the system architecture of the fault prediction system.

[0037] Figure 1 A system architecture diagram of a fault prediction system is provided for embodiments of the present application. As shown in the figure, the fault prediction system includes a central control module 100 and a node server 200. Figure 1

[0038] The node server 200 sends the running data of the detected graphics processor to the central control module 100, wherein the running data includes historical running data of the detected graphics processor before the current time and current running data of the current time.

[0039] The central control module 100 uses a pre-trained fault prediction model to perform data prediction on the historical running data and the current running data of the detected graphics processor, and determines the predicted running data of the detected graphics processor within a preset time length.

[0040] The central control module 100 determines the fault prediction result of the detected graphics processor based on the predicted running data and a preset threshold.

[0041] If the fault prediction result is that a fault will occur, the central control module 100 performs a fault prevention operation on the detected graphics processor.

[0042] In some embodiments, the node server further includes a baseboard management controller; the baseboard management controller collects the running data of the detected graphics processor and sends the running data of the detected graphics processor to the central control module.

[0043] The fault prediction system can include one or more node servers, and each node server is configured with a GPU and a BMC. Referring to Figure 1 Each node server 200 can be in communication connection with the central control module, and the GPU in each node server 200 can be regarded as a detected graphics processor. Specifically, the BMC in each node server 200 collects the running data of the detected graphics processor in real time and sends the collected running data to the central control module 100, and the central control module 100 can analyze the running data to predict whether the detected graphics processor will fail.

[0044] ​In some embodiments, the central control module 100 comprises a data acquisition unit 101, a fault prediction unit 102, and a control unit 103; the data acquisition unit 101 acquires historical running data and current running data of the detected graphics processor; the fault prediction unit 102 uses a pre-trained fault prediction model to perform data prediction on the historical running data and the current running data of the detected graphics processor, to determine predicted running data of the detected graphics processor within a preset time length; and the control unit 103 determines a fault prediction result of the detected graphics processor based on the predicted running data and a preset threshold, and performs a fault prevention operation on the detected graphics processor when the fault prediction result indicates that a fault will occur.

[0045] Specifically, the data acquisition unit 101 can perform preliminary checking and preprocessing on the historical running data and the current running data, such as removing outliers and filling missing values, to obtain the historical running data and the current running data after preliminary checking and preprocessing, so as to improve the analysis capability of the running data.

[0046] Optionally, the running data of the detected graphics processor includes, but is not limited to, one or a combination of the following: temperature, count, utilization rate, core frequency, and memory read-write rate.

[0047] In this way, through deep fusion and analysis of these multi-source data, the running state of the GPU can be more comprehensively and accurately described, and the accuracy and reliability of fault prediction can be greatly improved.

[0048] In some embodiments, the fault prediction unit 102 performs feature extraction processing on the historical running data and the current running data based on a feature extraction network in the pre-trained fault prediction model, to obtain a first feature vector; performs linear transformation processing and activation processing on the first feature vector based on an extraction and conversion network in the pre-trained fault prediction model, to obtain a second feature vector; and performs prediction processing on the second feature vector based on a prediction network in the pre-trained fault prediction model, to obtain the predicted running data.

[0049] Optionally, the pre-trained fault prediction model can be a time series prediction model, such as a multilayer perceptron (MLP). The feature extraction network in the pre-trained fault prediction model can be an input layer of the MLP model, the extraction and conversion network can be a hidden layer of the MLP model, and the prediction network can be an output layer of the MLP model.

[0050] In this way, the data prediction capability of the feature extraction network, the extraction and conversion network, and the prediction network in the fault prediction model is used to accurately determine the predicted running data.

[0051] Specifically, the fault prediction unit 102 can construct a pre-trained fault prediction model through learning and training of a large amount of data in advance, and then predict the possible faults of the GPU in a future period of time according to historical data and current real-time data of the GPU.

[0052] In order to continuously improve the prediction accuracy of the model, the fault prediction unit 102 will also regularly review and analyze historical data and prediction results, summarize lessons learned, find potential problems and improvement directions, so as to realize continuous optimization and improvement of system performance.

[0053] Specifically, the fault prediction unit 102 obtains the measured running data of the detected GPU within a preset time length; the fault prediction unit 102 calculates the difference between the predicted running data within the preset time length and the measured running data within the preset time length; and the fault prediction unit 102 adjusts the model parameters of the pre-trained fault prediction model based on the historical running data, the current running data and the measured running data if the difference is greater than a preset error threshold, to obtain an adjusted fault prediction model.

[0054] Thus, in combination with the machine learning algorithm, the threshold values of various parameters can be dynamically adjusted according to the actual situation. For example, when the GPU is in a high-load running state for a long time, the system will automatically increase the threshold values of temperature and utilization rate appropriately to avoid unnecessary alarms and operations due to misjudgment; and when the GPU appears abnormal fluctuations, the system can quickly reduce the threshold values to capture potential fault risks in time, thereby significantly improving the adaptability and accuracy of the system.

[0055] In this way, through this feedback and optimization mechanism, the fault prediction system can continuously adapt to changes in the GPU running environment and always maintain accurate prediction and efficient recovery capabilities for GPU faults.

[0056] Specifically, the control unit 103 has strong instruction sending capability, and if it is determined that the predicted running data is greater than a preset threshold, it is determined that the fault prediction result of the detected GPU is that a fault will occur, and the fault prevention operation on the detected GPU is performed, specifically by stable communication with the node BMC and sending various instructions to the GPU.

[0057] In one example, if the running data is temperature, if the control unit 103 determines that the predicted temperature exceeds the threshold value (such as 85°C), the fault prevention operation of the detected graphics processor is performed. Specifically, the fan speed is quickly adjusted by the BMC to increase the heat dissipation, increase the heat dissipation effect, ensure that the GPU temperature returns to the normal range as soon as possible, avoid hardware damage or performance degradation caused by overheating, for example, the native fan speed is adjusted to 100% by the BMC, and a yellow warning light and an interval alarm sound are triggered; if the temperature continues to rise after 3 minutes, start load reduction (limit GPU power to 70%).

[0058] In another example, if the running data is utilization, if the control unit 103 determines that the predicted utilization exceeds the threshold value (such as 90%), the fault prevention operation of the detected graphics processor is performed. Specifically, the control unit 103 interacts with the scheduling and arrangement system (Kubernetes), intelligently and reasonably reduces the load of the corresponding GPU according to the real-time utilization prediction value of each GPU, and distributes it to other GPUs with lower utilization prediction value, thereby optimizing the configuration of GPU resources, improving the operation efficiency of the whole system, and reducing the risk of failure caused by excessive load of GPU, for example, filtering GPU nodes with utilization < 60% in the resource pool, generating a task migration list; and transmitting task intermediate data (migration interruption < 50ms) through the Remote Direct Memory Access (RDMA) protocol, and the warning is removed after the migration is completed.

[0059] In yet another example, if the running data is count, if the control unit 103 determines that the count surges (such as ≥5 times in 1 minute), the fault prevention operation of the detected graphics processor is performed. Specifically, the control unit 103 immediately starts the standby GPU switching, triggers the red fault light, continuous alarm sound, and pushes the short message to the operation and maintenance personnel through the remote gateway; after switching is completed, the main GPU enters the isolation state and waits for repair.

[0060] For memory read / write speed, core frequency, etc., corresponding threshold values are also set, and targeted measures are taken when the predicted value exceeds the threshold value. For example, if the memory read / write speed abnormally decreases, it may mean that the memory has failed or there is a data transmission bottleneck, and the system can try to optimize the data read / write strategy, or prompt the operation and maintenance personnel to check the memory hardware connection, etc.

[0061] Thus, when it is predicted that the graphics processor may have a risk of failure due to overheating, the system will not only adjust the fan speed, but also cooperate with the energy management system to optimize the refrigeration strategy of the entire data center, improve the energy utilization efficiency, and at the same time guarantee the stable operation of the graphics processor. In cooperation with the network management system, when it is found that the high utilization rate of the graphics processor may lead to network congestion, the system can jointly adjust the network traffic distribution strategy with the network management system to avoid aggravating the failure risk of the graphics processor due to network problems. This intelligent collaborative operation mode breaks the limitation of traditional independent systems, realizes efficient cooperation between systems in the data center, and comprehensively improves the overall operation level and reliability of the data center.

[0062] In this way, the central control module combines the data acquisition unit and the failure prediction unit to accurately predict whether the detected graphics processor will fail, and when it is determined that the detected graphics processor will fail, the control module can quickly respond and perform corresponding failure prevention operations, thereby realizing intelligent management of the GPU.

[0063] In some embodiments, the central control module 100 generates a failure recovery instruction of the detected graphics processor when it is determined that the detected graphics processor will fail.

[0064] In some embodiments, the failure prediction system further comprises a failure recovery module.

[0065] The failure recovery module performs a recovery operation on the detected graphics processor based on the failure recovery instruction.

[0066] Specifically, the failure recovery module comprises a backup graphics processor in the node server, a switching circuit, and a redundant power supply. The backup graphics processor performs a state switching operation based on the failure recovery instruction and saves the data of the detected graphics processor. The switching circuit performs normal monitoring operation, switching operation, verification operation, and failure isolation operation based on the failure recovery instruction. The redundant power supply supplies power to the backup graphics processor.

[0067] When configuring the backup graphics processor, each node server is configured with one backup graphics processor of the same model as the main graphics processor, which shares a channel (such as a PCIe channel) with the main graphics processor through a specific key interconnection component (such as a PCIe switch). The backup state power consumption is ≤10W (the core voltage is reduced to 0.8V through BMC control), supports hot plugging (equipped with a special heat sink and an anti-mis-touch buckle).

[0068] The working process of the standby graphics processor is as follows: in the hot standby state, after the system starts, the standby graphics processor automatically enters the low-power standby mode (core voltage 0.8V, video memory self-refresh), power consumption <10W; the BMC collects parameters such as its temperature (<50℃) and power supply voltage (12V±5%) every 5 seconds to ensure that it is in a usable state; in the activation trigger state: when the main graphics processor fails or the predicted risk reaches the critical value, the control unit sends an activation instruction, and the standby graphics processor completes voltage lifting (1.05V), core initialization and firmware loading (synchronized with the main graphics processor configuration) within 20ms; in the data synchronization: during the activation process, the task context data of the main graphics processor is quickly synchronized (transmission rate ≥10GB / s) through a special cache chip (capacity 4GB) to ensure that the task continues to run at the breakpoint; in the state switching: after the switching is completed, the standby graphics processor returns a "ready signal" to the control unit, and the system updates the resource table (marks the main graphics processor as "faulty" and the standby graphics processor as "active"); the original main graphics processor is automatically powered off (only the BMC monitoring path is retained).

[0069] When configuring the switching circuit, a relay array and a voltage monitoring chip (such as INA219) are integrated on the node server mainboard to monitor the power supply voltage (accuracy ±1mV) of the main graphics processor and the link state of the channel (such as PCIe channel) in real time. When the main graphics processor fails (such as voltage drop, link interruption) or the control unit issues a switching instruction, the switching circuit disconnects the power supply of the main graphics processor within 30ms, connects the power supply of the standby graphics processor, and completes the link switching through the key interconnection components (such as PCIe switch), and at the same time feeds back the switching result (success / failure) to the control unit.

[0070] The working process of the switching circuit is as follows: perform normal monitoring operation, the channel (such as PCIe channel) switching circuit monitors the link state of the main graphics processor in real time (100 times per second), including link width (x16 / x8), transmission rate (8GT / s) and error rate (<1e-12). When performing switching operation, after receiving the fault recovery instruction of the control unit, three actions are completed within 30ms: ① disconnect the link (such as PCIe link) of the main graphics processor and send a reset signal; ② close the switch (such as PCIe switch chip switching channel) of the standby graphics processor; ③ notify the central processor to re-enum devices and trigger the hot plug event. When performing the verification operation, after the switching is completed, the circuit automatically performs the link integrity test (sends 1000 frames of test data packets), if the error rate >1e-9, it is determined that the switching fails, and immediately retries (up to 3 times). When performing fault isolation operation: when the short circuit (current >20A) of the main graphics processor is detected, the hardware protection (disconnect the power supply pipe) is automatically triggered to prevent the fault from spreading to other components of the mainboard.

[0071] In the configuration of redundant power supply, a 12V redundant power supply (power ≥ 200W) is configured for the standby graphics processor, which is isolated from the main power module by a diode, so as to ensure that the standby graphics processor can still start normally when the main power supply fails.

[0072] Therefore, through the redundant design introduced at the hardware level, a redundant graphics processor is provided on the key graphics processor node. When the main graphics processor fails, the system can automatically switch the service to the standby graphics processor in a very short time, ensuring the continuity of the service and almost causing no service interruption. At the same time, the system will immediately isolate and mark the failed graphics processor and start the fault diagnosis and repair process. Compared with the traditional single graphics processor architecture, this hardware redundancy and fast switching mechanism greatly improves the fault tolerance of the system and reduces the service interruption time caused by the failure of the graphics processor, providing a strong guarantee for application scenarios with extremely high requirements for service continuity.

[0073] In this way, when the graphics processor performs the recovery operation, through the hardware redundancy and fast switching mechanism in the fault recovery module, the service can be switched to the standby graphics processor in a very short time, ensuring that the service is almost not interrupted. At the same time, the system will immediately diagnose and repair the failed graphics processor, greatly shortening the fault recovery time and improving the availability and reliability of the system.

[0074] In some embodiments, the fault prediction system further comprises: an alarm module, which performs an alarm operation on the detected graphics processor if the fault prediction result is that a fault will occur; or an alarm module, which performs an alarm operation on the detected graphics processor based on the fault recovery instruction.

[0075] Specifically, the alarm module comprises: an indicator light alarm unit, an audio alarm unit, a remote alarm unit and an alarm control unit; the alarm control unit generates an alarm indication command; the indicator light alarm unit performs a display alarm operation based on the alarm indication command; the audio alarm unit performs an audio alarm operation based on the alarm indication command; and the remote alarm unit performs a remote alarm operation based on the alarm indication command.

[0076] In the configuration of the indicator light warning unit, the indicator light warning unit can specifically be a distributed indicator light array composed of node-level indicator lights and cabinet-level master control indicator lights. The node-level indicator light is embedded in the front panel of each node server and adopts a Φ5mm three-color LED (red / yellow / green). The brightness (0-100% adjustable) is controlled by the pulse width signal (PWM signal) of the BMC. It supports two modes: "constant" and "flashing" (frequency 2Hz). Red constant indicates "graphics processor has failed", red flashing indicates "failure recovery"; yellow constant indicates "high risk warning", yellow flashing indicates "warning continues and is not handled"; green constant indicates "normal operation". The cabinet-level master control indicator light is installed on the front end of the cabinet top and adopts a Φ10mm high-brightness LED. It synchronously displays the highest priority state in the current cabinet (failure > warning > normal) and supports manual switching of the display single node state through an infrared remote control.

[0077] The working process of the indicator light warning unit is as follows: in the initialization state, after the system starts, all indicator lights display green constant by default (indicating that the self-checking is completed). The warning triggering process: when the warning indication command issued by the warning control unit is a yellow warning instruction, the yellow light on the node server front panel is turned on, and the yellow light in the corresponding area of the cabinet master control panel is synchronously turned on. If it is not handled for 3 minutes, the indicator light switches to the flashing mode (frequency 2Hz). The failure triggering process: when a hardware failure is detected, the node panel red light is constantly on, and the red light in the corresponding area of the cabinet master control panel is turned on and flashed (frequency 1Hz). If the failure involves multiple nodes, the total control panel starts the area linkage display (the indicator lights of adjacent failure nodes are synchronously flashed). The recovery process: after the failure is eliminated, the warning control unit issues a release instruction as a warning indication command, and the indicator light transitions from red / yellow to green (constantly on after 3 seconds of flashing), and the completion state is reset. The abnormal monitoring process: the BMC detects the indicator light state every 10 seconds. If the instruction does not match the actual display (for example, the red light should be on but the green light is displayed), the fault code (such as "IND03_ERR") is automatically recorded and reported through the remote gateway.

[0078] In the configuration of the audio warning unit, the audio warning unit can specifically be a hierarchical audio warning device composed of a cabinet built-in loudspeaker and a node buzzer. The cabinet loudspeaker adopts a 2W full-frequency speaker (frequency response 200Hz-5kHz) and is installed at the ventilation hole of the cabinet side. It supports two kinds of warning audio effects: "warning sound" is a 1kHz sine wave with a short beep (0.5 seconds) every 2 seconds, and the volume is adjustable from 60-80dB; "failure sound" is an 800Hz square wave with continuous long beeping, and the volume is adjustable from 80-110dB, and has a design of anti-whistling (real-time feedback noise reduction through a microphone). The node buzzer (0.5W) is integrated in the server and is triggered only when the local node fails, forming a "near field + far field" audio coverage with the cabinet loudspeaker.

[0079] The working process of the audio alarm unit is as follows: mode activation process: after receiving the alarm indication command issued by the alarm control unit, switch to the corresponding mode within 100 ms, i.e., the pre-warning mode (1 kHz audio, 0.5 s response / 1 s pause cycle) or the fault mode (800 Hz audio, continuous output). Volume control process: the default pre-warning mode is 80 dB, and the fault mode is 100 dB; support step adjustment through the cabinet key (±10 dB per step, range 50-120 dB), and the adjustment parameters are automatically saved to the local storage (power failure does not lose). Mute mechanism: after triggering the alarm, the operation and maintenance personnel can execute the mute through the cabinet physical key or remote instruction (the mute duration can be set to 5 / 10 / 30 minutes); during the mute period, if the fault state is removed, the bell sound is automatically terminated; if it is not removed, the alarm is automatically restarted after reaching the set duration. Fault protection process: built-in loudspeaker overheat protection (automatic pause when temperature > 70°C, resume when temperature drops to 50°C), and monitor the audio output loop, automatically cut off the power supply and report "BUZZ01_ERR" fault when short-circuit.

[0080] When configuring the remote alarm unit, the remote alarm unit can be an alarm linkage control unit, which is a hardware module independent of the central control module (carrying an STM32F4 single-chip microcomputer), responsible for receiving the alarm indication command issued by the alarm control unit, analyzing and driving the indicator light and loudspeaker, and having an "alarm suppression" function: supporting manual pause of audio alarm through the physical key on the side of the cabinet (the pause duration can be set to 5-30 minutes), but the indicator light state remains unchanged; if the fault is not removed, the audio alarm is automatically restarted after the pause ends.

[0081] The working process of the remote alarm unit is as follows: information processing process: receiving structured fault information (including GPU number, fault type, and timestamp) sent by the alarm control unit as an alarm indication command, completing format conversion (adapting to SMS, email, and operation and maintenance platform protocols) within 1 second. Multi-channel pushing process: pre-warning information is pushed through email and operation and maintenance platform messages; fault information is pushed through "SMS + voice call + platform alarm" in three ways, and voice call supports 3 retries (interval 1 minute). Network switching process: by default, transmission is through Ethernet; when network interruption (ping gateway failure for 3 seconds) is detected, automatically switch to 4G / 5G module (switching time < 2 seconds) to ensure uninterrupted alarm. State feedback process: record the delivery state (success / failure) of each alarm, automatically switch to the backup contact list when failure (such as invalid mobile number), and save the alarm log locally (at least 90 days).

[0082] In this way, when a fault occurs or is recovered, sound and light alarms can be triggered in a timely manner according to the fault level, effectively and accurately reminding the user of the status of the graphics processing unit.

[0083] The fault prediction system provided in the application, in the process of monitoring the graphic processor in the server, the node server reports the running data of the detected graphic processor to the center control module, and then the center control module, based on the pre-trained fault prediction model, performs data prediction on the historical running data and the current running data contained in the running data, determines the predicted running data of the detected graphic processor within a preset time length, and then based on the predicted running data and the preset threshold, determines the fault prediction result of the detected graphic processor, and when the fault prediction result is that a fault will occur, performs the fault prevention operation on the detected graphic processor, so that the potential fault can be predicted based on the running data of the graphic processor, and necessary measures can be taken to prevent the fault of the graphic processor, solving the problems that the fault of the graphic processor cannot be predicted in advance and measures cannot be taken in advance to avoid the occurrence of the fault, thereby meeting the user's demand for GPU stability and reliability.

[0084] The embodiment of the application provides a fault prediction method, and the method is described in detail in combination with the execution process of the fault prediction method (the technical terms involved must be explained).

[0085] Figure 2 A flowchart of a fault prediction method provided by the embodiment of the application is applied to a center control module of a fault prediction system, and the fault prediction system further includes a node server, the node server includes a detected graphic processor, and specifically includes the following steps as shown in Figure 2

[0086] S210, obtaining running data of the detected graphic processor, wherein the running data includes historical running data of the detected graphic processor before the current time and current running data of the current time.

[0087] Optionally, the running data of the detected graphic processor includes but is not limited to one or a combination of the following: temperature, count, utilization rate, core frequency and memory read-write rate.

[0088] S220, using a pre-trained fault prediction model to perform data prediction on the historical running data and the current running data of the detected graphic processor, and determining predicted running data of the detected graphic processor within a preset time length.

[0089] In the model training stage, in order to comprehensively obtain the data of the GPU in different working states, first, make the GPU of the node server run in a plurality of different load states, and the data acquisition module synchronously collects and stores the GPU data generated in the working process. The data acquisition frequency is set to once per second to ensure that the subtle changes of the GPU running state can be captured in time. At the same time of collecting data, the corresponding time stamp is accurately recorded for subsequent time series analysis.​

[0090] Before using these data for training, a series of fine data processing operations are needed. First, the data is normalized to a specific value range using the Min-Max normalization method, eliminating the influence of different characteristic quantities due to the difference in dimension and value range, making the data easier for model learning and processing. Next, in order to convert the time series data into a form suitable for model training of supervised learning problems, the sliding window technique is used, specifically, a window size of 11 is selected, that is, the running data of 11 time steps contained in a window is taken as a sample, for each window, the data of the first 10 time steps is explicitly selected as the input, and the data of the last time step is selected as the output. After such processing, the original data collected is re-divided into a series of clear structure and clear feature samples.

[0091] In the model training using the above data, the pre-processed collected data is delivered to the fault prediction module for model training. The fault prediction module uses an advanced time series prediction model, in which MLP is applied to the time series prediction task as a simple but powerful neural network model. MLP can effectively capture the complex nonlinear relationship between input features through DenseLayer, and is particularly suitable for processing small or medium-sized time series data, which can fully play its advantages in the system to realize accurate prediction of GPU faults.

[0092] Referring to Figure 3 , the MLP model is mainly composed of three parts: input layer, hidden layer and output layer. Among them, the input layer is responsible for receiving input data, and the number of its neurons is usually exactly matched with the feature dimension of the input data. In this system, since the first 10 time steps of data are used as input in the sliding window data processing, the number of neurons in the input layer is set to 10. The hidden layer is the core component of MLP, which undertakes the task of deeply extracting features from input data and learning complex nonlinear relationships. The neurons in the hidden layer usually use nonlinear activation functions such as ReLU (Rectified Linear Unit), Sigmoid, Tanh, etc. These activation functions can greatly enhance the expression ability of the model, making the model better fit the complex data distribution. The output layer generates the final output according to the specific task requirements. In this system, since the next time step data is predicted, the number of neurons in the output layer is set to 1. In order to realize accurate prediction of different key indicators of GPU, an MLP model consistent with the number of categories of running data is constructed, which is specifically used for predicting the running data of these categories.

[0093] In all collected sample data, 70% is reasonably selected as training data for parameter learning and training of the model; 15% is selected as validation data for evaluating and selecting the performance of the model during the training process, including selecting key parameters such as the activation function type of the hidden layer, the number of layers, and the number of neurons, to ensure that the model has good generalization ability and prediction performance; the remaining 15% is selected as test data for objectively and accurately evaluating the final performance of the model after the model training is completed, to ensure that the model also performs excellent prediction ability on data that has never been seen. During the model training process, the loss function is defined as the mean squared error (MSE, Mean Squared Error), which can effectively measure the error between the predicted value and the true value of the model. The optimizer selects the stochastic gradient descent (SGD, Stochastic Gradient Descent) algorithm, which iteratively trains the data to gradually and accurately adjust the weights and biases of the model to minimize the loss function, thereby continuously improving the prediction performance of the model.

[0094] Optionally, the embodiment can also combine the MLP model with a deep learning model to predict historical and current running data and determine the predicted running data of the detected graphics processor within a preset time length, so as to improve the data prediction accuracy when multiple models are used for data prediction.

[0095] After the model training is completed and strictly tested and verified, a pre-trained fault prediction model is obtained, which is used to predict the historical and current running data of the detected graphics processor and determine the predicted running data of the detected graphics processor within a preset time length.

[0096] The specific implementation method of S220 includes but is not limited to the following method: based on the feature extraction network in the pre-trained fault prediction model, the historical and current running data are processed for feature extraction to obtain a first feature vector; based on the extraction and conversion network in the pre-trained fault prediction model, the first feature vector is processed for linear transformation and activation to obtain a second feature vector; and based on the prediction network in the pre-trained fault prediction model, the second feature vector is processed for prediction to obtain the predicted running data.

[0097] Optionally, when the pre-trained fault prediction model is an MLP model, the feature extraction network in the pre-trained fault prediction model can be the input layer of the MLP model, the extraction and conversion network can be the hidden layer of the MLP model, and the prediction network can be the output layer of the MLP model.

[0098] Thus, the feature extraction network, the extraction and conversion network, and the prediction network in the fault prediction model have learned the data prediction capability, and the predicted running data is accurately determined.

[0099] In some embodiments, after S220 is performed, the method further includes: obtaining the measured running data of the detected graphics processor within a preset time length; calculating a difference between the predicted running data within the preset time length and the measured running data within the preset time length; if the difference is greater than a preset error threshold, adjusting the model parameters of the pre-trained fault prediction model based on the historical running data, the current running data, and the measured running data to obtain an adjusted fault prediction model.

[0100] Thus, in combination with the machine learning algorithm, the threshold values of various parameters can be dynamically adjusted according to actual conditions. For example, when the graphics processor is in a high-load running state for a long time, the system automatically increases the threshold values of the temperature and the utilization rate to avoid unnecessary alarms and operations due to misjudgment; and when the graphics processor has abnormal fluctuations, the system can quickly reduce the threshold values to timely capture potential fault risks, thereby significantly improving the adaptability and accuracy of the system.

[0101] Thus, through the feedback and optimization mechanism, the fault prediction system can continuously adapt to changes in the GPU running environment and always maintain the ability to accurately predict and efficiently recover from GPU faults.

[0102] S230, determining a fault prediction result of the detected graphics processor based on the predicted running data and a preset threshold value.

[0103] In this embodiment, the node server compares the predicted running data with the preset threshold value. If the predicted running data is greater than the preset threshold value, it indicates that the fault prediction result of the detected graphics processor is that a fault will occur, otherwise, it is determined that the fault prediction result of the detected graphics processor is that a fault will not occur.

[0104] S240, if the fault prediction result is that a fault will occur, performing a fault prevention operation on the detected graphics processor.

[0105] In this embodiment, if the node server determines that the fault prediction result is that a fault will occur, the fault prevention operation on the detected graphics processor can be performed by adjusting the fan speed, cooperating with the energy management system, cooperating with the network management system, and the like.

[0106] After S240 is performed, the method further includes: generating a fault recovery instruction of the detected graphics processor when it is determined that the detected graphics processor will have a fault.

[0107] In one case, the fault recovery instruction is used to control a fault recovery module of the fault prediction system to perform a recovery operation on the detected graphics processor.

[0108] Specifically, the fault recovery module performs a state switching operation on the standby graphics processor based on the fault recovery instruction and saves data of the detected graphics processor; the fault recovery module performs a normal monitoring operation, a switching operation, a verification operation and a fault isolation operation based on the fault recovery instruction through a switching circuit; and the fault recovery module supplies power to the standby graphics processor through a redundant power supply.

[0109] In one case, the fault recovery instruction is used to control an alarm module of the fault prediction system to perform an alarm operation. Alternatively, the alarm module of the fault prediction system performs an alarm operation on the detected graphics processor when the fault prediction result is that a fault will occur.

[0110] Specifically, the alarm module generates an alarm indication command through an alarm control unit; the alarm module performs a display alarm operation based on the alarm indication command through an indicator alarm unit; the alarm module performs an audio alarm operation based on the alarm indication command through an audio alarm unit; and the alarm module performs a remote alarm operation based on the alarm indication command through a remote alarm unit.

[0111] The fault prediction method provided in the application can, in the process of monitoring the graphics processor in the server, receive running data of the detected graphics processor reported by a node server by a central control module of the fault prediction system, perform data prediction on historical running data and current running data contained in the running data based on a pre-trained fault prediction model, determine predicted running data of the detected graphics processor within a preset time length, then determine a fault prediction result of the detected graphics processor based on the predicted running data and a preset threshold, and perform a fault prevention operation on the detected graphics processor when the fault prediction result is that a fault will occur. Therefore, potential faults can be predicted based on the running data of the graphics processor, and necessary measures can be taken to prevent faults of the graphics processor, thus solving the problems of being unable to predict faults of the graphics processor in advance and being unable to take measures to avoid faults in advance, and meeting the needs of users for stability and reliability of GPUs.

[0112] Those skilled in the art can clearly understand from the description of the above embodiments that the method according to the above embodiments can be realized by means of software and a necessary general hardware platform, of course, it can also be realized by hardware, but in many cases, the former is a better embodiment.

[0113] The embodiments of the present application also provide a fault prediction device, the pure virtual device right item is not more than 1 item (except for special cases), the omitted virtual device right item content specification should be described, and the method right item corresponds one by one.

[0114] On the basis of the above-mentioned embodiments, Figure 4 A structural schematic diagram of a fault prediction device provided by the embodiments of the present application is shown in the figure, the fault prediction device is configured in a central control module of a fault prediction system, the fault prediction system further includes a node server, the node server includes a detected graphic processor, and the fault prediction device 400 includes a first acquisition module 401, a first determination module 402, and a first execution module 403.

[0115] The first acquisition module 401 is configured to acquire running data of the detected graphic processor, wherein the running data includes historical running data of the detected graphic processor before a current time and current running data of the current time.

[0116] The first determination module 402 is configured to use a pre-trained fault prediction model to perform data prediction on the historical running data and the current running data of the detected graphic processor, and determine predicted running data of the detected graphic processor within a preset time length.

[0117] The second determination module 403 is configured to determine a fault prediction result of the detected graphic processor based on the predicted running data and a preset threshold.

[0118] The first execution module 404 is configured to perform a fault prevention operation on the detected graphic processor if the fault prediction result is that a fault will occur.

[0119] In some embodiments of the present disclosure, the first determination module 402 includes:

[0120] The first processing unit is configured to perform feature extraction processing on the historical running data and the current running data based on a feature extraction network in the pre-trained fault prediction model, to obtain a first feature vector.

[0121] The second processing unit is configured to perform linear transformation processing and activation processing on the first feature vector based on an extraction and conversion network in the pre-trained fault prediction model, to obtain a second feature vector.

[0122] The third processing unit is configured to perform prediction processing on the second feature vector based on a prediction network in the pre-trained fault prediction model, to obtain the predicted running data.

[0123] In some embodiments of the present disclosure, the device further includes:

[0124] a second obtaining module, configured to obtain measured running data of the detected graphics processor in a preset time length;

[0125] a calculating module, configured to calculate a difference between the predicted running data in the preset time length and the measured running data in the preset time length;

[0126] an adjusting module, configured to, if the difference is greater than a preset error threshold, adjust model parameters of the pre-trained fault prediction model based on the historical running data, the current running data and the measured running data, to obtain an adjusted fault prediction model.

[0127] In some embodiments of the present disclosure, the apparatus further comprises:

[0128] a generating module, configured to generate a fault recovery instruction of the detected graphics processor if it is determined that the detected graphics processor will fail.

[0129] The features of the embodiments of the fault prediction apparatus can be referred to the related descriptions of the embodiments of the fault prediction method, which will not be repeated here.

[0130] Embodiments of the present application also provide an electronic device comprising the fault prediction system described in the above embodiments.

[0131] Embodiments of the present application also provide a computer readable storage medium, which stores a computer program, wherein the computer program is configured to execute the steps in any of the above fault prediction method embodiments when running.

[0132] In an example embodiment, the above computer readable storage medium can include, but is not limited to, a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store computer programs.

[0133] Embodiments of the present application also provide a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the steps in any of the above fault prediction method embodiments.

[0134] Embodiments of the present application also provide another computer program product, which comprises a non-volatile computer readable storage medium, and the non-volatile computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps in any of the above fault prediction method embodiments.

[0135] Those skilled in the art will further realize that the mere concepts, teachings, and embodiments described herein are merely meant to provide an enabling description of the claimed application. Accordingly, modifications and / or additions, other than those explicitly described herein, can be obvious to those skilled in the art in the light of this disclosure. The claimed application is intended to embrace all such modifications and / or additions.

[0136] The above has carried out detailed introduction to the fault prediction system and electronic equipment provided by the application. The principle and implementation mode of the application are described by applying specific examples in the present article, and the above description of the embodiments is only applicable to help understand the method and core idea of the application. It should be pointed out that for ordinary skilled in the art, some improvements and modifications can be made to the application without departing from the principle of the application, and these improvements and modifications also fall within the protection scope of the claims of the application.

Claims

1. A failure prediction system, characterized by, The system comprises a central control module and a node server, the node server comprises a detected graphics processor; The node server sends operation data of the detected graphics processor to the central control module, wherein the operation data comprises historical operation data of the detected graphics processor before the current time and current operation data of the current time; The central control module uses a pre-trained fault prediction model to predict the historical operation data and the current operation data of the detected graphics processor, and determines predicted operation data of the detected graphics processor within a preset time length; The central control module determines a fault prediction result of the detected graphics processor based on the predicted operation data and a preset threshold; If the fault prediction result is that a fault will occur, the central control module performs a fault prevention operation on the detected graphics processor. The central control module comprises a data acquisition unit, a fault prediction unit and a control unit; 2. The failure prediction system of claim 1, wherein The data acquisition unit acquires historical operation data and current operation data of the detected graphics processor; The fault prediction unit uses the pre-trained fault prediction model to predict the historical operation data and the current operation data of the detected graphics processor, and determines predicted operation data of the detected graphics processor within a preset time length; The control unit determines a fault prediction result of the detected graphics processor based on the predicted operation data and a preset threshold, and performs a fault prevention operation on the detected graphics processor if the fault prediction result is that a fault will occur.

3. The fault prediction system of claim 2, wherein The fault prediction unit performs feature extraction processing on the historical operation data and the current operation data based on a feature extraction network in the pre-trained fault prediction model to obtain a first feature vector; performs linear transformation processing and activation processing on the first feature vector based on an extraction and conversion network in the pre-trained fault prediction model to obtain a second feature vector; and performs prediction processing on the second feature vector based on a prediction network in the pre-trained fault prediction model to obtain the predicted operation data.

4. The fault prediction system of claim 2, wherein The fault prediction unit acquires actual measurement operation data of the detected graphics processor within a preset time length; The fault prediction unit calculates a difference between the predicted operation data within the preset time length and the actual measurement operation data within the preset time length; If the difference is greater than a preset error threshold, the fault prediction unit adjusts model parameters of the pre-trained fault prediction model based on the historical operation data, the current operation data and the actual measurement operation data to obtain an adjusted fault prediction model.

5. The fault prediction system of claim 1, wherein If the central control module determines that the detected graphics processor will have a fault, the central control module generates a fault recovery instruction for the detected graphics processor. ​ 6. The failure prediction system of claim 5, wherein, Also comprising: an alarm module, if the failure prediction result is that a failure will occur, performing an alarm operation on the detected graphics processor; or, an alarm module, based on the failure recovery instruction, performing an alarm operation on the detected graphics processor.

7. The failure prediction system of claim 6, wherein The alarm module comprises an indicator light alarm unit, an audio alarm unit, a remote alarm unit and an alarm control unit. The alarm control unit generates an alarm indication command. The indicator light alarm unit performs a display alarm operation based on the alarm indication command. The audio alarm unit performs an audio alarm operation based on the alarm indication command. The remote alarm unit performs a remote alarm operation based on the alarm indication command.

8. The failure prediction system of claim 5, wherein, Also comprising: a failure recovery module; The failure recovery module performs a recovery operation on the detected graphics processor based on the failure recovery instruction.

9. The failure prediction system of claim 8, wherein, The failure recovery module comprises a backup graphics processor in a node server, a switching circuit and a redundant power supply. The backup graphics processor performs a state switching operation based on the failure recovery instruction and saves the data of the detected graphics processor. The switching circuit performs a normal state monitoring operation, a switching operation, a verification operation and a failure isolation operation based on the failure recovery instruction. The redundant power supply supplies power to the backup graphics processor.

10. An electronic device, comprising: Comprising: The failure prediction system of any one of claims 1 to 9.