GPU fault detection and processing method, device and equipment and storage medium

Abstract

The invention discloses a GPU fault detection and processing method and device, equipment and a storage medium, and relates to the technical field of cloud computing, and the method comprises the steps: determining the position information of a video memory abnormal process based on the context switching frequency and verification error count of a GPU; compressing an association result obtained by dynamically associating each application performance index with the processor information of the GPU to obtain compressed data, and performing stream processing, time sequence prediction and twin mapping operation on the compressed data to obtain a detection result; processing the position information by using a target monitoring rule generated based on historical fault data and based on the target knowledge graph and the detection result to obtain a to-be-processed scheme; and generating a snapshot strategy by using an early warning signal generated based on the regional temperature distribution data of the GPU and the depth of the task queue, and processing the to-be-processed scheme based on the snapshot strategy by using the standby computing node to obtain a target scheme. Therefore, the GPU detection efficiency can be improved.

Description

Technical Field

[0001] The present invention relates to the field of cloud computing technology, and in particular to a GPU fault detection and processing method, device, equipment and storage medium. Background Art

[0002] Currently, the use of multi-machine, multi-GPU, or even 1,000-GPU GPU clusters to train deep learning models is mainly due to the urgent needs and significant advantages in the following aspects.

[0003] First, with the rapid development of deep learning technology, model complexity and parameter count have increased dramatically, posing unprecedented challenges to computing resources. Complex applications such as large language models and image recognition systems often require billions or even hundreds of billions of parameters. These massive models are nearly impossible to train effectively on a single machine with a single GPU. Multi-machine, multi-GPU clusters, on the other hand, fully leverage the computing power of multiple GPUs in the cluster. Through data or model parallelism, training tasks can be distributed across multiple GPUs for parallel processing, significantly accelerating training and shortening training cycles.

[0004] Secondly, deep learning model training requires the support of large-scale datasets. As data volumes continue to expand, the computing power of a single machine and single GPU is increasingly insufficient to process large datasets. Multi-machine, multi-GPU clusters can divide large datasets into multiple small batches, with each GPU processing a small batch of data, computing gradients and updating model parameters in parallel, thus enabling efficient processing of large datasets. This parallel processing approach not only improves data processing efficiency but also enables the model to more fully learn the features in the data, enhancing its generalization and performance.

[0005] Furthermore, multi-machine, multi-GPU clusters offer high scalability and flexibility. As computing demands grow, the cluster's computing power can be expanded simply by adding servers and GPUs to meet larger training tasks. Furthermore, GPU resources within the cluster can be dynamically allocated and scheduled based on task requirements, achieving load balancing and preventing situations where some GPUs are overloaded while others are idle. This flexible resource allocation makes multi-machine, multi-GPU clusters a valuable tool for addressing complex computing needs.

[0006] Furthermore, multi-machine, multi-GPU clusters offer high availability and fault tolerance. In a distributed training environment, if a node or GPU fails, other nodes can continue to operate, ensuring training continuity and stability. Furthermore, by introducing fault-tolerance mechanisms and backup strategies, the risk of single points of failure can be further reduced, ensuring a secure and reliable training process. In summary, using multi-machine, multi-GPU clusters, or even clusters with thousands of GPUs, to train deep learning models is an inevitable choice to meet current computing demands and complexity. It not only significantly improves training speed and efficiency, but also supports the processing of large-scale datasets and the training of complex models. With the continuous advancement of deep learning technology and the continued growth of computing demands, multi-machine, multi-GPU cluster training will become a mainstream trend in the deep learning field.

[0007] As can be seen from the above, how to improve the efficiency of detecting and processing GPU faults during the GPU fault detection and processing process is a problem that needs to be solved urgently. Summary of the Invention

[0008] In view of this, the purpose of the present invention is to provide a GPU fault detection and processing method, apparatus, device, and storage medium, which can improve the efficiency of detecting and processing GPU faults during the GPU fault detection and processing process, thereby improving the security of the generation process. The specific solution is as follows:

[0009] In a first aspect, the present application provides a GPU fault detection and processing method, comprising:

[0010] Collecting processor information corresponding to the GPU using a debug interface set, and collecting regional temperature distribution data of a heat dissipation structure using a preset collection device, then obtaining a number of context switches, a task queue depth, and a parity error count corresponding to the GPU, and determining location information of a video memory abnormal process using a preset positioning technology based on the number of context switches and the parity error count; the processor information includes streaming multiprocessor utilization and register file occupancy status in the GPU;

[0011] Dynamically associate each application performance indicator with the processor information to obtain an association result, compress the association result using a lightweight model to obtain compressed data, and then use a preset spatiotemporal fusion analysis model to perform stream processing, time series prediction, and twin mapping operations on the compressed data to obtain a detection result;

[0012] Construct a target knowledge graph containing historical fault cases, then use a preset agent to adjust the initial monitoring rules based on the historical fault data to obtain a target monitoring rule, and use the target monitoring rule to process the location information from several dimensions based on the target knowledge graph and the detection results to obtain a solution to be processed;

[0013] An early warning signal is generated based on the regional temperature distribution data and the task queue depth, a snapshot strategy is generated based on the early warning signal, and then the to-be-processed solution is processed using a spare computing node based on the snapshot strategy to obtain a target solution for processing the GPU failure.

[0014] Optionally, the method of collecting processor information corresponding to the GPU using a debug interface set, collecting regional temperature distribution data of a heat dissipation structure using a preset collection device, and then obtaining the number of context switches, task queue depth, and parity error count corresponding to the GPU, and determining the location information of the abnormal video memory process based on the number of context switches and the parity error count using a preset positioning technology, includes:

[0015] Integrate multiple debug interfaces to obtain a debug interface set, and collect processor information corresponding to the GPU based on the collection requirements and the debug interface set to obtain processor information; the processor information includes streaming multiprocessor utilization, register file occupancy status, and video memory controller throughput in the GPU;

[0016] Deploying a vibration sensor and a thermal imaging device in the GPU to capture vibration frequency fluctuation data and regional temperature distribution data corresponding to the heat dissipation structure using the vibration sensor and the thermal imaging device;

[0017] A preset kernel-level probe is used to capture the number of context switches, task queue depth and check error count corresponding to the GPU, and then eBPF technology is used to determine the location information corresponding to the abnormal video memory process based on the number of context switches and the check error count; wherein the number of context switches is the number of switches corresponding to CUDA in the GPU; the check error count is the ECRC check error count of the PCIe link layer.

[0018] Optionally, dynamically associating each application performance indicator with the processor information to obtain an association result, and compressing the association result using a lightweight model to obtain compressed data, includes:

[0019] Using the SDK interface and based on user needs, determine each application performance indicator, and dynamically associate each application performance indicator with the processor information using an association algorithm to obtain a correlation coefficient; wherein the application performance indicators include the gradient synchronization delay time corresponding to the deep learning framework and the AllReduce time required for distributed training;

[0020] determining whether the correlation coefficient is less than a preset coefficient threshold to obtain a correlation result; wherein, if the correlation result indicates that the correlation coefficient is less than the preset coefficient threshold, setting the processor information as information to be processed to process the information to be processed;

[0021] Determine information features corresponding to the information to be processed, and then use a lightweight model to filter and compress the association results based on the information features to obtain compressed data.

[0022] Optionally, the method of using a preset spatiotemporal fusion analysis model to perform stream processing, time series prediction, and twin mapping operations on the compressed data to obtain a detection result includes:

[0023] Developing a target operator based on the GPU using Flink, and then using the target operator to perform an identification operation of sudden abnormal data on the compressed data based on a preset sliding window and an isolation forest algorithm to obtain a first identification result;

[0024] determining a load curve, ambient temperature and humidity, and power grid fluctuation corresponding to the GPU, and processing the load curve, the ambient temperature and humidity, and the power grid fluctuation using a preset hybrid network to obtain a second recognition result;

[0025] Building a plurality of virtual images based on the GPU, and synchronizing the core voltage and memory frequency corresponding to each virtual image with the processor information to obtain hardware stress change information under different loads;

[0026] A detection result corresponding to the GPU is determined based on the first recognition result, the second recognition result, and the hardware stress change information.

[0027] Optionally, the target knowledge graph containing historical fault cases is constructed, and then the initial monitoring rules are adjusted based on the historical fault data using a preset agent to obtain target monitoring rules, and the location information is processed from several dimensions based on the target knowledge graph and the detection results using the target monitoring rules to obtain a solution to be processed, including:

[0028] Construct an initial knowledge graph based on historical failure cases, hardware manuals, and driver source code, parse the initial knowledge graph using a preset parsing model to obtain a target knowledge graph, and construct a preset intelligent agent based on a preset reinforcement learning algorithm;

[0029] Using the preset intelligent agent and based on historical fault data, the initial monitoring rules are adjusted to obtain target monitoring rules. The target monitoring rules and the target knowledge graph are used to locate the root cause of the fault in the location information from the time dimension, space dimension, and business dimension to obtain a fault location result. The detection results are then classified using preset fault classification rules to obtain a fault classification result.

[0030] If the fault classification result is characterized as a level 1 fault, a local self-healing operation is triggered to alleviate the load pressure; if the fault classification result is characterized as a level 2 fault, the current processing data is automatically migrated to the backup GPU; if the fault classification result is characterized as a level 3 fault, the current process is frozen and the hardware health diagnosis process is triggered based on the generated fault information.

[0031] Optionally, generating a warning signal based on the regional temperature distribution data and the task queue depth, generating a snapshot strategy based on the warning signal, and then processing the pending solution based on the snapshot strategy using a spare computing node to obtain a target solution for processing the GPU failure includes:

[0032] Using a preset transfer learning model and a preset hardware-level traceability tool, and based on the regional temperature distribution data and the task queue depth, generating an early warning signal; the early warning signal includes the remaining life of a component corresponding to the GPU and a component isolation strategy;

[0033] Determining a snapshot interval based on the early warning signal and the hardware health of the GPU, then determining a snapshot strategy based on the snapshot interval using a preset tiered storage architecture, and storing a hash value corresponding to the snapshot strategy in a preset blockchain using zero-copy technology, so as to monitor the GPU fault detection process using a preset three-dimensional monitoring sandbox;

[0034] The preset graph neural network is used to determine the solution to be processed based on the snapshot strategy and the preset differential recovery protocol to obtain a target solution for processing GPU failures.

[0035] Optionally, after generating an early warning signal based on the regional temperature distribution data and the task queue depth, generating a snapshot strategy based on the early warning signal, and then processing the pending solution based on the snapshot strategy using a spare computing node to obtain a target solution for processing the GPU failure, the method further includes:

[0036] Determine the full lifecycle events corresponding to the GPU using a preset kernel-level event tracking mechanism, and record the memory access characteristics and parallel stream execution sequence corresponding to the GPU; wherein the lifecycle events include the complete timing path corresponding to the GPU; the memory access characteristics include the cache hit status and memory bank conflict frequency information corresponding to the GPU; and the parallel stream execution sequence includes the inter-stream synchronization status;

[0037] Determining information to be processed corresponding to the GPU based on the full lifecycle events, the video memory access characteristics, and the parallel stream execution sequence, performing noise reduction processing on the information to be processed to obtain a noise reduction processing result, and then merging duplicate log entries in the noise reduction processing result using a clustering algorithm, so as to parse code information corresponding to the fault location in the merged result using a semantic parser;

[0038] Using an artificial intelligence engine and based on the code information, determining an application scenario corresponding to the fault location, and determining an optimization suggestion based on the application scenario; wherein, when the application scenario is characterized as a deep learning scenario, the optimization suggestion is to identify and process the degree of video memory fragmentation and kernel startup frequency corresponding to the merged result; and when the application scenario is characterized as a high-performance computing scenario, the optimization suggestion is to identify and process the cross-node communication delay and floating-point unit utilization status corresponding to the merged result;

[0039] Based on the optimization suggestions, a multi-dimensional analysis report including a hardware health status report, a performance bottleneck analysis report, and a fault correlation diagnosis report is generated, and a visual data cockpit is constructed, so as to use the visual data cockpit to construct an initial three-dimensional cluster topology view based on the multi-dimensional analysis report, and perform color-coded node load level marking operations, time series data waterfall flow superposition and display operations, and abnormal fluctuation point marking operations in the initial three-dimensional cluster topology view to obtain a three-dimensional cluster topology view to be processed;

[0040] A real interaction channel is established based on the three-dimensional cluster topology view to be processed, so as to overlay and display real-time operating parameters in the device entity corresponding to the GPU through the real interaction channel, and scale the historical data curve corresponding to the GPU after receiving gesture operation instructions to obtain a target three-dimensional cluster topology view including multi-node performance comparison results; the real interaction channel is used to maintain a data cache of a preset duration for the three-dimensional cluster topology view to be processed in an offline state.

[0041] In a second aspect, the present application provides a GPU fault detection and processing device, comprising:

[0042] a location information determination module configured to collect processor information corresponding to the GPU using a debug interface set, collect regional temperature distribution data of a heat dissipation structure using a preset acquisition device, and then obtain a number of context switches, a task queue depth, and a parity error count corresponding to the GPU, so as to determine the location information of a video memory abnormal process using a preset positioning technique and based on the number of context switches and the parity error count; the processor information includes streaming multiprocessor utilization and register file occupancy status in the GPU;

[0043] A detection result determination module is used to dynamically associate each application performance indicator with the processor information to obtain an association result, compress the association result using a lightweight model to obtain compressed data, and then use a preset spatiotemporal fusion analysis model to perform stream processing, time series prediction, and twin mapping operations on the compressed data to obtain a detection result;

[0044] A first solution determination module is configured to construct a target knowledge graph containing historical fault cases, and then use a preset agent to adjust initial monitoring rules based on historical fault data to obtain target monitoring rules. The target monitoring rules are then used to process the location information from multiple dimensions based on the target knowledge graph and the detection results to obtain a solution to be processed.

[0045] The second solution determination module is used to generate an early warning signal based on the regional temperature distribution data and the task queue depth, to generate a snapshot strategy based on the early warning signal, and then use the spare computing node and process the pending solution based on the snapshot strategy to obtain a target solution for handling GPU failures.

[0046] In a third aspect, the present application provides an electronic device, comprising:

[0047] Memory, used to store computer programs;

[0048] The processor is configured to execute the computer program to implement the aforementioned GPU fault detection and processing method.

[0049] In a fourth aspect, the present application provides a computer-readable storage medium for storing a computer program, wherein the computer program implements the aforementioned GPU fault detection and processing method when executed by a processor.

[0050] As can be seen from the above, before performing GPU fault detection and processing, the present application needs to use the debugging interface set to collect the processor information corresponding to the GPU, and use the preset acquisition device to collect the regional temperature distribution data of the heat dissipation structure, and then obtain the context switching number, task queue depth and check error count corresponding to the GPU, so as to use the preset positioning technology and determine the location information of the abnormal process of the video memory based on the context switching number and check error count; the processor information includes the streaming multi-processor utilization and register file occupancy status in the GPU; each application performance indicator is dynamically associated with the processor information to obtain the association result, and the association result is compressed using a lightweight model to obtain compressed data , and then use the preset spatiotemporal fusion analysis model to perform stream processing, time series prediction and twin mapping operations on the compressed data to obtain the detection results; construct a target knowledge graph containing historical fault cases, and then use the preset intelligent agent to adjust the initial monitoring rules based on the historical fault data to obtain the target monitoring rules, and use the target monitoring rules to process the location information from several dimensions based on the target knowledge graph and the detection results to obtain the solution to be processed; generate an early warning signal based on the regional temperature distribution data and the task queue depth, and generate a snapshot strategy based on the early warning signal, and then use the spare computing node to process the solution to be processed based on the snapshot strategy to obtain the target solution for handling GPU failures.

[0051] It can be seen that this application first needs to use the debugging interface set to collect the processor information corresponding to the GPU, and use the preset collection device to collect the regional temperature distribution data of the heat dissipation structure, and then obtain the context switching number, task queue depth and check error count corresponding to the GPU, so as to use the preset positioning technology and determine the location information of the abnormal memory process based on the context switching number and check error count; then, each application performance indicator is dynamically associated with the processor information to obtain the association result, and the association result is compressed using a lightweight model to obtain compressed data, and then the compressed data is subjected to stream processing, timing prediction and twin mapping operations using the preset spatiotemporal fusion analysis model to obtain the detection result; then, a target knowledge graph containing historical fault cases is constructed, and then the preset intelligent agent is used to adjust the initial monitoring rules based on the historical fault data to obtain the target monitoring rules, so as to use the target monitoring rules and the target knowledge graph and the detection results to process the location information from several dimensions to obtain the solution to be processed; finally, an early warning signal is generated based on the regional temperature distribution data and the task queue depth, and a snapshot strategy is generated based on the early warning signal, and then the backup computing node is used to process the solution to be processed based on the snapshot strategy to obtain a target solution for processing GPU failures. In this way, the efficiency of detecting and handling GPU faults is improved during the GPU fault detection and handling process, thereby improving the security of the build process. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0053] Figure 1 This is a flow chart of a GPU fault detection and processing method disclosed in this application;

[0054] Figure 2 This is a specific data flow diagram for parallel training disclosed in this application;

[0055] Figure 3 This is a flowchart of a specific data recovery based on snapshot disclosed in this application;

[0056] Figure 4 A schematic diagram of a specific process of GPU fault detection disclosed in this application;

[0057] Figure 5 This is a schematic diagram of the structure of a GPU fault detection and processing device disclosed in this application;

[0058] Figure 6 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0059] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0060] At present, the use of multi-machine, multi-card, or even 1,000-card GPU clusters to train deep learning models is mainly due to the urgent needs and significant advantages in the following aspects. First, with the rapid development of deep learning technology, the complexity and number of parameters of the model have increased dramatically, which has posed unprecedented challenges to computing resources. Complex applications such as large language models and image recognition systems often require billions or even hundreds of billions of parameters. These huge models are almost impossible to complete effective training in a single-machine, single-card environment. Multi-machine, multi-card GPU clusters can make full use of the computing power of multiple GPUs in the cluster, and distribute training tasks to multiple GPUs for parallel processing through data parallelism or model parallelism, thereby significantly improving training speed and shortening training cycles. Secondly, the training of deep learning models cannot be separated from the support of large-scale data sets. In addition, multi-machine, multi-card GPU clusters are also highly scalable, flexible, highly available, and fault-tolerant. To this end, the present application provides a GPU fault detection and processing method that can improve the efficiency of detecting and processing faults in the GPU during GPU fault detection and processing, thereby improving the security of the generation process. See. Figure 1 As shown, an embodiment of the present invention discloses a GPU fault detection and processing method, including:

[0061] Step S11: Using the debugging interface set to collect processor information corresponding to the GPU, and using a preset collection device to collect regional temperature distribution data of the heat dissipation structure, then obtaining the number of context switches, task queue depth and verification error count corresponding to the GPU, so as to use a preset positioning technology and determine the location information of the abnormal video memory process based on the number of context switches and the verification error count; the processor information includes the streaming multiprocessor utilization and register file occupancy status in the GPU.

[0062] In this embodiment, during GPU fault detection and processing, a three-level refined perception architecture is constructed to comprehensively capture the corresponding GPU operating status. The process for constructing this three-level refined perception architecture is as follows: First, based on the traditional NVML (NVIDIA Management Library), proprietary debugging interfaces such as NVIDIA Nsight Systems (NVIDIA performance analysis tool) are deeply integrated to achieve nanosecond-level high-frequency sampling of streaming multiprocessor (SM) utilization, register file occupancy, and memory controller throughput. MEMS (Micro-Electro-Mechanical Systems) vibration sensors and infrared thermal imaging equipment are also deployed to monitor abnormal heat sink vibration and core hotspot temperature gradients in real time, thereby accurately locating hardware-level anomalies. Subsequently, a kernel-level probe driver was developed to capture in real time the number of CUDA (Compute Unified Device Architecture) context switches, GPU task queue depth (a congestion warning will be triggered if it exceeds 1024), and PCIe (Peripheral Component Interconnect Express) link layer ECRC (Error Checking and Reporting Capability) check error counts. Memory page error events were dynamically tracked through eBPF (extended Berkeley Packet Filter) technology, thereby accurately locating the video memory leakage process.

[0063] Specifically, a debug interface set is used to collect processor information corresponding to a GPU, and a preset acquisition device is used to collect regional temperature distribution data of a heat dissipation structure. Then, the number of context switches, task queue depth, and check error count corresponding to the GPU are obtained, so as to determine the location information of a video memory abnormal process based on the number of context switches and the check error count using a preset positioning technology. This can include: integrating a number of debug interfaces to obtain a debug interface set, and collecting processor information corresponding to the GPU based on acquisition requirements and the debug interface set to obtain processor information; the processor information includes streaming multiprocessor utilization, register file occupancy status, and video memory controller throughput in the GPU; deploying a vibration sensor and a thermal imaging device in the GPU to capture vibration frequency fluctuation data and regional temperature distribution data corresponding to the heat dissipation structure using the vibration sensor and the thermal imaging device; using a preset kernel-level probe to capture the number of context switches, task queue depth, and check error count corresponding to the GPU, and then using eBPF technology to determine the location information corresponding to the video memory abnormal process based on the number of context switches and the check error count; wherein the number of context switches is the number of switches corresponding to CUDA in the GPU; and the check error count is the ECRC check error count of the PCIe link layer.

[0064] Step S12: Dynamically associate each application performance indicator with the processor information to obtain an association result, and use a lightweight model to compress the association result to obtain compressed data. Then, use a preset spatiotemporal fusion analysis model to perform stream processing, timing prediction, and twin mapping operations on the compressed data to obtain a detection result.

[0065] In this embodiment, application semantic layer adaptation is required. In one specific implementation, the process for performing application semantic layer adaptation is as follows: a standardized SDK (Software Development Kit) interface is designed to support user-defined monitoring indicators, such as the gradient synchronization delay of deep learning frameworks and the AllReduce (global reduction) time of distributed training. An algorithm automatically associates hardware indicators with application KPIs (Key Performance Indicators) (for example, optimization recommendations are triggered when the throughput-memory bandwidth correlation coefficient is less than 0.6). A lightweight TinyML model (with parameters less than 10MB) is then deployed on the edge to filter invalid data in real time. This increases the original data compression ratio to 1:20 while retaining 98% of key features, significantly reducing cloud transmission pressure.

[0066] Specifically, each application performance indicator is dynamically associated with the processor information to obtain an association result, and the association result is compressed using a lightweight model to obtain compressed data, which may include: using the SDK interface and based on user needs to determine each application performance indicator, and using the association algorithm to dynamically associate each application performance indicator with the processor information to obtain a correlation coefficient; wherein the application performance indicators include the gradient synchronization delay time corresponding to the deep learning framework and the AllReduce time consumed for distributed training; judging whether the correlation coefficient is less than a preset coefficient threshold to obtain an association result; wherein, if the correlation result indicates that the correlation coefficient is less than the preset coefficient threshold, the processor information is set as information to be processed to process the information to be processed; determining the information features corresponding to the information to be processed, and then using the lightweight model to filter and compress the association results based on the information features to obtain compressed data.

[0067] Furthermore, the embodiment of the present application needs to use a spatiotemporal fusion analysis model to deeply analyze the data, and the analysis process is as follows: First, perform real-time stream processing operations on the data: develop GPU-specific operators based on Flink (Apache Flink, a distributed stream processing framework), and use millisecond-level sliding windows and isolation forest algorithms to identify 3 in real time. The system detects unexpected abnormal data (such as a sudden temperature rise of 15°C / second) and keeps the false alarm rate below 0.3%. Subsequently, the system performs time series forecasting on the data: a hybrid LSTM-Attention (Long Short-Term Memory with Attention Mechanism) network is deployed. Inputting multi-dimensional data such as the previous 24 hours' load curve, ambient temperature and humidity, and power grid fluctuations, the network predicts computing bottlenecks within the next hour with 94% accuracy, thereby triggering resource preheating mechanisms in advance. Furthermore, the data is mapped using a digital twin: a virtual image is created for each GPU. By bidirectionally synchronizing physical parameters (core voltage, memory frequency) with operational data, the system simulates hardware stress changes under different loads in real time, providing 72 hours' advance warning of potential risks such as memory controller aging.

[0068] Specifically, the preset spatiotemporal fusion analysis model is used to perform stream processing, time series prediction and twin mapping operations on the compressed data to obtain detection results, which may include: using Flink and developing a target operator based on the GPU, and then using the target operator and based on the preset sliding window and isolation forest algorithm to perform sudden abnormal data identification operations on the compressed data to obtain a first identification result; determining the load curve, ambient temperature and humidity, and power grid fluctuations corresponding to the GPU, and using the preset hybrid network to process the load curve, ambient temperature and humidity, and power grid fluctuations to obtain a second identification result; constructing several virtual images based on the GPU, and synchronizing the core voltage and memory frequency corresponding to each virtual image with the processor information to obtain hardware stress change information under different loads; determining the detection result corresponding to the GPU based on the first identification result, the second identification result, and the hardware stress change information.

[0069] Step S13: construct a target knowledge graph containing historical fault cases, and then use the preset intelligent agent to adjust the initial monitoring rules based on the historical fault data to obtain the target monitoring rules, so as to use the target monitoring rules and process the location information from several dimensions based on the target knowledge graph and the detection results to obtain the solution to be processed.

[0070] In this embodiment, a dynamic threshold adjustment and hierarchical response mechanism are designed to improve fault handling efficiency. The process for optimizing dynamic thresholds is as follows: a reinforcement learning-based intelligent agent automatically adjusts monitoring rules based on historical fault data. For example, during low load conditions at night, this embodiment of the application requires lowering the temperature threshold from 85°C to 80°C to ensure performance while reducing heat dissipation.

[0071] In a specific implementation, different levels of faults correspond to different response operations: if it is a level one warning (yellow), local self-healing operations are triggered, such as dynamically reducing the frequency by 10%, starting video memory page recycling, and alleviating temporary load pressure; if it is a level two warning (orange), non-critical tasks are automatically migrated to the backup GPU, and 30% of the computing power is reserved for system self-checking to prevent the fault from spreading; if it is a level three fault (red), the current task is immediately frozen, a fault capsule (including register dump data) is generated with a volume compressed to 1 / 50 of the original, and the hardware health diagnosis process is simultaneously triggered.

[0072] Furthermore, embodiments of the present application require the construction of a fault knowledge graph and a three-dimensional positioning matrix to enable intelligent data analysis. In one specific implementation, the process for constructing the knowledge graph is as follows: More than 2,000 fault cases, hardware manuals, and driver source code are integrated to form a knowledge network consisting of 12 major categories and 87 subcategories. Subsequently, a BERT-ERNIE (idirectional Encoder Representations from Transformers-ERNIE: Enhanced Representation through Knowledge Integration) pre-trained model is used to parse unstructured logs, such as CUDA runtime errors, resulting in an entity recognition accuracy of 92% and the ability to automatically associate similar fault nodes. Furthermore, a three-dimensional positioning matrix is ​​used to cross-locate the root cause of the fault from three dimensions: time (sliding window analysis of abnormal parameter sequences 10 minutes before the fault), space (communication delay matrix to locate network or PCIe link faults), and business (task heatmap to identify CUDA kernel launch conflicts), significantly narrowing the scope of investigation.

[0073] Specifically, a target knowledge graph containing historical fault cases is constructed, and then the preset intelligent agent is used to adjust the initial monitoring rules based on the historical fault data to obtain the target monitoring rules, and the target monitoring rules are used to process the location information from several dimensions based on the target knowledge graph and the detection results to obtain the solution to be processed, which may include: constructing an initial knowledge graph based on historical fault cases, hardware manuals and driver source codes, parsing the initial knowledge graph with a preset parsing model to obtain the target knowledge graph, and constructing a preset intelligent agent based on a preset reinforcement learning algorithm; using the preset intelligent agent to adjust the initial monitoring rules based on the historical fault data to obtain Target monitoring rules are used, and target monitoring rules and target knowledge graphs are used to locate the root cause of the fault in location information from the time dimension, space dimension and business dimension respectively to obtain the fault location result, and then the detection results are graded using the preset fault classification rules to obtain the fault classification result; if the fault classification result is characterized as a level 1 fault, a local self-healing operation is triggered to relieve the load pressure; if the fault classification result is characterized as a level 2 fault, the current processing data is automatically migrated to the backup GPU; if the fault classification result is characterized as a level 3 fault, the current process is frozen, and the hardware health diagnosis process is triggered based on the generated fault information.

[0074] It's worth noting that embodiments of this application can enhance operational efficiency by developing augmented reality (AR) to enhance the monitoring system. In one specific implementation, this embodiment utilizes the Microsoft Hololens 2 (a mixed reality device) to allow maintenance personnel to visually visualize virtual data projections corresponding to the physical GPU. Green light indicates normal operation, while red flashes indicate faulty components. Furthermore, this embodiment supports gesture penetration to view the internal temperature distribution of the GPU, and voice commands to quickly retrieve historical fault-related logs. For example, "Show all data for the video memory overflow event from last Wednesday," thus enabling three-dimensional interactive diagnosis.

[0075] Step S14: Generate an early warning signal based on the regional temperature distribution data and the task queue depth, generate a snapshot strategy based on the early warning signal, and then use the spare computing node and process the pending solution based on the snapshot strategy to obtain a target solution for processing GPU failures.

[0076] In this embodiment, the embodiment of the present application can enhance the diagnostic capability by deploying a transfer learning model and a hardware-level traceability tool. Among them, the transfer learning model can be trained based on a large number of samples, and then only 50 new samples are needed to quickly adapt to the new cluster. Furthermore, the embodiment of the present application can also reproduce the fault scenario in a sandbox environment through Agent-Based (Agent-Based Modeling, i.e., agent-based modeling) modeling to predict the diffusion path of problems such as video memory leakage and generate isolation strategies in advance. Then, the process of hardware-level traceability includes: using the JTAG debugging interface to obtain register dump data, and combining waveform analysis to locate transistor-level faults (such as CUDA core floating-point unit errors). It is worth mentioning that the embodiment of the present application can calculate the remaining life of core components by establishing a hardware aging assessment model and based on the operating time and the number of temperature cycles.

[0077] It is worth mentioning that the data flow during parallel training is shown in the figure below: Figure 2 As shown, when performing data training, the embodiment of the present application needs to build an adaptive snapshot engine to improve recovery efficiency, and the flowchart of data recovery based on snapshots is as follows Figure 3As shown. In a specific embodiment, the embodiment of the present application needs to dynamically adjust the snapshot interval according to the complexity of the training task (gradient norm fluctuation) and the health of the hardware, wherein, during the stable period, the embodiment of the present application generates an incremental snapshot every 500 steps (the volume is reduced by 80%), and during the gradient explosion period, it is shortened to 100 steps and the complete calculation graph is recorded, and during the hardware warning period, a full snapshot including the register state is forced to be generated. In addition, if the local SSD in the embodiment of the present application stores the latest 10 snapshots and supports second-level recovery, the distributed storage architecture is used to retain 24 hours of data for process backtracking, and to archive and preserve key checkpoints (i.e., checkpoints) for a long time to support cross-cluster migration.

[0078] Specifically, an early warning signal is generated based on regional temperature distribution data and task queue depth, a snapshot strategy is generated based on the early warning signal, and then a backup computing node is used to process the pending solution based on the snapshot strategy to obtain a target solution for processing GPU failures, which may include: using a preset transfer learning model and a preset hardware-level traceability tool to generate an early warning signal based on regional temperature distribution data and task queue depth; the early warning signal includes the remaining life of the component corresponding to the GPU and the component isolation strategy; the snapshot interval is determined based on the early warning signal and the hardware health corresponding to the GPU, and then a preset hierarchical storage architecture is used to determine the snapshot strategy based on the snapshot interval, and the hash value corresponding to the snapshot strategy is stored in the preset blockchain using zero-copy technology, so as to monitor the GPU fault detection process using a preset three-dimensional monitoring sandbox; the pending solution is determined based on the snapshot strategy and the preset differentiated recovery protocol to process the pending solution to obtain a target solution for processing GPU failures.

[0079] It is worth mentioning that the flowchart of GPU fault detection is as follows: Figure 4 As shown, after GPU fault detection, the system automatically triggers knowledge graph matching and sandbox simulation verification operations to generate three sets of candidate solutions (prioritizing the optimal solution verified over 500 times, followed by a dynamic adjustment solution, and finally a manual intervention guidance solution), thereby achieving closed-loop management from detection to processing. Furthermore, embodiments of the present application can quantify the training progress delay (based on the checkpoint interval), the risk of model accuracy loss (gradient norm change), and the economic losses (wasted computing power and time cost) caused by computing failures, providing data support for operation and maintenance decisions.

[0080] Furthermore, the embodiment of the present application constructs a flexible fault recovery mechanism to develop a resource self-healing scheduler and acceleration technology, thereby reducing recovery time. Among them, the self-healing process includes automatically calling the spare node pool homogeneous GPU (warm-up < 30 seconds) when a fault occurs, and dynamically scaling the batch size (i.e., batch size) according to the remaining computing power (maintaining efficiency ≥ 85%), and then re-planning the AllReduce communication path through the graph neural network to avoid the faulty link. In addition, synchronous acceleration uses the zero-copy technology of RDMA (Remote Direct Memory Access) to increase the snapshot transmission bandwidth utilization to more than 90%, and uses a differentiated recovery protocol to synchronize the private parameters and public parameters of the faulty node, thereby significantly reducing the amount of data transmission.

[0081] Furthermore, the embodiments of this application require the introduction of blockchain evidence storage and three-dimensional monitoring technology to improve the reliability of detection results. In one specific implementation, blockchain evidence storage is used to generate a unique hash value for each snapshot, record the generated hash value in the blockchain, and support integrity verification within 200ms. The three-dimensional monitoring sandbox is used to visualize the faulty node replacement process in real time (flashing red → gray offline → green connected), dynamically demonstrate parameter synchronization data flow (different colors distinguish transmission status), and then provide a node recovery efficiency ranking to quantify the contribution.

[0082] This embodiment enables kernel-level event tracing and intelligent noise reduction. Specifically, it captures the entire GPU kernel startup lifecycle and records memory access patterns (L2 Cache hit rate and bank conflicts greater than 100 times / second trigger recommendations) and CUDA stream execution order to accurately locate inter-stream synchronization bottlenecks. Furthermore, this embodiment requires log optimization, using a TF-IDF (Term Frequency-Inverse Document Frequency) clustering algorithm to automatically merge 75% of duplicate logs and utilize a log semantic parser to map "GPU-side assertion" errors to specific lines of code, achieving an accuracy rate of 88%.

[0083] Furthermore, the embodiment of the present application needs to build an AI (Artificial Intelligence) driving engine and an automatic reporting system to optimize the suggestions and generate corresponding reports. In a specific embodiment, for deep learning scenarios, when the video memory fragmentation rate is greater than 40%, it is recommended to pre-allocate memory; when the GPU utilization rate is greater than 95% but the throughput is insufficient, the kernel (kernel function) startup storm (greater than 2000 times / second) is diagnosed; for HPC (High Performance Computing) scenarios, when the cross-node delay is greater than 50 NVLink (NVIDIA NVLink, NVIDIA's high-speed interconnect technology) is recommended for transmission, and the integer operation ratio is detected when the floating-point unit utilization is less than 60%. Furthermore, this embodiment of the application supports multi-dimensional templates such as daily operation and maintenance reports (hardware health), monthly development reports (performance bottlenecks), and fault recovery reports (automatically correlating monitoring, logs, and recovery data) to meet the needs of different roles.

[0084] In addition, this embodiment of the application chooses to create a data cockpit and mobile AR reporting to improve usability. The cockpit is used to construct a 3D (three-dimensional) cluster view, in which color-coded load levels can be used to click through to view the internal status of nodes. The time series data waterfall is used to dynamically display 7-day indicator fluctuations, and abnormal points can be automatically highlighted and pop-up diagnosis pop-up. The intelligent question and answer assistant supports natural language queries and returns related charts within 5 seconds. Furthermore, for mobile AR, this embodiment of the application can overlay and display real-time data (temperature, load, health) by scanning the device with a mobile phone, supporting gestures to zoom in and out of historical curves and double-finger comparison of node performance.

[0085] Specifically, an early warning signal is generated based on regional temperature distribution data and task queue depth, a snapshot strategy is generated based on the early warning signal, and then a backup computing node is used to process the pending solution based on the snapshot strategy. After obtaining a target solution for processing GPU failures, it can also include: using a preset kernel-level event tracking mechanism to determine the full life cycle events corresponding to the GPU, and recording the memory access characteristics and parallel stream execution sequence corresponding to the GPU; wherein the life cycle events include the complete timing path corresponding to the GPU; the memory access characteristics include the cache hit status and storage conflict frequency information corresponding to the GPU; the parallel stream execution sequence includes the inter-stream synchronization status; based on the full life cycle events, memory access characteristics and parallel stream execution sequence, the pending information corresponding to the GPU is determined, and the pending information is denoised to obtain a denoising result, and then a clustering algorithm is used to merge duplicate log entries in the denoising result, so as to use a semantic parser to parse the code information corresponding to the fault location in the merged result; using an artificial intelligence engine and based on the code information, the application scenario corresponding to the fault location is judged, so as to determine optimization suggestions based on the application scenario; wherein, when the application scenario is characterized as In deep learning scenarios, the optimization recommendation is to identify and process the degree of video memory fragmentation and kernel startup frequency corresponding to the merged results. When the application scenario is characterized as a high-performance computing scenario, the optimization recommendation is to identify and process the cross-node communication delay and floating-point unit utilization status corresponding to the merged results; based on the optimization recommendation, a multi-dimensional analysis report including a hardware health status report, a performance bottleneck analysis report, and a fault correlation diagnosis report is generated, and a visual data cockpit is constructed. The visual data cockpit is used to construct an initial three-dimensional cluster topology view based on the multi-dimensional analysis report, and color-coded node load level marking operations, time series data waterfall overlay and display operations, and abnormal fluctuation point marking operations are performed in the initial three-dimensional cluster topology view to obtain a three-dimensional cluster topology view to be processed; based on the three-dimensional cluster topology view to be processed, a real-world interaction channel is established to overlay and display real-time operating parameters in the device entity corresponding to the GPU through the real-world interaction channel, and after receiving gesture operation instructions, the historical data curve corresponding to the GPU is scaled to obtain a target three-dimensional cluster topology view including multi-node performance comparison results; the real-world interaction channel is used to maintain data cache for a preset time period for the three-dimensional cluster topology view to be processed in an offline state.

[0086] As can be seen from the above, the embodiment of the present application first needs to use the debug interface set to collect the processor information corresponding to the GPU, and use the preset collection device to collect the regional temperature distribution data of the heat dissipation structure, and then obtain the context switching number, task queue depth and check error count corresponding to the GPU, so as to use the preset positioning technology and determine the location information of the abnormal memory process based on the context switching number and check error count; then, each application performance indicator is dynamically associated with the processor information to obtain the association result, and the association result is compressed using a lightweight model to obtain compressed data, and then the compressed data is subjected to stream processing, timing prediction and twin mapping operations respectively using the preset spatiotemporal fusion analysis model to obtain the detection result; then, a target knowledge graph containing historical fault cases is constructed, and then the initial monitoring rules are adjusted based on the historical fault data using the preset intelligent agent to obtain the target monitoring rules, so as to use the target monitoring rules and the target knowledge graph and the detection results to process the location information from several dimensions to obtain the solution to be processed; finally, an early warning signal is generated based on the regional temperature distribution data and the task queue depth, and a snapshot strategy is generated based on the early warning signal, and then the solution to be processed is processed using the standby computing node based on the snapshot strategy to obtain the target solution for processing GPU faults. In this way, the efficiency of detecting and handling GPU faults is improved during the GPU fault detection and handling process, thereby improving the security of the build process.

[0087] Accordingly, see Figure 5 As shown, the present application also provides a GPU fault detection and processing device, including:

[0088] A location information determination module 11 is configured to collect processor information corresponding to the GPU using a debug interface set, collect regional temperature distribution data of a heat dissipation structure using a preset acquisition device, and then obtain the number of context switches, task queue depth, and parity error count corresponding to the GPU, so as to determine the location information of the abnormal video memory process using a preset positioning technology and based on the number of context switches and the parity error count; the processor information includes streaming multiprocessor utilization and register file occupancy status in the GPU;

[0089] A detection result determination module 12 is configured to dynamically associate each application performance indicator with the processor information to obtain an association result, compress the association result using a lightweight model to obtain compressed data, and then perform stream processing, time series prediction, and twin mapping operations on the compressed data using a preset spatiotemporal fusion analysis model to obtain a detection result;

[0090] A first solution determination module 13 is configured to construct a target knowledge graph containing historical fault cases, and then use a preset agent to adjust the initial monitoring rules based on the historical fault data to obtain a target monitoring rule, and then use the target monitoring rule to process the location information from multiple dimensions based on the target knowledge graph and the detection results to obtain a solution to be processed;

[0091] The second solution determination module 14 is used to generate an early warning signal based on the regional temperature distribution data and the task queue depth, to generate a snapshot strategy based on the early warning signal, and then use the spare computing node and process the pending solution based on the snapshot strategy to obtain a target solution for processing GPU failures.

[0092] As can be seen from the above, before performing GPU fault detection and processing, the embodiment of the present application first needs to use the debugging interface set to collect the processor information corresponding to the GPU, and use the preset acquisition device to collect the regional temperature distribution data of the heat dissipation structure, and then obtain the context switching number, task queue depth and check error count corresponding to the GPU, so as to use the preset positioning technology and determine the location information of the abnormal process of the video memory based on the context switching number and check error count; then, each application performance indicator is dynamically associated with the processor information to obtain the association result, and the association result is compressed using a lightweight model to obtain the compressed data, and then the preset spatiotemporal fusion analysis model is used to analyze the performance of the application. The compressed data is subjected to stream processing, time series prediction, and twin mapping operations to obtain the detection results. A target knowledge graph containing historical fault cases is then constructed. The initial monitoring rules are adjusted based on the historical fault data using a preset agent to obtain the target monitoring rules. The target monitoring rules are then used to process location information from several dimensions based on the target knowledge graph and the detection results to obtain the solution to be processed. Finally, an early warning signal is generated based on the regional temperature distribution data and the task queue depth. A snapshot strategy is generated based on the early warning signal. The solution to be processed is then processed using a backup computing node based on the snapshot strategy to obtain the target solution for handling GPU faults. This improves the efficiency of detecting and processing GPU faults during GPU fault detection and processing, thereby enhancing the security of the generation process.

[0093] In some specific implementations, the location information determining module 11 may specifically include:

[0094] a processor information acquisition unit, configured to integrate a plurality of debug interfaces to obtain a debug interface set, and to acquire processor information corresponding to the GPU based on acquisition requirements and the debug interface set to obtain processor information; the processor information includes streaming multiprocessor utilization, register file occupancy status, and video memory controller throughput in the GPU;

[0095] a heat dissipation structure information acquisition unit, configured to deploy a vibration sensor and a thermal imaging device in the GPU, so as to capture vibration frequency fluctuation data and regional temperature distribution data corresponding to the heat dissipation structure using the vibration sensor and the thermal imaging device;

[0096] A location information determination subunit is used to use a preset kernel-level probe to capture the number of context switches, task queue depth and check error count corresponding to the GPU, and then use eBPF technology to determine the location information corresponding to the abnormal memory process based on the number of context switches and the check error count; wherein the number of context switches is the number of switches corresponding to CUDA in the GPU; the check error count is the ECRC check error count of the PCIe link layer.

[0097] In some specific implementations, the detection result determination module 12 may specifically include:

[0098] A correlation coefficient determination unit, configured to determine each application performance indicator based on user needs using an SDK interface, and dynamically associate each application performance indicator with the processor information using an association algorithm to obtain a correlation coefficient; wherein the application performance indicators include a gradient synchronization delay time corresponding to a deep learning framework and an AllReduce time for distributed training;

[0099] a correlation result determination unit, configured to determine whether the correlation coefficient is less than a preset coefficient threshold, and obtain a correlation result; wherein, if the correlation result indicates that the correlation coefficient is less than the preset coefficient threshold, setting the processor information as information to be processed, so as to process the information to be processed;

[0100] The compressed data acquisition unit is used to determine the information features corresponding to the information to be processed, and then use a lightweight model to filter and compress the association results based on the information features to obtain compressed data.

[0101] In some specific implementations, the detection result determination module 12 may specifically include:

[0102] A first recognition result acquisition unit is configured to develop a target operator using Flink and based on the GPU, and then perform a burst abnormal data recognition operation on the compressed data using the target operator based on a preset sliding window and an isolation forest algorithm to obtain a first recognition result;

[0103] a second recognition result obtaining unit, configured to determine a load curve, ambient temperature and humidity, and power grid fluctuation corresponding to the GPU, and process the load curve, the ambient temperature and humidity, and the power grid fluctuation using a preset hybrid network to obtain a second recognition result;

[0104] a hardware stress change information acquisition unit, configured to construct a plurality of virtual images based on the GPU, and synchronize the core voltage and memory frequency corresponding to each virtual image with the processor information to obtain hardware stress change information under different loads;

[0105] A detection result determination subunit is configured to determine a detection result corresponding to the GPU based on the first recognition result, the second recognition result, and the hardware stress change information.

[0106] In some specific implementations, the first solution determination module 13 may specifically include:

[0107] A knowledge graph parsing unit is used to construct an initial knowledge graph based on historical fault cases, hardware manuals, and driver source code, parse the initial knowledge graph using a preset parsing model to obtain a target knowledge graph, and construct a preset intelligent agent based on a preset reinforcement learning algorithm;

[0108] a fault classification result determination unit, configured to use the preset intelligent agent and adjust the initial monitoring rules based on historical fault data to obtain a target monitoring rule, and use the target monitoring rule and the target knowledge graph to locate the root cause of the fault in the location information from the time dimension, the space dimension, and the business dimension to obtain a fault location result, and then use the preset fault classification rule to classify the detection results to obtain a fault classification result;

[0109] The diagnostic process triggering unit is used to trigger a local self-healing operation for alleviating load pressure if the fault classification result is characterized as a level 1 fault; trigger automatic migration of currently processed data to a backup GPU if the fault classification result is characterized as a level 2 fault; trigger freezing the current process if the fault classification result is characterized as a level 3 fault, and trigger a hardware health diagnostic process based on the generated fault information.

[0110] In some specific implementations, the second solution determination module 14 may specifically include:

[0111] an early warning signal generating unit, configured to generate an early warning signal based on the regional temperature distribution data and the task queue depth using a preset transfer learning model and a preset hardware-level traceability tool; the early warning signal including the remaining life of a component corresponding to the GPU and a component isolation strategy;

[0112] a snapshot strategy determination unit, configured to determine a snapshot interval based on the warning signal and the hardware health corresponding to the GPU, then determine a snapshot strategy based on the snapshot interval using a preset tiered storage architecture, and store a hash value corresponding to the snapshot strategy in a preset blockchain using zero-copy technology, so as to monitor the GPU fault detection process using a preset three-dimensional monitoring sandbox;

[0113] A solution processing unit is used to use a preset graph neural network and determine the solution to be processed based on the snapshot strategy and the preset differential recovery protocol to obtain a target solution for processing GPU failures.

[0114] In some specific implementations, the GPU fault detection and processing device may further include:

[0115] a full lifecycle event determination unit, configured to determine full lifecycle events corresponding to the GPU using a preset kernel-level event tracking mechanism, and record memory access characteristics and parallel stream execution sequences corresponding to the GPU; wherein the lifecycle events include a complete timing path corresponding to the GPU; the memory access characteristics include cache hit status and memory bank conflict frequency information corresponding to the GPU; and the parallel stream execution sequence includes inter-stream synchronization status;

[0116] an information denoising unit, configured to determine to-be-processed information corresponding to the GPU based on the full lifecycle events, the video memory access characteristics, and the parallel stream execution sequence, and to perform denoising on the to-be-processed information to obtain a denoising result; and then to merge duplicate log entries in the denoising result using a clustering algorithm, and to parse code information corresponding to a fault location in the merged result using a semantic parser;

[0117] an optimization suggestion determining unit, configured to utilize an artificial intelligence engine and, based on the code information, determine an application scenario corresponding to the fault location, and determine an optimization suggestion based on the application scenario; wherein, when the application scenario is characterized as a deep learning scenario, the optimization suggestion is to identify and process the degree of video memory fragmentation and kernel startup frequency corresponding to the merged result; and when the application scenario is characterized as a high-performance computing scenario, the optimization suggestion is to identify and process the cross-node communication delay and floating-point unit utilization status corresponding to the merged result;

[0118] a first three-dimensional cluster topology view determining unit, configured to generate a multi-dimensional analysis report including a hardware health status report, a performance bottleneck analysis report, and a fault correlation diagnosis report based on the optimization suggestion, and to construct a visual data cockpit, so as to utilize the visual data cockpit to construct an initial three-dimensional cluster topology view based on the multi-dimensional analysis report, and to perform color-coded node load level marking operations, time series data waterfall overlay and display operations, and abnormal fluctuation point marking operations in the initial three-dimensional cluster topology view, to obtain a three-dimensional cluster topology view to be processed;

[0119] A second three-dimensional cluster topology view determination unit is used to establish a real-life interaction channel based on the three-dimensional cluster topology view to be processed, so as to overlay and display real-time operating parameters in the device entity corresponding to the GPU through the real-life interaction channel, and to scale the historical data curve corresponding to the GPU after receiving gesture operation instructions to obtain a target three-dimensional cluster topology view including multi-node performance comparison results; the real-life interaction channel is used to maintain a data cache of a preset duration for the three-dimensional cluster topology view to be processed in an offline state.

[0120] Furthermore, the embodiment of the present application also discloses an electronic device, Figure 6 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the diagram should not be considered as any limitation on the scope of use of this application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps of the GPU fault detection and processing method disclosed in any of the aforementioned embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0121] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device. The communication protocol it follows is any communication protocol that can be applied to the technical solution of this application and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world. Its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0122] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or CD, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0123] The operating system 221 is used to manage and control the hardware devices on the electronic device 20 and the computer program 222. The operating system 221 can be Windows Server, NetWare, Unix, Linux, etc. In addition to including computer programs capable of implementing the GPU fault detection and processing method disclosed in any of the aforementioned embodiments and executed by the electronic device 20, the computer program 222 can further include computer programs capable of performing other specific tasks.

[0124] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when executed by a processor, the computer program implements the aforementioned GPU fault detection and handling method. The specific steps of this method can be referred to the corresponding content disclosed in the aforementioned embodiments and will not be repeated here.

[0125] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. Reference can be made to the descriptions of the identical or similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and the relevant parts can be referred to the descriptions of the methods.

[0126] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0127] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0128] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0129] The above is a detailed introduction to the technical solution provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the ideas of the present application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A GPU fault detection and processing method, characterized in that: include: Collecting processor information corresponding to the GPU using a debug interface set, and collecting regional temperature distribution data of the heat dissipation structure using a preset collection device, and then obtaining the number of context switches, task queue depth, and parity error count corresponding to the GPU, so as to determine the location information of the abnormal video memory process using a preset positioning technology and based on the number of context switches and the parity error count; The processor information includes streaming multiprocessor utilization and register file occupancy status in the GPU; Dynamically associate each application performance indicator with the processor information to obtain an association result, compress the association result using a lightweight model to obtain compressed data, and then use a preset spatiotemporal fusion analysis model to perform stream processing, time series prediction, and twin mapping operations on the compressed data to obtain a detection result; Construct a target knowledge graph containing historical fault cases, then use a preset agent to adjust the initial monitoring rules based on the historical fault data to obtain a target monitoring rule, and use the target monitoring rule to process the location information from several dimensions based on the target knowledge graph and the detection results to obtain a solution to be processed; An early warning signal is generated based on the regional temperature distribution data and the task queue depth, a snapshot strategy is generated based on the early warning signal, and then the to-be-processed solution is processed using a spare computing node based on the snapshot strategy to obtain a target solution for processing the GPU failure.

2. The GPU fault detection and processing method according to claim 1, characterized in that: The method includes collecting processor information corresponding to the GPU using a debug interface set, collecting regional temperature distribution data of a heat dissipation structure using a preset collection device, and then obtaining the number of context switches, task queue depth, and parity error count corresponding to the GPU, so as to determine the location information of the abnormal video memory process based on the number of context switches and the parity error count using a preset positioning technology, including: Integrate multiple debug interfaces to obtain a debug interface set, and collect processor information corresponding to the GPU based on the collection requirements and the debug interface set to obtain processor information; the processor information includes streaming multiprocessor utilization, register file occupancy status, and video memory controller throughput in the GPU; Deploying a vibration sensor and a thermal imaging device in the GPU to capture vibration frequency fluctuation data and regional temperature distribution data corresponding to the heat dissipation structure using the vibration sensor and the thermal imaging device; A preset kernel-level probe is used to capture the number of context switches, task queue depth and check error count corresponding to the GPU, and then eBPF technology is used to determine the location information corresponding to the abnormal video memory process based on the number of context switches and the check error count; wherein the number of context switches is the number of switches corresponding to CUDA in the GPU; the check error count is the ECRC check error count of the PCIe link layer.

3. The GPU fault detection and processing method according to claim 1, wherein: The dynamically associating each application performance indicator with the processor information to obtain an association result, and compressing the association result using a lightweight model to obtain compressed data, includes: Using the SDK interface and based on user needs, determine each application performance indicator, and dynamically associate each application performance indicator with the processor information using an association algorithm to obtain a correlation coefficient; wherein the application performance indicators include the gradient synchronization delay time corresponding to the deep learning framework and the AllReduce time required for distributed training; determining whether the correlation coefficient is less than a preset coefficient threshold to obtain a correlation result; wherein, if the correlation result indicates that the correlation coefficient is less than the preset coefficient threshold, setting the processor information as information to be processed to process the information to be processed; Determine information features corresponding to the information to be processed, and then use a lightweight model to filter and compress the association results based on the information features to obtain compressed data.

4. The GPU fault detection and processing method according to claim 1, wherein: The preset spatiotemporal fusion analysis model is used to perform stream processing, time series prediction, and twin mapping operations on the compressed data to obtain detection results, including: Developing a target operator based on the GPU using Flink, and then using the target operator to perform an identification operation of sudden abnormal data on the compressed data based on a preset sliding window and an isolation forest algorithm to obtain a first identification result; determining a load curve, ambient temperature and humidity, and power grid fluctuation corresponding to the GPU, and processing the load curve, the ambient temperature and humidity, and the power grid fluctuation using a preset hybrid network to obtain a second recognition result; Building a plurality of virtual images based on the GPU, and synchronizing the core voltage and memory frequency corresponding to each virtual image with the processor information to obtain hardware stress change information under different loads; A detection result corresponding to the GPU is determined based on the first recognition result, the second recognition result, and the hardware stress change information.

5. The GPU fault detection and processing method according to claim 1, wherein: The target knowledge graph containing historical fault cases is constructed, and then the initial monitoring rules are adjusted based on the historical fault data using a preset agent to obtain target monitoring rules. The target monitoring rules are used to process the location information from several dimensions based on the target knowledge graph and the detection results to obtain a solution to be processed, including: Construct an initial knowledge graph based on historical failure cases, hardware manuals, and driver source code, parse the initial knowledge graph using a preset parsing model to obtain a target knowledge graph, and construct a preset intelligent agent based on a preset reinforcement learning algorithm; Using the preset intelligent agent and based on historical fault data, the initial monitoring rules are adjusted to obtain target monitoring rules. The target monitoring rules and the target knowledge graph are used to locate the root cause of the fault in the location information from the time dimension, space dimension, and business dimension to obtain a fault location result. The detection results are then classified using preset fault classification rules to obtain a fault classification result. If the fault classification result is characterized as a level 1 fault, a local self-healing operation is triggered to alleviate the load pressure; if the fault classification result is characterized as a level 2 fault, the current processing data is automatically migrated to the backup GPU; if the fault classification result is characterized as a level 3 fault, the current process is frozen and the hardware health diagnosis process is triggered based on the generated fault information.

6. The GPU fault detection and processing method according to claim 1, wherein: The method further comprises generating a warning signal based on the regional temperature distribution data and the task queue depth, generating a snapshot strategy based on the warning signal, and then processing the pending solution based on the snapshot strategy using a spare computing node to obtain a target solution for processing the GPU failure, including: Using a preset transfer learning model and a preset hardware-level traceability tool, and based on the regional temperature distribution data and the task queue depth, generating an early warning signal; the early warning signal includes the remaining life of a component corresponding to the GPU and a component isolation strategy; Determining a snapshot interval based on the early warning signal and the hardware health of the GPU, then determining a snapshot strategy based on the snapshot interval using a preset tiered storage architecture, and storing a hash value corresponding to the snapshot strategy in a preset blockchain using zero-copy technology, so as to monitor the GPU fault detection process using a preset three-dimensional monitoring sandbox; The preset graph neural network is used to determine the solution to be processed based on the snapshot strategy and the preset differential recovery protocol to obtain a target solution for processing GPU failures.

7. The GPU fault detection and processing method according to any one of claims 1 to 6, characterized in that: After generating an early warning signal based on the regional temperature distribution data and the task queue depth, generating a snapshot strategy based on the early warning signal, and processing the pending solution based on the snapshot strategy using a spare computing node to obtain a target solution for processing the GPU failure, the method further includes: Determine the full lifecycle events corresponding to the GPU using a preset kernel-level event tracking mechanism, and record the memory access characteristics and parallel stream execution sequence corresponding to the GPU; wherein the lifecycle events include the complete timing path corresponding to the GPU; the memory access characteristics include the cache hit status and memory bank conflict frequency information corresponding to the GPU; and the parallel stream execution sequence includes the inter-stream synchronization status; Determining information to be processed corresponding to the GPU based on the full lifecycle events, the video memory access characteristics, and the parallel stream execution sequence, performing noise reduction processing on the information to be processed to obtain a noise reduction processing result, and then merging duplicate log entries in the noise reduction processing result using a clustering algorithm, so as to parse code information corresponding to the fault location in the merged result using a semantic parser; Using an artificial intelligence engine and based on the code information, determining an application scenario corresponding to the fault location, and determining an optimization suggestion based on the application scenario; wherein, when the application scenario is characterized as a deep learning scenario, the optimization suggestion is to identify and process the degree of video memory fragmentation and kernel startup frequency corresponding to the merged result; and when the application scenario is characterized as a high-performance computing scenario, the optimization suggestion is to identify and process the cross-node communication delay and floating-point unit utilization status corresponding to the merged result; Based on the optimization suggestions, a multi-dimensional analysis report including a hardware health status report, a performance bottleneck analysis report, and a fault correlation diagnosis report is generated, and a visual data cockpit is constructed, so as to use the visual data cockpit to construct an initial three-dimensional cluster topology view based on the multi-dimensional analysis report, and perform color-coded node load level marking operations, time series data waterfall flow superposition and display operations, and abnormal fluctuation point marking operations in the initial three-dimensional cluster topology view to obtain a three-dimensional cluster topology view to be processed; A real interaction channel is established based on the three-dimensional cluster topology view to be processed, so as to overlay and display real-time operating parameters in the device entity corresponding to the GPU through the real interaction channel, and scale the historical data curve corresponding to the GPU after receiving gesture operation instructions to obtain a target three-dimensional cluster topology view including multi-node performance comparison results; the real interaction channel is used to maintain a data cache of a preset duration for the three-dimensional cluster topology view to be processed in an offline state.

8. A GPU fault detection and processing device, characterized in that: include: a location information determination module, configured to collect processor information corresponding to the GPU using a debug interface set, collect regional temperature distribution data of a heat dissipation structure using a preset acquisition device, and then obtain a number of context switches, a task queue depth, and a parity error count corresponding to the GPU, so as to determine the location information of the abnormal video memory process using a preset positioning technology and based on the number of context switches and the parity error count; The processor information includes streaming multiprocessor utilization and register file occupancy status in the GPU; A detection result determination module is used to dynamically associate each application performance indicator with the processor information to obtain an association result, compress the association result using a lightweight model to obtain compressed data, and then use a preset spatiotemporal fusion analysis model to perform stream processing, time series prediction, and twin mapping operations on the compressed data to obtain a detection result; A first solution determination module is configured to construct a target knowledge graph containing historical fault cases, and then use a preset agent to adjust initial monitoring rules based on historical fault data to obtain target monitoring rules. The target monitoring rules are then used to process the location information from multiple dimensions based on the target knowledge graph and the detection results to obtain a solution to be processed. The second solution determination module is used to generate an early warning signal based on the regional temperature distribution data and the task queue depth, to generate a snapshot strategy based on the early warning signal, and then use the spare computing node and process the pending solution based on the snapshot strategy to obtain a target solution for handling GPU failures.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the GPU fault detection and processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that Used to store a computer program, wherein when the computer program is executed by a processor, it implements the GPU fault detection and processing method according to any one of claims 1 to 7.

Images