GPU fault diagnosis system, diagnosis method, equipment and readable storage medium

Abstract

The invention discloses a GPU fault diagnosis system, method and device and a readable storage medium. The system comprises a one-key log collection module used for achieving the one-key collection of system in-band logs, GPU fault logs and GPU operation state index files, a fault log inspection module used for inspecting GPU logs, outputting fault information and giving processing suggestions, a GPU real-time state detection module used for detecting the real-time running state of the GPU in a one-key mode, automatically discovering faults and giving out processing suggestions, a GPU pressure test module used for diagnosing GPU difficult faults, a GPU drive one-key replacement module used for one-key replacement of a GPU drive version, a log module used for outputting and storing logs, and a GPU driving module used for guaranteeing the operation of the GPU. The functions of one-key log collection, fault log inspection, GPU real-time state detection, GPU pressure testing, GPU drive one-key replacement, processing suggestion providing and the like can be achieved, and engineers can conveniently position faults on site and feed back the collected logs to the background for processing.

Description

technical field [0001] The invention relates to the technical field of GPU fault diagnosis, in particular to a GPU fault diagnosis system, a diagnosis method, a device and a readable storage medium. Background technique [0002] At present, the fields related to artificial intelligence are developing rapidly, and the number of AI server market has increased sharply. The rapid fault diagnosis of GPU (Graphics Processing Unit, English: Graphics Processing Unit, abbreviation: GPU) has also become an important part of server after-sales service. Diagnose the following problems: [0003] On-site operation and maintenance personnel / third-party engineers have different technical levels, and it takes a long time of training and practice to be competent for GPU fault diagnosis. There are problems such as long time for GPU fault diagnosis and low fault judgment accuracy. [0004] Customers are not allowed to perform GPU troubleshooting after logging in to the OS, and it is extremely ...

AI Technical Summary

Problems solved by technology

[0006] When judging GPU faults based on server out-of-band logs, because the current server out-of-band logs cannot effectively monitor the GPU running status, the accuracy of fault diagnosis is low

[0007] When judging GPU faults based on the in-band logs provided by customers, due to differences in customer technical levels, the in-band logs provided by customers are incompletely collected, and GPU faults cannot be accurately located.

[0008] When judging GPU faults based on customer repair descriptions, different customers describe GPU faults in different ways, and the description accuracy is poor, which brings great trouble to fault location.

[0009] In addition, most customers do not allow logging into the OS for troubleshooting and do not provide in-band logs

The technical level of on-site engineers is uneven, and the GPU fault diagnosis methods and tools are complicated to use

Images