Fault detection method and device for GPU (Graphic Processing Unit) and electronic equipment

Abstract

The invention provides a fault detection method and device for a GPU (Graphic Processing Unit), electronic equipment, a computer readable storage medium, a computer program product, an automatic driving system and a vehicle, and relates to the technical field of computers, in particular to the technical field of automatic driving. The fault detection method comprises the following steps: acquiring system log information related to a plurality of GPUs; at least according to the system log information, whether GPU hardware type faults exist in one or more GPUs in the multiple GPUs or not is determined, and the GPU hardware type faults are faults related to GPU hardware; and in response to determining that the GPU hardware fault exists, at least outputting prompt information indicating the GPU hardware fault. According to the method, GPU hardware faults can be detected and identified in time, and corresponding prompt information is provided, so that problems can be quickly responded and solved, and the reliability and the stability of a system are improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, in particular to the field of autonomous driving technology, and specifically to a fault detection method, device, electronic device, computer-readable storage medium, computer program product, autonomous driving system and vehicle of a graphics processing unit (GPU). Background Art

[0002] As autonomous driving technology continues to mature, more and more autonomous or driverless vehicles will appear on urban roads in the future, thereby improving urban transportation efficiency. The algorithms that implement autonomous driving technology are inseparable from powerful computing power, among which the Graphics Processing Unit (GPU) plays an important role in processing AI algorithms.

[0003] GPUs are good at parallel computing and graphics processing, which are critical for perception and visual processing tasks in autonomous driving technology. GPUs can process large amounts of data and perform parallel computing at the same time, providing high-speed image processing and real-time feedback, thereby achieving accurate target detection, obstacle recognition, and road identification.

[0004] By leveraging the powerful computing power of GPUs, autonomous driving technology can process data and make decisions more efficiently, enabling vehicles to perceive and understand the surrounding environment in real time and make accurate driving decisions and operations. This can improve driving safety, reduce the risk of traffic accidents, and bring higher efficiency and convenience to urban transportation.

[0005] The methods described in this section are not necessarily methods that have been previously conceived or employed. Unless otherwise indicated, it should not be assumed that any method described in this section is considered to be prior art simply because it is included in this section. Similarly, unless otherwise indicated, the issues mentioned in this section should not be considered to have been recognized in any prior art. Summary of the invention

[0006] The present disclosure provides a method, device, electronic device, computer-readable storage medium, computer program product, automatic driving system and vehicle for detecting faults of a graphics processing unit (GPU).

[0007] According to one aspect of the present disclosure, a fault detection method for a graphics processing unit (GPU) is provided, comprising: obtaining system log information related to multiple GPUs; determining whether one or more GPUs among the multiple GPUs has a GPU hardware type fault based at least on the system log information, wherein the GPU hardware type fault is a fault related to GPU hardware; and in response to determining that the GPU hardware type fault exists, at least outputting prompt information indicating the GPU hardware type fault.

[0008] According to another aspect of the present disclosure, a fault detection device for a graphics processing unit (GPU) is provided, comprising: a first module, for obtaining system log information related to multiple GPUs; a second module, for determining whether one or more GPUs among the multiple GPUs has a GPU hardware type fault based at least on the system log information, wherein the GPU hardware type fault is a fault related to GPU hardware; and a third module, for outputting at least prompt information indicating a GPU hardware type fault in response to determining that the GPU hardware type fault exists.

[0009] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned fault detection method.

[0010] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the above-mentioned fault detection method.

[0011] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program implements the above fault detection method when executed by a processor.

[0012] According to another aspect of the present disclosure, an autonomous driving system is provided, including: a plurality of GPUs; and the above-mentioned fault detection device or the above-mentioned electronic device.

[0013] According to another aspect of the present disclosure, a vehicle is provided, comprising: the above-mentioned automatic driving system.

[0014] According to one or more embodiments of the present disclosure, GPU hardware failures can be detected and identified in a timely manner, and corresponding prompt information can be provided to quickly respond to and solve the problem, thereby improving the reliability and stability of the system.

[0015] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0017] Figure 1 A schematic diagram showing an exemplary system in which the various methods described herein may be implemented according to an embodiment of the present disclosure;

[0018] Figure 2 A flowchart of a fault detection method for a graphics processing unit (GPU) according to an embodiment of the present disclosure is shown;

[0019] Figure 3 A flowchart of a fault detection method for a graphics processing unit GPU according to another embodiment of the present disclosure is shown;

[0020] Figure 4 A flowchart of a fault detection method for a graphics processing unit (GPU) according to another embodiment of the present disclosure is shown;

[0021] Figure 5 A flowchart of a fault detection method for a graphics processing unit GPU according to an embodiment of the present disclosure is shown;

[0022] Figure 6 is a schematic diagram showing the GPU being in place in the system log information according to an embodiment of the present disclosure;

[0023] Figure 7 is a schematic diagram showing GPU missing in system log information according to an embodiment of the present disclosure;

[0024] Figure 8 is a schematic diagram showing a GPU bandwidth value and a GPU rate value in system log information according to an embodiment of the present disclosure;

[0025] Fig. 9 is a schematic diagram showing the actual temperature of a GPU in system log information according to an embodiment of the present disclosure;

[0026] Fig.10 is a schematic diagram showing a first temperature threshold and a second temperature threshold in system log information according to an embodiment of the present disclosure;

[0027] Fig.11 is a schematic diagram showing error information in system log information according to an embodiment of the present disclosure;

[0028] Fig.12A structural block diagram of a fault detection device for a graphics processing unit (GPU) according to an embodiment of the present disclosure is shown; and

[0029] Fig.13 A structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0030] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0031] In the present disclosure, unless otherwise specified, the use of the terms "first", "second", etc. to describe various elements is not intended to limit the positional relationship, temporal relationship, or importance relationship of these elements, and such terms are only used to distinguish one element from another element. In some examples, the first element and the second element may refer to the same instance of the element, and in some cases, based on the description of the context, they may also refer to different instances.

[0032] The terms used in the description of various examples in this disclosure are only for the purpose of describing specific examples and are not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the element can be one or more. In addition, the term "and / or" used in this disclosure covers any one of the listed items and all possible combinations.

[0033] In the related technology, in the autonomous driving vehicle server system, each GPU has its corresponding functional algorithm and application division of labor. For example, GPU1 is responsible for 2D image processing (autonomous driving vehicle camera data processing), GPU2 is responsible for positioning algorithm processing, and GPU3 is responsible for 3D point cloud processing or algorithm fusion, etc. Any hardware failure of any GPU will directly block the implementation of the autonomous driving function. If the failure occurs before the autonomous driving starts the closed-loop behavior, the vehicle will not enter the autonomous driving behavior. If the failure occurs during the autonomous driving behavior, it will directly cause the vehicle behavior to be interrupted or the vehicle to lose control, and in severe cases, it may cause a traffic accident.

[0034] In the field of autonomous driving technology, if GPU hardware failures are not handled promptly and effectively, for example, the following serious consequences will occur. First, the system crashes. The phenomenon is similar to the blue screen crash on the PC side. The on-board server will instantly lose all perception and control, causing the vehicle to completely escape algorithm control and collide at will. Second, the algorithm is lost. During the closed-loop startup or closed-loop driving of autonomous driving, the vehicle will lose important algorithm functions (such as loss of positioning or navigation), causing the vehicle to drive in the wrong way. Third, the system algorithm processing performance is reduced. If the physical link initialized by the GPU is abnormal (such as the loss of part of the link), there will be a computing bottleneck or reduced efficiency when the actual computing power is executed, thereby reducing the autonomous driving capability or causing safety hazards of autonomous driving.

[0035] Therefore, the present disclosure provides a GPU hardware failure detection technology to ensure the safety of its system functions.

[0036] The embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0037] Figure 1 FIG. 1 is a schematic diagram of an exemplary system 100 in which various methods and apparatuses described herein may be implemented according to an embodiment of the present disclosure. Figure 1 , the system 100 includes a motor vehicle 110 , a server 120 , and one or more communication networks 130 coupling the motor vehicle 110 to the server 120 .

[0038] In an embodiment of the present disclosure, the motor vehicle 110 may include a computing device according to an embodiment of the present disclosure and / or be configured to perform a method according to an embodiment of the present disclosure.

[0039] The server 120 may run one or more services or software applications that enable the execution of a fault detection method for a graphics processing unit GPU. In some embodiments, the server 120 may also provide other services or software applications, which may include non-virtualized environments and virtualized environments. Figure 1 In the configuration shown, the server 120 may include one or more components that implement the functions performed by the server 120. These components may include software components, hardware components, or a combination thereof that can be executed by one or more processors. A user of the motor vehicle 110 may in turn utilize one or more client applications to interact with the server 120 to utilize the services provided by these components. It should be understood that a variety of different system configurations are possible, which may differ from the system 100. Therefore, Figure 1 is one example of a system for implementing the various methods described herein and is not intended to be limiting.

[0040] Server 120 may include one or more general purpose computers, dedicated server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and / or combination. Server 120 may include one or more virtual machines running virtual operating systems, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that may be virtualized to maintain a server's virtual storage device). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.

[0041] The computing units in the server 120 may run one or more operating systems including any of the above operating systems and any commercially available server operating systems. The server 120 may also run any of a variety of additional server applications and / or middle-tier applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.

[0042] In some embodiments, server 120 may include one or more applications to analyze and consolidate data feeds and / or event updates received from motor vehicle 110. Server 120 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of motor vehicle 110.

[0043] The network 130 may be any type of network known to those skilled in the art that may support data communications using any of a variety of available protocols, including, but not limited to, TCP / IP, SNA, IPX, etc. By way of example only, one or more networks 130 may be a satellite communications network, a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a blockchain network, a public switched telephone network (PSTN), an infrared network, a wireless network (including, for example, Bluetooth, WiFi), and / or any combination of these and other networks.

[0044] The system 100 may also include one or more databases 150. In some embodiments, these databases may be used to store data and other information. For example, one or more of the databases 150 may be used to store information such as audio files and video files. The data repository 150 may reside in various locations. For example, the data repository used by the server 120 may be local to the server 120, or may be remote from the server 120 and may communicate with the server 120 via a network-based or dedicated connection. The data repository 150 may be of different types. In some embodiments, the data repository used by the server 120 may be a database, such as a relational database. One or more of these databases may store, update, and retrieve data to and from the database in response to a command.

[0045] In some embodiments, one or more of the databases 150 may also be used by an application to store application data. The databases used by the application may be different types of databases, such as a key-value store, an object store, or a conventional store backed by a file system.

[0046] The motor vehicle 110 may include a sensor 111 for sensing the surrounding environment. The sensor 111 may include one or more of the following sensors: a visual camera, an infrared camera, an ultrasonic sensor, a millimeter wave radar, and a laser radar (LiDAR). Different sensors may provide different detection accuracy and range. The camera may be installed at the front, rear, or other location of the vehicle. The visual camera may capture the situation inside and outside the vehicle in real time and present it to the driver and / or passengers. In addition, by analyzing the images captured by the visual camera, information such as traffic light indications, intersection conditions, and the operating status of other vehicles may be obtained. The infrared camera may capture objects under night vision conditions. Ultrasonic sensors may be installed around the vehicle to measure the distance of objects outside the vehicle from the vehicle using the characteristics of strong ultrasonic directionality. Millimeter wave radars may be installed at the front, rear, or other location of the vehicle to measure the distance of objects outside the vehicle from the vehicle using the characteristics of electromagnetic waves. Laser radars may be installed at the front, rear, or other location of the vehicle to detect object edges and shape information, thereby identifying and tracking objects. Due to the Doppler effect, the radar device may also measure the speed change of the vehicle and the moving object.

[0047] The motor vehicle 110 may also include a communication device 112. The communication device 112 may include a satellite positioning module capable of receiving satellite positioning signals (e.g., Beidou, GPS, GLONASS, and GALILEO) from satellites 141 and generating coordinates based on these signals. The communication device 112 may also include a module for communicating with a mobile communication base station 142, and the mobile communication network may implement any suitable communication technology, such as GSM / GPRS, CDMA, LTE, and other current or evolving wireless communication technologies (e.g., 5G technology). The communication device 112 may also have a vehicle-to-vehicle or vehicle-to-everything (V2X) module, which is configured to implement, for example, vehicle-to-vehicle (V2V) communication with other vehicles 143 and vehicle-to-infrastructure (V2I) communication with infrastructure 144. In addition, the communication device 112 may also have a module configured to communicate with a user terminal 145 (including but not limited to a smartphone, a tablet computer, or a wearable device such as a watch) by, for example, using a wireless LAN or Bluetooth of the IEEE802.11 standard. Using the communication device 112, the motor vehicle 110 may also access the server 120 via the network 130.

[0048] The motor vehicle 110 may also include a control device 113. The control device 113 may include a processor that communicates with various types of computer-readable storage devices or media, such as a central processing unit (GPU) or a graphics processing unit (GPU), or other dedicated processors, etc. The control device 113 may include an automatic driving system for automatically controlling various actuators in the vehicle. The automatic driving system is configured to control the powertrain, steering system, and braking system of the motor vehicle 110 (not shown) via multiple actuators in response to inputs from multiple sensors 111 or other input devices to control acceleration, steering, and braking, respectively, without human intervention or limited human intervention. Some processing functions of the control device 113 can be implemented through cloud computing. For example, some processing can be performed using an on-board processor, while other processing can be performed using computing resources in the cloud. The control device 113 can be configured to perform a method according to the present disclosure. In addition, the control device 113 can be implemented as an example of a computing device on the motor vehicle side (client) according to the present disclosure.

[0049] Figure 1 The system 100 may be configured and operated in various ways to enable the application of various methods and apparatuses described in the present disclosure.

[0050] like Figure 2As shown, some embodiments of the present disclosure provide a fault detection method 200 for a graphics processing unit (GPU). The method 200 includes steps S201 to S202.

[0051] In step S201, system log information related to multiple GPUs is obtained.

[0052] In the example, the system log information is log data that records multiple GPU operations, states, and events.

[0053] In step S202, it is determined whether one or more GPUs among the multiple GPUs have a GPU hardware fault at least according to the system log information. The GPU hardware fault is a fault related to the GPU hardware.

[0054] In step S203, in response to determining that a GPU hardware fault exists, at least prompt information indicating a GPU hardware fault is output.

[0055] Method 200 performs fault judgment based on system log information, can detect and identify GPU hardware faults in a timely manner, and provide corresponding prompt information to quickly respond to and solve problems, improve system reliability and stability, and effectively prevent and avoid safety hazards caused by GPU hardware faults.

[0056] Before GPU hardware failures have a serious impact on the system, the autonomous driving behavior is terminated and the corresponding fault repair is completed to avoid the degradation of the autonomous driving system's computing performance or the crash of the algorithm due to GPU hardware failure, which further causes the autonomous driving vehicle to lose control, thereby avoiding the occurrence of such traffic accidents.

[0057] In some embodiments of the present disclosure, GPU hardware failure includes at least one of a GPU link failure, a GPU over-temperature failure, and a GPU operation failure.

[0058] In this example, the GPU link failure indicates that there is a fault in the link connection between the GPU and the bus. The GPU link failure requires hardware maintenance or physical repair to complete the fault repair.

[0059] For example, the link is a PCIe (Peripheral Component Interconnect Express) link. In traditional servers or autonomous driving vehicle server systems, GPU data communication is based on PCIe links. As a PCIe device in the system, the GPU can be directly mounted on the system Root Complex (root node), or it can communicate with the GPU or other devices through a PCIe switch.

[0060] In this example, the GPU overtemperature fault indicates that the actual GPU temperature exceeds the temperature limit. GPU overtemperature faults may cause GPU performance degradation or even damage. Specifically, GPUs generate heat during operation, which is usually dissipated by the cooling system. GPU overtemperature may be caused by cooling system failure, insufficient heat dissipation, poor package design, or excessively high ambient temperature. GPU overtemperature faults need to be repaired by cooling system maintenance or cooling strategy adjustment.

[0061] GPU overheating failure will have an impact on the computing performance of the entire system, although it will not directly damage the hardware function. But when the GPU overheats, it will perform power regulation internally to reduce the operating frequency to reduce heat generation and protect the hardware chip. However, during the autonomous driving process, this frequency reduction behavior is very dangerous. As the GPU main frequency is reduced, the computing efficiency will drop significantly, which will directly affect the vehicle's autonomous driving behavior, and may cause slow action or wrong decision-making, leading to traffic accidents. Therefore, keeping the GPU within a safe temperature range is very important to ensure the stability and reliability of the autonomous driving system.

[0062] In the example, the GPU operation failure indicates a hardware failure during GPU operation. The GPU operation failure may be related to the life of the GPU and the hardware quality. The GPU operation failure requires the system to be repaired by replacing the GPU.

[0063] This embodiment can effectively identify and monitor various GPU hardware failures by detecting GPU link failures, GPU over-temperature failures, and GPU operation failures, thereby improving system reliability and performance.

[0064] like Figure 3 As shown, some embodiments of the present disclosure provide a fault detection method 300 for a graphics processing unit (GPU). Figure 3 The situation that GPU hardware failure includes GPU link failure detection is shown. GPU link failure includes GPU missing failure, GPU bandwidth failure and GPU rate failure. System log information includes GPU access results. System log information can be, for example, device initialization information generated based on the Linux operating system. Device initialization information refers to device status, configuration and operation information recorded during device startup or reset.

[0065] In step S301, it is determined whether there is a GPU missing fault according to the GPU access result.

[0066] In the example, the GPU access result is obtained by accessing multiple GPUs according to the GPU identification list. The GPU identification list includes multiple GPU identifications corresponding to the multiple GPUs respectively. The GPU identification list includes GPU identifications that the system expects to access. The GPU missing fault indicates that there is a GPU that failed to be accessed.

[0067] For example, the GPU access result may include a GPU identifier of a successful access. When accessing each GPU, if the GPU device is successfully accessed, the GPU identifier is recorded. If the access fails or the identifier of the GPU device accessed does not match the preset identifier, the GPU identifier is not recorded.

[0068] For example, the GPU access result may include access statuses corresponding to multiple GPUs. The access status includes at least one of access success, access failure, and not detected. When accessing each GPU, if the GPU device is successfully accessed and matches the preset identifier, the access result of the GPU will be recorded as access success. Conversely, if the access fails or the identifier of the GPU device accessed does not match the preset identifier, the access result of the GPU will be recorded as access failure or not detected.

[0069] For example, the successfully accessed GPU ID in the GPU access result is matched with the GPU ID list. If they match, it means that multiple GPUs are accessed, that is, there is no GPU missing fault. If they do not match, it means that there is a GPU that failed to be accessed, that is, there is a GPU missing fault.

[0070] For example, determine whether there is an access failure in multiple access states in the GPU access result. If yes, it means that there is a GPU that failed to access, that is, there is a GPU missing fault. If not, it means that multiple GPUs have been accessed, that is, there is no GPU missing fault.

[0071] If it is determined in step S301 that there is no GPU missing fault, go to step S302 for execution. If it is determined in step S301 that there is a GPU missing fault, go to step S303 for execution.

[0072] In step S302 , in response to determining that there is no GPU missing fault, it is determined, based on system log information, whether one or more GPUs among the plurality of GPUs has a GPU bandwidth fault and / or a GPU rate fault.

[0073] In the example, the GPU bandwidth exception fault indicates that the actual link bandwidth value of the GPU does not match the expected bandwidth value. The GPU rate exception fault indicates that the actual link rate value of the GPU does not match the expected rate value.

[0074] This embodiment can comprehensively detect GPU link failures by determining GPU missing failures, GPU bandwidth failures, and GPU rate failures.

[0075] In some embodiments of the present disclosure, the system log information also includes the number of times the system is reinitialized. Accordingly, the method 300 also includes step S303 and step S304.

[0076] In step S303 , in response to determining that a GPU missing fault exists, the number of times the system is reinitialized is determined.

[0077] In the example, system reinitialization is an operation triggered in response to determining that the GPU is missing. In practice, the number of times the system is reinitialized can be recorded, and a counter can be increased at each reinitialization. Whenever a GPU is detected to be missing, system reinitialization can be triggered, and the number of reinitializations can be updated.

[0078] Reinitialization means that the system attempts to restart or reconfigure related hardware and software components to restore normal operating state.

[0079] In step S304, in response to determining that the number of system reinitializations is less than or equal to the number threshold, at least prompt information indicating a first risk level fault is output.

[0080] In the example, the prompt information indicating the first risk level fault indicates that the GPU link fault risk level is low, so as to prompt the operation and maintenance offline vehicle to take corresponding measures for maintenance or repair.

[0081] In the example, while outputting prompt information indicating a first risk level fault, reinitialization may continue to be performed to repair the GPU missing fault.

[0082] By automatically analyzing the number of system reinitialization times, the risk level of GPU hardware failures can be quickly determined and corresponding prompt information can be output.

[0083] In some embodiments of the present disclosure, the method 300 further includes step S305. In step S305, in response to determining that the number of system reinitializations is greater than a number threshold, at least outputting prompt information indicating a second risk level fault. The second risk level is greater than the first risk level.

[0084] In the example, the prompt information indicating the second risk level fault indicates that the GPU link fault risk level is high, prompting the need to prohibit the vehicle from entering the autonomous driving closed-loop operation and to perform maintenance or repair immediately.

[0085] By outputting multi-level prompt information, it is possible to distinguish GPU hardware faults of different risk levels and provide targeted fault responses, thereby improving the efficiency and accuracy of fault handling.

[0086] In some embodiments of the present disclosure, the GPU identification includes a bus device function value BDF. Accordingly, step S304 includes, in response to determining that the number of times the system is reinitialized is less than or equal to the number threshold, outputting a prompt message indicating a first risk level fault and a bus device function value of a failed access. The bus device function value of a failed access indicates a missing GPU.

[0087] BDF is a unique identifier used to describe devices on a bus. BDF usually consists of three parts: bus number (BusNumber), device number (Device Number) and function number (Function Number). The bus number is used to identify the bus to which the device is connected. The device number is used to identify different devices on the same bus. The function number is used to identify different functions on a multi-function device. For example, the BDF identifier is 1d:00.0, "1d" is the bus number, "00" is the device number. "0" is the function number. The BDF identifier "1d:00.0" represents a device or component connected to bus number 1d, device number 00, and function number 0. The BDF identifier uniquely identifies a specific bus device function in the system.

[0088] By outputting prompt information indicating the fault and the function value of the bus device where access failed, accurate and detailed fault information is provided, which helps users quickly find and locate faulty devices of the first risk level, thereby improving the efficiency and accuracy of fault handling.

[0089] In some embodiments of the present disclosure, the GPU identifier includes a BDF. Accordingly, step S305 includes, in response to determining that the number of times the system is reinitialized is greater than a number threshold, outputting a prompt message indicating a second risk level fault and a bus device function value of a failed access. The bus device function value of a failed access indicates a missing GPU.

[0090] By outputting prompt information indicating the fault and the function value of the bus device where access failed, accurate and detailed fault information is provided, which helps users quickly find and locate faulty devices at the second risk level, thereby improving the efficiency and accuracy of fault handling.

[0091] In some embodiments of the present disclosure, the system log information further includes a plurality of actual link bandwidth values ​​corresponding to the plurality of GPUs and / or a plurality of actual link rate values ​​corresponding to the plurality of GPUs.

[0092] Accordingly, step S302 includes at least one of step S3021 and step S3022.

[0093] In step S3021, based on the correspondence table between GPU identifiers and expected bandwidth values, it is determined whether the actual link bandwidth values ​​of the multiple GPUs match the corresponding expected bandwidth values. The mismatch between the actual link bandwidth values ​​of the multiple GPUs and the corresponding expected bandwidth values ​​indicates a GPU bandwidth failure.

[0094] In this example, B = {x|x∈B, x = 1, 2, 4, 8, 16}, and the set B is the PCIe link bandwidth of the GPU. i ∈B,B i It is determined after the hardware design is completed and is a known value of the system, representing the expected bandwidth value of the i-th GPU. i There are five possible values ​​(x1 / x2 / x4 / x8 / x16). |B|=N, the number of elements in set B is N, which represents the number of GPUs (multiple GPUs) in the system. i is the actual link bandwidth value of the i-th GPU.

[0095] For example, when the actual link bandwidth value of the i-th GPU does not match the corresponding expected bandwidth value, that is, b i ≠B i , indicating a GPU bandwidth failure.

[0096] In step S3022, based on the correspondence table between GPU identifiers and expected rate values, it is determined whether the actual rate values ​​of the multiple GPUs match the corresponding expected rate values. The mismatch between the actual link rate values ​​of the multiple GPUs and the corresponding expected rate values ​​indicates a GPU rate failure.

[0097] In this example, S = {x|x∈S, x = 2.5, 5, 8, 16}, and the set S is the PCIe link rate of the GPU. i ∈B,B i It is determined after the hardware design is completed and is a known value of the system, representing the expected rate value of the i-th GPU. i There are 4 possible values ​​(2.5GT / s, 5GT / s, 8GT / s and 16GT / s). |S| = N, the number of elements in the set S is N, which represents the number of GPUs (multiple GPUs) in the system. i is the actual link rate value of the i-th GPU.

[0098] For example, when the actual link rate value of the i-th GPU does not match the corresponding expected rate value, that is, s i ≠S i , indicating a GPU rate failure.

[0099] By establishing a corresponding relationship table and matching actual values ​​with expected values, an effective way is provided to determine whether there is a fault in the bandwidth and rate of the GPU, and problems can be discovered and diagnosed in a timely manner to maintain system performance and stability.

[0100] It should be noted that the order of GPU bandwidth detection and GPU bandwidth detection is not limited here. For example, GPU bandwidth detection is performed first, and then GPU rate detection is performed. For another example, GPU rate detection is performed first, and then GPU bandwidth detection is performed.

[0101] In one example, GPU bandwidth detection can be performed first, and then GPU rate detection can be performed. Since GPU bandwidth detection is simpler than GPU rate detection, if the GPU fails the bandwidth detection, there is no need to perform GPU rate detection to avoid wasting resources.

[0102] In some embodiments of the present disclosure, the method 300 further includes step S306. In step S306, in response to determining that the actual link bandwidth value of at least one GPU does not match the corresponding expected bandwidth value and / or the actual link rate value of at least one GPU does not match the corresponding expected rate value, at least prompt information indicating a second risk level fault is output.

[0103] In the example, the prompt information indicating the second risk level fault indicates that the GPU link fault risk level is high, prompting the need to prohibit the vehicle from entering the autonomous driving closed-loop operation and to perform maintenance or repair immediately.

[0104] In addition, when the actual link bandwidth values ​​of multiple GPUs match the corresponding expected bandwidth values ​​or the actual link rate values ​​of multiple GPUs match the corresponding expected rate values, the system is allowed to operate normally and the vehicle meets the requirements of the autonomous driving closed loop.

[0105] By outputting the prompt information of the second risk level, the user can quickly understand the severity of the fault and take appropriate measures to ensure the stability and performance of the system.

[0106] In some embodiments of the present disclosure, the GPU identifier includes BDF. Accordingly, step S306 includes outputting prompt information indicating a second risk level fault and an unmatched bus device function value in response to determining that the actual link bandwidth value of at least one GPU does not match the corresponding expected bandwidth value and / or the actual link rate value of at least one GPU does not match the corresponding expected rate value. The unmatched bus device function value indicates a GPU with abnormal bandwidth and / or abnormal rate.

[0107] By outputting the bus device function values ​​of GPUs with abnormal bandwidth and / or abnormal rate, users can quickly locate GPUs with abnormal bandwidth and / or abnormal rate and take appropriate measures to troubleshoot and repair the problem to ensure normal operation and performance of the system.

[0108] Method 300 determines GPU link faults based on system log information, and can timely detect and identify GPU link faults such as GPU missing faults, GPU bandwidth faults, and GPU rate faults, thereby achieving comprehensive detection of GPU link faults. In addition, prompt information of multiple risk levels is output to distinguish GPU hardware faults of different risk levels. In addition, providing the device number of the faulty GPU helps users quickly discover and locate the faulty GPU, thereby improving the efficiency and accuracy of fault handling.

[0109] like Figure 4 As shown, some embodiments of the present disclosure provide a fault detection method 400 for a graphics processing unit (GPU). Figure 4 The situation of GPU overtemperature fault detection is shown. The system log information includes multiple actual temperatures corresponding to multiple GPUs. The multiple actual temperatures corresponding to multiple GPUs can be obtained by polling multiple GPUs at regular intervals (for example, once per minute). The system log information may include polling information of information during GPU operation.

[0110] In step S401, for each GPU among the multiple GPUs: based on the correspondence table between the GPU identifier and the first temperature threshold and the second temperature threshold, the actual temperature of the GPU is compared with the corresponding first temperature threshold and the second temperature threshold. The second temperature threshold is greater than the first temperature threshold.

[0111] In this example, T = {x|x∈T, x>0}, and the set T is the real-time temperature of all GPUs (multiple GPUs) in the system, T i Represents the real-time temperature of the i-th GPU (T i >0). |T|=N, the number of elements of set T is the number of GPUs in the system N.

[0112] In the example, the first temperature threshold L i and the second temperature threshold H i is the two-level temperature threshold of the i-th GPU. For example, the first temperature threshold L i The second temperature threshold H may be reduced by 5 degrees based on the maximum operating temperature value (Max Operating Temp) of the GPU. i It can be reduced by 5 degrees based on the GPU's slowdown temperature value (Slowdown Temp).

[0113] Usually, GPU manufacturers usually provide the GPU's maximum operating temperature (Max Operating Temp) and slowdown temperature (Slowdown Temp) as its specification parameters. These parameters are known and are very important to ensure the normal and safe operation of the GPU.

[0114] If it is determined in step S401 that the actual temperature of the GPU is between the corresponding first temperature threshold and the corresponding second temperature threshold, the process proceeds to step S402. If it is determined in step S401 that the actual temperature of the GPU is greater than the second temperature threshold, the process proceeds to step S403. If it is determined in step S401 that the actual temperature of the GPU is less than the first temperature threshold, the process proceeds to step S404.

[0115] In step S402, in response to determining that the actual temperature of the GPU is between the corresponding first temperature threshold and the corresponding second temperature threshold, at least prompt information indicating a first risk level fault is output. The first risk level fault includes a first level temperature anomaly.

[0116] In the example, the prompt information indicating the first risk level fault indicates that the GPU overtemperature fault risk level is medium, so as to prompt the vehicle to terminate the autonomous driving behavior as soon as possible and the operation and maintenance needs to take the vehicle offline for inspection as soon as possible.

[0117] By setting different levels of temperature thresholds and corresponding three temperature intervals, and comparing them with the actual temperature of the GPU, it is possible to accurately determine faults of different risk levels and provide corresponding prompt information in a timely manner. This helps protect the health of the GPU, prevent faults caused by overheating, and improve system reliability and stability.

[0118] In some embodiments of the present disclosure, the GPU identifier includes BDF. Accordingly, step S402 includes outputting prompt information indicating a first risk level fault and a bus device function value of the GPU in response to determining that the actual temperature of the GPU is between a corresponding first temperature threshold and a corresponding second temperature threshold.

[0119] By outputting prompt information indicating faults and bus device function values ​​of temperature abnormalities, accurate and detailed fault information is provided, which helps users quickly find and locate faulty GPUs with first-level temperature abnormalities, thereby improving the efficiency and accuracy of fault handling.

[0120] In some embodiments of the present disclosure, the method 400 further includes step S403. In step S403, in response to determining that the actual temperature of the GPU is greater than the corresponding second temperature threshold, at least a prompt message indicating a second risk level fault is output. The second risk level fault includes a second level temperature anomaly. The second level temperature anomaly is greater than the first level temperature anomaly.

[0121] In the example, the prompt information indicating the second risk level fault indicates that the GPU overheating fault risk level is high, to prompt that the GPU is seriously overheated and that the autonomous driving behavior needs to be terminated immediately and the vehicle needs to be taken offline for inspection.

[0122] By outputting the prompt information of the second risk level, the user can quickly understand the severity of the fault and take appropriate measures to ensure the stability and performance of the system.

[0123] In some embodiments of the present disclosure, the GPU identifier includes BDF. Accordingly, step S403 includes outputting prompt information indicating a second risk level fault and a bus device function value of the GPU.

[0124] By outputting prompt information indicating faults and bus device function values ​​of temperature anomalies, accurate and detailed fault information is provided, which helps users quickly discover and locate faulty GPUs with second-level temperature anomalies, thereby improving the efficiency and accuracy of fault handling.

[0125] In some embodiments of the present disclosure, the method 400 further includes step S404. In step S404, in response to determining that the actual temperature of the GPU is less than the corresponding first temperature threshold, outputting prompt information instructing to enhance the heat dissipation and cooling system.

[0126] In the example, the prompt information indicating the enhancement of the heat dissipation and cooling system indicates that there is no need to interfere with the driving of the autonomous driving vehicle, and only the corresponding heat dissipation and cooling system needs to be enhanced.

[0127] By providing timely reminders of enhanced cooling systems, users can take proactive measures to optimize system temperature management and reduce the risk of failure.

[0128] Method 400 determines GPU overtemperature faults based on system log information. By distinguishing between first-level and second-level temperature anomalies, users can better understand the working status of the GPU and take appropriate measures to solve the problem when necessary, thereby improving the reliability and stability of the system. In addition, providing the device number of the faulty GPU helps users quickly discover and locate the GPU with abnormal temperature, thereby improving the efficiency and accuracy of fault handling.

[0129] like Figure 5 As shown, some embodiments of the present disclosure provide a fault detection method 500 for a graphics processing unit (GPU). Figure 5 A situation of GPU operation failure is shown. The system log information includes an error report of the GPU driver. The error report of the GPU driver includes at least one error message. The error message includes an error name and a corresponding GPU identifier. The error message can be obtained by polling multiple GPUs at a regular interval (e.g., once a minute).

[0130] In step S501, for each error message in at least one error message: based on a correspondence table between error types and error names, determine the error type corresponding to the error message.

[0131] In an example, the error type includes at least one of a non-hardware error and a hardware error. A hardware error refers to an error related to hardware that occurs during GPU operation. A non-hardware error refers to an error related to software, configuration, network, or other non-hardware factors that occurs during GPU operation.

[0132] In an example, the error type may be preset. For example, the error type may be reclassified based on conventional error categories provided by a GPU manufacturer, and the error type may be further divided into non-hardware errors and hardware errors.

[0133] For example, an error report can be an XID error (X Error ID). XID errors are error messages for Nvidia GPUs that indicate errors that occurred during GPU operation. Errors can be caused by a variety of reasons, including driver issues, hardware issues, application issues, and more. XID errors can contain a variety of error codes and messages, requiring further investigation and analysis to determine the specific source of the problem. XID error codes typically begin with "XID:" followed by a string of numbers. XID error codes can provide information about graphics driver or hardware issues, helping users and technical support personnel diagnose and resolve issues. Each XID error code corresponds to a specific error type, such as memory error, GPU error, display error, and more.

[0134] In step S502, in response to determining that the error type corresponding to the error information is a hardware error, at least prompt information indicating a second risk level fault is output.

[0135] In the example, the hardware error belongs to the blocking error, which may interrupt the normal operation of the system. Therefore, the prompt information indicating the second risk level fault indicates the error information to prompt that the automatic driving behavior needs to be terminated immediately and the vehicle needs to be taken offline for repair.

[0136] This embodiment can determine whether one or more GPUs among multiple GPUs have GPU operation failures according to the error type of the error information. By defining different error types, hardware failures occurring during GPU operation, i.e., GPU operation failures, are distinguished. And by outputting the prompt information of the second risk level, the user can quickly understand that the error information belongs to a hardware error of a greater severity, and take appropriate measures to ensure the stability and performance of the system.

[0137] In some embodiments of the present disclosure, the GPU identifier includes BDF. Accordingly, step S502 includes, in response to determining that the error type corresponding to the error information is a hardware error, outputting prompt information indicating a second risk level fault and a bus device function value of the GPU corresponding to the error information.

[0138] By providing the BDF of GPUs with hardware errors, the fault information is associated with specific hardware devices, making troubleshooting and repair more accurate and efficient.

[0139] In some embodiments of the present disclosure, the method 500 further includes step S503. In step S503, in response to determining that the error type of the error information is a non-hardware error, outputting prompt information indicating a first risk level fault.

[0140] In the example, non-hardware errors belong to non-blocking errors. Non-blocking errors refer to errors that occur during the operation of software or systems, which will not cause the system to completely stop or block, and the system can still continue to perform other tasks or operations. Therefore, by outputting the prompt information of the first risk level for prompt alarm, the error information can be notified to the R&D personnel to optimize the application layer code.

[0141] In some embodiments of the present disclosure, the method 500 further includes step S504. In step S504, in response to determining that the error type of the error information does not belong to a non-hardware error or a hardware error, a prompt message instructing to perform a risk assessment on the error information is output.

[0142] In the example, the error type does not belong to non-hardware errors and hardware errors, which are errors of the location type. Therefore, a prompt message indicating risk assessment of the error information is output to remind the user or system administrator to conduct risk assessment to fully understand the potential risks and possible impacts of the error. The error information can be notified to the R&D personnel for risk assessment or application code optimization and adaptation.

[0143] Method 500 determines GPU operation faults based on system log information, and distinguishes whether the error information belongs to GPU operation faults (hardware faults occurring during GPU operation) by defining error types of hardware faults and non-hardware faults. In addition, providing the device number of the faulty GPU helps users to quickly find and locate the faulty GPU, thereby improving the efficiency and accuracy of fault handling.

[0144] In one example, the fault detection method provided by the present disclosure can be applicable to various types of GPU devices such as Nvidia GPU. Since the characteristics of GPU hardware failures are generally applicable to devices of multiple GPU manufacturers, the method can be applied to GPUs of different brands and models. Through the analysis of system log information and GPU access results, hardware failures such as GPU link failures, GPU missing failures, GPU bandwidth failures, and GPU rate failures can be detected and diagnosed to improve the reliability and stability of the system. Whether it is a GPU from Nvidia or other manufacturers, the method can provide beneficial technical effects for GPU fault detection.

[0145] In one example, the prompt information output by any of the above embodiments can be output through a vehicle-mounted display device or a wireless network, notifying operation and maintenance personnel in the vehicle or cloud to perform effective maintenance on risky equipment, thereby intercepting the risk of GPU hardware failure and avoiding safety accidents that may be caused by such failure risks.

[0146] In one example, Figure 6 FIG. 1 is a schematic diagram showing the GPU in position in the system log information according to an embodiment of the present disclosure. Figure 6 As shown, the system log information includes the GPU identifier (such as BDF) that is accessed and exists. Figure 6 In the example, the BDFs that can be accessed include 1d:00.0.

[0147] In one example, Figure 7 FIG. 1 is a schematic diagram showing a GPU missing in system log information according to an embodiment of the present disclosure. Figure 7 As shown in FIG. 1 , the system log information includes GPU identifiers (such as BDF) that were not accessed, failed to access, or missing. Figure 7 In the case of a GPU access failure, only the bus number in the BDF (such as 1e, 1f) is accessed, but the corresponding device number and function number are not accessed, indicating that the GPU access failed.

[0148] In one example, Figure 8 Schematic diagram showing the GPU bandwidth value and GPU rate value in the system log information according to an embodiment of the present disclosure. Figure 8 As shown in Figure 1, the system log information includes bandwidth value (Width) and rate value (Speed). Figure 8 In the example, the bandwidth value (Width) is x8 and the rate value (Speed) is 2.5GT / s.

[0149] In one example, Fig. 9 FIG. 1 is a schematic diagram showing the actual temperature of a GPU in system log information according to an embodiment of the present disclosure. Fig. 9 As shown in the figure, the system log information includes the actual GPU temperature (Temp). Fig. 9 In the example, the actual temperature of the GPU with BDF 1B:00.0 is 52°C. The actual temperature of the GPU with BDF 1D:00.0 is 51°C.

[0150] In one example, Fig.10 is a schematic diagram showing the first temperature threshold and the second temperature threshold in the system log information according to an embodiment of the present disclosure. Fig.10 As shown in the figure, the system log information includes the maximum operating temperature value (MaxOperating Temp) and the slowdown temperature value (Slowdown Temp). Fig.10 In the example, the maximum operating temperature of the GPU is 91°C. The deceleration temperature of the GPU is 94°C. For example, the first temperature threshold L i The first temperature threshold L can be reduced by 5 degrees based on the maximum operating temperature value (MaxOperating Temp) of the GPU. i is 86. For example, the second temperature threshold H i The second temperature threshold H can be reduced by 5 degrees based on the GPU slowdown temperature value (Slowdown Temp). i It is 89.

[0151] In one example, Fig.11 FIG. 1 is a schematic diagram showing error information in system log information according to an embodiment of the present disclosure. Fig.11 As shown in , the system log information includes error information (such as XID error). Fig.11 In the "XID48" error, the GPU operation failure is a double-bit ECC (Error Correcting Code) error of the GPU memory particle, which is an unrecoverable memory transmission error. The system equipment must be taken offline in time for processing, otherwise it will directly affect the autonomous driving function and the safety of vehicle driving. "XID48" is a GPU operation failure.

[0152] Some embodiments of the present disclosure include Fig.12 FIG. 1 shows a structural block diagram of a fault detection device for a graphics processing unit (GPU) according to an embodiment of the present disclosure. Fig.12 As shown, the fault detection device 1200 may include a first module 1201 , a second module 1202 and a third module 1203 .

[0153] The first module 1201 is used to obtain system log information related to multiple GPUs.

[0154] The second module 1202 is used to determine whether one or more GPUs in the plurality of GPUs have a GPU hardware fault at least according to the system log information, wherein the GPU hardware fault is a fault related to the GPU hardware.

[0155] The third module 1203 is used to output at least prompt information indicating the GPU hardware fault in response to determining that there is a GPU hardware fault.

[0156] It should be understood that Fig.12 The various modules of the apparatus 1200 shown in FIG. 1 can be used in conjunction with the reference Figure 2 The various steps in the described method 200 correspond to each other. Therefore, the operations, features and advantages described above for method 200 are also applicable to device 1200 and the modules included therein. For the sake of brevity, certain operations, features and advantages are not repeated here.

[0157] Although specific functions are discussed above with reference to specific modules, it should be noted that the functions of the various modules discussed herein can be divided into multiple modules, and / or at least some functions of multiple modules can be combined into a single module. The specific module execution action discussed herein includes that the specific module itself performs the action, or alternatively the specific module calls or otherwise accesses another component or module that performs the action (or performs the action together with the specific module). Therefore, the specific module that performs an action can include the specific module itself that performs the action and / or the specific module calls or otherwise accesses another module that performs the action. For example, the second module 1202 / the third module 1203 described above can be combined into a single module in some embodiments.

[0158] It should also be understood that various techniques may be described herein in the general context of software hardware elements or program modules. Fig.12The modules described can be implemented in hardware or in hardware in combination with software and / or firmware. For example, these modules can be implemented as computer program code / instructions, which are configured to be executed in one or more processors and stored in a computer-readable storage medium. Alternatively, these modules can be implemented as hardware logic / circuit. For example, in some embodiments, one or more of the first module 1201, the second module 1202, and the third module 1203 can be implemented together in a system on chip (System on Chip, SoC). SoC may include an integrated circuit chip (which includes a processor (e.g., a central processing unit (CPU), a microcontroller, a microprocessor, a digital signal processor (DSP), etc.), a memory, one or more communication interfaces, and / or one or more components in other circuits), and may optionally execute the received program code and / or include embedded firmware to perform functions.

[0159] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned fault detection method.

[0160] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the steps of any one of the method embodiments described above.

[0161] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein the computer program implements the steps of any method embodiment described above when executed by a processor.

[0162] According to another aspect of the present disclosure, an autonomous driving system is provided, including: a plurality of GPUs; and the above-mentioned fault detection device or the above-mentioned electronic device.

[0163] According to another aspect of the present disclosure, a vehicle is provided, comprising: the above-mentioned automatic driving system.

[0164] refer to Fig.13, a block diagram of an electronic device 1300 that can be used as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0165] like Fig.13 As shown, the electronic device 1300 includes a computing unit 1301, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1302 or a computer program loaded from a storage unit 1308 into a random access memory (RAM) 1303. In the RAM 1303, various programs and data required for the operation of the electronic device 1300 can also be stored. The computing unit 1301, the ROM 1302, and the RAM 1303 are connected to each other via a bus 1304. An input / output (I / O) interface 1305 is also connected to the bus 1304.

[0166] Multiple components in the electronic device 1300 are connected to the I / O interface 1305, including: an input unit 1306, an output unit 1307, a storage unit 1308, and a communication unit 1309. The input unit 1306 can be any type of device that can input information to the electronic device 1300. The input unit 1306 can receive input digital or character information, and generate key signal input related to user settings and / or function control of the electronic device, and can include but is not limited to a mouse, a keyboard, a touch screen, a trackpad, a trackball, a joystick, a microphone, and / or a remote controller. The output unit 1307 can be any type of device that can present information, and can include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 1308 can include but is not limited to a disk and an optical disk. The communication unit 1309 allows the electronic device 1300 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and may include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver and / or a chipset, such as a Bluetooth device, a 1302.11 device, a WiFi device, a WiMax device, a cellular communication device and / or the like.

[0167] The computing unit 1301 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1301 include, but are not limited to, a central processing unit (GPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1301 performs the various methods and processes described above, such as method 200. For example, in some embodiments, the method 200 may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 1308. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 1300 via the ROM 1302 and / or the communication unit 1309. When the computer program is loaded into the RAM 1303 and executed by the computing unit 1301, one or more steps of the method 200 described above may be performed. Alternatively, in other embodiments, the computing unit 1301 may be configured to execute the method 200 in any other appropriate manner (eg, by means of firmware).

[0168] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0169] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0170] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0171] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0172] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0173] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0174] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0175] Although the embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-mentioned methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present invention is not limited by these embodiments or examples, but only by the claims after authorization and their equivalent scope. Various elements in the embodiments or examples can be omitted or replaced by their equivalent elements. In addition, each step can be performed in an order different from that described in the present disclosure. Further, the various elements in the embodiments or examples can be combined in various ways. It is important that with the evolution of technology, many elements described herein can be replaced by equivalent elements that appear after the present disclosure.

Claims

1. A fault detection method for a graphics processing unit (GPU), comprising: Get system log information related to multiple GPUs; Determine, at least according to the system log information, whether one or more GPUs among the multiple GPUs have a GPU hardware fault, wherein the GPU hardware fault is a fault related to GPU hardware; as well as In response to determining that the GPU hardware fault exists, at least prompt information indicating the GPU hardware fault is output.

2. The method according to claim 1, wherein: The GPU hardware failure includes at least one of the following: GPU link failure, which indicates that there is a failure in the link connection between the GPU and the bus; or GPU overtemperature fault, which indicates that the actual GPU temperature exceeds the temperature limit; or GPU operation failure, which indicates a hardware-related failure during GPU operation.

3. The method according to claim 2, wherein: The GPU hardware fault includes the GPU link fault, which includes a GPU missing fault, a GPU bandwidth fault, and a GPU rate fault, the system log information includes a GPU access result, and the GPU access result is obtained by accessing multiple GPUs according to a GPU identification list, the GPU identification list includes multiple GPU identifications corresponding to the multiple GPUs respectively, and determining whether one or more GPUs among the multiple GPUs have a GPU hardware fault at least based on the system log information includes: Determining whether there is a GPU missing fault according to the GPU access result; and In response to determining that there is no GPU missing fault, determining whether one or more GPUs among the plurality of GPUs has a GPU bandwidth fault and / or a GPU rate fault according to the system log information.

4. The method according to claim 3, wherein: The system log information also includes the number of times the system is reinitialized, where the system reinitialization is an operation triggered in response to determining that the GPU is missing, and in response to determining that the GPU hardware fault exists, at least outputting prompt information indicating the GPU hardware fault includes: In response to determining that the GPU missing fault exists, determining a number of times the system is reinitialized; and In response to determining that the number of times the system is reinitialized is less than or equal to a number threshold, at least prompt information indicating a first risk level fault is output.

5. The method according to claim 3, wherein: In response to determining that the GPU hardware fault exists, at least outputting prompt information indicating the GPU hardware fault further includes: In response to determining that the number of times the system is reinitialized is greater than the number threshold, at least prompt information indicating a second risk level fault is output, wherein the second risk level is greater than the first risk level.

6. The method according to claim 4, wherein: The GPU identification includes a bus device function value BDF, where the BDF is a unique identifier for describing a device on the bus, and the at least outputting prompt information indicating a first risk level fault includes: Outputting prompt information indicating the first risk level fault and a bus device function value of which access failed, wherein the bus device function value of which access failed indicates a missing GPU.

7. The method according to claim 5, wherein: The GPU identification includes a bus device function value BDF, where the BDF is a unique identifier for describing a device on the bus, and the at least outputting prompt information indicating a second risk level fault includes: Outputting prompt information indicating the second risk level fault and a bus device function value of which access failed, wherein the bus device function value of which access failed indicates a missing GPU.

8. The method according to claim 3, wherein: The system log information further includes a plurality of actual link bandwidth values ​​respectively corresponding to the plurality of GPUs and / or a plurality of actual link rate values ​​respectively corresponding to the plurality of GPUs, and determining, based on the system log information, whether one or more GPUs among the plurality of GPUs has a GPU bandwidth failure and / or a GPU rate failure includes at least one of the following: Determining whether actual link bandwidth values ​​of the multiple GPUs match corresponding expected bandwidth values ​​based on a correspondence table between GPU identifiers and expected bandwidth values, wherein a mismatch between the actual link bandwidth values ​​of the multiple GPUs and the corresponding expected bandwidth values ​​indicates a GPU bandwidth failure; or Based on a correspondence table between GPU identifiers and expected rate values, it is determined whether actual rate values ​​of the multiple GPUs match corresponding expected rate values, wherein a mismatch between actual link rate values ​​of the multiple GPUs and corresponding expected rate values ​​indicates a GPU rate failure.

9. The method according to claim 8, wherein: In response to determining that the GPU hardware fault exists, at least outputting prompt information indicating the GPU hardware fault includes: In response to determining that the actual link bandwidth value of at least one GPU does not match the corresponding expected bandwidth value and / or the actual link rate value of at least one GPU does not match the corresponding expected rate value, at least prompt information indicating a second risk level failure is output.

10. The method according to claim 9, wherein: The GPU identification includes a bus device function value BDF, where the BDF is a unique identifier for describing a device on the bus, and the at least outputting prompt information indicating a second risk level fault includes: Outputting prompt information indicating the second risk level fault and an unmatched bus device function value, wherein the unmatched bus device function value indicates a GPU with abnormal bandwidth and / or abnormal rate.

11. The method according to any one of claims 2 to 10, wherein: The GPU hardware fault includes the GPU overtemperature fault, the system log information includes a plurality of actual temperatures respectively corresponding to the plurality of GPUs, and determining whether one or more of the plurality of GPUs has a GPU hardware fault at least based on the system log information includes: For each GPU in the plurality of GPUs: Based on a correspondence table between a GPU identifier and a first temperature threshold and a second temperature threshold, comparing the actual temperature of the GPU with the corresponding first temperature threshold and second temperature threshold, wherein the second temperature threshold is greater than the first temperature threshold; and Based on the comparison, determining whether one or more GPUs among the plurality of GPUs has the GPU overtemperature fault, and Wherein, in response to determining that the GPU hardware fault exists, at least outputting prompt information indicating the GPU hardware fault includes: In response to determining that the actual temperature of the GPU is between a corresponding first temperature threshold and a corresponding second temperature threshold, at least prompt information indicating a first risk level fault is output, wherein the first risk level fault includes a first level temperature anomaly.

12. The method according to claim 11, wherein: The GPU identification includes a bus device function value BDF, where the BDF is a unique identifier for describing a device on the bus, and the at least outputting prompt information indicating a first risk level fault includes: Output prompt information indicating the first risk level fault and the bus device function value of the GPU.

13. The method according to claim 11, wherein: In response to determining that the GPU hardware fault exists, at least outputting prompt information indicating the GPU hardware fault further includes: In response to determining that the actual temperature of the GPU is greater than the corresponding second temperature threshold, at least prompt information indicating a second risk level fault is output, wherein the second risk level fault includes a second level temperature abnormality, and the second level temperature abnormality is greater than the first level temperature abnormality.

14. The method according to claim 11, wherein: The GPU identification includes a bus device function value BDF, where the BDF is a unique identifier for describing a device on the bus, and the at least outputting prompt information indicating a second risk level fault includes: Output prompt information indicating the second risk level fault and the bus device function value of the GPU.

15. The method according to claim 11, further comprising: In response to determining that the actual temperature of the GPU is less than the corresponding first temperature threshold, a prompt message instructing to enhance the heat dissipation and cooling system is output.

16. The method according to any one of claims 2 to 10, wherein: The GPU hardware fault includes the GPU operation fault, the system log information includes an error report of a GPU driver, the error report of the GPU driver includes at least one error message, the error message includes an error name and a corresponding GPU identifier, and determining whether one or more GPUs among the multiple GPUs have a GPU hardware fault at least based on the system log information includes: For each error message of the at least one error message: Based on the correspondence table between error types and error names, determine the error type corresponding to the error information, and The error type includes at least one of a non-hardware error and a hardware error, and in response to determining that the GPU hardware fault exists, at least outputting prompt information indicating the GPU hardware fault includes: In response to determining that the error type corresponding to the error information is the hardware error, at least prompt information indicating a second risk level fault is output.

17. The method according to claim 16, wherein: The GPU identification includes a bus device function value BDF, where the BDF is a unique identifier for describing a device on the bus, and the at least outputting prompt information indicating a second risk level fault includes: Output prompt information indicating the second risk level fault and a bus device function value of the GPU corresponding to the error information.

18. The method according to claim 16, further comprising: In response to determining that the error type of the error information is the non-hardware error, prompt information indicating a first risk level fault is output.

19. The method according to claim 16, further comprising: In response to determining that the error type of the error information does not belong to the non-hardware error and the hardware error, prompt information instructing to perform risk assessment on the error information is output.

20. A fault detection device for a graphics processing unit (GPU), comprising: The first module is used to obtain system log information related to multiple GPUs; A second module is used to determine whether one or more GPUs among the multiple GPUs have a GPU hardware fault at least according to the system log information, wherein the GPU hardware fault is a fault related to GPU hardware; as well as The third module is used to output at least prompt information indicating the GPU hardware fault in response to determining that the GPU hardware fault exists.

21. An electronic device, comprising: at least one processor; as well as a memory communicatively coupled to the at least one processor; in The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 19.

22. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-19.

23. A computer program product comprising a computer program, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 19 is implemented.

24. An automatic driving system comprising: Multiple GPUs; as well as The fault detection device as claimed in claim 20 or the electronic device as claimed in claim 21.

25. A vehicle comprising: An automated driving system as claimed in claim 24.

Images