GPU exception processing method and device, equipment and storage medium

Abstract

The invention discloses a GPU exception processing method and device, equipment and a storage medium, and the method comprises the steps: obtaining a first identifier of a first GPU when detecting that the first GPU is exceptional, and adding the first identifier to a target list used for storing the identifier of the exceptional GPU, the Kubernetes cluster cannot call the second GPU of which the identifier is stored in the target list; and when it is detected that the first GPU returns to normal, the first identifier is moved out of the target list, so that the Kubernetes cluster can call the first GPU. According to the method, the abnormal GPU is automatically isolated, and the normal GPU is automatically restarted, so that the efficiency of isolating the abnormal GPU and the efficiency of restarting the normal GPU are improved, and the resource utilization rate of the GPU is relatively high.

Description

Technical Field

[0001] The present application belongs to the field of information processing technology, and in particular, relates to a GPU exception processing method, device, equipment and storage medium. Background Art

[0002] In the prior art, a Kubernetes cluster is used to manage multiple GPUs (graphics processing units) and call the resources of multiple GPUs. When a Kubernetes cluster calls the resources of a healthy GPU, it often relies on a third-party device to detect whether an abnormality occurs in the GPU. When an abnormality is detected in the GPU, manual intervention is required to manually isolate the abnormal GPU to prevent the Kubernetes cluster from calling the resources of the abnormal GPU. When the abnormal GPU returns to normal, manual intervention is required to manually restart the normal GPU and use the resources of the normal GPU as the resources of a healthy GPU so that it can be called by the Kubernetes cluster. Manually isolating the abnormal GPU and restarting the normal GPU is inefficient, so the isolation efficiency and restart efficiency of the GPU are low, resulting in low GPU resource utilization. Summary of the invention

[0003] The embodiment of the present application provides an implementation solution different from the prior art to solve the technical problem of low resource utilization of GPU.

[0004] In a first aspect, the present application provides a GPU (Graphics Processing Unit) exception handling method applicable to a Kubernetes cluster, comprising: when an abnormality is detected in a first GPU, obtaining a first identifier of the first GPU, and adding the first identifier to a target list for storing identifiers of abnormal GPUs, wherein the Kubernetes cluster cannot call a second GPU whose identifier is stored in the target list; when it is detected that the first GPU has returned to normal, removing the first identifier from the target list so that the Kubernetes cluster can call the first GPU.

[0005] In a second aspect, the present application provides a GPU exception handling device, applicable to a Kubernetes cluster, comprising: a first detection unit, for obtaining a first identifier of the first GPU when an abnormality is detected in the first GPU, and adding the first identifier to a target list for storing identifiers of abnormal GPUs, wherein the Kubernetes cluster cannot call a second GPU whose identifier is stored in the target list; a second detection unit, for removing the first identifier from the target list when it is detected that the first GPU has returned to normal, so that the Kubernetes cluster can call the first GPU.

[0006] In a third aspect, the present application provides an electronic device comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any method in the first aspect or any possible implementation manner of the first aspect by executing the executable instructions.

[0007] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the computer program implements any method in the first aspect or any possible implementation manner of the first aspect.

[0008] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the method described in the first aspect or any possible implementation manner of the first aspect.

[0009] The solution provided by the present application is to obtain the first identifier of the first GPU when an abnormality is detected in the first GPU, and add the first identifier to a target list for storing identifiers of abnormal GPUs, wherein the Kubernetes cluster cannot call the second GPU whose identifier is stored in the target list; when the first GPU is detected to be restored to normal, the first identifier is removed from the target list, so that the Kubernetes cluster can call the solution of the first GPU. By adding the first identifier of the GPU detected to the target list, the GPU in the target list is not called when the GPU is called, so as to automatically isolate the abnormal GPU. When the abnormal first GPU returns to normal, the first identifier is deleted from the target list, so that the first GPU that has returned to normal can be called by the Kubernetes cluster, and then the GPU that has returned to normal is automatically restarted. Thereby, the efficiency of isolating abnormal GPUs and restarting and restoring normal GPUs is improved, so that the resource utilization rate of the GPU is higher. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings: Figure 1 A flowchart of a GPU exception handling method provided in one embodiment of the present application; Figure 2 A schematic diagram of a process of isolating a GPU by detecting parameter values ​​is provided for an embodiment of the present application; Figure 3 A schematic diagram of a process of isolating a GPU through an isolation request provided in an embodiment of the present application; Figure 4 A schematic diagram of the structure of a GPU exception handling device provided in one embodiment of the present application; Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0011] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, but cannot be understood as limiting the present application.

[0012] The terms "first" and "second" etc. in the specification, claims and drawings of the embodiments of the present application are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, the process, method, system, product or equipment comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0013] First, some terms in the embodiments of the present application are explained below to facilitate understanding by those skilled in the art.

[0014] Kubernetes (often referred to as K8s) is an open source container orchestration platform for automating the deployment, expansion, and management of computer containerized applications. Kubernetes can provide highly available service deployment, automated operations, and elastic scaling.

[0015] GPU (Graphics Processing Unit), also known as display core, visual processor, display chip, is a microprocessor dedicated to image computing on personal computers, workstations, game consoles and some mobile devices (such as tablets, smart phones, etc.).

[0016] In the prior art, the Kubernetes cluster is used to manage multiple GPUs (graphics processing units) and call the resources of multiple GPUs. When the Kubernetes cluster calls the resources of a healthy GPU, it often relies on third-party equipment to detect whether an abnormality occurs in the GPU. When an abnormality is detected in the GPU, manual intervention is required to manually isolate the abnormal GPU to prevent the Kubernetes cluster from calling the resources of the abnormal GPU. However, by manually isolating the abnormal GPU, it is easy to mistakenly isolate other healthy GPUs or isolate the abnormal GPU in time. Both will cause the Kubernetes cluster to call the abnormal GPU, resulting in incorrect calculation results when calling the resources of the abnormal GPU for calculation, resulting in a waste of GPU resources.

[0017] When the abnormal GPU returns to normal, manual intervention is required to manually restart the restored GPU and use the resources of the restored GPU as the resources of the healthy GPU so that it can be called by the Kubernetes cluster. However, manually restarting the restored GPU is prone to the problem of incorrectly restarting the abnormal GPU or restarting the restored GPU in time. As a result, the restored GPU cannot be called by the Kubernetes cluster or cannot be called by the Kubernetes cluster in time. The resources of the healthy GPU are idle and not utilized, resulting in a waste of GPU resources. Therefore, it can be seen that manually isolating the abnormal GPU and restarting the restored GPU is prone to errors, wastes resources, is inefficient, and takes a long time. Therefore, the isolation efficiency and restart efficiency of the GPU are low, resulting in low GPU resource utilization.

[0018] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0019] Figure 1 A flowchart of a GPU exception handling method provided by an exemplary embodiment of the present application, the execution subject of the method may be a Kubernetes cluster. The method at least includes the following steps S101-S102: S101. When an abnormality is detected in a first GPU, obtain a first identifier of the first GPU, and add the first identifier to a target list for storing identifiers of abnormal GPUs, wherein the Kubernetes cluster cannot call a second GPU whose identifier is stored in the target list.

[0020] Optionally, the target list is stored in an external database or a local database.

[0021] Specifically, storing the target list in a local database can save time for obtaining the target list. Storing the target list in an external database can realize a sharable target list and achieve data unification.

[0022] Optionally, in order to quickly obtain the target list and simultaneously realize data unification and data backup, when the target list in the local database is lost, the target list can be obtained from the external database. The method also includes storing the target list in the local database and synchronizing the target list to the external database.

[0023] Optionally, in order to improve the accuracy of determining whether the first GPU is abnormal, the method further includes: obtaining a detection parameter value indicating whether the first GPU is abnormal, and if the detection parameter value is not within a preset parameter interval, determining that the first GPU is abnormal. If the detection parameter value is within the preset parameter interval, determining that the first GPU is not abnormal.

[0024] Optionally, the detection parameter value of the first GPU may be collected by a data collection unit in the Kubernetes cluster. Specifically, the data collection unit may be a DCGM-Exporter, wherein the relevant content of the DCGM-Exporter may refer to the prior art, and will not be described in detail here.

[0025] Optionally, the detection parameter value includes: a temperature value of the first GPU and a computing power value of the first GPU.

[0026] Optionally, when the temperature value is not within a preset temperature interval and / or the computing power value is not within a preset computing power interval, the detection parameter value is deemed not to be within a preset parameter interval. When the temperature value is within a preset temperature interval and the computing power value is within a preset computing power interval, the detection parameter value is deemed to be within a preset parameter interval.

[0027] Optionally, the Kubernetes cluster calls the second GPU, specifically by calling the second GPU through a graphics card driver unit in the Kubernetes cluster, wherein the graphics card driver unit is used to allocate an available second GPU to the Kubernetes cluster.

[0028] Optionally, in order to improve the efficiency of the Kubernetes cluster calling available GPUs, the method also includes: in response to the target list being updated, the graphics card driver unit marks the third GPU corresponding to the identifier in the target list as unhealthy, indicating that the third GPU is in an unavailable state, and marks the fourth GPU corresponding to the identifier that is not in the target list as available.

[0029] In summary, when the target list is updated, the GPU corresponding to the identifier in the target list is marked in time to avoid calling abnormal GPUs. When the target list is not updated, there is no need to repeatedly query the identifier in the target list, and the GPU can be called directly through the mark of the GPU's available status, which improves the efficiency of the Kubernetes cluster calling available GPUs.

[0030] S102: When it is detected that the first GPU returns to normal, remove the first identifier from the target list, so that the Kubernetes cluster can call the first GPU.

[0031] In summary, by detecting whether the abnormal first GPU has returned to normal, the first identifier of the first GPU that has returned to normal is removed from the target list. This avoids the problem that after the abnormal first GPU returns to normal, it still cannot be called by the Kubernetes cluster, resulting in waste of resources and low resource utilization of the first GPU. This improves the utilization of the first GPU.

[0032] Optionally, in order to improve the accuracy of determining whether the first GPU has returned to normal, the method further includes: if it is detected that the detection parameter value changes from being outside the preset parameter interval to being within the preset parameter interval, determining that the first GPU has been detected to have returned to normal.

[0033] In summary, it can be seen that by comparing the detection parameter value obtained in real time with the preset parameter range, it is accurately determined whether the first GPU has recovered to normal.

[0034] Optionally, in order to save detection resources for detecting whether an abnormality occurs in the first GPU, the method further includes: obtaining a detection parameter value of whether an abnormality occurs in the first GPU every preset detection time.

[0035] For example, to more intuitively understand how the Kubernetes cluster automatically isolates the abnormal first GPU, see Figure 2 , Figure 2 A schematic diagram of a process of isolating a GPU by detecting parameter values ​​provided for an exemplary embodiment of the present application: By setting a time length, the detection parameter value of the first GPU is collected by the data collection unit at regular intervals, and it is determined whether the detection parameter value is within a preset parameter range. If so, the first GPU is determined to be a healthy GPU, and the first GPU is not isolated. If not, the first GPU is determined to be an abnormal GPU, and the first GPU is isolated.

[0036] Optionally, in order to allow relevant personnel to isolate or restore GPUs according to actual conditions, for example, a healthy GPU cannot be called by the Kubernetes cluster temporarily due to some business needs. The healthy GPU also needs to be isolated. When the isolated healthy GPU needs to be called by the Kubernetes cluster due to business needs, the isolated healthy GPU can be directly restored. The method further includes the following steps S001-S002: S001. When an isolation request for isolating the first GPU is detected, obtain a first identifier of the first GPU, and add the first identifier to the target list.

[0037] S002: When a recovery request for recovering the first GPU is detected, obtain a first identifier of the first GPU, and delete the first identifier from the target list.

[0038] For example, to more clearly understand how the relevant personnel isolate a specific GPU, see Figure 3 , Figure 3 A schematic diagram of a process of isolating a GPU through an isolation request provided for an exemplary embodiment of the present application: By obtaining the first identifier in the isolation request, the first identifier is added to the target list. Before the graphics card driver unit calls the corresponding GPU in the non-target list, it first determines the identifier recorded in the target list, and isolates the GPU corresponding to the fifth identifier recorded in the target list, wherein the fifth identifier includes the first identifier, and the GPU corresponding to the sixth identifier not recorded in the target list is not isolated, that is, the graphics card driver unit can call the GPU corresponding to the sixth identifier.

[0039] In summary, by sending an isolation request, relevant personnel can specify the GPU corresponding to the identifier carried in the isolation request for isolation, so that relevant personnel can determine the GPU that can be called according to actual business needs. This enables calling a more suitable GPU based on business needs and improves the efficiency of calling the GPU.

[0040] In summary, when an abnormality is detected in the first GPU, the first identifier of the first GPU is obtained, and the first identifier is added to the target list for storing the identifiers of the abnormal GPUs, wherein the Kubernetes cluster cannot call the second GPU whose identifier is stored in the target list; when the first GPU is detected to be restored to normal, the first identifier is removed from the target list, so that the Kubernetes cluster can call the solution of the first GPU. By adding the first identifier of the GPU that has been detected to be abnormal to the target list, the GPU in the target list is not called when the GPU is called, so that the abnormal GPU is automatically isolated. When the abnormal first GPU returns to normal, the first identifier is deleted from the target list, so that the first GPU that has returned to normal can be called by the Kubernetes cluster, and then the GPU that has returned to normal is automatically restarted. Thereby, the efficiency of isolating abnormal GPUs and restarting and restoring normal GPUs is improved, so that the resource utilization rate of the GPU is higher.

[0041] Specifically, by detecting a GPU abnormality, it can be automatically added to the target list and marked as unhealthy, indicating that the GPU is in an unavailable state, and the abnormal GPU can be isolated in time, and the incorrect isolation of the healthy GPU can be avoided, and the use of the abnormal GPU's resources for calculation can be avoided, resulting in incorrect calculation results, and avoiding wasting GPU resources. When it is detected that the GPU has returned to normal, the identifier corresponding to the GPU is deleted from the target list and marked as available, so that the isolated GPU can be automatically restored to an available state in time, avoiding the normal GPU from being still isolated, and avoiding the incorrect restoration of the abnormal GPU as a healthy GPU. The idle resources of the healthy GPU can be fully utilized. The resource utilization of the GPU is improved. In addition, the automatic isolation and recovery of the GPU is short in time and efficient, which reduces the workload of the relevant personnel and improves the management efficiency of the GPU equipment. In addition, the present application also provides the relevant personnel with the method of isolating the GPU by sending an isolation request or restoring the GPU by a recovery request, so that the relevant personnel can actively manage and schedule the GPU according to business needs.

[0042] Figure 4 A schematic diagram of a GPU exception handling device provided for an exemplary embodiment of the present application; wherein the device is applicable to a Kubernetes cluster, and the device includes: A first detection unit 41 is configured to obtain a first identifier of the first GPU when an abnormality is detected in the first GPU, and add the first identifier to a target list for storing identifiers of abnormal GPUs, wherein the Kubernetes cluster cannot call a second GPU whose identifier is stored in the target list; The second detection unit 42 is configured to remove the first identifier from the target list when detecting that the first GPU returns to normal, so that the Kubernetes cluster can call the first GPU.

[0043] Optionally, the device is further used to: obtain a detection parameter value indicating whether the first GPU is abnormal, and if the detection parameter value is not within a preset parameter range, determine that the first GPU is abnormal.

[0044] Optionally, the detection parameter value in the device includes: a temperature value of the first GPU and a computing power value of the first GPU.

[0045] Optionally, when the temperature value in the device is within a preset temperature range, and the computing power value is within a preset computing power range, the detection parameter value is deemed to be within a preset parameter range.

[0046] Optionally, the device is further configured to: if it is detected that the detection parameter value changes from being outside the preset parameter interval to being within the preset parameter interval, determine that the first GPU has been detected to be restored to normal.

[0047] Optionally, the target list in the device is stored in an external database or a local database.

[0048] Optionally, the apparatus is further configured to: when an isolation request for isolating the first GPU is detected, obtain a first identifier of the first GPU, and add the first identifier to the target list.

[0049] It should be understood that the device embodiment and the method embodiment may correspond to each other, and similar descriptions may refer to the method embodiment. To avoid repetition, no further description is given here. Specifically, the device may perform the above method embodiment, and the above and other operations and / or functions of each module in the device are the corresponding processes in each method in the above method embodiment, respectively, and no further description is given here for the sake of brevity.

[0050] The above describes the device of the embodiment of the present application from the perspective of the functional module in conjunction with the accompanying drawings. It should be understood that the functional module can be implemented in hardware form, can be implemented by instructions in software form, and can also be implemented by a combination of hardware and software modules. Specifically, the steps of the method embodiment in the embodiment of the present application can be completed by the hardware integrated logic circuit and / or software form instructions in the processor, and the steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to perform, or a combination of hardware and software modules in the decoding processor to perform. Optionally, the software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory, and completes the steps in the above method embodiment in conjunction with its hardware.

[0051] Figure 5 is a schematic block diagram of an electronic device provided in an embodiment of the present application, and the electronic device may include: The memory 301 and the processor 302, the memory 301 is used to store the computer program and transmit the program code to the processor 302. In other words, the processor 302 can call and run the computer program from the memory 301 to implement the method in the embodiment of the present application.

[0052] For example, the processor 302 may be configured to execute the above method embodiments according to instructions in the computer program.

[0053] In some embodiments of the present application, the processor 302 may include but is not limited to: General-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0054] In some embodiments of the present application, the memory 301 includes but is not limited to: Volatile memory and / or non-volatile memory. Among them, the non-volatile memory can be read-only memory (ROM), programmable ROM (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM) or flash memory. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct memory bus random access memory (Direct Rambus RAM, DR RAM).

[0055] In some embodiments of the present application, the computer program may be divided into one or more modules, which are stored in the memory 301 and executed by the processor 302 to complete the method provided by the present application. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program in the electronic device.

[0056] like Figure 5 As shown, the electronic device may also include: The transceiver 303 may be connected to the processor 302 or the memory 301 .

[0057] The processor 302 may control the transceiver 303 to communicate with other devices, specifically, to send information or data to other devices, or to receive information or data sent by other devices. The transceiver 303 may include a transmitter and a receiver. The transceiver 303 may further include an antenna, and the number of antennas may be one or more.

[0058] It should be understood that the various components in the electronic device are connected via a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus and a status signal bus.

[0059] The present application also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a computer, the computer can perform the method of the above method embodiment. In other words, the present application embodiment also provides a computer program product containing instructions, and when the instructions are executed by a computer, the computer can perform the method of the above method embodiment.

[0060] When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a server, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server, or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital video disc (DVD)), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0061] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0062] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the module is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0063] The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. For example, each functional module in each embodiment of the present application may be integrated into a processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0064] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A GPU (Graphics Processing Unit) exception handling method, characterized in that: Applicable to Kubernetes clusters, including: When an abnormality is detected in the first GPU, a first identifier of the first GPU is obtained, and the first identifier is added to a target list for storing identifiers of abnormal GPUs, wherein the Kubernetes cluster cannot call a second GPU whose identifier is stored in the target list; When it is detected that the first GPU returns to normal, the first identifier is removed from the target list, so that the Kubernetes cluster can call the first GPU.

2. The method according to claim 1, characterized in that The method further comprises: A detection parameter value indicating whether the first GPU is abnormal is obtained, and if the detection parameter value is not within a preset parameter interval, it is determined that the first GPU is abnormal.

3. The method according to claim 2, characterized in that The detection parameter values ​​include: a temperature value of the first GPU and a computing power value of the first GPU.

4. The method according to claim 3, characterized in that When the temperature value is within a preset temperature range and the computing power value is within a preset computing power range, it is deemed that the detection parameter value is within a preset parameter range.

5. The method according to claim 2, characterized in that: The method further comprises: If it is detected that the detection parameter value changes from not being within the preset parameter interval to being within the preset parameter interval, it is determined that the first GPU is detected to be restored to normal.

6. The method according to claim 1, characterized in that The target list is stored in an external database or a local database.

7. The method according to claim 1, characterized in that The method further comprises: When an isolation request for isolating the first GPU is detected, a first identifier of the first GPU is obtained, and the first identifier is added to the target list.

8. A GPU exception handling device, characterized in that: Applicable to Kubernetes clusters, including: A first detection unit is configured to, when detecting that an abnormality occurs in a first GPU, obtain a first identifier of the first GPU and add the first identifier to a target list for storing identifiers of abnormal GPUs, wherein the Kubernetes cluster cannot call a second GPU whose identifier is stored in the target list; The second detection unit is configured to remove the first identifier from the target list when detecting that the first GPU returns to normal, so that the Kubernetes cluster can call the first GPU.

9. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to perform the method of any one of claims 1 to 7 by executing the executable instructions.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images