Device failure handling method, electronic device, storage medium, and program product

Abstract

Embodiments of the present application provide a device fault handling method, electronic device, storage medium, and program product, relating to the field of computer technology. The method includes: when a target fault occurs in any graphics processor in a set of graphics processors, isolating access to the faulty graphics processor with the target fault, wherein the access isolation is used to block access tasks to the faulty graphics processor; and resetting the faulty graphics processor. If the reset is successful, the faulty graphics processor is restored to a target state, wherein the target state is a normal working state. The technical solution of the embodiments of the present application avoids data loss, achieves automated and rapid fault repair, and improves the efficiency of graphics processor fault repair.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a device fault handling method, electronic equipment, storage medium, and program product, which can be applied to the field of fault handling technology. Background Art

[0002] Graphics Processing Units (GPUs) are widely used in the field of artificial intelligence (AI) due to their superior parallel processing capabilities. For example, a single server is typically equipped with 8 to 16 GPUs during model training. However, GPUs can frequently experience failures during operation, with double-bit Error-Correcting Code (ECC) errors being a typical example. Because double-bit ECC errors are uncorrectable, meaning the GPU cannot self-correct, when a double-bit ECC error occurs in a GPU, the server containing that GPU is removed from the scheduling domain of the training cluster. This means that if a single GPU experiences a double-bit ECC error, all GPUs deployed on the server will be idle. Conventional technology addresses this issue by manually restarting the server to fix the GPU failure. However, restarting the server takes 10 to 20 minutes, resulting in low repair efficiency and potential data loss. Summary of the Invention

[0003] The embodiments of the present application provide a device fault handling method, electronic device, storage medium, and program product to alleviate or solve one or more technical problems existing in the prior art.

[0004] In a first aspect, an embodiment of the present application provides a device fault handling method, the method comprising: when a target fault occurs in any graphics processor in a graphics processor set, performing access isolation on the faulty graphics processor having the target fault, wherein the access isolation is used to block access tasks to the faulty graphics processor; resetting the faulty graphics processor, wherein, if the reset is successful, the faulty graphics processor is restored to a target state, so that the target state is a normal working state.

[0005] In a second aspect, an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor implements any method of the embodiment of the present application when executing the computer program.

[0006] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the method of any one of the embodiments of the present application is implemented.

[0007] In a fourth aspect, an embodiment of the present application provides a computer program product, including a computer program, which implements any method of the embodiments of the present application when executed by a processor.

[0008] In the device fault handling method provided in an embodiment of the present application, if any GPU in a set of GPUs experiences a target fault, access to the faulty GPU is isolated to block access tasks to it, and the faulty GPU is reset to restore it to its target state. This method not only improves the success rate of the reset, but also requires no human intervention. The entire process takes approximately 10 to 15 seconds, achieving automated and rapid fault repair without data loss, thereby improving the efficiency of GPU fault repair.

[0009] Please give an overview of the technical solution. In order to more clearly understand the technical means of this application, which can be implemented in accordance with the contents of the specification, and in order to make the above-mentioned and other purposes, features and advantages of this application more obvious and easy to understand, the specific implementation methods of this application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar components or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments according to the present application and should not be regarded as limiting the scope of the present application.

[0011] Figure 1 shows a schematic diagram of the fault distribution of a graphics processor;

[0012] Figure 2 A schematic diagram showing an application scenario of the device failure handling method according to an embodiment of the present application is shown;

[0013] Figure 3 A flow chart showing a method for handling equipment failures according to an embodiment of the present application is shown;

[0014] Figure 4 A schematic diagram showing a flow chart of a method for handling equipment failures according to an embodiment of the present application is shown;

[0015] Figure 5 A schematic diagram of a module of a device failure handling apparatus according to an embodiment of the present application is shown;

[0016] Figure 6 A block diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0017] Hereinafter, only certain exemplary embodiments are briefly described. As will be appreciated by those skilled in the art, the described embodiments may be modified in various ways without departing from the spirit or scope of the present application. Therefore, the drawings and description are to be regarded as illustrative in nature and not restrictive.

[0018] To facilitate understanding of the technical solutions of the embodiments of the present application, the following describes the related technologies of the embodiments of the present application. The following related technologies can be combined with the technical solutions of the embodiments of the present application as optional solutions, and all of them fall within the scope of protection of the embodiments of the present application.

[0019] The following terms will be used in the following text:

[0020] GPU: A processor that processes graphics and parallel computing tasks. It can be used to accelerate computing tasks that require high parallelism, such as graphics rendering, deep learning, scientific computing, and data analysis.

[0021] ECC: An error detection and correction technology that can detect and correct single-bit errors (SBE) during data storage and transmission. It can detect but not repair multi-bit errors (MBE) during data storage and transmission. Single-bit errors are also called single-bit ECC errors, and multi-bit errors are also called multi-bit ECC errors, such as double-bit ECC errors.

[0022] Reset refers to the reset operation of a system or device. It can be a hardware restart (such as pressing the restart button) or a software reset (such as restarting the system through a command) to restore the system to the target state and clear errors.

[0023] Reattach: Generally refers to the process of reconnecting or attaching to a device or resource. In computing, this may involve reconnecting to storage devices, network connections, or other external devices.

[0024] ECC Page Retirement: A memory page is marked as "retired" due to excessive ECC errors.

[0025] With the continuous development of artificial intelligence (AI) technology, the demand for high-performance computing resources continues to rise. GPUs, with their superior parallel processing capabilities, have become the primary device to meet this demand. For example, during model training, a server is typically configured with 8 to 16 GPUs. However, this reliance on GPUs also brings new challenges: GPUs can frequently experience various types of failures during operation.

[0026] like Figure 1As shown, GPU failures can include double-bit ECC errors (64.4%), single-bit ECC errors (9.2%), and other errors (26.4%). Double-bit ECC errors are a common GPU failure. Because double-bit ECC errors are uncorrectable, meaning the GPU cannot self-correct, a double-bit ECC error in a GPU will cause the server containing that GPU to be removed from the training cluster's scheduling domain. This means that if a single GPU experiences a double-bit ECC error, all GPUs deployed on the server will be idle.

[0027] Several solutions have been proposed. One is to manually restart the server. This reinitializes all hardware, including the GPU, and resolves issues such as double-bit ECC errors, memory leaks, and card failures. However, restarting the server takes 10 to 20 minutes, resulting in low repair efficiency and potential data loss.

[0028] Another approach is to directly reset the GPU when a double-bit ECC error occurs. However, if any process other than the reset process is using the GPU, the reset will fail. In real-world computing environments, various monitoring components are typically deployed to monitor GPU performance. These monitoring components continuously occupy the GPU. Even if all monitoring components are disabled, new processes will quickly access the GPU, resulting in a low reset success rate, currently less than 10%.

[0029] Another solution is reattachment. This involves uninstalling the GPU driver after all processes occupying the GPU exit. When a new process accesses the GPU, the GPU driver is reloaded to reattach the GPU. However, this approach has several drawbacks. For Ampere and later GPU architectures, the GPU supports row remapping. Each memory bank in HBM (High Bandwidth Memory) is equipped with a spare row in hardware. Unlike traditional page retirement, row remapping uses spare rows to replace memory cells with ECC errors, preventing gaps in memory when software manages video memory. However, row remapping requires a reset to take effect. Furthermore, after a successful reset, the relevant GPU processes must be manually restarted, requiring user intervention and increasing complexity.

[0030] Based on this, embodiments of the present application provide a device fault handling method, electronic device, storage medium, and program product, aiming to improve the fault repair efficiency of a graphics processor. Figure 2 A schematic diagram of an application scenario of a device fault handling method provided by an embodiment of the present application is shown. Figure 2 As shown, the scenario includes an electronic device, in which a graphics processing unit (GPU) cluster is configured, and the graphics processing unit cluster includes at least one graphics processing unit (GPU).

[0031] For example, the electronic device may be a terminal device or a server. The terminal device may be a mobile phone, tablet computer, desktop computer, portable notebook, vehicle-mounted terminal, etc. The server may be a physical server or a cloud server for cloud computing, etc. Figure 2 The electronic device is used as a physical server as an example for schematic illustration.

[0032] An electronic device maintains a fault handling strategy that, if a target fault occurs in any graphics processor in a set of graphics processors, isolates access to the faulty graphics processor and then resets the faulty graphics processor. Access isolation blocks access to the faulty graphics processor. If the faulty graphics processor is successfully reset, the graphics processor is restored to a target state, i.e., the state before the target fault occurred. Target faults include, but are not limited to, the double-bit ECC errors and single-bit ECC errors described above, as well as other types of errors described below.

[0033] By isolating access to the target GPU, blocking access to it, and resetting it to restore it to its target state, the system not only improves the success rate of the reset but also eliminates the need for human intervention. The entire process takes approximately 10 to 15 seconds, preventing data loss and enabling automated and rapid fault repair, thus improving GPU fault repair efficiency.

[0034] It should be noted that the above-mentioned application scenarios or application examples provided in the embodiments of this application are for ease of understanding, and the embodiments of this application do not specifically limit the application of the technical solution. In addition, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation portals for users to choose to authorize or refuse.

[0035] The following describes in detail the technical solution of this application and how it solves the aforementioned technical problems using specific embodiments. The several specific embodiments listed can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments. The following describes the embodiments of this application in detail with reference to the accompanying drawings.

[0036] Figure 3 A flow chart of a method for handling device failures according to an embodiment of the present application is shown. Figure 3 As shown, the method may include step S301 and step S302.

[0037] Step S301: when any graphics processor in a graphics processor set has a target fault, perform access isolation on the faulty graphics processor having the target fault, wherein the access isolation is used to block access tasks to the faulty graphics processor.

[0038] To promptly repair a faulty graphics processor that has a target fault, in some embodiments, the electronic device may be equipped with a repair tool that has both fault detection and repair capabilities. Upon successful startup of the electronic device, the electronic device may activate the repair tool and use it to perform fault detection on the set of graphics processors deployed in the electronic device.

[0039] In other embodiments, when the electronic device is successfully started, the electronic device may start a repair process with fault detection and fault repair functions, and perform fault detection on the graphics processor set deployed in the electronic device through the repair process.

[0040] Considering that resetting a GPU increases the success rate of resets when the GPU is not occupied, if the fault detection results indicate that any GPU in the set of GPUs has a target fault, the aforementioned repair tool or repair process is used to isolate access to the faulty GPU, thereby blocking tasks accessing the faulty GPU. Access tasks may include access processes, access threads, etc.

[0041] The target fault may be a fault caused by a double-bit ECC error, ECC page retirement, or remapped rows failure. The double-bit ECC error is used as an example for explanation below.

[0042] Step S302: Resetting the faulty graphics processor. If the reset is successful, the faulty graphics processor is restored to a target state, which is a state where the graphics processor can operate normally.

[0043] In some embodiments, the electronic device can use the aforementioned repair tool or repair process to execute a reset instruction based on the device identifier of the faulty graphics processor with the target fault, thereby resetting the faulty graphics processor and obtaining a reset result. If the reset result indicates a successful reset, the target fault has been repaired, and the corresponding faulty graphics processor has been restored to the target state. If the reset result indicates a reset failure, the electronic device can generate a reset failure record and report it to designated operations and maintenance personnel. The reset failure record can include information about the target fault and detailed information about the reset failure.

[0044] It should be noted that the reset instruction may vary depending on the manufacturer of the GPU, and this application does not specifically limit this. Accordingly, different reset instructions may also perform different operations on the GPU and correspond to different target states. For example, for a GPU manufactured by Manufacturer A, the operations performed on the corresponding faulty GPU after executing the reset instruction may include terminating all running CUDA (Compute Unified Device Architecture) processes, clearing video memory, and reinitializing the faulty GPU driver. Accordingly, the target state may be a normal operating state, such as no running CUDA tasks, cleared video memory, and the GPU driver is set to default parameters. For a GPU manufactured by Manufacturer B, the operations performed on the corresponding faulty GPU after executing the reset instruction may include terminating computing tasks, clearing video memory, and resetting error counters. Accordingly, the target state may be a normal operating state, such as no running computing tasks, cleared video memory, and resetting error counters.

[0045] It should be further pointed out that the target state in the embodiment of the present application is not the factory state, that is, the driver or physical hardware of the graphics processor will not be deleted. The configuration of the graphics processor is only restored to the basic configuration for normal operation, so that the graphics processor is in a normal working state.

[0046] Therefore, if any GPU in a set of GPUs experiences a target fault, access to the GPU with the target fault is isolated, blocking access tasks to it and resetting it to its target state. This not only improves the success rate of the reset, but also requires no human intervention. The entire process takes approximately 10 to 15 seconds, avoiding data loss and enabling automated and rapid fault repair, thus improving GPU fault repair efficiency.

[0047] In order to promptly discover a graphics processor with a target fault, in some implementations, such as Figure 4 As shown, fault detection can be performed on each graphics processor in the graphics processor set. That is, before step S301, the process may further include: obtaining operating status data of any graphics processor in the graphics processor set according to a preset detection cycle; and determining that the graphics processor is a faulty graphics processor having a target fault if the operating status data contains a preset fault identifier.

[0048] In some implementations, the operating status data may be log data, which may include fault logs, service logs, security logs, etc. of each graphics processor in the graphics processor set, and each log is arranged in chronological order of generation time. Accordingly, the electronic device may query candidate log data within the corresponding detection period according to a preset query path according to a preset detection period; or execute a second query instruction to obtain candidate log data within the corresponding detection period. For any graphics processor in the graphics processor set, query the target log data containing the device identifier of the graphics processor from the candidate log data. Determine whether the target log data contains a preset fault identifier. If the target log data contains a preset fault identifier, determine that the graphics processor is a faulty graphics processor with a target fault. The second query instruction is used to query the candidate log data of each graphics processor in the graphics processor set within the corresponding detection period.

[0049] In other implementations, the operating status data may be log data, and the log data may be divided into at least one log subset, wherein the at least one log subset corresponds one-to-one to at least one graphics processor in the graphics processor set, and the logs in any log subset are arranged in chronological order of generation time. Accordingly, for any graphics processor in the graphics processor set, the electronic device may obtain the log subset corresponding to the graphics processor from the log data according to the device identifier of the graphics processor according to the detection period. The target log data within the corresponding detection period is obtained from the log subset, and it is determined whether the target log data contains a preset fault identifier. If the target log data contains the preset fault identifier, the graphics processor is determined to be a faulty graphics processor with a target fault.

[0050] In some further implementations, the operating status data may be fault data. Accordingly, for any graphics processor in the set of graphics processors, the electronic device may execute a third query instruction according to a detection cycle to query the fault data of the graphics processor within the corresponding detection cycle. A determination is made as to whether the fault data contains a preset fault identifier. If the fault data contains the preset fault identifier, the graphics processor is determined to be a faulty graphics processor experiencing a target fault. The third query instruction is used to query the fault data of any graphics processor within the detection cycle.

[0051] Exemplarily, the target fault may be a fault caused by a double-bit ECC error. Accordingly, when the operating status data is log data, the preset fault identifier may be an identifier used to indicate a specific type related to the log data, such as "XID48"; when the operating status data is fault data, the preset fault identifier may be an identifier used to indicate the error type, such as "ECC Errors" (used to indicate ECC errors), "Uncorrectable Errors" (used to indicate uncorrectable errors), etc.

[0052] Therefore, by periodically acquiring the operating status data of each graphics processor in the graphics processor set during the corresponding detection cycle and determining whether the operating status data contains a preset fault identifier, effective detection of the target fault is achieved, which can provide a guarantee for timely fault repair.

[0053] Considering that in actual applications, when there are tasks occupying the graphics processor, the success rate of resetting the graphics processor is often relatively low. Based on this, in some implementations, such as Figure 4 As shown, isolating access to a GPU with a target fault includes both blocking new access tasks and terminating currently accessing tasks. Specifically, isolating access to the GPU in step S301 may include: configuring the access state of the faulty GPU to an inaccessible state, where the inaccessible state is used to block new access tasks to the faulty GPU; and, if the faulty GPU is currently in an accessed state, terminating the access tasks currently accessing the faulty GPU.

[0054] To prevent new access tasks from accessing a faulty graphics processor with a target fault when a graphics processor has a target fault, in some implementations, upon determining that a target fault exists on any graphics processor, the access state of the faulty graphics processor may be first configured as inaccessible. A detection is then made as to whether the faulty graphics processor is currently in an accessed state. If the faulty graphics processor is currently in an accessed state, indicating that an access task is currently accessing the faulty graphics processor, a termination signal is sent to the access task currently accessing the faulty graphics processor, thereby terminating the access task currently accessing the faulty graphics processor.

[0055] For example, the electronic device may call a kill command to send a termination signal to the access task that is accessing the faulty graphics processor, wherein the kill command is a command for managing processes in some operating systems and is used to send a signal to the process.

[0056] It should be noted that the order of executing the operation of configuring the access status of the faulty GPU to be inaccessible and the operation of detecting whether the faulty GPU is currently in an accessed state can be interchanged and can be executed simultaneously.

[0057] Therefore, for the graphics processor with a target fault, by blocking its new access tasks and terminating the access tasks being accessed, it is ensured that the graphics processor with a target fault is in an unoccupied state, providing favorable conditions for subsequent reset operations and improving the accuracy of the reset operation.

[0058] To ensure that a GPU without a target fault in the GPU set can be normally accessed, in some embodiments, configuring the access status of the faulty GPU to be inaccessible may include: querying a pointer to an access function based on the function name of the access function, where the access function is used to access any GPU in the GPU set; and modifying the pointer to a proxy function, where the proxy function includes the device identifier of the faulty GPU with the target fault, to indicate that the access status of the faulty GPU with the target fault is inaccessible.

[0059] The proxy function can include the device identifier of the GPU with the target fault in various ways. In some implementations, a proxy function and a corresponding target array can be predefined. The proxy function includes access request processing logic, and the target array includes the device identifiers and corresponding status identifiers of each GPU in the GPU set. Accordingly, upon determining that a target fault exists in any GPU in the GPU set, an update function interface can be called to update the status identifier corresponding to the device identifier of the faulty GPU with the target fault in the target array to preset data, thereby indicating that the access status of the corresponding faulty GPU is inaccessible. Furthermore, a query interface can be called based on the preset name of the access function to obtain a pointer to the access function. The pointer's current pointer (i.e., the access function) is modified to the proxy function. The query interface is used to query the corresponding function based on the function name. Exemplarily, the preset data is 1, and the access function currently pointed to by the pointer, "gpu_open," is modified to the proxy function's name, "gpu_dlopen."

[0060] In other implementations, a proxy function and its corresponding target array can be predefined. The proxy function includes access request processing logic, and the target array can include the device identifier of the faulty GPU experiencing the target fault. Accordingly, upon determining that a target fault exists in any GPU in the GPU set, an add function interface can be called to add the device identifier of the faulty GPU experiencing the target fault to the target array, thereby marking the access status of the faulty GPU experiencing the target fault as inaccessible. Furthermore, a query interface can be called based on the preset access function name to obtain a pointer to the access function. The current pointer (i.e., the access function) is then modified to point to the proxy function.

[0061] Therefore, by querying the pointer to the access function and modifying the object it points to, that is, hijacking the access function and redirecting it to the proxy function, and injecting the device identifier of the graphics processor with the target fault into the proxy function, it is possible to subsequently block new access tasks to the graphics processor with the target fault based on the proxy function, and ensure that other graphics processors in the graphics processor set (that is, graphics processors other than the graphics processor with the target fault) can be accessed normally.

[0062] Furthermore, in some embodiments, the method of the embodiment of the present application may also include: returning access failure information when the calling information of the proxy function is obtained and the proxy function contains the device identifier in the calling information; or calling the access function when the calling information of the proxy function is obtained and the proxy function does not contain the device identifier in the calling information.

[0063] Specifically, after the pointer to the access function is modified to point to the proxy function, any access task (such as an access process or access thread) must call the proxy function when accessing the graphics processor. Accordingly, upon receiving call information for the proxy function, the electronic device obtains the device identifier of the graphics processor to be accessed from the call information. Furthermore, if a target array corresponding to the access function is preset and the target array contains the device identifiers and corresponding status identifiers of each graphics processor, if the status identifier corresponding to the device identifier of the graphics processor to be accessed in the target array is preset data, the proxy function is determined to contain the device identifier in the call information, indicating that the graphics processor to be accessed is a faulty graphics processor with the target fault, and an access failure message is therefore returned. If the status identifier corresponding to the device identifier of the graphics processor to be accessed in the target array is not preset data, the proxy function is determined to not contain the device identifier in the call information, indicating that the graphics processor to be accessed is not a faulty graphics processor with the target fault, and the access function is therefore called to access the corresponding graphics processor without the target fault.

[0064] In the case where a target array corresponding to an access function is preset and the target array contains the device identifier of a graphics processor with a target fault, if the target array includes the device identifier of the graphics processor to be accessed, the proxy function is determined to contain the device identifier in the call information, indicating that the graphics processor to be accessed is a faulty graphics processor with the target fault, and therefore an access failure message is returned. If the target array does not contain the device identifier of the graphics processor to be accessed, the proxy function is determined to contain the device identifier in the call information, indicating that the graphics processor to be accessed is not a faulty graphics processor with the target fault, and therefore the access function is called to access the corresponding graphics processor without the target fault.

[0065] It can be seen that when the call information of the proxy function is obtained, by determining whether the proxy function contains the device identifier in the call information, it is possible to prevent new access tasks to the graphics processor with the target fault, and ensure that other graphics processors in the graphics processor set can be accessed normally.

[0066] To accurately determine whether the faulty graphics processor with the target fault is currently in an accessed state, in some embodiments, before terminating the access task that is accessing the faulty graphics processor, the method may further include: executing a first query instruction to obtain a query result; and if the query result indicates that there is an access task that is accessing the faulty graphics processor, determining that the graphics processor is currently in an accessed state.

[0067] Specifically, after determining that any graphics processor in the graphics processor set has a target fault, a first query instruction can be executed based on the device identifier of the faulty graphics processor with the target fault to obtain a query result. In some embodiments, the query result may include a task list. Accordingly, if the task list in the query result is non-empty, it indicates that there is an access task accessing the faulty graphics processor, that is, it is determined that the faulty graphics processor is currently in an accessed state. In other embodiments, the query result may include a task identifier. Accordingly, if the task identifier is non-empty, it indicates that there is an access task accessing the faulty graphics processor, that is, it is determined that the faulty graphics processor is currently in an accessed state. The specific form of the query result is not limited to the above description and can be set as needed in actual applications. The first query instruction is used to query the access tasks for the corresponding graphics processor based on the device identifier.

[0068] Therefore, by executing the first query instruction, it is possible to accurately determine whether the graphics processor is currently in an accessed state based on the query result, thereby providing an accurate basis for effective access isolation.

[0069] In some embodiments, the graphics processor set configured in the electronic device may be part or all of the graphics processors corresponding to high-performance computing tasks. For example, high-performance computing tasks include neural network training, neural network reasoning, etc. In order to accurately schedule each graphics processor, in some embodiments, such as Figure 4 As shown, the electronic device may further send fault information to the scheduling device when a target fault is detected; and send recovery information to the scheduling device when the reset is successful. That is, based on any of the above embodiments, the method may further include:

[0070] Fault information about a faulty GPU with a target fault is sent to a scheduling device of a GPU set. The fault information is used to instruct the GPU with the target fault to be removed from a scheduling domain, where the scheduling domain includes the GPU set. Furthermore, if the reset is successful, recovery information about the faulty GPU is sent to the scheduling device. The recovery information is used to instruct the GPU to be added to the scheduling domain.

[0071] Specifically, when it is determined that any graphics processor in the graphics processor set has a target fault, fault information can be sent to the scheduling device based on the device identification of the faulty graphics processor. When the scheduling device receives the fault information, it obtains the device identification from the fault information and marks the scheduling status of the faulty graphics processor corresponding to the device identification as invalid, so as to remove the faulty graphics processor from the scheduling domain. When the electronic device successfully resets the faulty graphics processor with the target fault, it sends recovery information to the scheduling device based on the device identification of the faulty graphics processor. The scheduling device obtains the device identification from the recovery information and marks the scheduling status of the faulty graphics processor corresponding to the device identification as valid, so as to add the faulty graphics processor to the scheduling domain. When the electronic device fails to reset the faulty graphics processor with the target fault, a reset failure record can be generated for subsequent analysis.

[0072] The scheduling device may be a device in a distributed cluster, which may include at least one electronic device. Any electronic device may be configured with a graphics processor set. The scheduling device is used to manage a scheduling domain including each graphics processor.

[0073] It should be pointed out that Figure 4 This is for illustration only and not limitation. The order of execution of some operation steps may be interchanged, and some operation steps may be performed simultaneously. For example, sending a fault message to the dispatching device may be performed after terminating the access task in progress; blocking new access tasks may also be performed simultaneously with terminating the access task in progress.

[0074] Thus, when a target fault occurs in a GPU, fault information is sent to the scheduling device, and when the GPU with the target fault is successfully reset, recovery information is sent to the scheduling device. This ensures that the scheduling device effectively manages the scheduling domain, thereby improving the accuracy of GPU scheduling.

[0075] To further illustrate the advantages of the device fault handling method provided in the embodiments of the present application (hereinafter referred to as the innovative solution), a comparison is made below with the centralized response method adopted in the related technologies described above, as shown in the following table:

[0076]

[0077] Based on the above comparison, it can be seen that the innovative solutions provided by the embodiments of the present application are superior to the solutions in the related technologies in terms of comprehensive time consumption, success rate, whether Row Remapping is supported, and whether Page Retirement is supported. They have the advantages of low time consumption, high success rate, and support for Row Remapping and Page Retirement.

[0078] It is understandable that the physical structure of the video memory has a combination of multiple storage units (rows / columns). When a row frequently fails due to hardware aging or other defects, the memory controller or firmware of the graphics processor will redirect the address of the row to a reserved spare row. This process is called Row Remapping. After the faulty graphics processor is successfully reset, the error checking mechanism of the faulty graphics processor at the hardware or firmware level will be triggered. When the result of the error check indicates that an error has occurred in a certain video memory row, the address of the row will be redirected to a reserved spare row through the memory controller or firmware of the faulty graphics processor. In addition, when the error check result indicates that an area of ​​the video memory (such as a video memory page) is irreversibly damaged, the driver or firmware of the faulty graphics processor will mark the area as retired so that it will no longer be allocated for use. Therefore, the embodiments provided in this application support Row Remapping and Page Retirement.

[0079] Corresponding to the application scenario and method of the method provided in the embodiment of the present application, the embodiment of the present application also provides a device fault processing device, which can be applied to Figure 2 The electronic equipment shown. Figure 5As shown, the apparatus includes: an isolation module 501 for isolating access to the faulty GPU if a target fault occurs in any GPU in a set of GPUs, wherein the access isolation is used to block access tasks to the faulty GPU; and a reset module 502 for resetting the faulty GPU. If the reset is successful, the faulty GPU is restored to a target state, which is a normal operating state.

[0080] Optionally, the isolation module 501 is specifically configured to configure the access state of the faulty graphics processor to an inaccessible state, where the inaccessible state is used to prevent new access tasks from being performed on the faulty graphics processor; and, when the faulty graphics processor is currently in an accessed state, terminate the access task that is accessing the faulty graphics processor.

[0081] Optionally, the isolation module 501 is further specifically configured to query a pointer to an access function based on a function name of the access function, wherein the access function is used to access any graphics processor in the graphics processor set; and modify the object pointed to by the pointer to a proxy function, wherein the proxy function includes a device identifier of the faulty graphics processor to indicate that the access state of the faulty graphics processor is the inaccessible state.

[0082] Optionally, the device also includes a processing module for returning an access failure message when the calling information of the proxy function is obtained and the proxy function contains the device identifier in the calling information; or, calling the access function when the calling information of the proxy function is obtained and the proxy function does not contain the device identifier in the calling information.

[0083] Optionally, the isolation module 501 is further configured to execute a first query instruction to obtain a query result before terminating the access task that is accessing the faulty graphics processor; and when the query result indicates that there is an access task that is accessing the faulty graphics processor, determine that the faulty graphics processor is currently in the accessed state.

[0084] Optionally, the device further includes a detection module configured to obtain, for any graphics processor in the graphics processor set, operating status data of the graphics processor according to a preset detection cycle before the isolation module 501 performs access isolation on the faulty graphics processor; and determine that the graphics processor is a faulty graphics processor having the target fault if the operating status data contains a preset fault identifier.

[0085] Optionally, the apparatus further comprises a sending module for sending fault information of the faulty graphics processor to a scheduling device of the graphics processor set, wherein the fault information is used to instruct removal of the faulty graphics processor from a scheduling domain, wherein the scheduling domain includes the graphics processor set; and if the reset is successful, sending recovery information of the faulty graphics processor to the scheduling device, wherein the recovery information is used to instruct addition of the faulty graphics processor to the scheduling domain.

[0086] The functions of each module in each device of the embodiment of the present application can be referred to the corresponding description in the above method, and have corresponding beneficial effects, which will not be repeated here. In addition, the device embodiments described above are merely schematic, wherein the modules described as separate components may or may not be physically separated, and the components illustrated as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application solution.

[0087] Figure 6 FIG. 1 is a block diagram of an electronic device for implementing an embodiment of the present application. Figure 6 As shown, the electronic device includes a memory 601 and a processor 602. The memory 601 stores a computer program executable by the processor 602. When the processor 602 executes the computer program, the method described in the above embodiment is implemented. The number of the memory 601 and the processor 602 can be one or more. In a specific implementation, the electronic device may also include a communication interface 603 for communicating with external devices and exchanging data.

[0088] In a specific implementation, if the memory 601, processor 602, and communication interface 603 are implemented independently, the memory 601, processor 602, and communication interface 603 can be connected to each other via a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0089] Optionally, in a specific implementation, if the memory 601 , the processor 602 , and the communication interface 603 are integrated on a chip, the memory 601 , the processor 602 , and the communication interface 603 may communicate with each other through an internal interface.

[0090] An embodiment of the present application provides a computer-readable storage medium storing a computer program, which implements the method provided in the embodiment of the present application when the program is executed by a processor.

[0091] An embodiment of the present application provides a computer program product, including a computer program, which implements the method provided in the embodiment of the present application when executed by a processor.

[0092] An embodiment of the present application also provides a chip, which includes a processor for calling and executing instructions stored in the memory from the memory, so that a communication device equipped with the chip executes the method provided in the embodiment of the present application.

[0093] An embodiment of the present application also provides a chip, including: an input interface, an output interface, a processor and a memory. The input interface, the output interface, the processor and the memory are connected through an internal connection path. The processor is used to execute the code in the memory. When the code is executed, the processor is used to execute the method provided in the embodiment of the application.

[0094] It should be understood that the processor described above may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor. It is worth noting that the processor may be a processor that supports the Advanced RISC Machines (ARM) architecture.

[0095] Furthermore, optionally, the aforementioned memory may include read-only memory and random access memory. The memory may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Non-volatile memory may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which serves as an external cache memory. By way of example and not limitation, many forms of RAM are available. For example, static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM) and direct memory bus random access memory (DR RAM).

[0096] In the above embodiments, all or part of the embodiments may be implemented using software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions according to the present application are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0097] In the description of this specification, the reference terms "one embodiment," "some embodiments," "example," "specific example," or "some examples" mean that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials, or characteristics described may be combined in any appropriate manner in any one or more embodiments or examples. In addition, those skilled in the art may combine and integrate different embodiments or examples described in this specification, as well as features of different embodiments or examples, unless they are mutually inconsistent.

[0098] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. Throughout the description of this application, "plurality" means two or more, unless otherwise specifically defined.

[0099] Any process or method described in the flowchart or otherwise described herein can be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a specific logical function or process. The scope of the preferred embodiments of the present application includes other implementations in which the functions may be performed in a different order than shown or discussed, including performing the functions substantially simultaneously or in reverse order depending on the functions involved.

[0100] The logic and / or steps described in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, apparatus or device (such as a computer-based system, a system including a processor, or other system that can fetch instructions from and execute instructions on an instruction execution system, apparatus or device), or used in conjunction with such instruction execution systems, apparatuses or devices.

[0101] It should be understood that various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. All or part of the steps of the above embodiment method can be completed by instructing the relevant hardware through a program, which can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0102] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing module, or each unit may exist physically separately, or two or more units may be integrated into a single module. The aforementioned integrated modules may be implemented in the form of hardware or in the form of software functional modules. If the aforementioned integrated modules are implemented in the form of software functional modules and sold or used as independent products, they may also be stored in a computer-readable storage medium. The storage medium may be a read-only memory, a magnetic disk, or an optical disk, etc.

[0103] The above is merely an exemplary embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily conceive of various modifications or substitutions within the technical scope described in this application, and such modifications or substitutions should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A method for handling equipment failure, the method comprising: In a case where a target fault occurs in any graphics processor in the graphics processor set, querying a pointer to an access function according to a function name of the access function, wherein the access function is used to access any graphics processor in the graphics processor set; Modifying the object pointed to by the pointer into a proxy function, wherein the proxy function includes a device identifier of the faulty graphics processor having the target fault, so as to indicate that the access status of the faulty graphics processor is inaccessible, and the inaccessible status is used to prevent new access tasks to the faulty graphics processor; Resetting the faulty graphics processor, wherein if the resetting is successful, the faulty graphics processor is restored to a target state, which is a normal working state.

2. The method according to claim 1, wherein The method further comprises: In a case where the faulty graphics processor is currently in an accessed state, an access task that is accessing the faulty graphics processor is terminated.

3. The method according to claim 1, wherein The method further comprises: When the calling information of the proxy function is obtained and the proxy function contains the device identifier in the calling information, an access failure message is returned; or When the calling information of the proxy function is acquired and the proxy function does not include the device identification in the calling information, the access function is called.

4. The method according to claim 2, wherein: Before terminating the access task that is accessing the faulty graphics processor, the method further includes: Execute the first query instruction to obtain the query result; If the query result indicates that there is an access task accessing the faulty GPU, it is determined that the faulty GPU is currently in the accessed state.

5. The method according to any one of claims 1 to 4, wherein: The method further comprises: For any graphics processor in the graphics processor set, obtaining operating status data of the graphics processor according to a preset detection period; In a case where the running status data includes a preset fault identifier, the graphics processor is determined to be a faulty graphics processor having the target fault.

6. The method according to any one of claims 1 to 4, wherein: The method further comprises: Sending fault information of the faulty graphics processor to a scheduling device of the graphics processor set, wherein the fault information is used to instruct to remove the faulty graphics processor from a scheduling domain, wherein the scheduling domain includes the graphics processor set; If the reset is successful, recovery information of the faulty graphics processor is sent to the scheduling device, where the recovery information is used to instruct the faulty graphics processor to be added to the scheduling domain.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory, wherein the processor implements the method according to any one of claims 1 to 6 when executing the computer program.

8. A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

9. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images