A GPU command abnormality recovery method

Through the CPU cache and GPU real-time monitoring methods, the fine management problem of GPU command exceptions is solved, and fast recovery and efficient command resend is achieved, which improves the system operation efficiency and user experience.

CN118503016BActive Publication Date: 2024-11-22WUHAN LINGJIU MICROELECTRONICS CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410953968.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-17
Publication Date
2024-11-22
Estimated Expiration
2044-07-17

AI Technical Summary

Technical Problem

In the prior art, when abnormal errors occur when GPU issues commands, it cannot be refined management, and usually only rough resets are performed, resulting in a long system recovery time and possible abnormal rendering results.

Method used

Through the CPU cache command package, the GPU monitors the command execution in real time, judges and feedbacks the location of abnormal commands, and the CPU resends abnormal commands and subsequent commands from the cache, ensuring that the error is corrected within the minimum impact range, and frees memory resources after the GPU is restored .

Benefits of technology

It realizes fine management of GPU command exceptions, reduces system recovery time, avoids unnecessary memory usage and application calls, and improves system efficiency and user experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118503016B_ABST
    Figure CN118503016B_ABST
Patent Text Reader

Abstract

The present invention is applicable to the field of GPU design, and provides a method for recovering GPU command exceptions, including: S1, command packet caching and sending; S2, exception judgment; S3, when no abnormal command occurs, execute the command and release the cached command packet; S4, when an abnormal command occurs, abnormal feedback and locate the command packet and command retransmission. The present invention caches the command packet through the CPU, and the GPU feeds back the abnormal command position, and then locates the position of the abnormal command, and only retransmits the abnormal command position and subsequent commands, corrects the problem of command execution errors within the minimum impact range, and clears the cache of the CPU after the FEP unit of the GPU is successfully executed, so as to solve the problem of GPU execution command exceptions with the minimum negative impact on the entire system. The work of the CPU and the GPU is coordinated and synchronized, and normal commands are restored when the GPU detects a command error, so as to ensure that the rendering result is consistent with expectations.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the technical field of GPU design, and in particular relates to a GPU command abnormality recovery method. Background Art

[0002] In the graphics system, the CPU (Central Processing Unit) controls and coordinates the execution of graphics rendering and computing tasks by issuing commands to the GPU (Graphic Processing Units). By issuing commands, the CPU can send instructions to the GPU, telling the GPU how to process data and perform specific computing tasks. These instructions include drawing graphics, calculating shading, executing computing tasks, etc. Through these instructions, the GPU can perform corresponding operations and processing according to the requirements of the CPU. The process of issuing commands is essentially the communication and collaboration between the CPU and the GPU, which enables the GPU to efficiently execute various graphics and computing tasks, thereby realizing complex graphics rendering and computing requirements.

[0003] The command processing flow of the GPU involves multiple key components and steps, including the memory subsystem, PCIe (Peripheral Component Interface Express) host interface, command buffer, instruction scheduler, etc. During long-term system operation, environmental factors such as voltage fluctuations and temperature drift, coupled with the fact that the command issuance process involves various complex systems, may cause command errors. If the command fails to recover in time after an error occurs, the rendering result will be abnormal, and some unexpected images will be displayed. In order to solve this problem, there are currently some publicly available patented technical solutions:

[0004] For example, patent publication number CN117707848A discloses a GPU device maintenance system, which adopts a scheme to restart abnormal GPU devices. The GPU device sends abnormal information to the driver. After the driver analyzes the abnormal information, it sends the analysis result to the monitoring program in a synchronous or asynchronous manner, and then the monitoring program notifies the service end, that is, the monitoring program sends a service suspension notification information to the service end. When the service end receives the service suspension notification information, it suspends the service corresponding to the GPU device, sends a reset notification information to the monitoring program, and collects the values ​​of all registers in the GPU device as the running status information. At this time, the running status information is stored in the memory of the service end to play a role in breakpoint protection, so that the subsequent GPU device can continue to execute the suspended service after the maintenance is completed. After receiving the reset notification information, the monitoring program sends the reset notification information to the driver, and the driver side generates a reset command and sends the reset command to the GPU device. After receiving the reset command, the GPU device performs a reset operation, and after the reset operation is completed, a reset completion information is formed. After detecting the reset completion information, the service end sends the running status information to the GPU device for register configuration, and the GPU device continues to execute the suspended service.

[0005] In this solution, the GPU device actively sends the abnormal information to the driver, so that the maintenance system can quickly respond to the maintenance of the GPU device, and the driver sends the reset command to the abnormal GPU device, and performs the reset operation on the GPU device side, without resetting the entire operating system layer, so that the abnormal GPU device can be restarted, and the business end collects the operating status information of the GPU device and reconfigures it after the GPU device is reset, so as to ensure that the business can continue to be executed after the GPU device maintenance. This solution only avoids restarting the operating system, but does not avoid restarting the GPU device, and does not provide fine control over the commands that need to be re-issued to the GPU.

[0006] For another example, patent publication number CN114564327A discloses a server multi-card control system based on the SMBUS bus. The solution adopted is to use a built-in watchdog to monitor whether the coprocessor has any abnormality, and to notify the server mainboard by lowering the potential of the alarm signal line when an abnormality occurs. When an abnormality occurs in the coprocessor of the GPU board, the watchdog in the GPU board pulls down the potential of the alarm signal line because it cannot be refreshed; after the server mainboard detects that the alarm signal line is at a low potential, it sends a response address command that complies with the SMBUS protocol to each GPU board. After receiving the command, each GPU board returns its own device address and restores the alarm signal line to a high potential; the server mainboard reads the status of each GPU board according to the obtained device address. If the coprocessor of a GPU board is in an abnormal state, a reset signal is sent to the GPU board. After receiving the reset signal, the GPU board The coprocessor is reset after the signal is received, and a corresponding interrupt signal is generated after the reset is completed; after the coprocessor is reset and restarted, it is informed of an abnormality that occurs in itself based on the interrupt signal, and the alarm signal line is pulled down to a low potential again; after the server mainboard detects that the alarm signal line is at a low potential, it re-issues a response address command that complies with the SMBUS protocol to each GPU board card, and each GPU board card returns its own device address after receiving the command, and restores the alarm signal line to a high potential at the same time; the server mainboard reads the status of each GPU board card again based on the obtained device address, and if there is no abnormality, the GPU board card reset is completed.

[0007] In this solution, the built-in watchdog of the GPU board monitors whether the coprocessor in the board has an abnormality. When an abnormality occurs, the server motherboard is notified by lowering the potential of the alarm signal line. In this way, the server no longer needs to send polling commands to query whether the GPU board has an abnormality, thereby reducing the workload of the server motherboard. This solution also simply restores the normal operation of the GPU by restarting the GPU device, and does not avoid restarting the GPU device, nor does it provide fine control over the commands that need to be re-issued to the GPU.

[0008] In general, it can be seen that the abnormality in the CPU sending commands to the GPU's FEP (Front End Processor) is a system problem. The existing technical solutions often only reset the entire system in a coarse-grained manner, and do not perform refined management on the specific error location. Summary of the invention

[0009] In view of the above problems, the purpose of the present invention is to provide a GPU command exception recovery method, aiming to solve the problem in the prior art that an abnormal error occurs when the GPU issues commands, resulting in the GPU being unable to work normally, and there is no fine management of command exceptions, but only a rough reset of the system.

[0010] The present invention adopts the following technical solution:

[0011] The GPU command abnormality recovery method comprises the following steps:

[0012] Step S1: When the CPU receives a command packet from an application, it caches the command packet into a memory space, sends the command packets one by one to the FEP unit of the GPU, and simultaneously sends the data required to execute the command packet to the GPU memory, wherein the command packet contains multiple commands;

[0013] Step S2: The GPU receives the command packet and data and executes the command packet. During the execution, the GPU monitors the execution of the commands in the command packet in real time to determine whether an abnormality occurs.

[0014] Step S3: If no exception occurs in the command execution, then for the command in the command package that is successfully executed, the GPU immediately sends a feedback notification of successful execution of the current command to the CPU. After the CPU receives the feedback notification of successful execution of all commands in the command package, it deletes the current command package from the memory space to release the memory space.

[0015] Step S4: If an exception occurs in the command execution, the GPU immediately saves the current working state and sends a feedback notification of the current command execution exception to the CPU. The CPU searches the memory space for the command packet corresponding to the abnormal command, and resends it and all subsequent sent and unsent commands to the GPU for execution again, starting from the abnormal command.

[0016] Furthermore, the format of the command packet is as follows from the front to the back: a command packet header, a command packet number, each command information in the command packet, and a command packet tail, wherein each command information includes a command number, a command instruction, and a check bit.

[0017] Furthermore, the following situations are used to determine whether the command execution is abnormal:

[0018] Whether the command legality and data integrity are abnormal: Check the legality of the input command. First, parse the command packet from the CPU to obtain at least the command packet number and command instructions. When the GPU executes the command, it records the command packet number and command instructions currently being executed in real time. If there is a problem with the legality of the command or the data integrity, the GPU sends the abnormal command packet number and the specific location of the abnormal command to the CPU. The CPU finds the command packet and resends it.

[0019] Whether the execution time is abnormal: If the CPU does not receive feedback notification from the GPU within a certain period of time, it is considered that the command execution time is abnormal, and the command for which no feedback notification is received is recorded as an abnormal command. The CPU searches the memory space for the command package corresponding to the abnormal command, and resends it and all subsequent sent and unsent commands to the GPU for execution again starting from the abnormal command; if the command execution time abnormality problem still exists, the CPU issues an instruction to reset the GPU.

[0020] Furthermore, the execution command will also monitor the following situations:

[0021] Whether the access rights are abnormal: Before executing a command, check whether the application from which the current command comes has access rights. If the rights are insufficient, the command will not be executed.

[0022] Is the device connection abnormal? Check the connection status of the GPU to ensure that the GPU is available. If the GPU connection is lost, the GPU will notify the CPU of the hardware connection information. The user will obtain this abnormal information to check the connection line status.

[0023] Is the driver abnormal? Verify whether the GPU driver is correctly installed and configured to ensure that the CPU can communicate with the GPU normally. If the driver does not configure the GPU correctly, the GPU will notify the CPU of the hardware connection information. After the user obtains this abnormal information, he needs to check the GPU driver.

[0024] Whether the GPU status is abnormal: The CPU monitors the working status of the GPU in real time. If the GPU crashes, it will not respond to any CPU commands. After the GPU crashes, it will not respond to the CPU's regular instructions. The CPU issues instructions to reset the GPU.

[0025] Furthermore, the exception feedback notification includes the command packet number of the abnormal command, the specific position of the abnormal command in the command packet, that is, the command number, and the command execution status. After the CPU obtains the feedback notification that the GPU executes the command with an execution exception, it searches the memory space for the command packet where the abnormal command is located and its specific position in the command packet. After finding the command packet, it resends the abnormal command from the position where the abnormal command is located and all the sent and unsent commands after the abnormal command position.

[0026] The beneficial effect of the present invention is that the GPU monitors the commands in real time, and the CPU saves the issued command packets and discards the command packets after receiving the feedback status of the GPU processing completion. Once the GPU finds that the command is abnormal, it saves the current working status and then feeds back the specific position of the command error to the CPU. After the CPU obtains the abnormal command position, it searches for the command packet where the error command is located in the command buffer, and after finding the command packet, it resends the command from the position where the error command is located. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 is a block diagram of a software and hardware system provided by an embodiment of the present invention;

[0028] Figure 2 is a flowchart of a GPU command abnormality recovery method provided by an embodiment of the present invention;

[0029] Figure 3 This is a diagram of the command packet format;

[0030] Figure 4 This is a schematic diagram of the CPU issuing commands and the GPU feeding back execution results. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0032] In a computer system, the CPU and GPU each have different tasks and functions. The CPU is usually responsible for handling general computing tasks, controlling system operation, and processing complex logic, while the GPU focuses on graphics processing and parallel computing. Therefore, by sending commands to the GPU, the CPU is actually assigning graphics processing and parallel computing tasks to the GPU to accelerate execution. By executing commands from the CPU, the GPU can efficiently handle large-scale graphics rendering, deep learning, scientific computing and other tasks, taking advantage of its powerful parallel computing capabilities. The collaboration between the CPU and the GPU enables the computer system to give full play to their respective advantages and achieve efficient task processing and reasonable allocation of resources.

[0033] Regarding the problem that the GPU cannot work properly due to an abnormal error when sending commands, the specific manifestation is that when the CPU continues to send commands to the GPU, the GPU finds that the received commands are abnormal and cannot execute the commands. In the existing technical solutions, there is no fine management of command abnormalities, and only a rough reset of the system. Resetting the GPU requires waiting time and completely suspends the command execution operation, which may cause screen distortion and poor user experience.

[0034] In order to solve this problem, this embodiment provides a GPU command abnormality recovery method, through the CPU cache command packet, the GPU feedback command abnormal position, then the abnormal command position is located, only the abnormal command position and subsequent commands are resent, the command execution error problem is corrected within the minimum impact range, and the CPU clears the cache after the GPU FEP unit is successfully executed, so as to solve the problem of GPU execution command abnormality with the minimum negative impact on the entire system. In order to illustrate the technical solution described in the present invention, it is described below through a specific embodiment.

[0035] like Figure 1 The software and hardware system block diagram of the method of this embodiment is shown. Based on this system block diagram, combined with Figure 2 As shown, the GPU command abnormality recovery method provided in this embodiment includes the following steps:

[0036] Step S1, command packet caching and sending: When the CPU receives a command packet from an application, it caches the command packet into memory space, sends the command packets one by one to the FEP unit of the GPU, and simultaneously sends the data required to execute the command packet to the GPU video memory, wherein the command packet contains multiple commands.

[0037] In computer graphics, the CPU communicates with the GPU through a driver. The GPU, like other hardware devices, is mapped to main memory through specific registers and programmable space. By configuring these registers, commands can be sent to the GPU, such as configuring the GPU for drawing operations. The CPU communicates with the GPU by writing data to specific memory addresses or ports to perform operations such as graphics processing and rendering. This communication method is implemented through system software and drivers, and the CPU does not have special instructions to communicate directly with the GPU. The command interaction process between the CPU and the GPU is the process of sending commands and data to the GPU through the CPU, allowing the GPU to perform specific tasks, and then the GPU returns the execution results to the CPU.

[0038] The organization form of the CPU sending commands to the GPU is usually carried out in the form of specific instructions or command streams. In modern computer systems, the CPU sends commands to the GPU through programming interfaces (such as CUDA, OpenCL, etc.) or graphics APIs (such as OpenGL, DirectX, etc.). These commands may include graphics rendering instructions, computing task instructions, data transfer instructions, etc., so that the GPU can perform corresponding operations. In this embodiment, the register operations, memory operations, etc. involved in the communication between the CPU and the GPU are not repeated here.

[0039] like Figure 1As shown, the driver is called during the execution of the application, and the CPU communicates with the GPU through the driver. For example, when performing a graphics rendering operation, after the CPU receives the command packet, it sends the data and command packet required for graphics rendering to the GPU memory and the FEP unit in the GPU respectively. After receiving the command and data from the CPU, the GPU performs the graphics rendering operation.

[0040] Before sending, in this step, the CPU caches the command packets from the application in the memory space for rollback and resending when an exception occurs in the GPU command execution. Then the command packets are sent one by one to the FEP unit of the GPU, and the data required to execute the command packets is sent to the GPU video memory.

[0041] A command package contains multiple commands. Figure 3 As shown in the figure, the command packet format is as follows, from the front to the back: command packet header, command packet number, each command information in the command packet, command packet tail, where each command information includes a command number, command instruction and check bit. The command packet header and command packet number are located at the packet header position, the command packet tail is located at the packet tail position, and each command information is in the middle.

[0042] Step S2, abnormality judgment: The GPU receives the command packet and data and executes the command packet. During the execution process, the GPU monitors the execution status of the commands in the command packet in real time to judge whether an abnormality occurs.

[0043] After receiving commands from the CPU, the GPU first needs to decode these commands. The GPU identifies the type of command, determines the operation to be performed, and the data and resources involved. The GPU then schedules the decoded commands and arranges them to be executed on the appropriate execution unit. During the execution of the command, the GPU may need to read data from the system memory or other storage, perform calculations, and then write the results back to the memory. The GPU monitors the execution of commands in real time and makes exception judgments on the executed commands. The exception judgment mainly includes whether the command legality and data integrity are abnormal, whether the execution time is abnormal, etc. Among them:

[0044] Whether the legality of the command and the integrity of the data are abnormal: Perform a legality check on the input command, including whether the command format, parameters, etc. comply with the specifications. First, parse the command packet from the CPU, and at least obtain the command packet number and command instructions. When the GPU executes the command, it records the command packet number and command instructions currently being executed in real time. If there is a problem with the legality of the command or the integrity of the data, this abnormal situation needs to be handled according to the scheme of this embodiment, that is, the GPU sends the abnormal command packet number and the specific location of the abnormal command to the CPU, and the CPU finds the command packet and resends it.

[0045] Whether the execution time is abnormal: After the CPU sends a command packet to the GPU, it needs to obtain the execution result of the GPU. If the CPU does not receive a feedback notification from the GPU within a certain period of time, the command execution times out, and the command execution time is considered abnormal. This abnormal situation also needs to be handled according to the scheme of this embodiment. The command that has not received a feedback notification is recorded as an abnormal command. The CPU searches for the command packet corresponding to the abnormal command from the memory space, and resends it and all subsequent sent and unsent commands to the GPU for execution again starting from the abnormal command; if the problem of abnormal command execution time still exists, other processing methods need to be added, such as the CPU issuing an instruction to reset the GPU. Resetting the GPU is to restore it to a reliable state so that it can continue to execute subsequent commands, which may involve operations such as clearing error status and resetting the command buffer.

[0046] In addition, it is also possible to monitor whether access permissions are abnormal, device connections are abnormal, driver programs are abnormal, GPU status is abnormal, etc. The details are as follows:

[0047] Whether the access rights are abnormal: Before executing a command, check whether the application from which the current command comes has access rights. If the rights are insufficient, the command will not be executed. This abnormality does not need to be handled.

[0048] Is the device connection abnormal? Check the connection status of the GPU to ensure that the GPU is available. If the GPU connection is lost, the GPU will notify the CPU of the hardware connection information. The user will obtain this abnormal information to check the connection line. For example, you need to manually check the device connection line, power supply, etc., and try to reconnect the device.

[0049] Is the driver abnormal? Verify whether the GPU driver is correctly installed and configured to ensure that the CPU can communicate with the GPU normally. If the driver does not configure the GPU correctly, the GPU will notify the CPU of the hardware connection information. After the user obtains this abnormal information, he needs to check the GPU driver.

[0050] Whether the GPU status is abnormal: The CPU monitors the working status of the GPU in real time. If the GPU crashes, it will not respond to any CPU commands. After the GPU crashes, it will not respond to the CPU's regular instructions. The CPU issues instructions to reset the GPU.

[0051] Step S3, execute the command and release the cached command package: If there is no exception in the command execution, for the command in the command package that is successfully executed, the GPU immediately sends a feedback notification of successful execution of the current command to the CPU. After the CPU receives the feedback notification of successful execution of all commands in the command package, it deletes the current command package from the memory space to release the memory space.

[0052] If there is no exception in the command execution, for the command successfully executed by the GPU, the GPU will immediately send a feedback notification of the successful execution of the current command to the CPU. After receiving the feedback notification of the successful execution of the GPU, the CPU will discard the successfully executed command packet to free up memory space.

[0053] Step S4, abnormal feedback and positioning command package and command resending: If an abnormality occurs in command execution, the GPU immediately saves the current working state and sends a feedback notification of the current command execution abnormality to the CPU. The CPU searches for the command package corresponding to the abnormal command from the memory space, and resends it and all subsequent sent and unsent commands to the GPU for execution again starting from the abnormal command.

[0054] The abnormal feedback notification includes the command packet number of the abnormal command, the specific position of the abnormal command in the command packet, that is, the command number, and the command execution status. After the abnormal command occurs, the GPU reports the abnormal position of the command execution to the CPU in real time through interrupts and registers. After the CPU obtains the feedback notification of the execution abnormality of the GPU execution command, it searches the memory space for the command packet where the abnormal command is located and the specific position in the command packet. After finding the command packet, it resends the abnormal command and all the sent and unsent commands after the abnormal command position from the abnormal command position.

[0055] In this step, the CPU uses the following strategy to quickly find the location of the abnormal command from the commands that need to be sent in the memory space:

[0056] Using a hash table data structure, commands and their status (such as whether they have been executed) are stored in a hash table, so that specific command objects can be quickly located by command ID or other unique identifiers. Each command object contains detailed information about the command and the execution status.

[0057] For ease of understanding, combined Figure 4 The diagram shows the CPU sending commands and the GPU feeding back the execution results. There are 5 consecutive command packets, each with 8 commands. The fourth command of command packet 4 has been sent to the diagram, and the third command of command packet 2 has been successfully executed. If a notification of abnormal feedback about the fourth command of command packet 3 is received, all 4 commands starting from the third command of command packet 3 and the commands sent by command packet 4 are resent.

[0058] In summary, the technical solution of the present invention has the following characteristics:

[0059] First, the CPU caches the command packet for resending when an abnormal command occurs on the GPU. Since the command packet from the application is cached in advance, there is no need to resend the command from the application when the command needs to be re-executed, thus avoiding unnecessary application calls and improving the operating efficiency of the system.

[0060] Secondly, when the command execution ends normally, the GPU reports the successful execution end position of the command to the CPU through interrupts and registers. Since the GPU reports the status immediately after successfully executing the command, the CPU can release the command packet cached in the memory space immediately, thus avoiding unnecessary memory resource occupation.

[0061] Third, the GPU reports the error location of the command execution to the CPU in real time through interrupts and registers. Since the CPU can receive the error status of abnormal commands in real time, the system can avoid long-term abnormal operation.

[0062] Finally, the GPU feeds back the position of the abnormally executed command in the command packet and the command packet where the abnormal command is located to the CPU in real time through interrupts and registers. Since the CPU can accurately locate the position where the command needs to be resent after obtaining this position, it avoids repeatedly sending a large number of executed commands from the beginning, thereby improving the operating efficiency of the system.

[0063] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A GPU command abnormality recovery method, characterized in that: The method comprises the following steps: Step S1: When the CPU receives a command packet from an application, it caches the command packet into a memory space, sends the command packets one by one to the FEP unit of the GPU, and simultaneously sends the data required to execute the command packet to the GPU memory, wherein the command packet contains multiple commands; Step S2: The GPU receives the command packet and data and executes the command packet. During the execution, the GPU monitors the execution of the commands in the command packet in real time to determine whether an abnormality occurs. Step S3: If no exception occurs in the command execution, then for the command in the command package that is successfully executed, the GPU immediately sends a feedback notification of successful execution of the current command to the CPU. After the CPU receives the feedback notification of successful execution of all commands in the command package, it deletes the current command package from the memory space to release the memory space. Step S4: If an exception occurs in the command execution, the GPU immediately saves the current working state and sends a feedback notification of the current command execution exception to the CPU. The CPU searches the memory space for the command packet corresponding to the abnormal command, and resends the abnormal command and all subsequent sent and unsent commands to the GPU for re-execution. The format of the command packet is as follows: a command packet header, a command packet number, each command information in the command packet, and a command packet tail, wherein each command information includes a command number, a command instruction, and a check bit; In step S2, it is determined whether the command execution is abnormal in the following situations: Whether the command legality and data integrity are abnormal: Check the legality of the input command. First, parse the command packet from the CPU to obtain at least the command packet number and command instructions. When the GPU executes the command, it records the command packet number and command instructions currently being executed in real time. If there is a problem with the legality of the command or the data integrity, the GPU sends the abnormal command packet number and the specific location of the abnormal command to the CPU. The CPU finds the command packet and resends it. Whether the execution time is abnormal: If the CPU does not receive feedback notification from the GPU within a certain period of time, it is considered that the command execution time is abnormal, and the command for which no feedback notification is received is recorded as an abnormal command. The CPU searches the memory space for the command package corresponding to the abnormal command, and resends it and all subsequent sent and unsent commands to the GPU for execution again starting from the abnormal command. If the command execution time is still abnormal, the CPU issues an instruction to reset the GPU. The execution command also monitors the following: Whether the access rights are abnormal: Before executing a command, check whether the application from which the current command comes has access rights. If the rights are insufficient, the command will not be executed. Is the device connection abnormal? Check the connection status of the GPU to ensure that the GPU is available. If the GPU connection is lost, the GPU will notify the CPU of the hardware connection information. The user will obtain this abnormal information to check the connection line status. Is the driver abnormal? Verify whether the GPU driver is correctly installed and configured to ensure that the CPU can communicate with the GPU normally. If the driver does not configure the GPU correctly, the GPU will notify the CPU of the hardware connection information. After the user obtains this abnormal information, he needs to check the GPU driver. Is the GPU status abnormal? The CPU monitors the working status of the GPU in real time. If the GPU crashes, it will not respond to any CPU commands. After the GPU crashes, it will not respond to the CPU's regular commands. The CPU issues a command to reset the GPU. In step S4, the feedback notification of the execution exception includes the command packet number of the abnormal command, the specific position of the abnormal command in the command packet, that is, the command number, and the command execution status. After the CPU obtains the feedback notification that the GPU executes the command with an execution exception, it searches the memory space for the command packet where the abnormal command is located and its specific position in the command packet. After finding the command packet, it resends the abnormal command from the position where the abnormal command is located and all the sent and unsent commands after the abnormal command position.