Method and device for recovering from hang of operation core, electronic equipment and storage medium

Abstract

The application provides an operation core hang-up recovery method and device, electronic equipment and a storage medium, relates to the chip design and manufacturing technical field, and comprises the following steps: obtaining and sending tasks from a first task queue to an operation core, backing up the tasks to a second task queue, and marking the tasks in the second task queue as valid; in response to task execution information sent by the operation core, marking the task execution information in the second task queue as invalid; in response to hang-up recovery information, pausing the task obtaining from the first task queue, and sending the tasks marked as valid in the second task queue to the operation core; canceling the hang-up recovery information and resuming the task obtaining from the first task queue and sending the tasks to the operation core. The method and device provided by the application realize fast and efficient task distribution recovery in the case of operation core hang-up, shorten the delay from hang-up to recovery, and improve the operation efficiency and performance of the entire graphics processor.

Description

Technical Field

[0001] This application relates to the field of chip design and manufacturing technology, and in particular to a method, apparatus, electronic device, and storage medium for recovering from a computer core crash. Background Technology

[0002] Graphics Processing Units (GPUs) need to handle a massive number of tasks. These tasks are parsed and broken down by the Command Processor (CP) inside the processor, and ultimately scheduled to be executed on hundreds or thousands of GPU cores. During task execution, one or more GPU cores may become stuck, meaning they stop responding and cannot continue executing subsequent instructions. This local failure often causes the instruction pipeline to block, which can then cause the entire GPU to stop running, severely impacting the stability of the GPU and the continuity of computational tasks.

[0003] Related technologies typically employ two recovery methods. One method is a process-level recovery scheme. This scheme involves not only the coordination of multiple hardware modules but also deep software involvement. The entire recovery process is time-consuming, causing the graphics processor to remain idle for extended periods, resulting in low processing efficiency. The other method is a hardware reset of the command processor. This scheme not only loses all currently processed context information but also has an extremely long recovery time, leading to a severe performance sag in the graphics processor.

[0004] Therefore, how to quickly and efficiently restore task distribution in scenarios where the computing core freezes, and improve the operating efficiency and performance of graphics processors, has become a technical problem that the industry urgently needs to solve. Summary of the Invention

[0005] This application provides a method, apparatus, electronic device, and storage medium for recovering from a suspended computing core, which addresses the technical problem of how to quickly and efficiently resume task distribution in scenarios where a computing core is suspended, thereby improving the operating efficiency and performance of the graphics processor.

[0006] This application provides a method for recovering from a suspended computing core, including: The task is retrieved from the first task queue and sent to the computing core. At the same time, the task is backed up to the second task queue and marked as valid in the second task queue. In response to the task execution information sent by the computing core, the task corresponding to the task execution information in the second task queue is marked as invalid; In response to the hang-up recovery information, the process of acquiring tasks from the first task queue is paused, and tasks marked as valid in the second task queue are sent to the computing core. If there are no valid tasks in the second task queue, the hang-up recovery information is revoked, and tasks are retrieved from the first task queue and sent to the computing core.

[0007] In some embodiments, the hang-up recovery information is represented by the status value of the target register; In response to the hang-up recovery information, the target register is set to a first preset value; The cancellation of the hang recovery information corresponds to the target register being set to the second preset value.

[0008] In some embodiments, the step of retrieving and sending tasks from the first task queue to the computing core includes: Obtain the status value of the target register and the current number of tasks in the second task queue; When the state value of the target register is the second preset value and the current number of tasks is less than the queue depth of the second task queue, a task is retrieved from the first task queue and sent to the computing core.

[0009] In some embodiments, the step of retrieving and sending tasks from a first task queue to the computing core, while simultaneously backing up the tasks to a second task queue, includes: Determine the backup location of the task in the second task queue; The index information of the backup location is associated with the task and sent to the computing core, while the task is stored in the backup location in the second task queue.

[0010] In some embodiments, the step of marking the task corresponding to the task execution information in the second task queue as invalid in response to the task execution information sent by the computing core includes: The task execution information is parsed to obtain index information; Based on the index information, determine the corresponding task in the second task queue; Mark the corresponding task as invalid.

[0011] In some embodiments, the method of sending the tasks marked as valid in the second task queue to the computing core further includes: Keep the task's label in the second task queue unchanged; Upon receiving the task execution information of the task sent by the computing core, the task is marked as invalid in the second task queue.

[0012] In some embodiments, the queue depth of the second task queue is determined based on the maximum number of tasks that the computing core can concurrently process.

[0013] This application provides a device for recovering from a computer core being stuck, comprising: The task sending module is used to obtain and send tasks from the first task queue to the computing core, and at the same time back up the tasks to the second task queue and mark the tasks as valid in the second task queue; The task marking module is used to mark the task corresponding to the task execution information in the second task queue as invalid in response to the task execution information sent by the computing core; The task resend module is used to respond to the hang-up recovery information, pause the acquisition of tasks from the first task queue, and send the tasks marked as valid in the second task queue to the computing core. The hang-up recovery module is used to cancel the hang-up recovery information and resume the acquisition and sending of tasks from the first task queue to the computing core when there are no tasks marked as valid in the second task queue.

[0014] This application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the aforementioned method for recovering from a hangup of the computing core.

[0015] This application provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the aforementioned method for recovering from a hangup of the computing core.

[0016] The computing core hang recovery method, apparatus, electronic device, and storage medium provided in this application, by setting a second task queue, retrieves and sends tasks from the first task queue to the computing core when the computing chip is not hanged, and simultaneously backs up the tasks to the second task queue; when the computing chip hangs and recovery is required, the tasks marked as valid in the second task queue are sent to the computing core; through automatic task backup and automatic resending, fast and efficient task distribution recovery is achieved in the scenario where the computing core hangs, without having to re-execute the entire instruction processing pipeline, reducing the involvement of hardware modules, and eliminating the need for complex software intervention, greatly shortening the delay from hang to recovery, reducing the idle time of the computing core, and thus significantly improving the operating efficiency and performance of the entire graphics processor. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0018] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is one of the flowcharts illustrating the method for recovering from a suspended computing core provided in this application.

[0020] Figure 2 This is a schematic diagram of the hardware structure of the task distribution module provided in this application.

[0021] Figure 3 This is the second flowchart of the method for recovering from a hangup in the computing core provided in this application.

[0022] Figure 4 This is a schematic diagram of the structure of the computing core hang-up recovery device provided in this application.

[0023] Figure 5 This is a schematic diagram of the structure of the electronic device provided in this application.

[0024] Figure label: 200: Task distribution module; 210: Instruction packet splitting module; 220: First task queue; 230: Second task queue; 240: Target register; 300: Computational core; 410: Task sending module; 420: Task marking module; 430: Task resending module; 440: Hangout recovery module; 510: Processor; 520: Communication interface; 530: Memory; 540: Communication bus. Detailed Implementation

[0025] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present application.

[0026] It should be noted that the terms "first," "second," etc., used in this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that comprises a series of steps, units, or modules is not necessarily limited to those explicitly listed, but may include other steps, units, or modules not explicitly listed or inherent to such processes, methods, products, or devices.

[0027] When a graphics processing unit (GPU) core hangs during task execution, two methods are used to restore task distribution in this scenario. One method is to restore at the entire process level. This requires stopping, draining, and resetting all instruction packets (pkts) associated with the process, followed by software re-fetching instructions from external memory. This process is cumbersome, involves multiple hardware modules and software interventions, and is time-consuming. The other method is to directly perform a hardware reset of the entire command processor. While this method is thorough, the recovery cost is extremely high, equivalent to a cold start of the entire command processor. It requires a complete reconfiguration of the software, resulting in a prolonged GPU downtime and extremely low efficiency.

[0028] In order to address the shortcomings of related technologies, Figure 1 This is one of the flowcharts illustrating the method for recovering from a suspended operating core provided in this application, such as... Figure 1 As shown, the method includes steps 110, 120, 130 and 140.

[0029] Step 110: Obtain and send the task from the first task queue to the computing core, back up the task to the second task queue, and mark the task as valid in the second task queue.

[0030] Specifically, the execution subject of the computing core hang-up recovery method provided in this application embodiment is a computing core hang-up recovery device. This device can be implemented in software, such as a computing core hang-up recovery program; or it can be a hardware device that executes the computing core hang-up recovery method, such as a task distribution module in a command processor, a graphics processor containing the command processor, and a terminal, computer, or server equipped with the graphics processor, etc.

[0031] In this application, the chip refers to a processor that includes multiple computing cores for high-performance parallel computing, such as a graphics processor, a neural processing unit (NPU), or an artificial intelligence (AI) accelerator.

[0032] A processing core can be understood as a hardware processing unit within a chip responsible for executing specific computational tasks. A task is the smallest working unit assigned to the processing core for execution, typically derived from higher-level modules based on more macroscopic instruction packets (Packets, pkts) or commands (Commands, cmds).

[0033] Generally, the Central Processing Unit (CPU) issues computation commands, and the Command Processor (CP) in the Graphics Processing Unit (GPU) receives and parses these commands. Its internal Command Processor Dispatcher (CPD) decomposes the macro-tasks and assigns them to one or more GPU-Cores.

[0034] In a normal task execution process, the task distribution module retrieves tasks to be processed from the first task queue. This first task queue can be a First-In-First-Out (FIFO) hardware buffer used to temporarily store tasks generated by upstream modules and awaiting distribution to the computing core. In a specific embodiment, the first task queue can be referred to as the Jobpool FIFO.

[0035] When a task is retrieved from the first task queue and prepared to be sent to the computing core for execution, the task distribution module also performs a parallel backup operation. Specifically, this involves storing a copy of the same task in a dedicated backup queue, which is referred to as the second task queue in this embodiment. The second task queue can be a hardware-implemented storage array or buffer. In a specific embodiment, the second task queue can be referred to as a copy tab.

[0036] While backing up a task to the second task queue, a status flag needs to be created for that backup task, marking it as valid. This valid flag indicates that the task has been assigned to the computing core, but its execution result has not yet been confirmed. This flag can be a dot associated with the backup task entry; for example, it can be set to a preset value (e.g., "1"), indicating that the backup entry is valid and its corresponding task is waiting for the execution result.

[0037] Step 120: In response to the task execution information sent by the computing core, mark the task corresponding to the task execution information in the second task queue as invalid.

[0038] Specifically, after the computing core completes the execution of a task, it sends a task execution message to the upstream module (i.e., the task distribution module) to indicate that the task has been completed. The task execution message can be a signal or data packet containing a specific identifier. In a specific embodiment, the task execution message can be a job done signal, which clearly indicates which task has been completed.

[0039] Upon receiving the task execution information, the task distribution module, responsible for managing the second task queue, will locate the corresponding backup task entry in the second task queue based on this information. Once found, it will update the entry's status flag, marking it as invalid. Marking a task as invalid means that the task has been successfully executed by the computing core, its backup no longer needs to be retained, and the corresponding storage space can be released for backing up subsequent new tasks. For example, the previously valid flag can be cleared or set to another preset value (e.g., "0").

[0040] Through the cyclical execution of the above steps, the second task queue always maintains a valid backup of all tasks that have been issued but not yet confirmed as completed.

[0041] Step 130: In response to the hang-up recovery information, pause the acquisition of tasks from the first task queue and send the tasks marked as valid in the second task queue to the computing core.

[0042] Specifically, when the processing core hangs while executing a task (i.e., the processing core stops responding, cannot continue execution, and cannot return task execution information), the upper-level management logic of the graphics processor (such as the driver software) will detect this abnormal state. At this time, the upper-level management logic can perform a partial reset on the processing core, restoring it from the hung state to an idle state that can receive new tasks.

[0043] After the computing core recovers, the upper-level management logic or related hardware will issue a hang-up recovery message. This message is a control signal or instruction whose purpose is to trigger the hang-up recovery process.

[0044] In response to the hangup recovery information, the task distribution module performs two key actions. The first is to pause the acquisition of new tasks. Specifically, this involves controlling the first task queue to temporarily stop sending new tasks to the computing core's path. This prevents new tasks from interfering with the recovery process and avoids sending tasks to computing cores that may not yet be fully stable during recovery. The second action is to resend incomplete tasks. Specifically, the task distribution module, which manages the second task queue, begins scanning all task entries within it to identify all tasks still marked as valid. These valid tasks are those that were issued before the computing core hangup but did not receive completion information. Subsequently, the task distribution module resends these valid tasks to the recovered computing core in a specific order (e.g., according to their order of storage in the queue).

[0045] Step 140: If there are no valid tasks in the second task queue, cancel the hang-up recovery information and resume retrieving and sending tasks from the first task queue to the computing core.

[0046] Specifically, valid tasks in the second task queue are resent to the computing core one by one. After successfully executing these resent tasks, the computing core returns task execution information as normal. Upon receiving this information, the task distribution module, which manages the second task queue, continues to mark the corresponding backup tasks as invalid.

[0047] When all valid tasks in the second task queue have been successfully resent and executed (i.e., there are no longer any task entries marked as valid in the second task queue), it indicates that all unprocessed tasks before the hang has been completed. At this point, the task distribution module automatically cancels the hang recovery information, for example, restoring the control signals that previously triggered the recovery process to their normal state.

[0048] Once the hang-up recovery information is revoked, the pause restriction on the first task queue is lifted. The task distribution module resumes its normal workflow, that is, it resumes retrieving new tasks from the first task queue and sending them to the computing core for execution, while simultaneously performing backups. The entire process returns to its normal loop.

[0049] The computing core hang recovery method provided in this application embodiment sets up a second task queue. When the computing chip is not hanged, tasks are retrieved from the first task queue and sent to the computing core, while the tasks are backed up to the second task queue. When the computing chip hangs and recovery is required, the tasks marked as valid in the second task queue are sent to the computing core. Through automatic task backup and automatic resending, task distribution is quickly and efficiently restored in the scenario where the computing core hangs. There is no need to re-execute the entire instruction processing pipeline, reducing the involvement of hardware modules and eliminating the need for complex software intervention. This greatly shortens the delay from hang to recovery, reduces the idle time of the computing core, and thus significantly improves the operating efficiency and performance of the entire graphics processor.

[0050] It should be noted that each implementation method of this application can be freely combined, rearranged, or executed individually, and does not need to rely on or depend on a fixed execution order.

[0051] In some embodiments, hang-up recovery information is represented by the status value of the target register; In response to hang-up recovery information, the target register is set to the first preset value; Cancel the hang recovery information, which corresponds to the target register being set to the second preset value.

[0052] Specifically, hang recovery information can be represented by the status value of a target register. This target register can be a dedicated, software-readable and writable hardware register, and its physical location can be set within the task distribution module so that its status value can be easily accessed by relevant functional units. In one specific embodiment, this target register can be referred to as the Retry register.

[0053] The trigger and undo operations for hang-up recovery information specifically correspond to writing to the target register to change its status value.

[0054] When the upper-level management logic (such as driver software) detects that the computing core has hung and successfully performs a partial reset operation on the computing core, the software needs to initiate the hardware-level task recovery process. At this time, the software writes a first preset value to the target register. The first preset value is a pre-defined specific value used to start the recovery mode, such as a logic high level "1".

[0055] Once the target register's status value changes to the first preset value, this status value is sent as a control signal (hang-up recovery information) to the relevant hardware module. Upon receiving this signal, the control logic of the first task queue suspends its task dequeueing operation, while the control logic of the second task queue starts scanning and resending valid tasks.

[0056] During the hang-up recovery process, once the control logic of the second task queue detects that all tasks marked as valid within it have been successfully resent and has received task execution information returned by the computing core (i.e., all unexecuted tasks have been cleared), the control logic will proactively write a second preset value to the target register. This second preset value is a pre-defined specific value used to represent the normal operating mode, such as a logic low level "0".

[0057] Once the target register's status value is restored to the second preset value, it indicates that the hang-up recovery process has ended. Upon detecting this status change, the control logic of the first task queue will release the paused state, resume retrieving new tasks from the queue, and send them to the processing core.

[0058] The hang-up recovery method for the computing core provided in this application realizes the triggering and cancellation of hang-up recovery by setting the state value of the target register. This allows the upper-layer software to trigger the complex hardware recovery process simply by writing a specific value to the register at the appropriate time, without having to worry about the internal details of the recovery process or the timing of the recovery end. The end of the recovery is automatically determined by the hardware and the state switch is completed. This greatly simplifies the development logic of software-driven systems, reduces the coupling between software and hardware, and ensures the timeliness and efficiency of the recovery process, thereby improving the overall operating efficiency and performance of the graphics processor.

[0059] In some embodiments, retrieving and sending tasks from the first task queue to the computing core includes: Get the status value of the target register and the number of current tasks in the second task queue; If the target register's status value is the second preset value and the current number of tasks is less than the queue depth of the second task queue, then the task is retrieved from the first task queue and sent to the computing core.

[0060] Specifically, before retrieving a new task from the first task queue, the task dispatch module first obtains the status value of the target register and the current number of tasks in the second task queue.

[0061] The status value of the target register can indicate whether the processing core is in normal working mode or hang-up recovery mode.

[0062] The current number of tasks in the second task queue refers to the number of backup tasks that are currently stored and marked as valid. This number reflects the occupancy status of the second task queue.

[0063] A new task can be dequeued from the first task queue only if both of the following conditions are met simultaneously.

[0064] Condition one is that the state value of the target register is the second preset value.

[0065] The second preset value indicates that the graphics processor is in normal operating mode. Checking this condition ensures that the hang-up recovery process has higher priority. If the target register value is set to the first preset value, it means the graphics processor is performing a hang-up recovery; in this case, new task issuance must be paused, and resent tasks in the second task queue must be processed first. Only after recovery is complete and the target register is automatically restored to the second preset value by the hardware can the new task issuance process continue.

[0066] Condition two is that the current number of tasks is less than the queue depth of the second task queue.

[0067] The queue depth of the second task queue refers to the maximum number of backup tasks that the queue can hold. This condition is essentially a back pressure mechanism. Every task issued from the first task queue must have a corresponding backup in the second task queue. If the second task queue is full (i.e., the current number of tasks equals its queue depth), it means there is no space to back up new tasks. At this point, even if the graphics processor is in normal operating mode, it must pause acquiring tasks from the first task queue until the processing core completes some existing tasks, thus freeing up space in the second task queue.

[0068] The task distribution module will only retrieve a new task from the first task queue and send it to the computing core when the graphics processor is in normal operating mode (not hang-up recovery mode) and there is still space in the backup queue. If either of these two conditions is not met, the control logic of the first task queue will remain in a waiting state until both conditions are met simultaneously.

[0069] The computing core hang recovery method provided in this application embodiment not only ensures the priority execution of the hang recovery process and avoids conflicts between new and old task flows, but also ensures that each executed task has a reliable backup through the back pressure mechanism, laying a solid foundation for rapid recovery, thereby greatly improving the stability and reliability of the entire graphics processor.

[0070] In some embodiments, retrieving and sending tasks from a first task queue to the computing core, while simultaneously backing up the tasks to a second task queue, includes: Determine the backup location of the task in the second task queue; The index information of the backup location is associated with the task and sent to the computing core. At the same time, the task is stored in the backup location in the second task queue.

[0071] Specifically, before retrieving a new task from the first task queue and preparing to distribute and back it up, the task distribution module needs to reserve a storage location for that task in the second task queue. This process is called determining the backup location.

[0072] The backup location is a specific physical storage unit in the second task queue, such as a row (Entry) in the queue. Each location has a unique identifier, which is referred to as index information in this embodiment. The index information can be an integer, such as a specific row number.

[0073] There are several ways to determine the backup location. For example, the control logic of the second task queue can maintain a write pointer that always points to the next available free location. Whenever a new task needs to be backed up, the location currently pointed to by this pointer is used as the backup location, and the pointer is subsequently updated (e.g., incremented or pointed to the next location marked as invalid).

[0074] After determining the backup location and its index information, the task distribution module will perform two parallel operations.

[0075] One operation involves associating index information with a task and sending it to the processing core. Specifically, this association can involve embedding the index information into the data structure of the task to be sent. For example, a task's data packet can contain multiple fields, one of which can be reserved specifically for carrying the task's backup index in the second task queue. Subsequently, this task, carrying its own backup location index information, is sent to the processing core for processing. The main purpose of sending the index information to the processing core is to ensure that after completing the task, the processing core can retrieve this index verbatim in its returned task execution information, thus clearly informing the upstream module which backup task can be cleared.

[0076] The second step is to store the task in a backup location. Simultaneously with sending the task to the computing core, a complete copy of the task is written to the backup location in the second task queue. This ensures that throughout the entire lifecycle of the task being received and executed by the computing core, a perfectly corresponding and recoverable backup exists in the second task queue.

[0077] The computing core hang recovery method provided in this application determines the backup location and index information for each task, so that when it is necessary to query or update the task status, there is no need to go through complex content comparison. The task can be quickly located based solely on the index information, which greatly shortens the delay from hang to recovery and significantly improves the operating efficiency and performance of the entire graphics processor.

[0078] In some embodiments, in response to task execution information sent by the computing core, marking the task corresponding to the task execution information in the second task queue as invalid includes: Parse the task execution information to obtain the index information; The corresponding task is determined in the second task queue based on the index information; Mark the corresponding task as invalid.

[0079] Specifically, after a computing core successfully completes a task, it sends task execution information to the upstream task distribution module. When processing a task, the task itself carries its index information in the second task queue. Therefore, after completing the task, the computing core includes this received index information in its sent task execution information and returns it as is.

[0080] Upon receiving the task execution information, the task distribution module, responsible for handling task completion confirmation, parses it to extract index information. For example, if the task execution information is a data packet, the module reads a specific field reserved for index information within the data packet to obtain a specific value (such as a line number).

[0081] After successfully extracting the index information from the task execution information, this index information is used for addressing in the second task queue. Since the second task queue is a structured memory, and each storage location has a unique index, the corresponding backup task's storage unit can be located directly and quickly based on the obtained index information. This index-based direct addressing method is far more efficient than searching for backup tasks by comparing task content one by one, especially when there are a large number of backup tasks in the second task queue.

[0082] After accurately locating the backup task entry in the second task queue using the index information, the task distribution module updates the entry's status. Specifically, it marks the task corresponding to the entry as invalid. Once the task is marked as invalid, the storage space occupied by the backup entry is released and can be used to store backups for subsequent new tasks. This signifies that the entire lifecycle of the task (from distribution, backup, execution to confirmation of completion) has ended.

[0083] The computing core hang-up recovery method provided in this application embodiment utilizes the index information transmitted during the task lifecycle to construct an efficient closed loop from task issuance to completion confirmation, thereby improving the operational efficiency of the second task queue and significantly enhancing the overall operating efficiency and performance of the graphics processor.

[0084] In some embodiments, the method of sending tasks marked as valid in the second task queue to the computing core further includes: Keep the task's tag in the second task queue unchanged; Upon receiving the task execution information for the task from the computing core, the task is marked as invalid in the second task queue.

[0085] Specifically, after the hang-up recovery process is triggered, the control logic of the second task queue scans and identifies all tasks marked as valid. When it resends one of the valid tasks to the recovered computing core, it does not immediately modify the task's status flag in the second task queue at the time of resending. In other words, the backup entry for that task continues to maintain its valid status.

[0086] Since it cannot be guaranteed that the computing core will not hang again when executing this retransmission task after being reset, if the backup is marked as invalid at the beginning of the retransmission, the backup of the task will be lost if it hangs again during the recovery process, causing the task to be permanently skipped, which may lead to data processing errors or other anomalies.

[0087] After the task is resent, the control logic of the second task queue will wait for the computing core to return the task execution information, just like the normal process.

[0088] Only after receiving a completion confirmation message from the computing core for this resend task will the control logic of the second task queue parse the index in the task execution information, find the corresponding backup entry based on the index, and then update its status mark to invalid.

[0089] This means that, regardless of whether a task is sent for the first time or resent during the recovery process, the only condition for marking its backup as invalid is that task execution information must be received from the computing core.

[0090] The computing core hang recovery method provided in this application embodiment ensures that even if an anomaly occurs again during the recovery process, the system can still maintain the correct task state and has the opportunity to try to recover again, thereby improving the fault tolerance and reliability of computing core hang recovery.

[0091] In some embodiments, the queue depth of the second task queue is determined based on the maximum number of tasks that the computing core can process concurrently.

[0092] Specifically, the queue depth of the second task queue refers to the maximum number of backup task entries that the queue can hold in total.

[0093] The maximum number of concurrent tasks a computing core can process refers to the maximum number of tasks that a computing core can process in parallel at any given time. This is a key performance indicator reflecting the hardware design and processing power of the computing core, and is technically often referred to as its outstanding unprocessed request capacity.

[0094] Setting the queue depth of the second task queue equal to the maximum number of tasks that the computing core can handle concurrently ensures that as long as the computing core has the capacity to accept new tasks, the second task queue will always have space to provide backup. Backpressure signals in the instruction pipeline will only be generated by the computing core when it truly reaches its processing capacity limit, and will not be erroneously triggered by insufficient capacity in the backup queue.

[0095] The method for recovering a suspended computing core provided in this application embodiment ensures that the computing core can be fully utilized, thereby maximizing the task processing throughput of the entire graphics processor and significantly improving the operating efficiency and performance of the entire graphics processor.

[0096] Figure 2 This is a schematic diagram of the hardware structure of the task distribution module provided in this application, as shown below. Figure 2 As shown, the task distribution module 200 in the command processor includes an instruction packet splitting module 210, a first task queue 220, a second task queue 230, and a target register 240. The task distribution module 200 is connected to the computing core 300.

[0097] Figure 3 This is the second flowchart of the method for recovering from a suspended computing core provided in this application, as shown below. Figure 3 As shown, the method includes: Step 310: Set the queue depth of the second task queue.

[0098] The depth of the second task queue is equal to the maximum number of concurrent tasks that the downstream computing cores can handle. This ensures that when sending tasks, the backpressure will only occur because the downstream computing cores cannot execute too many tasks simultaneously, rather than because the upstream command processor is limited in its ability to send tasks.

[0099] Step 320: Check the conditions for issuing tasks in the first task queue.

[0100] Condition one is that the target register's status value is the second preset value. Condition two is that the current number of tasks is less than the queue depth of the second task queue.

[0101] Step 330: Obtain and send tasks from the first task queue to the computing core, and back up the tasks to the second task queue at the same time.

[0102] When the first task queue issues a task, it includes the index of the next available free space in the second task queue within the task and sends it to the computing core. The task issued by the first task queue is simultaneously sent to the backup in the second task queue, placed in the storage space corresponding to the identified index. A prefix for this space is written with the task type and a validity flag. When the downstream computing core executes normally, after each task is completed, it sends task execution information back to the second task queue, including the index of its backup in the second task queue and the type of the current task. After successfully comparing the task type of the corresponding index, the second task queue sets the validity flag in the corresponding prefix to invalid, indicating that the current index is invalid in the second task queue and the task does not need to be backed up. The next task can then be placed at this position.

[0103] Step 340: Trigger the hang-up recovery process based on the status value of the target register.

[0104] When a downstream computing core hangs while executing certain tasks, meaning it cannot correctly execute the received tasks (at this time, the downstream computing core will exert back pressure, and the command processor cannot send tasks to the downstream computing core), the upper-layer software can choose to reset the downstream computing core separately, directly releasing the computing core from the hang state and restoring it to the idle state.

[0105] The upper-layer software continuously monitors the completion of the de-reset of the computing core; otherwise, it waits for the de-reset to complete. Once the upper-layer software detects that the computing core has completed its reset, it configures the target register (Retry register), which sends a high-level signal to both the first and second task queues.

[0106] When the first task queue receives a high-level signal, it will not send out any more tasks, even if the processing core does not apply back pressure at this time.

[0107] When the second task queue receives a high-level signal, it begins sending its valid tasks to the processing core. While sending tasks, the second task queue does not pull down the front-end indicator signal (maintaining a valid flag); it waits for the processing core to return task execution information before pulling down the corresponding indicator signal (changing it to an invalid flag). Once all valid tasks in the second task queue have been sent to the processing core, the second task queue actively configures the status value of the target register (Retry register), allowing the first task queue to resend tasks.

[0108] The apparatus provided in the embodiments of this application is described below. The apparatus described below can be referred to in correspondence with the method described above.

[0109] Figure 4 This is a schematic diagram of the structure of the computing core hang-up recovery device provided in this application, as shown below. Figure 4 As shown, the device includes: The task sending module 410 is used to obtain and send tasks from the first task queue to the computing core, while backing up the tasks to the second task queue and marking the tasks as valid in the second task queue; The task marking module 420 is used to mark the task corresponding to the task execution information in the second task queue as invalid in response to the task execution information sent by the computing core. The task resending module 430 is used to respond to the hang-up recovery information, pause the acquisition of tasks from the first task queue, and send the tasks marked as valid in the second task queue to the computing core. The hang-up recovery module 440 is used to cancel the hang-up recovery information and resume the acquisition and sending of tasks from the first task queue to the computing core when there are no tasks marked as valid in the second task queue.

[0110] The computing core hang recovery device provided in this application embodiment, by setting a second task queue, obtains and sends tasks to the computing core from the first task queue when the computing chip is not hanged, and at the same time backs up the tasks to the second task queue; when the computing chip hangs and is being recovered, the tasks marked as valid in the second task queue are sent to the computing core; through automatic backup and automatic resending of tasks, fast and efficient task distribution recovery is achieved in the scenario where the computing core hangs, without having to re-execute the entire instruction processing pipeline, reducing the involvement of hardware modules, and eliminating the need for complex software intervention, greatly shortening the delay from hang to recovery, reducing the idle time of the computing core, and thus significantly improving the operating efficiency and performance of the entire graphics processor.

[0111] Figure 5 This is a schematic diagram of the structure of the electronic device provided in this application, such as... Figure 5 As shown, the electronic device may include: a processor 510, a communications interface 520, a memory 530, and a communications bus 540, wherein the processor, communications interface, and memory communicate with each other via the communications bus. The processor can invoke logical commands stored in the memory to execute the methods described in the above embodiments, for example: The system retrieves and sends tasks from the first task queue to the computing core, while simultaneously backing up the tasks to the second task queue and marking them as valid in the second task queue. In response to task execution information sent by the computing core, the system marks the corresponding task in the second task queue as invalid. In response to hang-up recovery information, the system suspends the retrieval of tasks from the first task queue and sends the valid tasks from the second task queue to the computing core. If there are no valid tasks in the second task queue, the system cancels the hang-up recovery information and resumes retrieving and sending tasks from the first task queue to the computing core.

[0112] Furthermore, the logical commands in the aforementioned memory can be implemented as software functional units and sold or used as independent products, and can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several commands to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0113] The processor in the electronic device provided in this application embodiment can call logical instructions in the memory to implement the above method. Its specific implementation method is the same as the aforementioned method implementation method and can achieve the same beneficial effects, which will not be repeated here.

[0114] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the methods provided in the above embodiments.

[0115] The specific implementation method is the same as the aforementioned method implementation method and can achieve the same beneficial effects, so it will not be repeated here.

[0116] This application provides a computer program product, including a computer program that, when executed by a processor, implements the method described above.

[0117] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0118] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0119] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for recovering from a suspended computing core, characterized in that, include: The task is retrieved from the first task queue and sent to the computing core. At the same time, the task is backed up to the second task queue and marked as valid in the second task queue. In response to the task execution information sent by the computing core, the task corresponding to the task execution information in the second task queue is marked as invalid; In response to the hang-up recovery information, the process of acquiring tasks from the first task queue is paused, and tasks marked as valid in the second task queue are sent to the computing core. If there are no valid tasks in the second task queue, the hang-up recovery information is revoked, and tasks are retrieved from the first task queue and sent to the computing core.

2. The method for recovering from a suspended computing core according to claim 1, characterized in that, The hang-up recovery information is represented by the status value of the target register; In response to the hang-up recovery information, the target register is set to a first preset value; The cancellation of the hang recovery information corresponds to the target register being set to the second preset value.

3. The method for recovering from a suspended computing core according to claim 2, characterized in that, The step of retrieving and sending tasks from the first task queue to the computing core includes: Obtain the status value of the target register and the current number of tasks in the second task queue; When the state value of the target register is the second preset value and the current number of tasks is less than the queue depth of the second task queue, a task is retrieved from the first task queue and sent to the computing core.

4. The method for recovering from a suspended computing core according to claim 1, characterized in that, The step of retrieving and sending tasks from the first task queue to the computing core, and simultaneously backing up the tasks to the second task queue, includes: Determine the backup location of the task in the second task queue; The index information of the backup location is associated with the task and sent to the computing core, while the task is stored in the backup location in the second task queue.

5. The method for recovering from a suspended computing core according to claim 4, characterized in that, The step of responding to the task execution information sent by the computing core by marking the task corresponding to the task execution information in the second task queue as invalid includes: The task execution information is parsed to obtain index information; Based on the index information, determine the corresponding task in the second task queue; Mark the corresponding task as invalid.

6. The method for recovering from a suspended computing core according to claim 1, characterized in that, The method of sending the tasks marked as valid in the second task queue to the computing core further includes: Keep the task's label in the second task queue unchanged; Upon receiving the task execution information of the task sent by the computing core, the task is marked as invalid in the second task queue.

7. The method for recovering from a suspended computing core according to any one of claims 1 to 6, characterized in that, The queue depth of the second task queue is determined based on the maximum number of tasks that the computing core can process concurrently.

8. A device for recovering from a suspended computing core, characterized in that, include: The task sending module is used to obtain and send tasks from the first task queue to the computing core, and at the same time back up the tasks to the second task queue and mark the tasks as valid in the second task queue; The task marking module is used to mark the task corresponding to the task execution information in the second task queue as invalid in response to the task execution information sent by the computing core; The task resend module is used to respond to the hang-up recovery information, pause the acquisition of tasks from the first task queue, and send the tasks marked as valid in the second task queue to the computing core. The hang-up recovery module is used to cancel the hang-up recovery information and resume the acquisition and sending of tasks from the first task queue to the computing core when there are no tasks marked as valid in the second task queue.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the operation core hang-up recovery method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the operation core hang-up recovery method according to any one of claims 1 to 7.

Images