Processor exception recovery method and apparatus, computing device, storage medium, and product

Abstract

The present disclosure relates to a processor exception recovery method and device, a computing device, a storage medium and a product. The processor exception recovery method comprises: in response to an exception event of a processor, determining at least one exception object related to the exception event and obtaining state data of the at least one exception object; determining an association relationship between the at least one exception object based on the state data to construct an object relationship view of the at least one exception object; obtaining forward advancing state data of the processor; determining a fault domain of the processor based on the object relationship view and the forward advancing state data; determining a recovery granularity corresponding to the fault domain; and performing at least one exception recovery operation of the processor based on the recovery granularity, thereby identifying a real impact range of the exception and determining the exception recovery operation by using the association relationship view constructed around the exception object, improving the accuracy of the fault domain and the exception recovery operation, and further improving the effect of the exception recovery.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology. More specifically, this disclosure relates to a processor exception recovery method and apparatus, a computing device, a storage medium, and a computer program product. Background Technology

[0002] With the rapid development of large-scale online model services, retrieval-enhanced generation, streaming decoding inference, multi-tenant model hosting, and edge inference scenarios, a single processor (e.g., but not limited to, a graphics processing unit (GPU)) often simultaneously handles multiple inference requests, multiple queues, multiple contexts, and multiple model instances. Such artificial intelligence (AI) inference workloads have high requirements for latency, throughput, continuity, and resource utilization. If a processor (e.g., a GPU) malfunctions, it can easily cause other normal inference tasks to be interrupted, thereby affecting the overall stability of the service. Summary of the Invention

[0003] One or more exemplary embodiments of this disclosure provide a processor anomaly recovery method and apparatus, which can identify the true scope of the anomaly's impact and determine the anomaly recovery operation by using an association view built around the anomaly object, thereby improving the accuracy of the fault domain and the anomaly recovery operation, and thus improving the processor anomaly recovery effect.

[0004] According to one or more example embodiments of this disclosure, a processor exception recovery method is provided, comprising: in response to the occurrence of an exception event of the processor, determining at least one exception object related to the exception event and obtaining state data of the at least one exception object, wherein the state data represents the state of the at least one exception object in the task processing chain of the processor; determining the association relationship between the at least one exception object based on the state data to construct an object relationship view of the at least one exception object; obtaining forward progress state data of the processor, wherein the forward progress state data represents the state of the task processing chain of the processor continuously progressing in the completion direction within an expected time; determining a fault domain of the processor based on the object relationship view and the forward progress state data; determining a recovery granularity corresponding to the fault domain; and performing at least one exception recovery operation of the processor based on the recovery granularity.

[0005] In an example embodiment, determining at least one abnormal object related to the abnormal event may include: determining an initial abnormal object based on the type of the abnormal event; determining associated abnormal objects associated with the initial abnormal object by performing an association traversal on the initial abnormal object; and determining the initial abnormal object and the associated abnormal objects as the at least one abnormal object.

[0006] In an example embodiment, the associated exception object may include at least one of a synchronization object, a task object, a queue object, a context object, a video memory object, and an execution engine object associated with the initial exception object, wherein the initial exception object is directly associated with the synchronization object or the task object, the synchronization object or the task object belongs to the queue object, the queue object belongs to the context object, the queue object is bound to the video memory object, and the context object maps to the execution engine object.

[0007] In an example embodiment, the association may include at least one of the following: the dependency relationship between the synchronization object and the task object; the affiliation relationship between the task object and the queue object; the mapping relationship between the queue object and the context object; the scheduling relationship between the context object and the execution engine object; and the binding relationship between the queue object and the video memory object. The forward progress status data may include at least one of the following: whether the synchronization object completes within a preset time; whether the completion sequence number of the queue object advances; whether the completion pointer or heartbeat of the execution engine object is updated; and whether the newly submitted test task is completed within a preset time.

[0008] In the example embodiment, the fault domain can be divided into queue-level faults, context-level faults, execution engine-level faults, and device-level faults according to the fault level from low to high.

[0009] In an example embodiment, determining the processor's fault domain based on the object relationship view and the forward propagation state data may include: when the number of queue objects meets the queue-level fault condition, the fault domain can be determined as the queue-level fault; when the number of queue objects does not meet the queue-level fault condition, and there is at least one context object in the context objects whose queue object anomaly rate meets the context-level fault condition, the fault domain can be determined as the context-level fault; when there is no context object in the context objects whose queue object anomaly rate meets the context-level fault condition, and there is at least one execution engine object in the execution engine objects whose forward propagation loss rate meets the execution engine-level fault condition, the fault domain can be determined as the execution engine-level fault; when there is no forward propagation in the execution engine objects... A fault domain can be identified as a device-level fault when the loss rate of at least one execution engine object satisfies the execution engine-level fault condition, and the number of execution engine objects included in the at least one abnormal object satisfies the device-level fault condition. The queue-level fault condition may include the number of queue objects being less than or equal to a first predetermined number and other queue objects on the same execution engine object as the queue objects maintaining forward propagation. The context-level fault condition may include the exception rate of the queue objects being greater than or equal to an exception rate threshold. The execution engine-level fault condition may include a forward propagation loss rate greater than or equal to a forward propagation loss rate threshold. The device-level fault condition may include the number of execution engine objects included in the at least one abnormal object being greater than or equal to a second predetermined number, wherein the second predetermined number is based on the number of execution engine objects associated with the processor.

[0010] In an example embodiment, the step of performing at least one exception recovery operation of the processor based on the recovery granularity may include: upgrading the recovery granularity in response to the recovery granularity upgrade condition being met; determining at least one exception recovery operation corresponding to the upgraded recovery granularity according to a pre-stored recovery operation mapping table, and performing the at least one exception recovery operation, wherein the recovery granularity upgrade condition includes at least one of the same queue object, the same context object, or the same execution engine object repeatedly triggering exceptions within a preset time window.

[0011] In an example embodiment, the method further includes: acquiring an object relationship snapshot of the processor at the time of the abnormal event, wherein the object relationship snapshot stores object relationships in the processor's task processing chain, and executing at least one abnormal recovery operation of the processor may include: executing the at least one abnormal recovery operation based on the object relationships in the object relationship snapshot.

[0012] In an example embodiment, determining the recovery granularity corresponding to the fault domain may include: obtaining the reference source of the queue object from the object relationship snapshot based on the fact that the fault domain is a queue-level fault and the lifecycle of the queue object is abnormal; reconstructing the logical reference relationship of the queue object based on the reference source; and performing a queue object lifecycle recovery operation based on the logical reference relationship to restore the lifecycle of the queue object.

[0013] In an example embodiment, determining the recovery granularity corresponding to the fault domain may further include: when the lifecycle of the queue object is not successfully recovered, determining whether abnormal isolation is required; when it is determined that abnormal isolation is required, determining the recovery granularity corresponding to the queue-level fault in the pre-stored fault recovery mapping table as the recovery granularity; when it is determined that abnormal isolation is not required, upgrading the recovery granularity corresponding to the queue-level fault in the pre-stored fault recovery mapping table by a predetermined number of levels to obtain the upgraded recovery granularity, and determining the upgraded recovery granularity as the recovery granularity.

[0014] In an example embodiment, the method may further include: after performing the exception recovery operation of the processor, verifying whether the exception recovery was successful by using the object relationship snapshot; in response to the failure of the exception recovery, upgrading the recovery granularity; and performing the exception recovery operation of the processor based on the upgraded recovery granularity.

[0015] In an example embodiment, verifying whether the anomaly recovery was successful may include: performing at least one of an object consistency verification operation and a forward push verification operation, wherein the object consistency verification operation is used to verify whether the object relationships associated with the at least one abnormal object are consistent before and after the anomaly recovery using the object relationship snapshot, and the forward push verification operation is used to verify whether the processor's task processing chain is progressing in the completion direction; based on at least one of the results of the object consistency verification operation and the forward push verification operation, it is determined whether the anomaly recovery was successful.

[0016] According to one or more example embodiments of this disclosure, a processor exception recovery apparatus is provided, comprising: a state acquisition unit configured to, in response to an exception event of the processor, determine at least one exception object related to the exception event and acquire state data of the at least one exception object, wherein the state data represents the state of the at least one exception object in the task processing chain of the processor; an association relationship determination unit configured to, based on the state data, determine the association relationship between the at least one exception object to construct an object relationship view of the at least one exception object; a forward progress state acquisition unit configured to acquire forward progress state data of the processor, wherein the forward progress state data represents the state of the task processing chain of the processor continuously progressing in the completion direction within an expected time; a fault domain determination unit configured to, based on the object relationship view association relationship and the forward progress state data of the processor, determine the fault domain of the processor; a recovery granularity determination unit configured to, determine the recovery granularity corresponding to the fault domain; and a recovery operation execution unit configured to, based on the recovery granularity, execute at least one exception recovery operation of the processor.

[0017] In an example embodiment, the state acquisition unit may be configured to: determine an initial exception object based on the type of the exception event; determine associated exception objects associated with the initial exception object by performing an association traversal on the initial exception object; and determine the initial exception object and the associated exception objects as the at least one exception object.

[0018] In an example embodiment, the associated exception object may include at least one of a synchronization object, a task object, a queue object, a context object, a video memory object, and an execution engine object associated with the initial exception object, wherein the initial exception object is directly associated with the synchronization object or the task object, the synchronization object or the task object belongs to the queue object, the queue object belongs to the context object, the queue object is bound to the video memory object, and the context object maps to the execution engine object.

[0019] In an example embodiment, the association may include at least one of the following: the dependency relationship between the synchronization object and the task object; the affiliation relationship between the task object and the queue object; the mapping relationship between the queue object and the context object; the scheduling relationship between the context object and the execution engine object; and the binding relationship between the queue object and the video memory object. The forward progress status data may include at least one of the following: whether the synchronization object completes within a preset time; whether the completion sequence number of the queue object advances; whether the completion pointer or heartbeat of the execution engine object is updated; and whether the newly submitted test task is completed within a preset time.

[0020] In the example embodiment, the fault domain can be divided into queue-level faults, context-level faults, execution engine-level faults, and device-level faults according to the fault level from low to high.

[0021] In an example embodiment, the fault domain determination unit can be configured to: determine the fault domain as a queue-level fault when the number of queue objects meets the queue-level fault condition; determine the fault domain as a context-level fault when the number of queue objects does not meet the queue-level fault condition, and there is at least one context object in the context objects whose exception rate meets the context-level fault condition; determine the fault domain as an execution engine-level fault when there is no context object in the context objects whose exception rate meets the context-level fault condition, and there is at least one execution engine object in the execution engine objects whose forward propagation loss rate meets the execution engine-level fault condition; and determine the fault domain as an execution engine-level fault when there is no execution engine object whose forward propagation loss rate meets the execution engine-level fault condition. A fault domain can be identified as a device-level fault when at least one execution engine object meets the conditions, and the number of execution engine objects included in the at least one abnormal object satisfies the device-level fault conditions. The queue-level fault conditions may include the number of queue objects being less than or equal to a first predetermined number and other queue objects on the same execution engine object as the queue objects maintaining forward propagation. The context-level fault conditions may include the exception rate of the queue objects being greater than or equal to an exception rate threshold. The execution engine-level fault conditions may include a forward propagation loss rate greater than or equal to a forward propagation loss rate threshold. The device-level fault conditions may include the number of execution engine objects included in the at least one abnormal object being greater than or equal to a second predetermined number, where the second predetermined number is based on the number of execution engine objects associated with the processor.

[0022] In an example embodiment, the recovery operation execution unit may be configured to: upgrade the recovery granularity in response to the recovery granularity upgrade condition being met; determine at least one abnormal recovery operation corresponding to the upgraded recovery granularity according to a pre-stored recovery operation mapping table, and execute the at least one abnormal recovery operation, wherein the recovery granularity upgrade condition includes at least one of the same queue object, the same context object, or the same execution engine object repeatedly triggering an exception within a preset time window.

[0023] In an example embodiment, the apparatus may further include: a snapshot acquisition unit configured to: acquire an object relationship snapshot of the processor at the time of the abnormal event, wherein the object relationship snapshot stores object relationships in the task processing chain of the processor, and wherein the recovery operation execution unit may be configured to: execute the at least one abnormal recovery operation based on the object relationships in the object relationship snapshot.

[0024] In an example embodiment, the recovery granularity determination unit may be configured to: obtain the reference source of the queue object from the object relationship snapshot based on the fact that the fault domain is a queue-level fault and the lifecycle of the queue object is abnormal; reconstruct the logical reference relationship of the queue object based on the reference source; and perform a queue object lifecycle recovery operation based on the logical reference relationship to restore the lifecycle of the queue object.

[0025] In an example embodiment, the recovery granularity determination unit may further be configured to: determine whether abnormal isolation is required when the lifecycle of the queue object is not successfully recovered; when it is determined that abnormal isolation is required, determine the recovery granularity corresponding to the queue-level fault in the pre-stored fault recovery mapping table as the recovery granularity; when it is determined that abnormal isolation is not required, upgrade the recovery granularity corresponding to the queue-level fault in the pre-stored fault recovery mapping table by a predetermined number of levels to obtain the upgraded recovery granularity, and determine the upgraded recovery granularity as the recovery granularity.

[0026] In an example embodiment, the apparatus may further include: a verification unit configured to verify whether the abnormal recovery was successful by using the object relationship snapshot after performing the abnormal recovery operation of the processor; a granularity upgrade unit configured to upgrade the recovery granularity in response to the failure of the abnormal recovery; and a re-recovery unit configured to perform the abnormal recovery operation of the processor based on the upgraded recovery granularity.

[0027] In an example embodiment, the verification unit may be configured to: perform at least one of an object consistency verification operation and a forward push verification operation, wherein the object consistency verification operation is used to verify whether the object relationships associated with the at least one abnormal object are consistent before and after the abnormality recovery using the object relationship snapshot, and the forward push verification operation is used to verify whether the processor's task processing chain is progressing in the completion direction; and based on at least one of the results of the object consistency verification operation and the forward push verification operation, determine whether the abnormality recovery is successful.

[0028] According to one or more example embodiments, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements a processor exception recovery method according to one or more example embodiments.

[0029] According to one or more example embodiments, a computing device is provided, comprising: at least one processor; at least one memory storing a computer program that, when executed by the at least one processor, implements a processor exception recovery method according to one or more example embodiments.

[0030] According to one or more example embodiments, a computer program product is provided, wherein instructions in the computer program product are executable by a processor of a computer device to perform a processor exception recovery method according to one or more example embodiments.

[0031] According to embodiments of the present disclosure, the processor anomaly recovery method and apparatus, computing device, storage medium, and computer program product construct an object relationship view of at least one anomaly object by determining the association between at least one anomaly object based on the state data of at least one anomaly object, thereby identifying the true impact range of the anomaly; determine the fault domain of the processor based on the object relationship view and forward state data, thereby improving the accuracy of the fault domain; and improve the accuracy of the anomaly recovery operation by determining the recovery granularity corresponding to the fault domain and determining at least one anomaly recovery operation corresponding to the recovery granularity, thereby improving the effectiveness of the processor anomaly recovery.

[0032] Further aspects and / or advantages of the general concept of this disclosure will be set forth in part in the description which follows, and in part will be clear from the description or may be learned by practice of the general concept of this disclosure. Attached Figure Description

[0033] The above and other objects and features of one or more exemplary embodiments will become clearer from the following description taken in conjunction with the accompanying drawings, which exemplarily illustrate the embodiments.

[0034] Figure 1 A flowchart of a processor exception recovery method is shown.

[0035] Figure 2 A block diagram of a processor exception recovery device is shown.

[0036] Figure 3 A flowchart illustrating a processor exception recovery method according to one or more example embodiments is shown.

[0037] Figure 4 The flowchart illustrates a process for constructing an object relationship view by identifying abnormal objects and state data, according to one or more example embodiments.

[0038] Figure 5 A flowchart illustrating the determination of a processor's fault domain according to one or more example embodiments is shown.

[0039] Figure 6 This diagram illustrates the determination of recovery granularity in a queue lifecycle anomaly scenario according to one or more example embodiments.

[0040] Figure 7 A flowchart illustrating verification exception recovery according to one or more example embodiments is shown.

[0041] Figure 8 A schematic diagram of a processor exception recovery system according to one or more example embodiments is shown.

[0042] Figure 9 A flowchart illustrating a processor exception recovery method according to one or more example embodiments is shown.

[0043] Figure 10 A block diagram of a processor exception recovery apparatus according to one or more example embodiments is shown.

[0044] Figure 11 shows a schematic diagram of a computing device according to one or more example embodiments. Detailed Implementation

[0045] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0046] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following examples do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0047] It should be noted that the phrase "at least one of several items" in this disclosure refers to three parallel cases: "any one of the several items", "a combination of any number of the several items", and "all of the several items". For example, "including at least one of A and B" includes the following three parallel cases: (1) including A; (2) including B; (3) including A and B. Another example is "performing at least one of step one and step two", which means the following three parallel cases: (1) performing step one; (2) performing step two; (3) performing both step one and step two.

[0048] Existing processor (e.g., GPU) anomaly recovery methods typically revolve around timeout detection, engine reset, or device reset. For example, upon detecting fence timeouts, heartbeat pauses, or Timeout Detection and Recovery (TDR) timeouts, the driver framework often directly triggers queue resets, engine resets, or even device resets. While such solutions can restore device availability to some extent, their core logic is usually based on "detecting timeouts—performing a reset," making it difficult to accurately characterize the scope of anomaly propagation.

[0049] For AI inference workloads, processors (e.g., GPUs) often involve complex relationships such as multiple concurrent requests, multiple queue objects sharing an execution engine object, a single context object holding multiple queue objects, and multiple task objects sharing video memory and synchronization objects. Resetting directly based on a single timeout event can easily lead to the following problems: First, the recovery granularity is too coarse, amplifying what could have been a local recovery into a full-card interruption; second, object relationships are not preserved before recovery, resulting in dangling references, incorrect attachments, or inconsistent context records after recovery; third, only verifying whether the reset is complete, without verifying whether forward propagation has truly resumed after recovery.

[0050] In one existing technical solution, starting with a timeout or watchdog event, when a synchronization object times out, a queue is not completed for a long time, the execution engine heartbeat stops, or a system time domain reflection (TDR) event is detected, a preset recovery action is directly executed, such as queue reset, execution engine reset, or processor (e.g., GPU) device reset.

[0051] Figure 1 A flowchart illustrating a processor exception recovery method is shown. Figure 1 As shown, in step S101, a fence timeout or heartbeat stagnation is detected. In step S102, the queue or engine that triggered the timeout is located. In step S103, the queue object is reset or the engine object is reset according to a preset strategy. In step S104, if necessary, a device reset is performed. In step S105, subsequent tasks are received again.

[0052] Figure 1 This existing technical solution is suitable for handling general timeout failures and exception recovery in general graphics stacks. Figure 1 The existing technical solution in this paper has the following main drawbacks: 1. Insufficient identification of the scope of anomaly propagation: It often directly maps a single timeout event to a reset action, lacking a comprehensive judgment on the relationship between synchronization objects, task objects, queue objects, context objects, and memory objects; 2. Coarse-grained recovery: When the problem is essentially limited to a single queue or a single context, it may directly escalate to engine-level recovery or device-level recovery, resulting in an excessively large impact. 3. Lack of relationship protection before restoration: The object relationship snapshot was not saved before the reset, making it difficult to verify whether the logical references, attachment states and mapping states are consistent after restoration; 4. Insufficient post-recovery verification: Usually, it only checks whether the reset action has been completed, but it is difficult to confirm whether the object relationship has been restored to consistency and whether the task chain has truly resumed forward progress.

[0053] In another existing technical solution, recovery actions are determined through an exception classification framework. Specifically, processor exceptions are first categorized into several types based on error codes, timeout types, page fault types, engine exception types, etc., and then corresponding handling actions are configured for each type of exception. The processing framework then selects the recovery action or operation according to the category. This existing technical solution facilitates the formation of a unified recovery framework and also makes it easy to expand to new exception categories.

[0054] Figure 2 A block diagram of a processor exception recovery device is shown. Figure 2 As shown, the processor exception recovery device in the prior art includes an exception detection module 210, an exception classification module 220, an action mapping module 230, a queue reset module 240, an engine reset module 250, and a device reset module 260. The exception detection module 210 is used to detect processor exceptions. The exception classification module 220 is used to classify processor exceptions into several categories based on error codes, timeout types, page fault types, engine exception types, etc. The action mapping module 230 is used to configure corresponding processing actions for each type of exception. The queue reset module 240 is used to perform queue reset operations. The engine reset module 250 is used to perform engine reset operations. The device reset module 260 is used to perform device reset operations.

[0055] Figure 2 This existing technical solution facilitates the formation of a unified recovery framework and also makes it easier to expand new anomaly categories. Figure 2 The existing technical solution in this paper has the following main drawbacks: 1. Classification results are not completely equivalent to fault domains: The actual impact range of the same type of anomaly may be different under different object relationship topologies, and it is difficult to accurately determine the recovery granularity based solely on the anomaly category; 2. Ignore object lifecycle exceptions: When problems occur in lifecycle paths such as queue destruction, reference leaks, or errors in the delayed garbage collection chain, simple classification is insufficient to support recovery; 3. Insufficient adaptation to AI inference scenarios: AI inference workloads emphasize minimal business interruption and continuous progress after recovery. Relying solely on classification and action mapping is insufficient to meet the dual requirements of object consistency verification and forward-progress verification.

[0056] In view of the above-mentioned shortcomings of the prior art, this disclosure mainly addresses the following technical problems: 1. How to identify the true scope of the exception's impact by building an object relationship view around the exception object, rather than relying solely on timeouts or error codes, after a processor (e.g., GPU) exception occurs; 2. How to jointly determine the fault domain based on the object relationship view and the forward progress status, and select the queue-level, context-level, execution engine-level, or device-level recovery granularity accordingly; 3. How to save a snapshot of the relationships related to the abnormal object before the recovery action is executed, so as to avoid the loss of logical references, disordered attachment relationships or inconsistent context records after recovery; 4. How to reconstruct the actual reference source and logical reference relationship of the target queue object in response to abnormal queue lifecycle scenarios, so as to support accurate recovery or safe isolation; 5. How to simultaneously verify object relationship consistency and processor (e.g., GPU) forward state after the recovery action is executed, and escalate the recovery action hierarchically in case of failure; 6. How to maintain the execution of unrelated tasks as much as possible in AI inference workload scenarios, and reduce the impact of single point of failure on other inference tasks on the same card.

[0057] This disclosure proposes a processor anomaly recovery method that, instead of simply resetting after a timeout, constructs a relational view around the anomaly object, identifies fault domains and recovery granularity by combining forward state analysis, saves a snapshot of object relationships before recovery, and simultaneously performs object consistency verification and forward state analysis after recovery, progressively escalating the recovery granularity as necessary. See below for further details. Figures 3 to 11 A processor exception recovery method and apparatus according to embodiments of the present disclosure are described in detail.

[0058] Figure 3 A flowchart illustrating a processor exception recovery method according to one or more example embodiments is shown. Figure 3 The processor exception recovery method in the document can be used to recover from exceptions for various processors (e.g., but not limited to, graphics processing units (GPUs)). Figure 3The processor exception recovery method described herein can be applied to online or offline inference tasks, such as, but not limited to, AI inference workload scenarios. AI inference workload scenarios include online or offline inference tasks such as model forward execution, streaming decoding, continuous batch processing, dynamic batch processing, and multi-request shared processor (e.g., GPU).

[0059] Reference Figure 3 In step S301, in response to an abnormal event occurring in the processor, at least one abnormal object associated with the abnormal event is identified and status data of the at least one abnormal object is obtained. Here, the status data represents the status of the at least one abnormal object in the processor's task processing chain. In one or more example embodiments, the status data may include at least one of completion sequence number, reference count, scheduling attachment, and history recovery record.

[0060] In one or more example embodiments, the exception triggering event may include at least one of the following: processor synchronization object timeout, watchdog notification, user-mode recovery request, kernel exception notification, and execution engine state exception. In one or more example embodiments, the exception object may include at least one of the following: synchronization object, task object, queue object, context object, execution engine object, and video memory object. Synchronization objects can be used to represent task completion, waiting, dependency fences, semaphores, events, or other synchronization primitives. Task objects can be used to represent batches of commands, inference requests, execution descriptors, or scheduling units submitted to the processor (e.g., GPU) for execution. Queue objects can be used to represent execution queues that sequentially submit and complete tasks in the processor's task processing chain. Context objects can be used to represent the processor (e.g., GPU) execution context associated with a process, tenant, model instance, or session of a task in the processor's task processing chain. Execution engine objects can be used to represent the computation, copying, media, or other command execution engines within the processor (e.g., GPU). Graphical memory objects (GMMOs) are dedicated memory in processors (e.g., GPUs) used to store graphics rendering and computation data, and their performance directly affects the training speed and inference efficiency of the model.

[0061] In one or more example embodiments, determining at least one exception object associated with an exception event may include: determining an initial exception object based on the type of the exception event; determining associated exception objects associated with the initial exception object by performing an association traversal on the initial exception object; and determining the initial exception object and the associated exception objects as at least one exception object.

[0062] As an example, when a fence timeout occurs, the initial exception object can be the timed-out synchronization object; when a queue forward progress stalls, the initial exception object can be the stalled queue object; when a lifecycle exception occurs, the initial exception object can be the queue object to be destroyed or the queue object to which the exception is attached.

[0063] In one or more example embodiments, the associated exception object may include at least one of a synchronization object, task object, queue object, context object, video memory object, and execution engine object associated with the initial exception object, wherein the initial exception object is directly associated with a synchronization object or task object, the synchronization object or task object belongs to a queue object, the queue object belongs to a context object, the queue object is bound to a video memory object, and the context object maps to an execution engine object. As an example, determining the associated exception object by performing an association traversal on the initial exception object may include: determining the synchronization object and / or task object associated with the initial exception object, determining the queue object of the synchronization object and / or task object, determining the context object of the queue object, determining the video memory object bound to the queue object, determining the execution engine mapped to the context object, and determining at least one of the synchronization object, task object, queue object, context object, video memory object, and execution engine as the associated exception object.

[0064] In step S302, based on the state data, the association relationships between at least one abnormal object are determined to construct an object relationship view of at least one abnormal object. Here, the object relationship view can be used to characterize the association relationship view built around the abnormal object, and is used to characterize the association relationships between at least one abnormal object, such as, but not limited to, dependency, attribution, mapping, scheduling, binding, etc.

[0065] In one or more example embodiments, the association may include at least one of the following: a dependency relationship between a synchronization object and a task object, an ownership relationship between a task object and a queue object, a mapping relationship between a queue object and a context object, a scheduling relationship between a context object and an execution engine object, and a binding relationship between a queue object and a video memory object.

[0066] Object relationship views can be implemented using various structures that can express relationships such as dependencies, affiliations, and mappings between objects (e.g., but not limited to, graph structures, relational table structures, or multi-index mapping structures).

[0067] As an example, an object relationship view can be a graph structure including edges and nodes, with each abnormal object as a node and different types of associations as edges. Furthermore, attributes such as current status, completion sequence number, reference count, attachment status, and recovery count can be attached to nodes as status data, facilitating subsequent precise determination of the fault domain and recovery granularity.

[0068] Figure 4 The flowchart illustrates a process for constructing an object relationship view by identifying abnormal objects and state data, according to one or more example embodiments.

[0069] As an example, such as Figure 4 As shown, in step S401, the initial exception object is determined. In step S402, the synchronization object and / or task object associated with the initial exception object are located. In step S403, the queue object to which the synchronization object and / or task object belongs is located. In step S404, the context object of the queue object is located. In step S405, the video memory object bound to the queue object is located. In step S406, the execution engine mapped to the context object is located. In step S407, the dependency wait chain of the synchronization object and / or task object is counted. In step S408, based on at least one of the synchronization object, task object, queue object, context object, video memory object, and execution engine, and the dependency wait chain of the synchronization object and / or task object, an object relationship view is constructed.

[0070] Return to reference Figure 3 In step S303, the processor's forward propagation state data is acquired. Here, the forward propagation state data can represent the state of the processor's task processing chain continuously progressing towards completion within the expected time. Forward propagation can be used to characterize indicators that the processor's (e.g., GPU) task processing chain continuously progresses towards completion within the expected time, such as completion sequence number updates, heartbeat advances, or successful completion of test tasks. Furthermore, at least one of the following can be used as an indicator of forward propagation: hardware counter changes, completion pointer advances, scheduling queue consumption rate, etc.

[0071] In one or more example embodiments, the forward progress status data may include at least one of the following: whether the synchronization object completes within a preset time, whether the completion sequence number of the queue object advances, whether the completion pointer or heartbeat of the execution engine object is updated, and whether the newly submitted test task completes within a preset time.

[0072] In step S304, the fault domain of the processor is determined based on the object relationship view and forward state data, thereby improving the accuracy of the fault domain. Here, the fault domain can be used to characterize the recovery object level corresponding to the actual impact range of the processor's anomaly.

[0073] In one or more example embodiments, the fault domain can be divided into queue-level faults, context-level faults, execution engine-level faults, and device-level faults according to the fault level from low to high.

[0074] In one or more example embodiments, determining the processor's fault domain based on the object relationship view and forward propagation state data may include: determining the fault domain as a queue-level fault when the number of queue objects meets the queue-level fault condition; determining the fault domain as a context-level fault when the number of queue objects does not meet the queue-level fault condition, and there is at least one context object in the context objects whose queue object exception rate meets the context-level fault condition; determining the fault domain as an execution engine-level fault when there is no context object in the context objects whose queue object exception rate meets the context-level fault condition, and there is at least one execution engine object in the execution engine objects whose forward propagation loss rate meets the execution engine-level fault condition; and determining the fault domain as a device-level fault when there is no execution engine object in the execution engine objects whose forward propagation loss rate meets the execution engine-level fault condition, and the number of execution engine objects included in at least one exception object meets the device-level fault condition. Here, queue-level fault conditions may include a queue object having a number less than or equal to a first predetermined number and other queue objects on the same execution engine object keeping forward. Context-level fault conditions may include a queue object's exception rate being greater than or equal to an exception rate threshold. Execution engine-level fault conditions may include a forward propagation loss rate being greater than or equal to a forward propagation loss rate threshold. Device-level fault conditions may include at least one exception object having a number of execution engine objects greater than or equal to a second predetermined number, where the second predetermined number is based on the number of execution engine objects associated with the processor. By using the object relationship view and forward propagation state data to determine the processor's fault domains as described above, the accuracy of the fault domains can be improved, and interference with the processor's tasks other than exception tasks can be reduced.

[0075] For example, when a processor exception is associated with only a single queue object, and other queue objects on the same execution engine object are still advancing, it is determined to be a queue-level failure; when multiple queue objects in the same context object are in an abnormal state, it is determined to be a context-level failure; when multiple context objects on the same execution engine object lose their forward advance, it is determined to be an execution engine-level failure; and when multiple execution engine objects are in an abnormal state, it is determined to be a device-level failure.

[0076] As an example, determining the fault domain of a processor based on an object relationship view and forward state data may include: using the object relationship view and forward state data, determining whether queue-level fault conditions to device-level fault conditions are met in the order of queue-level fault conditions to device-level fault conditions, until the condition that is met first among the queue-level fault conditions to device-level fault conditions is determined, and the fault corresponding to the condition that is met first is determined as the fault domain.

[0077] As an example, determining the processor's fault domain based on the object relationship view and forward propagation state data may include: determining whether the number of queue objects satisfies the queue-level fault condition; when the number of queue objects satisfies the queue-level fault condition, determining the fault domain as a queue-level fault; when the number of queue objects does not satisfy the queue-level fault condition, determining the exception rate of the queue objects in each context object using association relationships; when there is at least one context object in the context object whose queue object exception rate satisfies the context-level fault condition, determining the fault domain as a context-level fault. When there is no context object in the context object whose exception rate satisfies the context-level fault condition, the forward propagation loss rate of the context object on each execution engine object in the execution engine object is determined by using association relationships and forward propagation state data; when there is at least one execution engine object in the execution engine object whose forward propagation loss rate satisfies the execution engine-level fault condition, the fault domain is determined as an execution engine-level fault; when there is no execution engine object in the execution engine object whose forward propagation loss rate satisfies the execution engine-level fault condition, the number of execution engine objects included in the exception object is determined; when the number of execution engine objects included in the exception object satisfies the device-level fault condition, the fault domain is determined as a device-level fault.

[0078] Figure 5 A flowchart illustrating the determination of a processor's fault domain according to one or more example embodiments is shown.

[0079] As an example, such as Figure 5As shown, in step S501, the input object relationship view and forward propagation status data are processed. In step S502, it is determined whether the abnormal object includes only a single queue object. If it includes only a single queue object (determined to be yes in step S502), then in step S503, the fault domain is determined to be a queue-level fault; if it does not include only a single queue object (determined to be no in step S502), then in step S504, it is determined whether there are multiple abnormal queue objects in the same context object by using the association relationship. If there are multiple abnormal queue objects in the same context object (determined to be yes in step S504), then in step S505, the fault domain is determined to be a context-level fault. If there are no multiple abnormal queue objects in the same context object (determined to be no in step S504), then in step S506, it is determined whether there are multiple context objects that have lost forward propagation in the same execution engine object by using the association relationship and forward propagation status data. If multiple context objects that have lost forward propagation in the same execution engine object (determined to be yes in step S506), then in step S507, the fault domain is determined to be an execution engine-level fault. If no multiple context objects in the same execution engine object lose forward propagation (determined in step S506), then in step S508, the fault domain is determined to be a device-level fault.

[0080] Return to reference Figure 3 In step S305, the recovery granularity corresponding to the fault domain is determined. For example, the recovery granularity corresponding to the fault domain can be determined according to a pre-stored fault recovery mapping table. Here, the fault recovery mapping table stores the mapping relationship between each fault domain and its corresponding recovery granularity.

[0081] In cases of abnormal queue lifecycle scenarios, this disclosure further introduces a mechanism for reconstructing the actual source of references.

[0082] In one or more example embodiments, determining the recovery granularity corresponding to the fault domain may include: based on the fault domain being a queue-level fault and the queue object's lifecycle being abnormal, obtaining the reference source of the queue object from an object relationship snapshot; reconstructing the logical reference relationship of the queue object based on the reference source; and performing a queue object lifecycle recovery operation based on the logical reference relationship to restore the queue object's lifecycle. Determining the recovery granularity corresponding to the fault domain may further include: when the queue object's lifecycle is not successfully restored, determining whether abnormal isolation is required; when abnormal isolation is required, determining the recovery granularity corresponding to the queue-level fault in a pre-stored fault recovery mapping table as the recovery granularity; when abnormal isolation is not required, upgrading the recovery granularity corresponding to the queue-level fault in the pre-stored fault recovery mapping table by a predetermined number of levels to obtain an upgraded recovery granularity, and determining the upgraded recovery granularity as the recovery granularity. By obtaining the queue object's reference source from an object relationship snapshot to reconstruct the queue object's logical reference relationship, and then performing a queue object lifecycle recovery operation based on the logical reference relationship to restore the queue object's lifecycle, the problem of erroneous reclamation or retention caused by relying solely on software reference counting for judgment is avoided. This effectively supports scenarios of reference misalignment, delayed reclamation anomalies, and attachment anomalies.

[0083] As an example, a queue object reference may come from at least one of the following: a connection between a scheduling list and a queue object; a holding relationship between an unfinished task object and a queue object; a record relationship between a context object and a queue object; a temporary relationship between a delayed recycling chain and a queue object; a temporary relationship between an error recovery path and a queue object; or an association between a user-space file handle or channel object and a queue object.

[0084] Figure 6 This diagram illustrates the determination of recovery granularity in a queue lifecycle anomaly scenario according to one or more example embodiments. As an example, such as... Figure 6 As shown, in step S611, a queue lifecycle anomaly is detected. In step S612, the actual reference source of the queue object is obtained from the object relationship snapshot. In step S613, the logical reference relationship of the queue object is reconstructed based on the reference source. In step S614, it is determined whether the reference relationship is recoverable. If the reference relationship is recoverable, then in step S615, scheduling is restored and service continues. If the reference relationship is not recoverable, then in step S616, it is determined whether anomaly isolation is required. If anomaly isolation is required, then in step S617, the recovery granularity corresponding to the queue-level failure is determined as the recovery granularity. If anomaly isolation is not required, then in step S618, the recovery granularity corresponding to the queue-level failure is upgraded by a predetermined number of levels as the recovery granularity.

[0085] Return to reference Figure 3In step S306, based on the recovery granularity, at least one exception recovery operation of the processor is executed. For example, at least one exception recovery operation corresponding to the recovery granularity can be determined according to a pre-stored recovery operation mapping table, and at least one exception recovery operation is executed. Here, the recovery operation mapping table stores the mapping relationship between each recovery granularity and the corresponding at least one exception recovery operation. In the recovery operation mapping table, the at least one exception recovery operation corresponding to each recovery granularity is the at least one exception recovery operation with the smallest impact range among all exception recovery operations applicable to each recovery granularity from multiple candidate recovery actions. Through the minimum impact recovery strategy, other inference tasks unrelated to the exception can be kept running as much as possible, improving overall business continuity. The minimum impact recovery strategy is used to keep the impact of at least one exception recovery operation on the processor to a minimum.

[0086] As an example, several candidate recovery actions include: queue object freeze operation, synchronization object isolation and / or marking operation, queue object recycling or reconstruction operation, context object reconstruction operation, execution engine object reset operation, and device reset operation. The impact on the processor increases sequentially from queue object freeze operation, synchronization object isolation operation, queue object recycling or reconstruction operation, context object reconstruction operation, execution engine object reset operation, to device reset operation. For example, a queue object freeze operation may include freezing new task submissions to the queue object. For example, a synchronization object isolation and / or marking operation may include cutting off the blocking propagation of an abnormal synchronization object to subsequent task objects and marking the task object associated with the abnormal synchronization object as failed. For example, a queue object recycling or reconstruction operation may include recycling and reconstructing an abnormal queue object. For example, a context object reconstruction operation may include reconstructing an abnormal context object.

[0087] When determining at least one exception recovery operation corresponding to the recovery granularity, the decision can also be made by combining the historical recovery count and the number of exceptions occurring within a preset time window. If the same queue object, the same context object, or the same execution engine object repeatedly triggers exceptions within the preset time window, the exception recovery operation corresponding to the recovery granularity of the current fault domain in the recovery operation mapping table can be skipped, and the exception recovery operation corresponding to the recovery granularity of the higher-level fault domain in the recovery operation mapping table can be directly selected to reduce the jitter caused by repeated recovery.

[0088] In one or more example embodiments, performing at least one exception recovery operation of the processor based on the recovery granularity may include: upgrading the recovery granularity in response to the fulfillment of the recovery granularity upgrade condition; determining at least one exception recovery operation corresponding to the upgraded recovery granularity according to a pre-stored recovery operation mapping table, and performing at least one exception recovery operation, wherein the recovery granularity upgrade condition includes at least one of the same queue object, the same context object, or the same execution engine object repeatedly triggering exceptions within a preset time window, thereby improving the effectiveness of the exception recovery operation, avoiding the need for repeated recovery due to recovery failure, and reducing the jitter caused by repeated recovery.

[0089] At least one exception recovery operation can be implemented internally by the kernel driver or jointly by the driver and the user-mode runtime. Furthermore, after executing at least one exception recovery operation, the execution of other tasks unrelated to the processor exception continues, thereby reducing the impact of processor exception recovery on other inference tasks on the same processor (e.g., GPU).

[0090] In one or more example embodiments, the processor exception recovery method may further include: acquiring a snapshot of the processor's object relationships at the time of the exception event, wherein the object relationship snapshot stores object relationships in the processor's task processing chain. Performing at least one exception recovery operation on the processor may include: performing at least one exception recovery operation based on the object relationships in the object relationship snapshot, thereby reducing the risk of object relationship corruption and making the recovery process safer. The object relationship snapshot may be stored in memory, a circular buffer, a log area, or a dedicated recovery context, and this disclosure is not limited thereto.

[0091] In one or more example embodiments, the object relationship snapshot may include at least one of the following: association state between processor-related objects, reference source, scheduling hook, and memory binding. Furthermore, the object relationship snapshot may also include unfinished task objects of the processor.

[0092] As an example, an object relationship snapshot may include at least one of the following: a list of timeout synchronization objects for the processor and dependencies between synchronization objects in the list of timeout synchronization objects; a holding relationship between unfinished task objects of the processor and processor-related queue objects; a hook relationship between processor-related queue objects in the processor's scheduling domain; a mapping relationship between processor-related context objects and processor-related queue objects; the identifier and binding status of memory objects related to exception objects; historical recovery paths; and the number of historical recovery attempts.

[0093] In one or more example embodiments, the processor exception recovery method may further include: after performing a processor exception recovery operation, verifying whether the exception recovery was successful by using an object relationship snapshot; in response to a failure of exception recovery, upgrading the recovery granularity; and performing a processor exception recovery operation based on the upgraded recovery granularity. When a low-granularity recovery fails, it can be upgraded to a higher-granularity recovery action to improve the overall system recoverability.

[0094] In one or more example embodiments, verifying whether anomaly recovery was successful may include: performing at least one of an object consistency verification operation and a forward push verification operation, wherein the object consistency verification operation is used to verify whether the object relationships associated with at least one anomalous object are consistent before and after anomaly recovery using an object relationship snapshot, and the forward push verification operation is used to verify whether the processor's task processing chain is progressing in the completion direction; based on at least one of the results of the object consistency verification operation and the forward push verification operation, it is determined whether the anomaly recovery was successful. By simultaneously verifying object consistency and forward push status, the problem of "actions being executed but business not being restored" is avoided.

[0095] As an example, an object consistency verification operation may include at least one of the following: verifying whether the software reference count of a queue object is consistent with the logical reference count of the queue object; verifying whether a queue object is still held by a dangling path other than an exception object; and verifying whether the set of queue objects recorded in the context object is consistent with the actual set of attached queue objects in the processor's scheduling domain.

[0096] As an example, a forward verification operation may include at least one of the following: whether the completion sequence number of the queue object is advanced, whether the completion pointer or heartbeat of the execution engine object is updated, and whether the test task used for verification is completed within a preset time.

[0097] Figure 7 A flowchart illustrating verification exception recovery according to one or more example embodiments is shown.

[0098] As an example, such as Figure 7 As shown, after the exception recovery operation is completed, in step S701, the reference count and the queue object set are verified. In step S702, the test task is submitted. In step S703, the test task completion result is returned. In step S704, object consistency and forward propagation are determined. In step S705, if verification fails, an upgrade of the recovery granularity is requested. In step S706, a higher-granularity recovery action is executed.

[0099] Figure 8 shows a schematic diagram of a processor exception recovery system according to one or more example embodiments. (Refer to...) Figure 8The processor anomaly recovery system includes an anomaly acquisition module 810, a status acquisition module 820, a relationship construction module 830, a fault domain determination module 840, a snapshot protection module 850, a recovery execution module 860, a recovery verification module 870, and an upgrade recovery module 880.

[0100] The exception acquisition module 810 is used to acquire processor (GPU) exception triggering information, such as, but not limited to, at least one of synchronization object timeout, watchdog notification, user-mode recovery request, kernel exception notification, and execution engine state exception.

[0101] The status acquisition module 820 is used to identify at least one exception object related to the exception event and acquire the status data of at least one exception object. The exception object may include at least one of the following: synchronization object, task object, queue object, context object, execution engine object, and video memory object.

[0102] The relationship building module 830 is used to determine the association relationships between at least one abnormal object based on state data, so as to construct an object relationship view of at least one abnormal object. The association relationships may include at least one of the following: dependency relationship between synchronization object and task object, ownership relationship between task object and queue object, mapping relationship between queue object and context object, scheduling relationship between context object and execution engine object, and binding relationship between queue object and video memory object.

[0103] The fault domain determination module 840 is used to determine the processor's fault domains based on the object relationship view and forward propagation state data, thereby improving the accuracy of fault domain determination. Fault domains can be classified into queue-level faults, context-level faults, execution engine-level faults, and device-level faults, according to their fault level from low to high.

[0104] The snapshot protection module 850 is used to save a snapshot of the processor's object relationships at the time of an abnormal event. The object relationship snapshot may include at least one of the following: association state between processor-related objects, reference source, scheduling attachment, and memory binding. In addition, the object relationship snapshot also includes the processor's unfinished task objects.

[0105] The recovery execution module 860 is used to perform minimal impact exception recovery operations, such as, but not limited to, queue object freezing operations, synchronization object isolation and / or marking operations, queue object recycling or reconstruction operations, context object reconstruction operations, execution engine object reset operations, and device reset operations.

[0106] The recovery verification module 870 verifies the success of the exception recovery operation by using an object relationship snapshot after the processor has performed its exception recovery operation. It executes at least one of an object consistency verification operation and a forward push verification operation. The object consistency verification operation verifies the consistency of object relationships associated with at least one exception object before and after the exception recovery using the object relationship snapshot. The forward push verification operation verifies whether the processor's task processing chain is progressing in the completion direction. Based on at least one of the results of the object consistency verification operation and the forward push verification operation, it determines whether the exception recovery was successful. By simultaneously verifying object consistency and forward push status, it avoids the false recovery problem of "actions being executed but business not being restored."

[0107] The upgrade recovery module 880 is used to upgrade the recovery granularity in response to unsuccessful abnormal recovery, so that the abnormal recovery operation of the processor is performed based on the upgraded recovery granularity. When low-granularity recovery fails, it can be upgraded to a higher-granularity recovery action, thereby improving the overall recoverability of the system.

[0108] Figure 9 A flowchart illustrating a processor exception recovery method according to one or more example embodiments is shown.

[0109] Reference Figure 9 In step S901, the abnormal event triggering information of the processor is obtained. In step S902, at least one abnormal object related to the abnormal event is identified, and the state data of at least one abnormal object is obtained. In step S903, based on the state data, the association relationship between at least one abnormal object is determined to construct an object relationship view of at least one abnormal object. In step S904, the forward propagation state data of the processor is obtained, and the fault domain of the processor is determined based on the object relationship view and the forward propagation state data. In step S905, a snapshot of the processor's object relationship at the time the abnormal event occurs is saved. In step S906, a minimum-impact abnormal recovery operation is performed. In step S907, an object consistency verification operation and a forward propagation verification operation are performed. If the verification passes, in step S908, the processor's business processing is resumed. If the verification fails, in step S909, the recovery granularity is upgraded, and the process returns to step S906.

[0110] If the verification is successful, it means that the recovery action has both repaired the object relationships and restored the continuous progress of the execution chain, and the system can resume normal scheduling. If the verification fails, the upgraded recovery module will execute higher-granularity recovery actions according to the scope of impact, from smallest to largest, until the recovery is successful or the processor (GPU) is placed in an isolated state.

[0111] In AI inference workloads, this disclosure can maintain the continued execution of other tasks unrelated to the anomalous object after the recovery action is performed, thereby reducing the impact of anomalous recovery on other inference tasks on the same processor (GPU). Especially in scenarios involving continuous batch processing, dynamic batch processing, streaming decoding, and multiple model instances sharing a processor (GPU), this disclosure can significantly improve business continuity.

[0112] The above has been combined Figures 3 to 9 A processor exception recovery method according to one or more example embodiments has been described. Hereinafter, reference will be made to... Figure 10 A processor exception recovery apparatus and its components according to one or more example embodiments are described.

[0113] Figure 10 A block diagram of a processor exception recovery apparatus according to one or more example embodiments is shown.

[0114] Reference Figure 10 The processor fault recovery device 1000 includes a status acquisition unit 1001, an association relationship determination unit 1002, a progress status acquisition unit 1003, a fault domain determination unit 1004, a recovery granularity determination unit 1005, and a recovery operation execution unit 1006.

[0115] The status acquisition unit 1001 is configured to, in response to an abnormal event occurring in the processor, identify at least one abnormal object associated with the abnormal event and acquire status data of at least one abnormal object, wherein the status data represents the status of at least one abnormal object in the processor's task processing chain.

[0116] In one or more example embodiments, the state acquisition unit 1001 may be configured to: determine an initial exception object based on the type of the exception event; determine associated exception objects associated with the initial exception object by performing an association traversal on the initial exception object; and determine the initial exception object and the associated exception objects as at least one exception object.

[0117] In one or more example embodiments, the associated exception object may include at least one of a synchronization object, a task object, a queue object, a context object, a video memory object, and an execution engine object associated with the initial exception object, wherein the initial exception object is directly associated with the synchronization object or the task object, the synchronization object or the task object belongs to the queue object, the queue object belongs to the context object, the queue object is bound to the video memory object, and the context object is mapped to the execution engine object.

[0118] The association determination unit 1002 is configured to determine the association between at least one abnormal object based on state data, so as to construct an object relationship view of at least one abnormal object.

[0119] In one or more example embodiments, the association may include at least one of the following: a dependency relationship between a synchronization object and a task object, an ownership relationship between a task object and a queue object, a mapping relationship between a queue object and a context object, a scheduling relationship between a context object and an execution engine object, and a binding relationship between a queue object and a video memory object.

[0120] The forward progress state acquisition unit 1003 is configured to acquire the processor's forward progress state data, wherein the forward progress state data represents the state in which the processor's task processing chain continues to progress in the completion direction within the expected time.

[0121] In one or more example embodiments, the forward progress status data may include at least one of the following: whether the synchronization object completes within a preset time, whether the completion sequence number of the queue object advances, whether the completion pointer or heartbeat of the execution engine object is updated, and whether the newly submitted test task completes within a preset time.

[0122] The fault domain determination unit 1004 is configured to determine the fault domain of the processor based on the object relationship view association and the processor's forward state data information.

[0123] In one or more example embodiments, the fault domain can be divided into queue-level faults, context-level faults, execution engine-level faults, and device-level faults according to the fault level from low to high.

[0124] In one or more example embodiments, the fault domain determination unit 1004 may be configured to: determine a fault domain as a queue-level fault when the number of queue objects meets the queue-level fault condition; determine a fault domain as a context-level fault when the number of queue objects does not meet the queue-level fault condition, and there is at least one context object in the context objects whose exception rate meets the context-level fault condition; determine a fault domain as an execution engine-level fault when there is no context object in the context objects whose exception rate meets the context-level fault condition, and there is at least one execution engine object in the execution engine objects whose forward propagation loss rate meets the execution engine-level fault condition; and determine a fault domain as an execution engine-level fault when there is no execution engine object whose forward propagation loss rate meets the execution engine-level fault condition. A fault domain can be identified as a device-level fault when at least one execution engine object of a fault condition and the number of execution engine objects included in at least one exception object satisfies a device-level fault condition. A queue-level fault condition may include a queue object number less than or equal to a first predetermined number and other queue objects on the same execution engine object maintaining forward propagation. A context-level fault condition may include a queue object exception rate greater than or equal to an exception rate threshold. An execution engine-level fault condition may include a forward propagation loss rate greater than or equal to a forward propagation loss rate threshold. A device-level fault condition may include at least one exception object containing an execution engine object greater than or equal to a second predetermined number, wherein the second predetermined number is based on the number of execution engine objects associated with the processor.

[0125] The recovery granularity determination unit 1005 is configured to determine the recovery granularity corresponding to the fault domain.

[0126] In one or more example embodiments, the recovery granularity determination unit 1005 may be configured to: obtain the reference source of the queue object from the object relationship snapshot based on the fact that the fault domain is a queue-level fault and the lifecycle of the queue object is abnormal; reconstruct the logical reference relationship of the queue object based on the reference source; perform a queue object lifecycle recovery operation based on the logical reference relationship to restore the lifecycle of the queue object; and determine whether abnormal isolation is required when the lifecycle of the queue object is not successfully restored. The recovery granularity determination unit 1005 may also be configured to: when it is determined that abnormal isolation is required, determine the recovery granularity corresponding to the queue-level fault in the pre-stored fault recovery mapping table as the recovery granularity; when it is determined that abnormal isolation is not required, upgrade the recovery granularity corresponding to the queue-level fault in the pre-stored fault recovery mapping table by a predetermined number of levels to obtain an upgraded recovery granularity, and determine the upgraded recovery granularity as the recovery granularity.

[0127] The recovery operation execution unit 1006 is configured to perform at least one exception recovery operation of the processor based on the recovery granularity.

[0128] In one or more example embodiments, the recovery operation execution unit 1006 may be configured to: upgrade the recovery granularity in response to the recovery granularity upgrade condition being met; determine at least one abnormal recovery operation corresponding to the upgraded recovery granularity according to a pre-stored recovery operation mapping table, and execute at least one abnormal recovery operation, wherein the recovery granularity upgrade condition includes at least one of the same queue object, the same context object, or the same execution engine object repeatedly triggering an exception within a preset time window.

[0129] In one or more example embodiments, the processor exception recovery apparatus 1000 may further include: a snapshot acquisition unit (not shown), configured to: acquire a snapshot of the processor's object relationships at the time of the exception event, wherein the object relationship snapshot stores object relationships in the processor's task processing chain. A recovery operation execution unit 1006 may be configured to: perform at least one exception recovery operation based on the object relationships in the object relationship snapshot.

[0130] In one or more example embodiments, the processor exception recovery apparatus 1000 may further include: a verification unit (not shown), configured to verify whether the exception recovery was successful by using an object relationship snapshot after performing the exception recovery operation of the processor; a granularity upgrade unit (not shown), configured to upgrade the recovery granularity in response to the failure of the exception recovery; and a re-recovery unit (not shown), configured to perform the exception recovery operation of the processor based on the upgraded recovery granularity.

[0131] In one or more example embodiments, the verification unit may be configured to: perform at least one of an object consistency verification operation and a forward push verification operation, wherein the object consistency verification operation is used to verify whether the object relationships associated with at least one abnormal object are consistent before and after the anomaly recovery using an object relationship snapshot, and the forward push verification operation is used to verify whether the processor's task processing chain is progressing in the completion direction; and determine whether the anomaly recovery is successful based on at least one of the results of the object consistency verification operation and the results of the forward push verification operation.

[0132] In addition, according to one or more example embodiments, a computer-readable storage medium is provided having a computer program stored thereon that, when executed, implements a processor exception recovery method according to one or more example embodiments.

[0133] In one or more example embodiments, the computer-readable storage medium may carry one or more computer programs that, when executed, perform the following steps: in response to an abnormal event occurring in the processor, identify at least one abnormal object associated with the abnormal event and obtain state data of the at least one abnormal object, wherein the state data represents the state of the at least one abnormal object in the processor's task processing chain; based on the state data, determine the association relationships between the at least one abnormal object to construct an object relationship view of the at least one abnormal object; obtain forward progress state data of the processor, wherein the forward progress state data represents the state of the processor's task processing chain continuously progressing in the completion direction within an expected time; based on the object relationship view and the forward progress state data, determine the processor's fault domain; determine the recovery granularity corresponding to the fault domain; and based on the recovery granularity, perform at least one abnormal recovery operation of the processor.

[0134] Computer-readable storage media can be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In embodiments of this disclosure, a computer-readable storage medium can be any tangible medium that contains or stores a computer program that can be used by or in conjunction with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable storage medium can be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof. A computer-readable storage medium can be included in any apparatus; it can also exist independently without being assembled into that apparatus.

[0135] Furthermore, according to one or more example embodiments, a computer program product is also provided, wherein instructions in the computer program product are executable by a processor of a computer device to perform a processor exception recovery method according to one or more example embodiments.

[0136] The above has been combined Figure 10 A processor exception recovery apparatus according to one or more example embodiments has been described. Next, in conjunction with… Figure 11 A computing device according to one or more example embodiments is described.

[0137] Figure 11A schematic diagram of a computing device 1100 according to one or more example embodiments is shown.

[0138] Reference Figure 11 A computing device 1100 according to one or more example embodiments includes a memory 1110 and a processor 1120, wherein the memory 1110 stores a computer program that, when executed by the processor 1120, implements a processor exception recovery method according to one or more example embodiments.

[0139] As an example, computing device 1100 may be a computer, smartphone, tablet, personal digital assistant, or other electronic terminal capable of executing the aforementioned computer programs. In computing device 1100, processor 1120 may include a central processing unit (CPU), graphics processing unit (GPU), programmable logic device, dedicated processor system, microcontroller, or microprocessor. As an example, and not a limitation, processor may also include analog processors, digital processors, microprocessors, multi-core processors, processor arrays, network processors, etc. Processor 1120 may execute instructions or code stored in memory 1110, which may also store data. Instructions and data may also be sent and received via a network through a network interface device, which may employ any known transmission protocol. Memory 1110 may be integrated with processor 1120, for example, by arranging RAM or flash memory within an integrated circuit microprocessor. Furthermore, memory 1110 may include separate devices, such as external disk drives, storage arrays, or other storage devices usable by any database system. The memory 1110 and the processor 1120 may be operatively coupled or may communicate with each other, for example, through I / O ports, network connections, etc., so that the processor 1120 can read files stored in the memory.

[0140] In addition, the computing device 1100 may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, touch input device, etc.). All components of the computing device 1100 may be interconnected via a bus and / or a network.

[0141] The above has been referred to Figures 1 to 11 A processor exception recovery method and apparatus according to one or more example embodiments are described. However, it should be understood that: Figure 10 The processor exception recovery device and its units shown can be configured as software, hardware, firmware, or any combination thereof to perform specific functions. Figure 11 The computing device shown is not limited to the components shown above, but some components may be added or removed as needed, and the above components may also be combined.

[0142] According to embodiments of the present disclosure, the processor anomaly recovery method and apparatus, computing device, storage medium, and computer program product construct an object relationship view of at least one anomaly object by determining the association between at least one anomaly object based on the state data of at least one anomaly object, thereby identifying the true impact range of the anomaly; determine the fault domain of the processor based on the object relationship view and forward state data, thereby improving the accuracy of the fault domain; and improve the accuracy of the anomaly recovery operation by determining the recovery granularity corresponding to the fault domain and determining at least one anomaly recovery operation corresponding to the recovery granularity, thereby improving the effectiveness of the processor anomaly recovery.

[0143] The processor exception recovery method and apparatus, computing device, storage medium and computer program product according to embodiments of the present disclosure can identify the true propagation range of exceptions by constructing an exception object relationship view, rather than directly resetting based on a single timeout event, thereby improving the accuracy of fault identification.

[0144] According to the processor exception recovery method and apparatus, computing device, storage medium and computer program product of the present disclosure, the fault domain determination mechanism can preferentially select the recovery operation corresponding to the queue level or context level recovery granularity, reduce interference with other tasks and make the recovery granularity more refined.

[0145] According to the processor exception recovery method and apparatus, computing device, storage medium and computer program product of the present disclosure, by saving a snapshot of object relationships before recovery, a basis can be provided for recovery execution and post-recovery verification, reducing the risk of object relationship damage and making the recovery process safer.

[0146] The processor exception recovery method and apparatus, computing device, storage medium and computer program product according to the embodiments of this disclosure can effectively support scenarios of reference disorder, delayed reclamation exception and attachment exception by introducing an actual reference source reconstruction mechanism for queue lifecycle exceptions, thereby handling lifecycle exception situations.

[0147] The processor anomaly recovery method and apparatus, computing device, storage medium and computer program product according to embodiments of the present disclosure avoid the false recovery problem of "actions were executed but services were not restored" by simultaneously verifying object consistency and forward progress status, making the recovery results more reliable.

[0148] The processor exception recovery method and apparatus, computing device, storage medium and computer program product according to the embodiments of this disclosure can maintain the continuous operation of other inference tasks unrelated to the exception as much as possible through the minimum impact recovery strategy, thereby improving the overall business continuity and making it more suitable for AI inference business.

[0149] The processor exception recovery method and apparatus, computing device, storage medium and computer program product according to the embodiments of the present disclosure can support upgrading the recovery operation by recovery granularity. When a low-granularity recovery fails, it can be upgraded to a higher-granularity recovery action, thereby improving the recovery effect.

[0150] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0151] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A processor exception recovery method, characterized in that, The method includes: In response to an abnormal event occurring in the processor, at least one abnormal object associated with the abnormal event is identified and state data of the at least one abnormal object is obtained, wherein the state data represents the state of the at least one abnormal object in the task processing chain of the processor. Based on the state data, the association relationships between the at least one abnormal object are determined to construct an object relationship view of the at least one abnormal object; Acquire the forward progress state data of the processor, wherein the forward progress state data represents the state of the processor's task processing chain continuously progressing in the completion direction within the expected time; Based on the object relationship view and the forward propagation state data, the fault domain of the processor is determined; Determine the recovery granularity corresponding to the fault domain; Based on the recovery granularity, at least one exception recovery operation of the processor is performed.

2. The method according to claim 1, characterized in that, Determining at least one anomalous object related to the anomalous event includes: Determine the initial exception object based on the type of the exception event; The associated exception objects are determined by performing an associative traversal on the initial exception object. The initial abnormal object and the associated abnormal object are identified as at least one abnormal object.

3. The method according to claim 2, characterized in that, The associated exception object includes at least one of the following: a synchronization object, a task object, a queue object, a context object, a video memory object, and an execution engine object associated with the initial exception object. The initial exception object is directly associated with the synchronization object or the task object, the synchronization object or the task object belongs to the queue object, the queue object belongs to the context object, the queue object is bound to the video memory object, and the context object maps to the execution engine object.

4. The method according to claim 3, characterized in that, The association relationship includes at least one of the following: the dependency relationship between the synchronization object and the task object; the ownership relationship between the task object and the queue object; the mapping relationship between the queue object and the context object; the scheduling relationship between the context object and the execution engine object; and the binding relationship between the queue object and the video memory object. The forward progress status data includes at least one of the following: whether the synchronization object completes within a preset time, whether the completion sequence number of the queue object advances, whether the completion pointer or heartbeat of the execution engine object is updated, and whether the newly submitted test task completes within a preset time.

5. The method according to claim 3, characterized in that, The fault domains are divided into queue-level faults, context-level faults, execution engine-level faults, and device-level faults according to their fault levels from low to high.

6. The method according to claim 5, characterized in that, Determining the fault domain of the processor based on the object relationship view and the forward propagation state data includes: When the number of queue objects meets the queue-level fault conditions, the fault domain is determined as the queue-level fault. When the number of queue objects does not meet the queue-level fault condition, and there is at least one context object in the context object whose exception rate meets the context-level fault condition, the fault domain is determined as the context-level fault. When there is no context object in the context object whose exception rate satisfies the context-level fault condition, and there is at least one execution engine object in the execution engine object whose forward push loss rate satisfies the execution engine-level fault condition, the fault domain is determined as an execution engine-level fault. When there is no execution engine object whose forward propagation loss rate satisfies the execution engine-level fault condition, and the number of execution engine objects included in the at least one abnormal object satisfies the device-level fault condition, the fault domain is determined to be a device-level fault. The queue-level fault conditions include situations where the number of queue objects is less than or equal to a first predetermined number and other queue objects on the same execution engine object as the queue objects are advancing forward. The context-level fault condition includes an exception rate of the queue object that is greater than or equal to an exception rate threshold. The execution engine-level failure conditions include a forward propulsion loss rate greater than or equal to a forward propulsion loss rate threshold. The device-level fault condition includes the fact that the number of execution engine objects included in the at least one abnormal object is greater than or equal to a second predetermined number, wherein the second predetermined number is based on the number of execution engine objects associated with the processor.

7. The method according to claim 1, characterized in that, The step of performing at least one exception recovery operation on the processor based on the recovery granularity includes: In response to the condition for upgrading the recovery granularity being met, the recovery granularity is upgraded. Based on a pre-stored recovery operation mapping table, at least one abnormal recovery operation corresponding to the upgraded recovery granularity is determined, and the at least one abnormal recovery operation is executed. The recovery granularity upgrade conditions include at least one of the same queue object, the same context object, or the same execution engine object repeatedly triggering an exception within a preset time window.

8. The method according to claim 3, characterized in that, The method further includes: Obtain a snapshot of the object relationships of the processor at the time the abnormal event occurs, wherein the snapshot stores the object relationships in the processor's task processing chain. The execution of at least one exception recovery operation of the processor includes: Perform the at least one exception recovery operation based on the object relationships in the object relationship snapshot.

9. The method according to claim 8, characterized in that, Determining the recovery granularity corresponding to the fault domain includes: Based on the fact that the fault domain is a queue-level fault and the lifecycle of the queue object is abnormal, the reference source of the queue object is obtained from the object relationship snapshot; Reconstruct the logical reference relationship of the queue object based on the reference source; Perform a queue object lifecycle recovery operation based on the logical reference relationship to restore the lifecycle of the queue object.

10. The method according to claim 9, characterized in that, Determining the recovery granularity corresponding to the fault domain further includes: If the lifecycle of the queue object fails to recover successfully, determine whether abnormal isolation is required; When it is determined that abnormal isolation is required, the recovery granularity corresponding to the queue-level fault in the pre-stored fault recovery mapping table is determined as the recovery granularity; When it is determined that abnormal isolation is not required, the recovery granularity corresponding to the queue-level fault in the pre-stored fault recovery mapping table is upgraded by a predetermined number of levels to obtain the upgraded recovery granularity, and the upgraded recovery granularity is determined as the recovery granularity.

11. The method according to claim 10, characterized in that, The method further includes: After performing the exception recovery operation of the processor, the success of the exception recovery is verified by using the object relationship snapshot; In response to an unsuccessful recovery, the recovery granularity is increased; Based on the upgraded recovery granularity, the abnormal recovery operation of the processor is performed.

12. The method according to claim 11, characterized in that, The verification of whether the anomaly recovery was successful includes: Perform at least one of an object consistency verification operation and a forward push verification operation, wherein the object consistency verification operation is used to verify whether the object relationship associated with the at least one abnormal object is consistent before and after the abnormality recovery using the object relationship snapshot, and the forward push verification operation is used to verify whether the task processing chain of the processor is advancing in the completion direction; Based on at least one of the results of the object consistency verification operation and the forward push verification operation, determine whether the anomaly recovery was successful.

13. A processor exception recovery device, the device comprising: A status acquisition unit is configured to, in response to an abnormal event occurring in the processor, determine at least one abnormal object associated with the abnormal event and acquire status data of the at least one abnormal object, wherein the status data represents the status of the at least one abnormal object in the processor's task processing chain. The association relationship determination unit is configured to determine the association relationship between the at least one abnormal object based on the state data, so as to construct an object relationship view of the at least one abnormal object; The forward progress state acquisition unit is configured to acquire the forward progress state data of the processor, wherein the forward progress state data represents the state in which the processor's task processing chain continues to progress in the completion direction within an expected time. The fault domain determination unit is configured to determine the fault domain of the processor based on the object relationship view association and the processor's forward propagation state data information; The recovery granularity determination unit is configured to determine the recovery granularity corresponding to the fault domain; The recovery operation execution unit is configured to perform at least one exception recovery operation of the processor based on the recovery granularity.

14. A computer-readable storage medium storing a computer program, wherein, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 12.

15. A computing device, comprising: At least one processor; At least one memory storing a computer program that, when executed by the at least one processor, implements the method of any one of claims 1 to 12.

16. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 12.

Images