Kernel compilation method and device for hardware accelerator, and running method

Abstract

This invention provides a kernel compilation and execution method and apparatus for hardware accelerators, relating to the field of deep learning technology. The method includes: decomposing each operator in the computation graph into multiple fine-grained tasks; constructing a task dependency graph by treating each fine-grained task as a task node and each preceding task synchronization condition of a task node as an event node, based on the data dependencies between these tasks; performing event fusion, normalization, and linearization on the task dependency graph to obtain a linearized task dependency graph; and generating a kernel for the target model based on the linearized task dependency graph. This kernel is deployed on the hardware accelerator and executes the inference of the target model. This invention compiles the entire computation graph into a single persistent kernel and statically divides the physical computation core of the hardware accelerator into scheduling units and working units, which can shorten the scheduling critical path of the deep learning model, reduce inference latency, and improve the inference efficiency of the deep learning model.

Description

Technical Field

[0001] This invention relates to the field of deep learning technology, and in particular to a kernel compilation and execution method and apparatus for hardware accelerators. Background Technology

[0002] Currently, inference latency in large language models has become a critical factor determining user experience. Whether it's the response speed of a large language model or the real-time interaction of an intelligent assistant, every millisecond of latency directly impacts product usability and user satisfaction. Simultaneously, the scarcity of accelerator computing resources makes cost control for each inference operation crucial. Current deep learning inference systems generally employ an execution model where each operator corresponds to one kernel. That is, the inference engine represents the model as a computation graph, where nodes represent various operators (such as matrix multiplication, attention mechanisms, normalization, etc.), and edges represent tensor data. During inference, the main control processor submits kernel startup requests for each operator to the hardware accelerator sequentially according to the topological order of the computation graph. After each kernel completes execution, synchronization is performed through kernel barriers to ensure correct data dependencies before the main control processor starts the kernel for the next operator.

[0003] This type of execution model has two drawbacks: First, kernel barriers confine the software pipeline to a single operator. While the computation and data handling units of the hardware accelerator could potentially work asynchronously and in parallel, the presence of kernel barriers forces the next operator to wait for the entire operator to complete before it can begin, even if it only depends on a small portion of the output from the previous operator, creating pipeline bubbles. Second, kernel startup overhead accumulates significantly with model size. For example, a single inference iteration of a large language model requires the startup of hundreds to thousands of kernels. The startup of each kernel involves communication between the main processor and the accelerator, resulting in substantial latency. Furthermore, in the operator-by-operator startup execution model, kernel scheduling and execution follow a master-slave pattern, where the main processor assumes all scheduling responsibilities, and the hardware accelerator passively executes kernels issued by the main processor. After each operator kernel completes execution on the accelerator, it must return a completion status to the main processor, which then decides and issues the next kernel based on the topological order of the computation graph. Frequent communication between the main processor and the accelerator further exacerbates the performance bottleneck. Instruction transmission latency and synchronization wait time accumulate with the number of cores. Moreover, the main processor cannot perceive the real-time load of each computing unit within the accelerator and can only issue instructions sequentially according to a predetermined topology, resulting in unbalanced load. As the model size increases and the granularity of operator decomposition becomes finer, the number of cores further increases, and the impact of the above scheduling mode on the inference efficiency of deep learning models becomes increasingly serious. Summary of the Invention

[0004] This invention provides a kernel compilation and execution method and apparatus for hardware accelerators to address the problem of low inference efficiency in deep learning models.

[0005] In a first aspect, embodiments of the present invention provide a kernel compilation method for hardware accelerators, including: Obtain the computational graph of the target model; Each operator in the computation graph is decomposed into multiple fine-grained tasks; Based on the data dependencies between each fine-grained task, each fine-grained task is treated as a task node, and each preceding task synchronization condition of a task node is treated as an event node, thus constructing a task dependency graph. The task dependency graph is subjected to event fusion, normalization, and linearization to obtain a linearized task dependency graph. The kernel of the target model is generated based on the linearized task dependency graph. The kernel is used to deploy on a hardware accelerator and perform inference on the target model.

[0006] In a second aspect, embodiments of the present invention provide a kernel running method for hardware accelerators, which is applied to hardware accelerators. The computing units of the hardware accelerators are divided into working units and scheduling units. The hardware accelerators are loaded with a kernel of a target model obtained by the method of the first aspect or any possible implementation of the first aspect. The method includes: Each scheduling unit maintains an event queue, monitors the activation status of event nodes in the kernel, and when any event node is activated, dispatches the task node triggered by that event node to the task queue. Each work unit maintains at least one task queue, retrieves tasks from the task queue for execution, and sends a notification to the scheduling unit after the task execution is completed to trigger the activation of the corresponding event node.

[0007] Thirdly, embodiments of the present invention provide a kernel compilation apparatus for hardware accelerators, comprising: The acquisition module is used to acquire the computational graph of the target model; The decomposition module is used to decompose each operator in the computation graph into multiple fine-grained tasks; The building module is used to construct a task dependency graph based on the data dependencies between each fine-grained task, treating each fine-grained task as a task node and each preceding task synchronization condition of a task node as an event node. The adjustment module is used to perform event fusion, normalization, and linearization on the task dependency graph to obtain a linearized task dependency graph. The generation module is used to generate the kernel of the target model based on the linearized task dependency graph. The kernel is deployed on the hardware accelerator and performs inference on the target model.

[0008] Fourthly, embodiments of the present invention provide a kernel running device for a hardware accelerator, applied to a hardware accelerator, wherein the computing unit of the hardware accelerator is divided into a working unit and a scheduling unit, and the hardware accelerator is loaded with a kernel of a target model obtained by the method of the first aspect or any possible implementation of the first aspect. The device includes: The scheduling module is used to control each scheduling unit to maintain an event queue, monitor the activation status of event nodes in the kernel, and when any event node is activated, distribute the task nodes triggered by that event node to the task queue. The work module is used to control each work unit to maintain at least one task queue, retrieve tasks from the task queue for execution, and send a notification to the scheduling unit after the task is completed to trigger the activation of the corresponding event node.

[0009] The kernel compilation and execution method and apparatus for hardware accelerators provided in this invention express fine-grained synchronization relationships through a dependency graph composed of task nodes and event nodes. The entire computation graph is compiled into a single persistent kernel, eliminating global kernel barriers at operator boundaries. Event nodes provide a graph structure foundation for event-driven scheduling at runtime, refining the synchronization granularity between tasks from the operator level to the task level. This allows the preloading and computation phases of adjacent tasks to overlap in time, avoiding pipeline bubbles caused by waiting for the entire operator to complete. The physical computing core of the hardware accelerator is statically divided into scheduling units and working units. An event queue is maintained within the scheduling unit to achieve event-driven distribution, ensuring that all scheduling logic is completed in a closed loop within the hardware accelerator. The main control processor does not need to participate in any runtime scheduling decisions, eliminating inter-chip scheduling communication latency in master-slave mode. The working unit maintains its task queue locally and sends a notification to the scheduling unit directly after task completion. Synchronization overhead is only transmitted between cores within the accelerator. The scheduling unit senses the task queue status of each working unit and distributes tasks accordingly, achieving dynamic load balancing among working units. These mechanisms together shorten the critical path of scheduling, reduce inference latency, and improve the inference efficiency of deep learning models. Attached Figure Description

[0010] Figure 1 This is a flowchart illustrating the implementation of a kernel compilation method for hardware accelerators provided in an embodiment of the present invention. Figure 2 This is a comparison diagram of the execution logic between the traditional operator-by-operator execution model and the execution model provided by this invention; Figure 3This is a flowchart illustrating the implementation of a kernel compilation method for hardware accelerators provided in another embodiment of the present invention. Figure 4 This is a schematic diagram of the normalization transformation of the task dependency graph provided in an embodiment of the present invention; Figure 5 This is a flowchart illustrating the implementation of the kernel compilation method for hardware accelerators provided in this embodiment of the invention. Figure 6A This is a layout diagram of the physical computing unit of the hardware accelerator provided in an embodiment of the present invention; Figure 6B This is a schematic diagram of the event-driven execution timing provided in an embodiment of the present invention; Figure 7A This is a timing diagram of a cross-task software pipeline provided in an embodiment of the present invention; Figure 7B This is a schematic diagram of paged on-chip memory management provided in an embodiment of the present invention; Figure 8 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0011] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0012] See Figure 1 The document illustrates a flowchart of the implementation of a kernel compilation method for hardware accelerators provided in an embodiment of the present invention, detailed below: Step 101: Obtain the computational graph of the target model.

[0013] Step 102: Decompose each operator in the computation graph into multiple fine-grained tasks.

[0014] Step 103: Based on the data dependencies between each fine-grained task, each fine-grained task is treated as a task node, and each preceding task synchronization condition of a task node is treated as an event node, thus constructing a task dependency graph.

[0015] Step 104: Perform event fusion, normalization, and linearization on the task dependency graph to obtain the linearized task dependency graph.

[0016] Step 105: Generate the kernel of the target model based on the linearized task dependency graph. The kernel is used to deploy on the hardware accelerator and execute the inference of the target model.

[0017] In this embodiment, some terms involved in the present invention will be explained first. A computation graph is a directed acyclic graph structure used by deep learning frameworks to express neural network models. Each node in the graph represents an operator, such as a matrix multiplication operator, attention mechanism operator, normalization operator, or fully reduced communication operator. The directed edges in the graph represent tensor data flowing between operators. A hardware accelerator refers to a dedicated processor designed specifically for tensor operations, which typically includes multiple physical computing cores, on-chip high-speed temporary storage, and interfaces for accessing external device storage. In this invention, the kernel does not refer to the kernel at the operating system level, but rather to a program image that can be directly loaded and executed by the hardware accelerator. This program image contains all the computational tasks and their scheduling logic required to complete the entire target model inference.

[0018] During step 101, the compiler receives the trained target model from the deep learning framework and parses it into a framework-independent intermediate representation. This intermediate representation retains all the topological information of the original computation graph, namely the types of operators, the parameter configurations of the operators, and the tensor flow directions between operators.

[0019] In step 102, the compiler further decomposes each operator in the computation graph into multiple fine-grained tasks. A task here refers to an independent computational operation performed on a single physical computing unit. For a matrix multiplication operator, its input tensor is typically divided into multiple blocks, each processed in parallel by a different physical computing unit. The compiler determines the optimal decomposition strategy for each operator based on the number of physical computing cores in the hardware accelerator and the on-chip storage capacity. Specifically, the compiler evaluates the size of the operator's output tensor in each dimension and, considering the maximum parallelism that the hardware computing unit can handle, divides the computational load of the entire operator into several subtasks, ensuring that each subtask can be completed independently on a single computing unit and that the required data can be fully loaded into the on-chip storage of that unit. For communication operators, such as full reduction or broadcast operations, the compiler similarly decomposes them into multiple asynchronous communication subtasks, each responsible for transmitting a portion of the tensor data. The granularity of operator decomposition directly determines the effectiveness of subsequent scheduling optimizations; coarse granularity fails to expose sufficient parallelism and pipeline opportunities, while excessively fine granularity may lead to excessive scheduling overhead. The compiler automatically determines the appropriate decomposition granularity based on hardware parameters and model size.

[0020] See Figure 2The diagram illustrates a comparison between the traditional operator-by-operator execution model and the persistent kernel execution model provided by this invention. In the traditional model, the main processor needs to sequentially distribute the independent kernels for each operator to the accelerator according to the topological order of the computation graph. After each kernel completes execution, synchronization is achieved through a kernel barrier to ensure the correctness of data dependencies. However, this barrier also blocks the pipeline across operators. In contrast, this invention integrates the entire model into a single persistent kernel, requiring only a single distribution to complete all inference computations, and is unaffected by kernel barriers.

[0021] See Figure 3 This diagram illustrates the overall flowchart of the compilation stage provided by an embodiment of the present invention. The diagram more intuitively demonstrates the complete transformation path from the original computation graph to the final persistent kernel. From left to right, the flowchart includes: inputting the model computation graph, decomposing operators into fine-grained tasks, constructing a task dependency graph containing task nodes and event nodes, event fusion optimization, task dependency graph normalization, task dependency graph linearization, and finally, persistent kernel code generation. The steps are explained as follows: Step 1: Decompose each operator into multiple fine-grained tasks, each task corresponding to a job on a physical computation unit; Step 2: Construct a task dependency graph containing task nodes and event nodes based on data dependencies; Step 3: Merge multiple events with the same prerequisite dependencies to reduce synchronization overhead; Step 4: Introduce empty task nodes so that each task depends on at most one event and triggers at most one event; Step 5: Perform breadth-first search sorting to ensure that tasks triggered by the same event have consecutive indices.

[0022] During step 103, the compiler constructs a task dependency graph based on the data dependencies between the fine-grained tasks. The essence of data dependencies is the tensor read-write dependency between producer and consumer tasks; that is, the output tensor produced by one task is the input tensor necessary for the execution of another task. The compiler establishes dependency edges between tasks by tracing the source of the input tensors and the destination of the output tensors for each task. Specifically, for each tensor flow path, the tensor producer task must complete the write to the tensor before the consumer task begins; this constraint is called read-after-write dependency. The compiler abstracts each fine-grained task as a task node in the task dependency graph, and simultaneously abstracts the synchronization condition of all preceding tasks being completed as an event node. The activation condition of an event node is that all its preceding task nodes have been executed. Once activated, the event node triggers the execution of one or more of its corresponding subsequent task nodes. It is worth noting that the same task may depend on the synchronization conditions represented by multiple different event nodes. For example, an attention mechanism task may need to wait for the completion of the preceding matrix multiplication task and mask generation task before it can begin execution. Similarly, after a task is completed, it may trigger multiple subsequent event nodes simultaneously, allowing different subsequent tasks to begin execution in parallel. In this way, the coarse-grained synchronization that was uniformly enforced by kernel barriers in the traditional execution model is decomposed into fine-grained event synchronization in the task dependency graph, laying the foundation for subsequent cross-operator pipeline optimization.

[0023] In the initially constructed task dependency graph, a task node may depend on or trigger multiple event nodes. This many-to-many dependency relationship requires complex counting and matching logic at runtime for correct scheduling, significantly increasing the processing overhead of the scheduling unit. Therefore, in step 104, the compiler transforms the task dependency graph into a regularized form through three sub-steps: event fusion, normalization, and linearization. This regularized form ensures that each task node depends on at most one event node and triggers at most one event node. This regularized form greatly simplifies the scheduling logic; the scheduling unit only needs to maintain a counter for each event. When the counter reaches zero, the event is activated, and then all ready tasks are directly located and distributed based on the consecutive indices allocated in the linearization phase. The specific implementation of step 104 will be further explained in subsequent embodiments.

[0024] Following this, in step 105, the compiler generates the kernel of the target model based on the linearized task dependency graph. This kernel is a complete, independently executable program image containing the topology information of the task dependency graph, the device code for each task node, and scheduling logic. The device code for each task node is generated by the compiler, and the generation process includes selecting a suitable computation unit mapping scheme for each task, allocating register resources, and generating the corresponding device instruction sequence. The compiler links the device code of all task nodes together to form a single executable image. This kernel is designed to be resident on the hardware accelerator; that is, the kernel is not immediately unloaded after startup but remains resident in the accelerator's instruction store until the inference computation of the entire target model is completed. This persistent design eliminates the instruction transfer overhead caused by the repeated loading and unloading of each operator kernel in the traditional operator-by-operator execution model.

[0025] In one possible implementation, the task dependency graph is subjected to event fusion, normalization, and linearization to obtain a linearized task dependency graph, including: Multiple event nodes with the same set of preceding task nodes in the task dependency graph are merged into one event node to obtain the task dependency graph after event fusion. In the task dependency graph after event fusion, empty task nodes are introduced and equivalent transformations are performed so that each task node depends on at most one event node and triggers at most one event node, resulting in a normalized task dependency graph. The normalized task dependency graph is topologically sorted using a breadth-first search algorithm, and a continuous index is assigned to each task node to obtain a linearized task dependency graph.

[0026] In this embodiment, event fusion is the first step in graph transformation. In the initially constructed task dependency graph, there may be multiple event nodes with the exact same set of preceding tasks, meaning these event nodes are all waiting for the same set of tasks to complete. For example, after a matrix multiplication task is completed, its output tensor may be used by both subsequent normalization and residual connection tasks. In this case, the event nodes corresponding to the two subsequent tasks each use the matrix multiplication task as their sole preceding task. In this scenario, merging multiple event nodes into one event node does not change the semantics of the task dependency graph because all events waiting for the same synchronization condition can be triggered simultaneously at the same point. The benefit of event fusion is reduced runtime synchronization overhead, i.e., the length of the event queue that the scheduling unit needs to maintain is shortened, and the number of event matching and dispatching operations is correspondingly reduced. The compiler scans each event node one by one, comparing whether its set of preceding tasks is completely identical. If they are identical, these event nodes are merged.

[0027] Normalization is the second and most crucial step in graph transformation. The goal of normalization is to ensure that each task node depends on at most one event node and triggers at most one event node. This constraint is not inherent to the task dependency graph but is introduced to significantly simplify runtime scheduling logic. In the normalized graph, the scheduling decision for any task only needs to check if a unique event is active, and the completion of any task only needs to trigger a unique event. This allows the scheduling unit to complete dependency checks and task distribution in constant time. For original task nodes that do not satisfy the above constraints, the compiler performs an equivalent transformation by introducing empty task nodes. An empty task node is a special type of task node that does not contain any actual computation or communication operations; it is only used to transmit event triggering relationships in the graph.

[0028] See Figure 4 This diagram illustrates a task dependency graph normalization transformation provided by an embodiment of the present invention. The upper part of the diagram shows the task dependency graph before normalization. After task node T0 completes execution, it needs to simultaneously trigger event nodes e1, e2, and e3, which violates the constraint that each task can trigger at most one event. The lower part of the diagram shows the task dependency graph after the normalization equivalence transformation. T0 only triggers a newly introduced event node e', which then triggers three empty task nodes T1, T2, and T3. These three empty task nodes respectively trigger the original event nodes e1, e2, and e3. This transformation preserves the data dependency semantics of the original graph because e' is triggered only after T0 completes, e' triggers the empty tasks, and the empty tasks then trigger their respective events. The entire causal chain is logically equivalent to T0 directly triggering all events in the original graph. For symmetrical cases where task nodes depend on multiple events, the transformation method is similar: multiple empty task nodes are introduced, each receiving a preceding event trigger, and then collectively triggering a new event node to start the original task node.

[0029] Linearization is the third step in graph transformation. After normalization, each event node in the task dependency graph may trigger one or more task nodes. The compiler uses a breadth-first search-based topology sorting algorithm to linearize and sort the normalized task dependency graph. This algorithm traverses the hierarchical structure of the task dependency graph layer by layer, ensuring that the position of each task node in the sort is no earlier than the position of the event node it depends on. The traversal order of the breadth-first search ensures that all task nodes triggered by the same event node are assigned consecutive index numbers in the sort. This property allows the triggering range of each event node to be compactly represented using only two integers: the index of the first task node triggered by the event node and the index of the last task node. Compared to explicitly storing the complete list of task nodes triggered by each event node, this index range representation significantly reduces the storage overhead of scheduling information in the persistent kernel, and also allows the scheduling unit to quickly locate the task to be dispatched at runtime through simple range comparisons.

[0030] In one possible implementation, an empty task node is introduced into the task dependency graph after event fusion and an equivalent transformation is performed, such that each task node depends on at most one event node and triggers at most one event node, resulting in a normalized task dependency graph, including: For each task node, if the task node triggers multiple event nodes, a new event node and multiple empty task nodes are introduced, so that the task node triggers the new event node, the new event node triggers multiple empty task nodes, and each empty task node triggers an original event node. For each task node, if the task node depends on multiple event nodes, multiple empty task nodes and a new event node are introduced, so that each event node triggers an empty task node, multiple empty task nodes trigger a new event node, and the new event node triggers the task node.

[0031] In this embodiment, the above transformation strategy provides specific construction methods for two scenarios that violate the normalization constraint. For the first scenario, where a task node triggers multiple event nodes, the core idea of ​​the transformation is to insert an indirection layer between the original task node and the multiple original event nodes. First, a new event node is created, and the triggering target of the original task node is changed to trigger only this new event node. Then, an equal number of empty task nodes are created, each corresponding to one original event node. The new event node triggers all empty task nodes via broadcast, and each empty task node, upon being triggered, immediately triggers its corresponding original event node. After this transformation, the original task node still triggers at most one event, and each empty task node also satisfies the constraint of triggering at most one event, but the semantics of the original task node triggering multiple events are fully preserved.

[0032] For the second scenario, where a task node depends on multiple event nodes, a symmetrical approach is adopted. First, a corresponding empty task node is created for each preceding event node. Each preceding event node triggers its corresponding empty task node. Then, a new event node is created, and all empty task nodes trigger this new event node. Finally, this new event node triggers the original task node. After this transformation, each empty task node depends on at most one event and triggers at most one event, and the original task node depends on only one new event node. The overall semantics are equivalent to the original graph.

[0033] The two transformations described above can be applied iteratively until there are no nodes in the task dependency graph that violate the normalization constraints.

[0034] In one possible implementation, the kernel of the target model is generated based on the linearized task dependency graph, including: Task nodes whose execution time depends on the data are marked as instant start mode, and task nodes with fixed execution time are marked as prefetch start mode.

[0035] In this embodiment, the compiler performs a startup mode marking step before generating kernel code. Different operators may have different actual execution times on hardware accelerators. The execution time of some operators varies with the characteristics of the input data. For example, the computational cost of the attention mechanism operator is directly related to the length of the input sequence, and the sequence length may differ between different inference requests. Therefore, the runtime of the attention mechanism is difficult to predict precisely during the compilation phase. Task nodes corresponding to these data-dependent operators are marked as immediate startup mode. For task nodes in immediate startup mode, their actual execution start time is delayed until the dependent event nodes are fully activated, i.e., after the scheduling unit confirms that all preceding tasks of the task have been completed and received the corresponding event activation signal, the task is then distributed to the work unit. This strategy can tolerate fluctuations in execution time because even if the preceding tasks take different times due to changes in data characteristics, subsequent tasks can always be scheduled and executed as soon as the preceding tasks are actually completed, without unnecessary idle waiting.

[0036] See Figure 5 This document illustrates a flowchart of the implementation of a kernel execution method for hardware accelerators provided in an embodiment of the present invention. Applied to a hardware accelerator, the hardware accelerator's computing unit is divided into a working unit and a scheduling unit. The hardware accelerator is loaded with a kernel representing the target model obtained through the aforementioned kernel compilation method for hardware accelerators. Details are as follows: Step 501: Each scheduling unit maintains an event queue, monitors the activation status of event nodes in the kernel, and when any event node is activated, distributes the task node triggered by that event node to the task queue.

[0037] Step 502: Each work unit maintains at least one task queue, retrieves tasks from the task queue for execution, and sends a notification to the scheduling unit after the task execution is completed to trigger the activation of the corresponding event node.

[0038] In this embodiment, the physical resource allocation of the hardware accelerator is first described. The hardware accelerator contains multiple physical computing cores, which are structurally identical and all capable of performing tensor operations and data transfer. During persistent kernel startup, these physical computing cores are statically divided into two roles: scheduling units and working units. The number of scheduling units is typically one or more, depending on the total number of cores in the hardware accelerator and design trade-offs. Scheduling units are responsible for maintaining event queues, listening for event activations, and distributing tasks to working units. Working units occupy the remaining computing cores, each maintaining its own task queue and responsible for actually executing the distributed computing or communication tasks. This division of scheduling units and working units is determined at kernel startup and remains unchanged throughout the kernel's execution. This static division avoids the overhead of dynamic role switching at runtime and simplifies hardware design, as each core knows clearly at startup whether it is responsible for scheduling or computing.

[0039] The scheduling unit maintains an event queue at runtime, which stores active but unprocessed event nodes. At the start of the persistent kernel, all event nodes in the task dependency graph that do not depend on any preceding tasks—the starting event nodes—are pre-activated and added to the event queue. The scheduling unit continuously retrieves active event nodes from the event queue and, based on the index range of the task nodes associated with each event node, distributes all ready task nodes within that range to the task queues of each work unit. Distribution can employ round-robin or heuristic strategies based on the current load of the work unit to achieve load balancing among work units. The scheduling unit's distribution operation only involves writing the task node's identifier and its metadata to the work unit's task queue; it does not involve actual data movement or computation.

[0040] See Figure 6A , Figure 6B It shows the layout of the runtime scheduling unit and the working unit and the event-driven execution timing diagram provided in the embodiment of the present invention. Figure 6A The layout of the physical computing core of the hardware accelerator is shown. The physical computing core is divided into scheduling units and working units. The scheduling units are organized in the form of fine-grained thread groups, and each scheduling thread group maintains an independent event queue. Figure 6BThis diagram illustrates the event-driven execution sequence, with the horizontal axis representing the direction of time progression and the vertical axis representing the activity status of each work unit. As shown, after the initial event node is activated, the scheduling unit immediately distributes tasks to each work unit. Each work unit executes its received tasks in parallel, and upon completion, sends a completion notification to the scheduling unit. The scheduling unit then updates the preceding completion count of the corresponding subsequent event nodes. When the count of an event node reaches zero, that event node is activated and added to the event queue. Subsequently, the scheduling unit continues to distribute the next batch of tasks triggered by that event. Throughout this process, each work unit operates in parallel, relying solely on event synchronization, thus avoiding global kernel barriers.

[0041] The work unit retrieves task nodes from its own task queue and executes them. The execution of each task node is divided into two phases: a preloading phase and a computation phase. The preloading phase is responsible for moving the tensor data required for the task from the hardware accelerator's device storage to on-chip storage. This phase only involves issuing data transfer instructions and does not perform actual computation operations. The computation phase performs the actual computation operations based on the data already existing in the on-chip storage. After a task node completes execution, the work unit sends a completion notification to the scheduling unit, which includes identification information of the completed task. Upon receiving this notification, the scheduling unit determines the subsequent event nodes triggered by the task based on the task dependency graph and adds the task to the corresponding event node's preceding task completion count. When all preceding task nodes of an event node have been completed, the event node is activated and added to the scheduling unit's event queue, awaiting processing in subsequent scheduling loops.

[0042] It is worth emphasizing that the entire scheduling and execution process is completed within the hardware accelerator. The main control processor only needs to send a startup command to the accelerator once when the persistent kernel starts. All subsequent event listening, task distribution, task execution, and synchronization notification are coordinated and completed by the scheduling and working units within the accelerator. This decentralized design eliminates frequent communication between the main control processor and the accelerator, freeing up the main control processor's computing resources, and also avoids the additional latency caused by the main control processor's participation in scheduling.

[0043] In one possible implementation, the task queue includes an immediate queue and a prefetch queue. Tasks in the immediate queue correspond to task nodes whose execution time is data-dependent, while tasks in the prefetch queue correspond to task nodes with fixed execution times. Each work unit maintains at least one task queue and retrieves tasks from the task queues for execution, including: Each work unit maintains an immediate queue and a prefetch queue. If the immediate queue is not empty, tasks are retrieved from the immediate queue for execution; if the immediate queue is empty, tasks are retrieved from the prefetch queue for execution.

[0044] In this embodiment, each work unit maintains two independent task queues: an immediate queue and a prefetch queue. When distributing tasks, the scheduling unit determines which queue to place the task in based on the task node's startup mode flag. Task nodes marked as having an immediate startup mode are distributed to the immediate queue, while those marked as having a prefetch startup mode are distributed to the prefetch queue. The work unit employs a fixed priority strategy when executing tasks, meaning the immediate queue has a strictly higher priority than the prefetch queue. The work unit first checks if the immediate queue is empty. If there are tasks waiting to be executed in the immediate queue, the work unit retrieves one and executes it immediately, regardless of whether there are any tasks waiting to be executed in the prefetch queue. Only when the immediate queue is empty will the work unit check the first task in the prefetch queue to determine if its dependent event node has been confirmed as active by the scheduling unit. If it is active, the task is executed; otherwise, it continues to wait. This design reflects the higher timeliness requirements of immediate startup mode tasks, as the execution time of these tasks fluctuates with data changes. Once the preconditions are met, execution should begin immediately to avoid pipeline stalls due to delays. In contrast, tasks in the prefetch queue have fixed execution times and can occupy queue positions in advance even before prerequisites are met, without preempting the execution resources of immediate tasks. This dual-queue design with strict priorities allows tasks in both startup modes to coexist harmoniously on the same work unit, ensuring a rapid response to tasks with uncertain delays while hiding scheduling overhead by utilizing the prefetch capability of tasks with fixed delays.

[0045] In one possible implementation, each work unit maintains at least one task queue, and tasks are retrieved from the task queue for execution, including: Each work unit retrieves the current task from the task queue, executes the preloading phase of the current task, and moves the tensor data required by the current task from the device storage to the on-chip storage of the hardware accelerator. Each work unit executes the computation phase of the current task based on the tensor data required by the task. If the current task has issued all data transfer instructions and the available space in the on-chip storage is greater than the tensor data required by the next task, then the preloading phase of the next task is executed simultaneously.

[0046] In this embodiment, when each work unit executes a task, it first performs the preloading phase of that task, i.e., sends a data transfer instruction to the device storage interface of the hardware accelerator to read the input tensor data required by the task from off-chip device storage to on-chip fast storage. Since the data transfer operation is completed by an independent data transfer engine, the computing core of the work unit does not need to block and wait for the data transfer to complete after issuing the data transfer instruction, but can continue to execute other operations. Based on this, the present invention realizes cross-task software pipeline. Specifically, after the current task's computing phase begins execution, the work unit determines whether the pipeline overlap conditions are met. The first condition is that the current task has issued all the data transfer instructions it needs, which means that the data prefetching work of the current task has been fully started, and the work unit no longer needs to initiate a new data transfer for this task. The second condition is that the currently available free space in the on-chip storage can accommodate all the data to be prefetched by the next task. When the above two conditions are met simultaneously, the work unit starts the preloading phase of the next task in advance while the current task's computing phase is still executing, transferring the data required by the next task from device storage to the unoccupied free pages in the on-chip storage. In this way, the data movement phase of the subsequent task overlaps with the computation phase of the preceding task in time, eliminating the waiting time forced into serialization due to kernel barriers in the traditional operator-by-operator execution model. It should be noted that this overlap is predicated on a conflict-free constraint for data movement instructions; that is, the data movement of the next task and the data movement currently in progress in the current task will not conflict with the data movement in the same memory region. The work unit ensures that this constraint is satisfied through the page allocation status of on-chip memory.

[0047] See Figure 7A , Figure 7B This illustration shows a schematic diagram of cross-task software pipeline and paging on-chip memory management provided by an embodiment of the present invention. Figure 7A The execution timing relationship of two adjacent tasks T1 and T2 within a single work unit is shown. After the preloading phase of T1 is completed, it enters the computation phase. During the execution of the computation phase of T1, the preloading phase of T2 starts simultaneously. The two phases overlap on the timeline. Figure 7B The on-chip fast storage paging management mechanism is demonstrated. The entire on-chip storage space is divided into multiple fixed-size pages. During execution, T1 occupies several pages to store its input and intermediate data. When T1's calculation phase is completed, these pages are immediately released. T2's preloading phase then acquires these released pages to prefetch the data it needs.

[0048] In one possible implementation, the on-chip memory of the hardware accelerator is divided into multiple fixed-size pages; each work unit maintains at least one task queue, from which tasks are retrieved and executed, including: Each work unit retrieves the current task from the task queue, dynamically acquires pages based on the storage space required by the current task, releases the corresponding pages after completing the current task, and does not acquire pages for the current task again; While releasing the page corresponding to the current task, pre-allocate available pages for the next task and perform data pre-fetching.

[0049] In this embodiment, the on-chip fast storage of the hardware accelerator is divided into multiple fixed-size pages. The size of each page is a pre-configured hardware parameter, which can be set to match the burst transfer size of the hardware accelerator's data transfer engine to achieve the highest data transfer efficiency. When generating the device code for each task, the compiler estimates the amount of on-chip storage space required by the task during execution and encodes this information in the task node's metadata. When a work unit starts executing a task, it requests an integer number of pages from the storage management unit based on the storage space required by the task. The allocated pages are used to store the input tensor data transferred from device storage during the preloading phase and the intermediate results generated during the computation phase. During execution, the task can also dynamically acquire more pages as needed, for example, when the size of some intermediate results is only fully determined at runtime. However, once the task begins to release the pages it occupies, it means that the task has entered the final stage of execution and no longer needs more on-chip storage resources. At this point, the task is no longer allowed to acquire any new pages. This monotonic usage pattern simplifies the page allocation and reclamation logic and avoids complex circular dependencies and deadlock problems. When the current task releases the pages it occupies, the storage management unit immediately marks these pages as free and pre-allocates these available pages for the next task to be executed. This pre-allocation operation can be performed in parallel with the remaining computation phase of the current task; that is, while the current task is still computing, the data prefetching of the next task has already begun using the just-released pages. Through the combination of the paging mechanism and the monotonic usage mode described above, time-division multiplexing of on-chip storage resources is achieved among multiple tasks. Different tasks use the same on-chip storage area at different times without resource conflicts. This fine-grained storage multiplexing method significantly improves the utilization rate of on-chip storage, enabling multiple tasks that would otherwise not be able to be loaded into on-chip storage simultaneously to be executed smoothly through time-division sharing, providing the necessary storage resource guarantee for the aforementioned cross-task software pipeline.

[0050] This invention expresses fine-grained synchronization relationships through a dependency graph composed of task nodes and event nodes. The entire computation graph is compiled into a single persistent kernel, eliminating global kernel barriers at operator boundaries. Event nodes provide the graph structure foundation for runtime event-driven scheduling, refining the synchronization granularity between tasks from the operator level to the task level. This allows the preloading and computation phases of adjacent tasks to overlap in time, avoiding pipeline bubbles caused by waiting for the entire operator to complete. The physical computing core of the hardware accelerator is statically divided into scheduling units and working units. An event queue is maintained within the scheduling unit to achieve event-driven distribution, ensuring that all scheduling logic is completed in a closed loop within the hardware accelerator. The master processor does not need to participate in any runtime scheduling decisions, eliminating inter-chip scheduling communication latency in master-slave mode. Working units maintain task queues locally, sending notifications directly to the scheduling unit after task completion. Synchronization overhead is only transmitted between cores within the accelerator. The scheduling unit senses the task queue status of each working unit and distributes tasks accordingly, achieving dynamic load balancing among working units. These mechanisms collectively shorten the scheduling critical path, reduce inference latency, and improve the inference efficiency of deep learning models.

[0051] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0052] The following are device embodiments of the present invention. For details not described in detail, please refer to the corresponding method embodiments described above.

[0053] A kernel compilation apparatus for hardware accelerators includes: The acquisition module is used to acquire the computational graph of the target model; The decomposition module is used to decompose each operator in the computation graph into multiple fine-grained tasks; The building module is used to construct a task dependency graph based on the data dependencies between each fine-grained task, treating each fine-grained task as a task node and each preceding task synchronization condition of a task node as an event node. The adjustment module is used to perform event fusion, normalization, and linearization on the task dependency graph to obtain a linearized task dependency graph. A generation module is used to generate the kernel of the target model based on the linearized task dependency graph. The kernel is used to deploy on a hardware accelerator and execute the inference of the target model.

[0054] In one possible implementation, the adjustment module is specifically used for: Multiple event nodes with the same set of preceding task nodes in the task dependency graph are merged into one event node to obtain the task dependency graph after event fusion. In the task dependency graph after event fusion, empty task nodes are introduced and equivalent transformations are performed so that each task node depends on at most one event node and triggers at most one event node, resulting in a normalized task dependency graph. The normalized task dependency graph is topologically sorted using a breadth-first search algorithm, and a continuous index is assigned to each task node to obtain a linearized task dependency graph.

[0055] In one possible implementation, the adjustment module is specifically used for: For each task node, if the task node triggers multiple event nodes, a new event node and multiple empty task nodes are introduced, so that the task node triggers the new event node, the new event node triggers multiple empty task nodes, and each empty task node triggers an original event node. For each task node, if the task node depends on multiple event nodes, multiple empty task nodes and a new event node are introduced, so that each event node triggers an empty task node, multiple empty task nodes trigger a new event node, and the new event node triggers the task node.

[0056] In one possible implementation, the generation module is specifically used for: Task nodes whose execution time depends on the data are marked as instant start mode, and task nodes with fixed execution time are marked as prefetch start mode.

[0057] A kernel compilation device for hardware accelerators is applied to a hardware accelerator, wherein the computing unit of the hardware accelerator is divided into a working unit and a scheduling unit, and the hardware accelerator is loaded with a kernel of a target model obtained by the aforementioned kernel compilation method for hardware accelerators. The device includes: The scheduling module is used to control each scheduling unit to maintain an event queue, monitor the activation status of event nodes in the kernel, and when any event node is activated, distribute the task node triggered by that event node to the task queue. The work module is used to control each work unit to maintain at least one task queue, retrieve tasks from the task queue for execution, and send a notification to the scheduling unit after the task is completed to trigger the activation of the corresponding event node.

[0058] In one possible implementation, the task queue includes an immediate queue and a prefetch queue. Tasks in the immediate queue correspond to data-dependent task nodes, while tasks in the prefetch queue correspond to task nodes with fixed execution times. The working module is specifically used for: Each work unit maintains an immediate queue and a prefetch queue. If the immediate queue is not empty, a task is retrieved from the immediate queue for execution; if the immediate queue is empty, a task is retrieved from the prefetch queue for execution.

[0059] In one possible implementation, the working module is specifically used for: Each work unit is controlled to retrieve the current task from the task queue, execute the preloading phase of the current task, and move the tensor data required by the current task from the device storage to the on-chip storage of the hardware accelerator. Each working unit is controlled to execute the computation phase of the current task based on the tensor data required by the task. If the current task has issued all data transfer instructions and the available space in the on-chip storage is greater than the tensor data required by the next task, the preloading phase of the next task is executed simultaneously.

[0060] In one possible implementation, the on-chip memory of the hardware accelerator is divided into multiple fixed-size pages; the working module is specifically used for: Each work unit is controlled to retrieve the current task from the task queue, dynamically acquire pages according to the storage space required by the current task, release the corresponding pages after the current task is completed, and no longer acquire pages for the current task. While releasing the page corresponding to the current task, pre-allocate available pages for the next task and perform data pre-fetching.

[0061] This invention expresses fine-grained synchronization relationships through a dependency graph composed of task nodes and event nodes. The entire computation graph is compiled into a single persistent kernel, eliminating global kernel barriers at operator boundaries. Event nodes provide the graph structure foundation for runtime event-driven scheduling, refining the synchronization granularity between tasks from the operator level to the task level. This allows the preloading and computation phases of adjacent tasks to overlap in time, avoiding pipeline bubbles caused by waiting for the entire operator to complete. The physical computing core of the hardware accelerator is statically divided into scheduling units and working units. An event queue is maintained within the scheduling unit to achieve event-driven distribution, ensuring that all scheduling logic is completed in a closed loop within the hardware accelerator. The master processor does not need to participate in any runtime scheduling decisions, eliminating inter-chip scheduling communication latency in master-slave mode. Working units maintain task queues locally, sending notifications directly to the scheduling unit after task completion. Synchronization overhead is only transmitted between cores within the accelerator. The scheduling unit senses the task queue status of each working unit and distributes tasks accordingly, achieving dynamic load balancing among working units. These mechanisms collectively shorten the scheduling critical path, reduce inference latency, and improve the inference efficiency of deep learning models.

[0062] Figure 8 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. For example... Figure 8As shown, the electronic device 8 of this embodiment includes a processor 80 and a memory 81. The memory 81 stores a computer program 82. When the processor 80 executes the computer program 82, it implements the steps in the various method embodiments described above. Alternatively, when the processor 80 executes the computer program 82, it implements the functions of each module / unit in the various device embodiments described above.

[0063] For example, computer program 82 may be divided into one or more modules / units, which are stored in memory 81 and executed by processor 80 to complete the present invention. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of computer program 82 in electronic device 8.

[0064] Electronic device 8 may include, but is not limited to, processor 80 and memory 81. Those skilled in the art will understand that... Figure 8 This is merely an example of electronic device 8 and does not constitute a limitation on electronic device 8. It may include more or fewer components than shown, or combine certain components, or different components. For example, electronic device 8 may also include input / output devices, network access devices, buses, etc.

[0065] The processor 80 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0066] The memory 81 can be an internal storage unit of the electronic device 8, such as a hard disk or RAM. The memory 81 can also be an external storage device of the electronic device 8, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory 81 can include both internal and external storage units of the electronic device 8. The memory 81 is used to store the computer program 82 and other programs and data required by the electronic device 8. The memory 81 can also be used to temporarily store data that has been output or will be output.

[0067] For the sake of simplicity and clarity, only the above-described functional modules / units are used as examples. In practical applications, the functions described above can be assigned to different functional modules / units as needed. These modules / units can be implemented in hardware, software, or a combination of both.

[0068] This invention also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the methods described in the above-described method embodiments.

[0069] This invention also provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements the methods described in the above-described method embodiments.

[0070] Computer programs include computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. Computer-readable media can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0071] In the above embodiments, the descriptions of each embodiment have their own emphasis. Parts not detailed or described in a particular embodiment can be referred to in the relevant descriptions of other embodiments. Unless otherwise specified or in conflict with logic, the terminology and / or descriptions between different embodiments are consistent and can be referenced interchangeably. Technical features in different embodiments can be combined to form new embodiments based on their inherent logical relationships.

[0072] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A kernel compilation method for hardware accelerators, characterized in that, include: Obtain the computational graph of the target model; Each operator in the computation graph is decomposed into multiple fine-grained tasks; Based on the data dependencies between each fine-grained task, each fine-grained task is treated as a task node, and each preceding task synchronization condition of a task node is treated as an event node, thus constructing a task dependency graph. The task dependency graph is subjected to event fusion, normalization, and linearization to obtain a linearized task dependency graph. The kernel of the target model is generated based on the linearized task dependency graph. The kernel is used to deploy on a hardware accelerator and perform inference of the target model.

2. The kernel compilation method for hardware accelerators according to claim 1, characterized in that, The process of performing event fusion, normalization, and linearization on the task dependency graph to obtain a linearized task dependency graph includes: Multiple event nodes with the same set of preceding task nodes in the task dependency graph are merged into one event node to obtain the event-fused task dependency graph. In the task dependency graph after event fusion, empty task nodes are introduced and equivalent transformations are performed so that each task node depends on at most one event node and triggers at most one event node, resulting in a normalized task dependency graph. The normalized task dependency graph is topologically sorted using a breadth-first search algorithm, and a continuous index is assigned to each task node to obtain a linearized task dependency graph.

3. The kernel compilation method for hardware accelerators according to claim 2, characterized in that, The process of introducing empty task nodes into the task dependency graph after event fusion and performing equivalent transformations, such that each task node depends on at most one event node and triggers at most one event node, results in a normalized task dependency graph, including: For each task node, if the task node triggers multiple event nodes, a new event node and multiple empty task nodes are introduced, so that the task node triggers the new event node, the new event node triggers the multiple empty task nodes, and each empty task node triggers an original event node. For each task node, if the task node depends on multiple event nodes, multiple empty task nodes and a new event node are introduced, so that each event node triggers an empty task node, the multiple empty task nodes trigger the new event node, and the new event node triggers the task node.

4. The kernel compilation method for hardware accelerators according to claim 1, characterized in that, The process of generating the kernel of the target model based on the linearized task dependency graph includes: Task nodes whose execution time depends on the data are marked as instant start mode, and task nodes with fixed execution time are marked as prefetch start mode.

5. A kernel execution method for hardware accelerators, characterized in that, Applied to a hardware accelerator, wherein the computing unit of the hardware accelerator is divided into a working unit and a scheduling unit, and the kernel of the target model obtained by the method described in any one of claims 1 to 4 is loaded on the hardware accelerator. The method includes: Each scheduling unit maintains an event queue, monitors the activation status of event nodes in the kernel, and when any event node is activated, distributes the task node triggered by that event node to the task queue. Each work unit maintains at least one task queue, retrieves tasks from the task queue for execution, and sends a notification to the scheduling unit after the task execution is completed to trigger the activation of the corresponding event node.

6. The kernel execution method for hardware accelerators according to claim 5, characterized in that, The task queue includes an immediate queue and a prefetch queue. Tasks in the immediate queue correspond to task nodes whose execution time is data-dependent, while tasks in the prefetch queue correspond to task nodes with fixed execution time. Each work unit maintains at least one task queue, and retrieves tasks from the task queue for execution, including: Each work unit maintains an immediate queue and a prefetch queue. If the immediate queue is not empty, tasks are retrieved from the immediate queue for execution; if the immediate queue is empty, tasks are retrieved from the prefetch queue for execution.

7. The kernel execution method for hardware accelerators according to claim 5, characterized in that, Each work unit maintains at least one task queue, and retrieves tasks from the task queue for execution, including: Each work unit retrieves the current task from the task queue, performs the preloading phase of the current task, and moves the tensor data required by the current task from the device storage to the on-chip storage of the hardware accelerator. Each work unit executes the computation phase of the current task based on the tensor data required by the task. If the current task has issued all data transfer instructions and the available space in the on-chip storage is greater than the tensor data required by the next task, then the preloading phase of the next task is executed simultaneously.

8. The kernel execution method for hardware accelerators according to claim 7, characterized in that, The on-chip memory of the hardware accelerator is divided into multiple fixed-size pages; each working unit maintains at least one task queue, and retrieves tasks from the task queue for execution, including: Each work unit retrieves the current task from the task queue, dynamically acquires pages based on the storage space required by the current task, releases the corresponding pages after completing the current task, and does not acquire pages for the current task again; While releasing the page corresponding to the current task, pre-allocate available pages for the next task and perform data pre-fetching.

9. A kernel compilation device for hardware accelerators, characterized in that, include: The acquisition module is used to acquire the computational graph of the target model; The decomposition module is used to decompose each operator in the computation graph into multiple fine-grained tasks; The building module is used to construct a task dependency graph based on the data dependencies between each fine-grained task, treating each fine-grained task as a task node and each preceding task synchronization condition of a task node as an event node. The adjustment module is used to perform event fusion, normalization, and linearization on the task dependency graph to obtain a linearized task dependency graph. A generation module is used to generate the kernel of the target model based on the linearized task dependency graph. The kernel is used to deploy on a hardware accelerator and execute the inference of the target model.

10. A kernel compilation device for hardware accelerators, characterized in that, Applied to a hardware accelerator, wherein the computing unit of the hardware accelerator is divided into a working unit and a scheduling unit, and the kernel of the target model obtained by the method described in any one of claims 1 to 4 is loaded on the hardware accelerator. The device includes: The scheduling module is used to control each scheduling unit to maintain an event queue, monitor the activation status of event nodes in the kernel, and when any event node is activated, distribute the task node triggered by that event node to the task queue. The work module is used to control each work unit to maintain at least one task queue, retrieve tasks from the task queue for execution, and send a notification to the scheduling unit after the task is completed to trigger the activation of the corresponding event node.

Images