Dynamic binding methods for task flows and hardware submission queues, computer devices, and storage media.
By dynamically binding task flows to hardware submission queues, the problem of low resource utilization of artificial intelligence chips under static binding is solved, realizing efficient utilization of hardware resources and high-concurrency computing power scheduling.
Patent Information
- Application Number
- CN202611274142.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-20
- Publication Date
- 2026-11-10
AI Technical Summary
In existing technologies, the static binding of task flow and hardware submission queue in artificial intelligence chips results in low resource utilization, which cannot meet the needs of high-concurrency computing power scheduling, and causes problems such as head-of-line blocking and idle hardware resources.
A dynamic binding method between task flow and hardware submission queue is adopted. By determining the rebinding conditions before the task flow command is submitted, the current binding relationship is removed and the task flow is migrated to an idle hardware submission queue, thereby realizing the dynamic adjustment of task flow and hardware submission queue.
It improves the overall resource utilization of artificial intelligence chips, solves the head-of-line blocking problem, realizes efficient use of hardware resources, and meets the needs of high-concurrency computing power scheduling.
Smart Images

Figure CN123068098A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, and in particular to a method for dynamically binding task flows to hardware submission queues, computer devices, and storage media. Background Technology
[0002] In the computing architecture of artificial intelligence (AI) chips, the host computer submits various computing tasks to the AI chip, such as kernel function startup, memory copying, and memory filling, based on task flows. Task flows, as a software-layer task submission abstraction, require instruction issuance and execution through the hardware submission queue at the AI chip's hardware layer. In scenarios such as deep learning, high-performance computing, and scientific computing, AI chips typically need to handle a large number of concurrent task scheduling tasks. The mapping and matching efficiency between task flows and hardware submission queues directly affects the utilization rate of AI chip hardware resources and the performance of concurrent task execution.
[0003] In related technologies, AI chip driver architectures employ a static binding mapping mechanism. The correspondence between task flows and hardware submission queues is determined during the task flow creation phase and remains fixed throughout the entire runtime lifecycle of the task flow. Drivers typically use polling or modulo methods to complete the mapping, binding each task flow to its corresponding hardware submission queue. When the number of user-created task flows exceeds the total number of hardware submission queues, multiple task flows will share the same hardware submission queue.
[0004] However, this static binding method has obvious drawbacks. Commands from multiple task streams that share the same hardware submission queue can only be executed serially. As a result, the lengthy tasks at the beginning will block the execution of all subsequent tasks. This not only significantly reduces the command submission efficiency in multi-stream concurrent scenarios, but also causes the hardware submission queue resources to be idle. Ultimately, this leads to low overall resource utilization of the AI chip, which cannot meet the needs of high-concurrency computing power scheduling. Summary of the Invention
[0005] Therefore, it is necessary to provide a method, computer device, and storage medium for dynamically binding task flow and hardware submission queue that can improve the resource utilization of artificial intelligence chips, in order to address the above-mentioned technical problems.
[0006] Firstly, this application provides a method for dynamically binding a task flow to a hardware submission queue, the method comprising:
[0007] For a target task flow, before the command of the target task flow is submitted to the first hardware submission queue, it is determined whether the rebinding condition is met. The first hardware submission queue is the hardware submission queue currently bound to the target task flow. The rebinding condition includes the target task flow being in an idle state and the first hardware submission queue being in a busy state.
[0008] When the rebinding condition is met, find the second hardware submission queue that is currently idle from each hardware submission queue, and remove the binding relationship between the target task flow and the first hardware submission queue, and establish the binding relationship between the target task flow and the second hardware submission queue.
[0009] The commands of the target task flow are submitted to the second hardware submission queue.
[0010] In one embodiment, determining whether the rebinding condition is currently met includes:
[0011] Determine the state of the target task flow and the state of the first hardware submission queue;
[0012] When the target task flow is in an idle state and the first hardware submission queue is in a busy state, it is determined that the rebinding condition is met.
[0013] In one embodiment, the method further includes:
[0014] When the target task flow is in a busy state and / or the first hardware submission queue is in an idle state, it is determined that the rebinding condition is not currently met.
[0015] The commands of the target task flow are submitted to the first hardware submission queue.
[0016] In one embodiment, determining the state of the target task flow includes:
[0017] Obtain the task count value corresponding to the target task flow and the first count value corresponding to the first hardware submission queue. The task count value is the total number of commands in the first hardware submission queue when the target task flow last submitted a command to the first hardware submission queue, and the first count value is the total number of commands that have been completed in the first hardware submission queue.
[0018] When the task count value is less than or equal to the first count value, the target task flow is determined to be in an idle state; or, when the task count value is greater than the first count value, the target task flow is determined to be in a busy state.
[0019] In one embodiment, determining the state of the first hardware submission queue includes:
[0020] Obtain the first count value and the second count value corresponding to the first hardware submission queue. The first count value is the total number of commands completed in the first hardware submission queue, and the second count value is the total number of commands currently submitted in the first hardware submission queue.
[0021] When the first count value equals the second count value, the first hardware submission queue is determined to be in an idle state; or, when the first count value is less than the second count value, the first hardware submission queue is determined to be in a busy state.
[0022] In one embodiment, the step of searching for the second hardware submission queue that is currently idle from each hardware submission queue includes:
[0023] Traverse each hardware submission queue. When the currently traversed hardware submission queue is in an idle state, end the traversal operation and designate the currently traversed hardware submission queue as the second hardware submission queue.
[0024] In one embodiment, the method further includes:
[0025] When creating the target task flow, if multiple hardware submission queues have been created in the queue pool, the reference counting mapping table is traversed. The reference counting mapping table is used to record the reference count corresponding to each hardware submission queue. The reference count is used to represent the number of task flows bound to the hardware submission queue.
[0026] The first hardware submission queue is determined based on the reference count of each hardware submission queue, and the binding relationship between the target task flow and the first hardware submission queue is established.
[0027] The reference count of the first hardware submission queue in the reference count mapping table is updated based on the incrementing rule.
[0028] In one embodiment, determining the first hardware submission queue based on the reference count of each of the hardware submission queues includes:
[0029] The traversal operation ends when the reference count of the currently traversed hardware submission queue is 0, and the currently traversed hardware submission queue is designated as the first hardware submission queue; or,
[0030] If there is no hardware commit queue with a reference count of 0, the hardware commit queue with the smallest reference count among all the hardware commit queues shall be designated as the first hardware commit queue.
[0031] In one embodiment, the method further includes:
[0032] After ending the binding relationship between the target task flow and the first hardware submission queue, the reference count of the first hardware submission queue in the reference count mapping table is updated based on the decrement rule;
[0033] After establishing the binding relationship between the target task flow and the second hardware submission queue, the reference count of the second hardware submission queue in the reference count mapping table is updated based on the increment rule.
[0034] Secondly, this application also provides a dynamic binding device for task flow and hardware submission queue, comprising:
[0035] The first determining module is used to determine whether the rebinding conditions are met for the target task flow before the command of the target task flow is submitted to the first hardware submission queue. The first hardware submission queue is the hardware submission queue currently bound to the target task flow. The rebinding conditions include the target task flow being in an idle state and the first hardware submission queue being in a busy state.
[0036] The rebinding module is used to, when the rebinding condition is met, search for the second hardware submission queue that is currently idle from each hardware submission queue, and remove the binding relationship between the target task flow and the first hardware submission queue, and establish the binding relationship between the target task flow and the second hardware submission queue.
[0037] The first submission module is used to submit the commands of the target task flow to the second hardware submission queue.
[0038] In one embodiment, determining whether the rebinding condition is currently met includes:
[0039] Determine the state of the target task flow and the state of the first hardware submission queue;
[0040] When the target task flow is in an idle state and the first hardware submission queue is in a busy state, it is determined that the rebinding condition is met.
[0041] In one embodiment, the device further includes:
[0042] The second determining module is used to determine that the rebinding condition is not currently met when the state of the target task flow is busy and / or the state of the first hardware submission queue is idle.
[0043] The second submission module is used to submit the commands of the target task flow to the first hardware submission queue.
[0044] In one embodiment, determining the state of the target task flow includes:
[0045] Obtain the task count value corresponding to the target task flow and the first count value corresponding to the first hardware submission queue. The task count value is the total number of commands in the first hardware submission queue when the target task flow last submitted a command to the first hardware submission queue, and the first count value is the total number of commands that have been completed in the first hardware submission queue.
[0046] When the task count value is less than or equal to the first count value, the target task flow is determined to be in an idle state; or, when the task count value is greater than the first count value, the target task flow is determined to be in a busy state.
[0047] In one embodiment, determining the state of the first hardware submission queue includes:
[0048] Obtain the first count value and the second count value corresponding to the first hardware submission queue. The first count value is the total number of commands completed in the first hardware submission queue, and the second count value is the total number of commands currently submitted in the first hardware submission queue.
[0049] When the first count value equals the second count value, the first hardware submission queue is determined to be in an idle state; or, when the first count value is less than the second count value, the first hardware submission queue is determined to be in a busy state.
[0050] In one embodiment, the step of searching for the second hardware submission queue that is currently idle from each hardware submission queue includes:
[0051] Traverse each hardware submission queue. When the currently traversed hardware submission queue is in an idle state, end the traversal operation and designate the currently traversed hardware submission queue as the second hardware submission queue.
[0052] In one embodiment, the device further includes:
[0053] The traversal module is used to traverse the reference count mapping table when creating the target task flow if multiple hardware submission queues have been created in the queue pool. The reference count mapping table is used to record the reference count corresponding to each hardware submission queue, and the reference count is used to represent the number of task flows bound to the hardware submission queue.
[0054] A binding module is used to determine the first hardware submission queue based on the reference count of each hardware submission queue, and to establish a binding relationship between the target task flow and the first hardware submission queue.
[0055] The first update module is used to update the reference count of the first hardware submission queue in the reference count mapping table based on an incrementing rule.
[0056] In one embodiment, determining the first hardware submission queue based on the reference count of each of the hardware submission queues includes:
[0057] The traversal operation ends when the reference count of the currently traversed hardware submission queue is 0, and the currently traversed hardware submission queue is designated as the first hardware submission queue; or,
[0058] If there is no hardware commit queue with a reference count of 0, the hardware commit queue with the smallest reference count among all the hardware commit queues shall be designated as the first hardware commit queue.
[0059] In one embodiment, the device further includes:
[0060] The second update module is used to update the reference count of the first hardware submission queue in the reference count mapping table based on a decrementing rule after the binding relationship between the target task flow and the first hardware submission queue is terminated; and,
[0061] After establishing the binding relationship between the target task flow and the second hardware submission queue, the reference count of the second hardware submission queue in the reference count mapping table is updated based on the increment rule.
[0062] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the dynamic binding method of task flow and hardware submission queue of any of the above.
[0063] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the dynamic binding method of task flow and hardware submission queue of any of the above.
[0064] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements a method for dynamically binding the task flow and hardware submission queue of any of the above.
[0065] The aforementioned method, computer device, and storage medium for dynamically binding task flows to hardware submission queues, for a target task flow, before the command of the target task flow is submitted to the first hardware submission queue, determine whether the rebinding conditions are currently met. The first hardware submission queue is the hardware submission queue currently bound to the target task flow. The rebinding conditions include the target task flow being in an idle state and the first hardware submission queue being in a busy state. If the rebinding conditions are met, find the second hardware submission queue that is currently in an idle state from among the hardware submission queues, and remove the binding relationship between the target task flow and the first hardware submission queue, and establish the binding relationship between the target task flow and the second hardware submission queue, and submit the command of the target task flow to the second hardware submission queue. Using the dynamic binding method, computer device, and storage medium for task flows to hardware submission queues provided in this application embodiment, the binding relationship between task flows and hardware submission queues can be dynamically adjusted. When the hardware submission queue bound to a task flow is in a busy state, a task flow that can be migrated can be rebound to an idle hardware submission queue. In this way, efficient utilization of hardware submission queue resources can be achieved, improving the overall resource utilization rate of artificial intelligence chips, thereby meeting the needs of high-concurrency computing power scheduling. Attached Figure Description
[0066] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0067] Figure 1 This is a flowchart illustrating a method for dynamically binding task flow to a hardware submission queue in one embodiment.
[0068] Figure 2 This is a schematic diagram of the heterogeneous hardware platform in one embodiment;
[0069] Figure 3 This is a schematic diagram of the structure of an artificial intelligence chip in one embodiment;
[0070] Figure 4 This is a flowchart illustrating the rebinding condition determination process in one embodiment;
[0071] Figure 5 This is a flowchart illustrating the task flow state determination process in one embodiment;
[0072] Figure 6 This is a flowchart illustrating the hardware submission queue status determination process in one embodiment;
[0073] Figure 7 This is a flowchart illustrating the initial binding process in one embodiment;
[0074] Figure 8 This is a flowchart illustrating the process of updating the reference count map in one embodiment;
[0075] Figure 9 Here is a flowchart of the initial queue binding process during task flow creation in one embodiment;
[0076] Figure 10 This is a flowchart illustrating the process of rebinding the task flow to the hardware submission queue in one embodiment;
[0077] Figure 11 This is a structural block diagram of a dynamic binding device for task flow and hardware submission queue in one embodiment;
[0078] Figure 12 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation
[0079] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0080] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.
[0081] In the traditional driver architecture of AI chips, the mapping between task streams (also represented as streams) and hardware submission queues (also represented as submit queues) is determined when the task stream is created and remains unchanged throughout the task stream's lifecycle. This static binding method involves the driver assigning a fixed hardware submission queue to each task stream upon creation. The allocation strategy typically employs a simple round-robin or modulo mapping, mapping the i-th task stream to the i-th mod N (i-th task stream, modulo the total number of hardware submission queues N) hardware submission queue.
[0082] When the number of created task flows exceeds the total number of hardware commit queues, multiple task flows can share the same hardware commit queue. In this case, commands from all task flows sharing the same hardware commit queue are executed sequentially in the order of commit. This static binding method has the following technical drawbacks:
[0083] (1) Head-of-line blocking problem: When multiple task flows share the same hardware submission queue, if a task flow A has submitted a command that has not yet been completed, and a task flow B submits a new command after task flow A, then task flow B's command must wait for task flow A's command to be completed before it can be executed by the AI chip. More seriously, when all commands of task flow A have been completed (task flow A is in an idle state), but there are still commands from other task flows being executed in the hardware submission queue, task flow A is still forced to wait for the hardware submission queue to be emptied when submitting a new command, even if other hardware submission queues may be completely idle at this time. This phenomenon of "idle task flows being forced to wait for busy hardware submission queues" is a direct manifestation of head-of-line blocking.
[0084] (2) Low utilization of hardware submission queue resources: Under static binding, the load between hardware submission queues is uneven. Some hardware submission queues may be overloaded due to binding multiple active task flows, while other hardware submission queues may be completely idle. Since the binding relationship is immutable, the driver cannot schedule subsequent commands of idle task flows to idle hardware submission queues, resulting in a waste of hardware parallelism capabilities.
[0085] (3) Lack of runtime adaptability: The workload of AI chips changes dynamically during runtime. For example, a task flow may be highly active for a period of time and then become idle. Static binding cannot perceive such runtime load changes and make corresponding adjustments, resulting in the scheduling strategy always being based on static information at the time of creation, rather than the current actual state.
[0086] (4) Performance degradation in multi-stream creation scenarios: In high-concurrency scenarios (such as deep learning inference servers processing multiple requests simultaneously), users often create a large number of task streams to achieve task parallelism. When the number of task streams far exceeds the number of hardware queues, the mapping between the statically bound multi-task streams and the single hardware submission queue causes the head-of-queue blocking problem to worsen sharply, and the system throughput decreases as the number of streams increases.
[0087] To address the aforementioned technical issues, this application provides a method for dynamically binding task flows to hardware submission queues. In this method, the binding relationship between task flows and hardware submission queues can be dynamically adjusted. When the hardware submission queue to which a task flow is bound is busy, the task flow that can be migrated can be rebound to an idle hardware submission queue. In this way, efficient utilization of hardware submission queue resources can be achieved, improving the overall resource utilization of artificial intelligence chips and thus meeting the needs of high-concurrency computing power scheduling.
[0088] like Figure 1 As shown, a method for dynamically binding task flows to hardware submission queues is provided, applicable to heterogeneous hardware platforms equipped with CPUs (Central Processing Units) and AI chips. (Refer to...) Figure 2 As shown. In this embodiment of the application, the artificial intelligence chip can be any one of GPU (Graphics Processing Unit), TPU (Tensor Processing Unit), NPU (Neural Network Processing Unit), DPU (Deep Learning Processing Unit), APU (Accelerated Processing Unit), and GPGPU (General-Purpose Graphics Processing Unit). This embodiment of the application does not make specific limitations on this one, and the following description uses GPGPU as an example.
[0089] Reference Figure 3 The diagram shows a schematic of a GPGPU. A GPGPU is actually an array of Streaming Processor Clusters (SPCs), including, for example,... Figure 3 The diagram shows streaming processor clusters 1, ..., M, where M is a positive integer greater than 1. In a graphics processing unit (GPU), one streaming processor cluster processes one computational task, or multiple streaming processor clusters process one computational task. Multiple streaming processor clusters share data through a global cache or global memory.
[0090] like Figure 3 As shown, taking streaming processor cluster 1 as an example, one streaming processor cluster includes multiple computing units, such as... Figure 3The system is structured as Computation Unit 1, Computation Unit 2, ..., Computation Unit N, where N is a positive integer. Each Computation Unit (CU) performs arithmetic and logical operations other than matrix calculations such as matrix multiplication and convolution, including operations like accumulation, reduction, and standard addition, subtraction, multiplication, and division. A Computation Unit contains multiple cores (also called computational kernels), each including an Arithmetic Logic Unit (ALU), a floating-point unit, etc., which are used to execute specific computational tasks. Furthermore, the Computation Unit also includes registers (e.g., ...). Figure 3 The register file and shared cache in a computing unit are used to store source and destination data related to computing tasks in a hierarchical manner. The shared cache in a computing unit is used to share data between the cores of that computing unit.
[0091] In parallel computing, computational tasks are typically executed by multiple threads. These threads are divided into multiple thread blocks before execution in a general-purpose graphics processor (or parallel computing processor), and then dispatched via a thread block distribution module. Figure 3 (Not shown in the image) Multiple thread blocks are distributed to various computation units. All threads in a thread block must be assigned to the same computation unit for execution. Simultaneously, thread blocks are broken down into minimum execution thread bundles (or simply warps), each containing a fixed number (or less than this fixed number) of threads, for example, 32 threads. Multiple thread blocks can execute in the same computation unit or in different computation units.
[0092] In each computing unit, the thread beam scheduling / distribution module ( Figure 3 (Not shown in the diagram) Thread bundles are scheduled and allocated so that multiple computing cores within the computing unit can run thread bundles. Depending on the number of computing cores in the computing unit, multiple thread bundles within a thread block can be executed concurrently or in a time-sharing manner. Multiple threads within each thread bundle execute the same instructions. Memory execution instructions are issued to the shared cache within the computing unit or further issued to intermediate-level caches, global caches, or global memory for read and write operations, etc.
[0093] like Figure 3 As shown, the streaming processor cluster 1 also includes a tensor operation unit, which is used to perform tensor calculations, such as matrix multiplication, convolution operations, etc.
[0094] In one exemplary embodiment, such as Figure 1 As shown, a method for dynamically binding task flows to hardware submission queues is provided, including the following steps 102, 104, and 106. Wherein:
[0095] Step 102: For the target task flow, before the command of the target task flow is submitted to the first hardware submission queue, determine whether the rebinding conditions are met. The first hardware submission queue is the hardware submission queue currently bound to the target task flow. The rebinding conditions include the target task flow being in an idle state and the first hardware submission queue being in a busy state.
[0096] The method for dynamically binding task flows and hardware submission queues provided in this application can be applied to scenarios such as deep learning training, high-performance computing of artificial intelligence chips, and scientific computing to alleviate the head-of-queue blocking problem under concurrent scheduling of multiple task flows. To clearly describe this application, the following embodiments will use a deep learning inference task in an artificial intelligence chip as an example. The host generates multiple task flows carrying computational instructions, each containing commands such as kernel function startup, device memory copying, and video memory filling. The artificial intelligence chip hardware is configured with multiple hardware submission queues as channels for receiving commands from the host and sending them to the hardware computing unit.
[0097] Before the command for a task flow is issued to the hardware submission queue, the driver first distinguishes and determines the type of the task flow. For example, a task flow can include specified task flows and non-specified task flows. Specified task flows carry computational tasks with fixed priority and uninterrupted flow, and do not support dynamic adjustments to the hardware submission queue mapping relationship; that is, they are subject to binding lock constraints. Therefore, in this embodiment, the target task flows participating in the rebinding condition determination are all non-specified task flows that are not subject to binding lock constraints. Based on the confirmation that a task flow is a non-specified task flow, this task flow is used as the target task flow, and the driver performs rebinding condition determination on this target task flow.
[0098] In one exemplary embodiment, reference is made to Figure 4 As shown, in step 102, determining whether the rebinding condition is currently met can include the following steps 402 and 404, wherein:
[0099] Step 402: Determine the status of the target task flow and the status of the first hardware submission queue;
[0100] Step 404: When the target task flow is in an idle state and the first hardware submission queue is in a busy state, determine that the rebinding condition is met.
[0101] For example, the driver can determine the state of the target task flow and the state of the first hardware submission queue to which the target task flow is currently bound, and determine whether the rebinding condition is met based on the states of both. The rebinding condition includes the target task flow being in an idle state and the first hardware submission queue being in a busy state. The target task flow being in an idle state means that there are no commands submitted by the target task flow to be executed in the first hardware submission queue (i.e., commands submitted by the target task flow to the first hardware submission queue that have not yet been executed); the first hardware submission queue being in a busy state means that there are commands to be executed in the first hardware submission queue (i.e., commands that have not yet been executed).
[0102] In this embodiment, when the first hardware submission queue is in a busy state, it means that if the target task flow submits a command, it needs to wait for the first hardware submission queue to be empty. When the target task flow is in an idle state, it means that all commands submitted by the target task flow have been completed. At this time, it is not subject to hardware constraints and can perform queue migration. Therefore, at this time, the rebinding operation of the subsequent hardware submission queue can be triggered to rebind the target task flow to the idle hardware submission queue, so as to execute the commands submitted by the target task flow in a timely manner. This can improve the computing efficiency of the artificial intelligence chip while improving the utilization of submission resources.
[0103] Step 104: When the rebinding condition is met, find the second hardware submission queue that is currently idle from each hardware submission queue, and remove the binding relationship between the target task flow and the first hardware submission queue, and establish the binding relationship between the target task flow and the second hardware submission queue.
[0104] In this embodiment of the application, when it is determined that the rebinding condition is met, that is, the target task flow is currently in an idle state and the first hardware submission queue is currently in a busy state, the hardware submission queue that is currently in an idle state can be used as the second hardware submission queue and rebinded with the target task flow so that the commands of the target task flow can be submitted to the idle hardware submission queue for timely processing and avoid blocking.
[0105] In one exemplary embodiment, searching for a second hardware submission queue that is currently idle from the various hardware submission queues may include:
[0106] It can iterate through each hardware submission queue. When the currently traversed hardware submission queue is in an idle state, the traversal operation ends and the currently traversed hardware submission queue is used as the second hardware submission queue.
[0107] For example, all hardware submission queues in the system can be traversed, and the hardware submission queue without any queuing commands can be selected as the second hardware submission queue. For example, if the total number of hardware submission queues is N, the driver will sequentially traverse the states of the hardware condition queues numbered 0 to N-1, and select the first queue in the idle state as the second hardware submission queue.
[0108] In this embodiment of the application, after completing the filtering of the second hardware submission queue, the driver can clear the original mapping relationship between the target task flow and the first hardware submission queue, that is, remove the binding relationship between the target task flow and the first hardware submission queue, and generate a mapping relationship between the target task flow and the second hardware submission queue, that is, establish a binding relationship between the target task flow and the second hardware submission queue.
[0109] Step 106: Submit the commands of the target task flow to the second hardware submission queue.
[0110] In this embodiment of the application, after the binding relationship of the target task flow is updated, the commands to be issued in the target task flow are no longer sent to the original first hardware submission queue, but are uniformly submitted to the second hardware submission queue that has been rebounded. The second hardware submission queue then distributes the commands to the hardware computing unit of the artificial intelligence chip for execution.
[0111] In one exemplary embodiment, the above method may further include:
[0112] If the target task flow is in a busy state and / or the first hardware submission queue is in an idle state, determine that the rebinding conditions are not met; submit the commands of the target task flow to the first hardware submission queue.
[0113] In this embodiment of the application, when the target task flow is in a busy state, it indicates that there are commands submitted by the target task flow that have not yet been completed in the current first hardware submission queue. Therefore, the current target task flow is constrained by hardware resources and cannot perform task migration or rebinding of the hardware submission queue. Thus, regardless of the state of the first hardware submission queue currently bound to the target task flow, commands can only be submitted directly to the first hardware submission queue for processing.
[0114] When the first hardware submission queue is in an idle state, the command representing the current target task flow is submitted to the first hardware submission queue without any blocking phenomenon. It can be immediately distributed to the artificial intelligence chip for calculation. Therefore, regardless of whether the target task flow can be migrated at present, there is no need to rebind the hardware submission queue. The command can be directly submitted to the first hardware submission queue for processing.
[0115] The dynamic binding method for task flows and hardware submission queues provided in this application embodiment determines whether rebinding conditions are met before the command of the target task flow is submitted to the first hardware submission queue. The first hardware submission queue is the hardware submission queue currently bound to the target task flow. The rebinding conditions include the target task flow being idle and the first hardware submission queue being busy. If the rebinding conditions are met, the method searches for a second hardware submission queue that is currently idle, removes the binding relationship between the target task flow and the first hardware submission queue, establishes a binding relationship between the target task flow and the second hardware submission queue, and submits the command of the target task flow to the second hardware submission queue. Using the dynamic binding method for task flows and hardware submission queues provided in this application embodiment, the binding relationship between task flows and hardware submission queues can be dynamically adjusted. When the hardware submission queue bound to a task flow is busy, a task flow capable of task migration can be rebound to an idle hardware submission queue. This achieves efficient utilization of hardware submission queue resources, improves the overall resource utilization of AI chips, and meets the needs of high-concurrency computing power scheduling.
[0116] In one exemplary embodiment, reference is made to Figure 5 As shown, in step 402, determining the state of the target task flow may include the following steps 502 and 504, wherein:
[0117] Step 502: Obtain the task count value corresponding to the target task flow and the first count value corresponding to the first hardware submission queue. The task count value is the total number of commands in the first hardware submission queue when the target task flow last submitted a command to the first hardware submission queue, and the first count value is the total number of commands that have been completed in the first hardware submission queue.
[0118] Step 504: When the task count value is less than or equal to the first count value, determine that the target task flow is in an idle state; or, when the task count value is greater than the first count value, determine that the target task flow is in a busy state.
[0119] In this embodiment, when a target task flow submits a command to the first hardware submission queue, the total number of commands in the first hardware submission queue can be recorded. For example, assuming that when the target task flow submits command 'a' to the first hardware submission queue, the total number of commands in the first hardware submission queue is n1, then the task count value corresponding to this target task flow is recorded as n1; when the target task flow continues to submit command 'b' to the first hardware submission queue, the total number of commands in the first hardware submission queue is n2, then the task count value corresponding to this target task flow is updated to n2; that is, the task count value corresponding to the target task flow is always the total number of commands in the first hardware submission queue when the command was last submitted to the first hardware submission queue.
[0120] The hardware submission queue also records the total number of commands in the queue (i.e., the second count) and the total number of completed commands (i.e., the first count) in real time. After receiving a command submitted by a task flow, the hardware submission queue increments the total number of commands in the queue by 1 to obtain the updated second count; and after executing a command, the total number of completed commands in the hardware submission queue is incremented by 1 to obtain the updated first count. For example, assuming the current second count value corresponding to the first hardware submission queue is m1, after task flow A submits command a to the first hardware submission queue, the second count value is updated to m1+1; assuming the current first count value corresponding to the first hardware submission queue is m2, after command a is executed, the first count value is updated to m2+1.
[0121] The system can obtain the task count value corresponding to the target task flow and the first count value corresponding to the first hardware submission queue to determine whether the execution progress of the current first hardware submission queue has reached the command last submitted by the target task flow. When the task count value is less than or equal to the first count value, it can be determined that the execution progress of the current first hardware submission queue has reached the command last submitted by the target task flow, that is, all commands submitted by the target task flow have been executed, and there are no pending tasks for the target task flow in the current first hardware submission queue, so it can be determined that the target task flow is in an idle state; or, when the task count value is greater than the first count value, it can be determined that the execution progress of the current first hardware submission queue has not yet reached the command last submitted by the target task flow, that is, the command last submitted by the target task flow is still waiting to be executed in the first hardware submission queue, so it can be determined that the target task flow is in a busy state.
[0122] In one exemplary embodiment, reference is made to Figure 6 As shown, in step 402, determining the status of the first hardware submission queue may include the following steps 602 and 604, wherein:
[0123] Step 602: Obtain the first count value and the second count value corresponding to the first hardware submission queue. The first count value is the total number of commands completed in the first hardware submission queue, and the second count value is the total number of commands currently submitted in the first hardware submission queue.
[0124] Step 604: When the first count value equals the second count value, determine that the first hardware submission queue is in an idle state; or, when the first count value is less than the second count value, determine that the first hardware submission queue is in a busy state.
[0125] In this embodiment, the method for determining the first count value and the second count value corresponding to the first hardware submission queue can be referred to the foregoing embodiments, and will not be repeated here.
[0126] After obtaining the first count value and the second count value corresponding to the first hardware submission queue, the first count value and the second count value can be compared. When the first count value equals the second count value, it indicates that all commands in the first hardware submission queue have been executed, that is, the first hardware submission queue is in an idle state; when the first count value is less than the second count value, it indicates that there are still commands in the first hardware submission queue that have not been executed, that is, the first hardware submission queue is in a busy state.
[0127] In this way, the status of the first hardware submission queue and the status of the target task flow can be determined based on the task count value of the target task flow and the first and second count values of the first hardware submission queue. This allows for timely rebinding of the target task flow and the hardware submission queue when the target task flow is idle and the first hardware submission queue is busy. This effectively utilizes the idle hardware submission queue, improves hardware resource utilization, and enhances the computational efficiency of the artificial intelligence chip.
[0128] In one exemplary embodiment, reference is made to Figure 7 As shown, the above method may further include steps 702, 704, and 706, wherein:
[0129] Step 702: When creating the target task flow, if multiple hardware submission queues have been created in the queue pool, the reference counting mapping table is traversed. The reference counting mapping table is used to record the reference count corresponding to each hardware submission queue. The reference count is used to represent the number of task flows bound to the hardware submission queue.
[0130] Step 704: Determine the first hardware submission queue based on the reference count of each hardware submission queue, and establish a binding relationship between the target task flow and the first hardware submission queue;
[0131] Step 706: Update the reference count of the first hardware submission queue in the reference count mapping table based on the increment rule.
[0132] In this embodiment, the system pre-initializes a fixed number of hardware submission queues, which are stored uniformly in a queue pool to provide command submission channels for all host-side task flows. The system maintains an independent reference count mapping table, which records the reference count corresponding to each hardware submission queue in real time. This reference count represents the number of task flows bound to the hardware submission queue. For example, if a hardware submission queue is bound to task flow A and task flow B, its corresponding reference count is 2. The reference count of the hardware submission queue can be increased or decreased accordingly as the bound task flows change in real time.
[0133] It should be noted that if the hardware submission queue has not been initialized when a new task flow is created, that is, if multiple hardware submission queues have not been created in the queue pool, then multiple hardware submission queues can be created first.
[0134] If a ready-to-use hardware commit queue already exists in the queue pool at the moment a new task flow is created, the driver can actively traverse the complete reference counting map, read the current reference count of all hardware commit queues, and select the appropriate hardware commit queue as the first hardware commit queue based on the size of the reference count. For example, the hardware commit queue with the smallest reference count can be selected first, as this type of hardware commit queue currently has the fewest bound task flows, resulting in lower load pressure on the hardware commit queue.
[0135] In one exemplary embodiment, determining a first hardware submission queue based on the reference counts of each hardware submission queue may include:
[0136] When the reference count of the currently traversed hardware commit queue is 0, the traversal operation ends and the currently traversed hardware commit queue is designated as the first hardware commit queue; or, if there is no hardware commit queue with a reference count of 0, the hardware commit queue with the smallest reference count among all hardware commit queues is designated as the first hardware commit queue.
[0137] In this embodiment, the reference counting map can be traversed. If a hardware submission queue with a reference count of 0 is encountered, it means that the hardware submission queue has not yet been bound to any task flow, so the traversal can be ended, and the hardware submission queue can be directly used as the first hardware submission queue and bound to the newly created task flow. If the reference count encountered is greater than 0, the traversal continues until the entire reference counting map is traversed. The hardware submission queue with the smallest reference count is then used as the first hardware submission queue and bound to the newly created task flow.
[0138] After the first hardware submission queue is determined, the system can perform initial mapping binding, associating the newly created task flow with the first hardware submission queue. Subsequent command submission actions of the task flow will all point to the first hardware submission queue.
[0139] After completing the binding operation between the new task flow and the first hardware submission queue, since the first hardware submission queue has added a bound task flow and its load status has changed, the system can update the reference count of the first hardware submission queue according to the increment rule, such as: incrementing the reference count by 1, or after determining the increment magnitude according to the activity identifier of the task flow, updating the reference count according to the increment magnitude, and synchronously modifying the reference count corresponding to the first hardware submission queue in the reference count mapping table.
[0140] In the task flow creation phase, this application embodiment can implement preferential binding of hardware submission queues based on a reference counting mapping table, replacing the traditional static allocation method of fixed modulo and round-robin. By using the number of task flows bound to each hardware submission queue in real time as the allocation basis, newly created task flows are preferentially bound to low-load hardware submission queues, thus achieving load balancing during the task flow initialization phase.
[0141] In one exemplary embodiment, reference is made to Figure 8 As shown, the above method may further include steps 802 and 804, wherein:
[0142] Step 802: After ending the binding relationship between the target task flow and the first hardware submission queue, update the reference count of the first hardware submission queue in the reference count mapping table based on the decrement rule;
[0143] Step 804: After establishing the binding relationship between the target task flow and the second hardware submission queue, update the reference count of the second hardware submission queue in the reference count mapping table based on the increment rule.
[0144] In this embodiment, after the target task flow completes the rebinding operation, that is, the binding relationship between the target task flow and the first hardware submission queue ends and a new binding relationship between the target task flow and the second hardware submission queue is established, the reference count of the first hardware submission queue in the reference count mapping table can be updated according to a decrementing rule, and the reference count of the second hardware submission queue in the reference count mapping table can be updated according to an incrementing rule. The decrementing rule may include: decrementing the reference count by 1, or determining the decrementing magnitude according to the activity identifier of the task flow and updating the reference count according to the decrementing magnitude. The incrementing rule may include: incrementing the reference count by 1, or determining the incrementing magnitude according to the activity identifier of the task flow and updating the reference count according to the incrementing magnitude.
[0145] In this way, the reference count of each hardware submission queue can be updated in real time, which can guide the subsequent creation of task flows to prioritize binding to low-load hardware submission queues, thereby achieving load balancing during the task flow initialization phase.
[0146] To enable those skilled in the art to better understand the embodiments of this application, the embodiments of this application are described below through specific examples.
[0147] This application proposes a dynamic binding method for task flow and hardware submission queue. The core idea is that at each command submission moment, based on the real-time status of task flow and hardware submission queue, idle task flow is allowed to migrate from busy hardware submission queue to idle hardware submission queue, thereby realizing adaptive dynamic adjustment of the mapping relationship between task flow and hardware submission queue.
[0148] In this context, a task flow (which can be represented as a Stream) is a logical abstract unit for user-level task submission in the AI chip driver. It represents an ordered sequence of AI chip tasks. Tasks within the same task flow are executed in the order of submission, while tasks in different task flows can be executed in parallel. In this embodiment, the task flow is the subject of dynamic rebinding, holding a reference to the currently bound hardware submission queue. The hardware submission queue (which can be represented as a Submit Queue) is a physical channel used by the AI chip hardware to receive and execute commands. It corresponds to a command submission port of the AI chip hardware and has an independent command buffer and completion counter. In this embodiment, the hardware submission queue is the object of dynamic rebinding. Multiple task flows can share the same hardware submission queue, and task flows can switch to different hardware submission queues at runtime.
[0149] In this embodiment of the application, the fact that the task flow is idle but the hardware submission queue is busy is a sufficient condition for the binding state to be migrated: when all the submitted commands of the task flow have been completed (the task flow can be considered to be idle), but there are still commands of other task flows being executed in the hardware submission queue to which it is bound (the hardware submission queue can be considered to be busy), the task flow does not have any unfinished hardware dependencies, so it can be safely switched to other hardware submission queues.
[0150] In this embodiment, a hardware-based completion counter provides lock-free busy detection, including: the AI chip hardware can record the command completion progress in real time using an atomic counter (i.e., the total number of completed commands in the hardware submission queue, which can be represented as gpu_counter, and is asynchronously incremented by the AI chip hardware); it can record the maximum progress of submitted commands in the queue using a queue counter (i.e., the total number of commands submitted to the hardware submission queue by each bound task flow, which can be represented as gpu_wait_counter, and is incremented by the CPU software when a command is submitted); and it can record the maximum progress of submitted commands in the queue using a task flow counter (i.e., the total number of commands submitted to the hardware submission queue by each bound task flow at the time of the most recent command submission, which can be represented as wait_counter, and is incremented by the CPU software when a command is submitted); the software can determine whether the hardware submission queue is busy and whether the task flow is idle without acquiring any locks by comparing the current gpu_counter, gpu_wait_counter, and wait_counter.
[0151] In this embodiment, the command submission time of the task flow is the optimal rebinding time. Before each command submission, the task flow must be in a deterministic state (idle or busy). At this time, performing the rebinding check can ensure state consistency and will not introduce additional synchronization overhead.
[0152] Reference Figure 9 The diagram illustrates the initial queue binding process during task flow creation. It includes: upon receiving a task flow creation request, determining if the queue pool has been initialized. If the queue pool is not yet initialized, it can be initialized, i.e., N hardware submission queues are created. If the queue pool is initialized or has completed initialization, the reference counting map is traversed to find the hardware submission queue corresponding to the lowest reference count. This includes: checking if a hardware submission queue with a reference count of 0 exists; if so, directly selecting that hardware submission queue with a reference count of 0 as the task flow's binding queue; otherwise, selecting the hardware submission queue with the lowest reference count as the task flow's binding queue. The reference count of that hardware submission queue in the reference counting map is incremented by 1, and the hardware submission queue is bound to the task flow. Subsequently, commands from the task flow can be submitted to this bound hardware submission queue.
[0153] Figure 9 The flowchart shown illustrates the initial queue allocation process during task flow creation. The core strategy of the algorithm is least reference priority: it prioritizes selecting idle hardware submission queues (with a reference count of 0) that are not bound to any task flows; if none exist, it selects the hardware submission queue with the lowest reference count to maximize the load balance among queues. An early exit mechanism (stopping the search upon finding a hardware submission queue with a reference count of 0) effectively ensures allocation efficiency.
[0154] Reference Figure 10 The diagram illustrates the process of rebinding a task flow to the hardware submission queue. It includes: before submitting a command for the task flow, determining whether the task flow is the default flow (i.e., the specified task flow). If so, rebinding is not allowed, and the process returns to the hardware submission queue to which the task flow is bound, submitting the task flow's command to the currently bound hardware submission queue. If not, it further determines whether the task flow is in an idle state.
[0155] If the task flow is not in an idle state (i.e., in a busy state), rebinding is not allowed. The task flow is returned to the hardware submission queue to which it is bound, and the command of the task flow is submitted to the currently bound hardware submission queue. If the task flow is in an idle state, it is further determined whether the hardware submission queue to which the task flow is currently bound is in a busy state.
[0156] If the hardware submission queue is not busy (i.e., it is idle), rebinding is not allowed. The task flow is returned to the hardware submission queue it is bound to, and the task flow's commands are submitted to the currently bound hardware submission queue. If the hardware submission queue is busy, the system further searches for an idle hardware submission queue.
[0157] If no hardware submission queue is available, rebinding is not allowed. The task flow is returned to the hardware submission queue it is currently bound to, and the task flow's commands are submitted to the currently bound hardware submission queue. If an available hardware submission queue is available, the task flow's historical completion events (i.e., the task count value corresponding to the task flow) are cleared, the currently bound hardware submission queue is released, the reference count of the hardware submission queue is decremented by 1, the task flow is bound to the available hardware submission queue, the task flow counter is reset, and the task flow's commands are submitted to the newly bound hardware submission queue.
[0158] Figure 10 This demonstrates the complete decision-making process for dynamic rebinding. The algorithm executes before each command submission, determining whether rebinding is needed through a cascading check of three preconditions (default stream → stream idle → queue busy), thus avoiding unnecessary queue switching overhead. This includes:
[0159] Triple precondition checks: Default flow is skipped (ensuring the stability of the default flow); busy flow is skipped (unfinished work cannot be migrated); idle queue is skipped (no migration required). These three conditions together ensure the safety and necessity of rebinding. When rebinding is determined, an idle queue scan is performed: all hardware submission queues are traversed, and the completion counter is compared with the queue counter to determine if the hardware submission queue is idle. The first idle hardware submission queue encountered is selected as the hardware submission queue for rebinding. During the switching of hardware submission queues for rebinding: under the protection of the task flow's resource mutex lock, the following steps are executed sequentially: clearing historical completion events (i.e., clearing the task count value of the task flow for the previously bound hardware submission queue to avoid confusion between cross-queue counters), releasing the old queue reference, binding the new queue, and resetting the task flow counter, ensuring the atomicity of the switching process.
[0160] The following describes the process of determining the status of the task flow and the hardware submission queue, including:
[0161] Assume the hardware submission queue set in the queue pool is The task flow set is Where N is the number of hardware submission queues and M is the number of task flows.
[0162] For hardware submission queues Its busyness level is defined as:
[0163]
[0164] in, Represents the hardware submission queue The second count value, Represents the hardware submission queue The first count value; when At that time, it indicates the hardware submission queue. Busy; when At that time, it indicates the hardware submission queue. idle.
[0165] For task flow Its bound hardware submission queue is The idle time of a task flow is defined as:
[0166]
[0167] in, Represents task flow Task count value, Represents task flow The bound hardware submission queue is The first count value, in When =1, it indicates a task flow. Idle; otherwise, it indicates a task flow. busy.
[0168] Define the rebinding conditions. Task Flow Rebinding is required if and only if:
[0169]
[0170] in, When =1, it indicates a task flow. idle, Indicates the hardware submission queue is The level of busyness Indicates the hardware submission queue is Busy means that the target stream is idle but its bound hardware submission queue is busy (caused by other task streams that share the same hardware submission queue).
[0171] When the rebinding condition is met, the target queue (That is, the newly bound hardware submission queue) should satisfy:
[0172]
[0173] That is, select the first completely empty queue as the rebinding target.
[0174] The key difference between gpu_wait_counter and wait_counter is: It records the task flow. The task flow counter value at the time of the last command submission, and It may be larger due to the submission of task flows from other shared hardware submission queues. Therefore, it may occur... but This is precisely the condition that triggers rebinding.
[0175] In this embodiment of the application, after the initial allocation adopts the least reference strategy, the reference count of each hardware submission queue is... satisfy:
[0176]
[0177] That is, the difference in reference count between each hardware submission queue does not exceed 1, thus achieving an approximately uniform initial load distribution.
[0178] After dynamic rebinding, the idle task flow is migrated from the busy hardware commit queue to the idle hardware commit queue, which is equivalent to performing local balancing of the reference count:
[0179]
[0180] in, For the reference count of the old queue, The reference count for the new queue. Since rebinding only occurs... (The old queue is busy) and This occurs when the new queue becomes idle. This operation shifts the workload from the busy queue to the idle queue, achieving positive feedback regulation of load balancing on a macro level.
[0181] The dynamic binding method for task flows and hardware submission queues provided in this application can eliminate head-of-queue blocking in idle task flows. Idle task flows are no longer forced to wait for busy hardware submission queues to empty, but can directly switch to the idle hardware submission queue to submit new tasks, significantly reducing command submission latency. It also improves the parallelism of the hardware submission queues. Dynamic rebinding balances the workload from the overloaded hardware submission queue to the idle hardware submission queue, making full use of the multiple concurrent channels of the AI chip hardware and improving overall throughput. Furthermore, it has zero additional synchronization overhead. The rebinding check uses the existing atomic counters of the AI chip hardware for lock-free busy detection, without introducing additional synchronization primitives or context switching.
[0182] Furthermore, it can adapt at runtime. Unlike static binding, the embodiments of this application dynamically adjust the mapping relationship according to the real-time status of the task flow and the hardware submission queue to adapt to the dynamic changes in workload. The method is safe and reversible. Triple preconditions (flow idle, queue busy, non-vector dump mode) ensure that rebinding is only performed under safe conditions. When no idle queue can be found, it naturally degenerates to static binding behavior without functional loss. It improves the hardware resource utilization of artificial intelligence chips. By balancing queue load, it reduces the idle time of artificial intelligence chip hardware channels, enabling more computing tasks to be completed under the same hardware configuration, which can reduce the unit computing power cost.
[0183] The dynamic binding method for task flow and hardware submission queue provided in this application embodiment can be applied to high-concurrency computing scenarios, such as deep learning inference servers, scientific computing clusters, graphics rendering farms, and other scenarios that require scheduling of a large number of concurrent artificial intelligence chip tasks; it can be compatible with existing programming models, and users can obtain the benefits of dynamic binding without modifying any code.
[0184] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.
[0185] Based on the same inventive concept, this application also provides a device for dynamically binding task flows and hardware submission queues to implement the above-described method for dynamically binding task flows and hardware submission queues. The solution provided by this device is similar to the solution described in the above method. Therefore, the specific limitations in one or more embodiments of the device for dynamically binding task flows and hardware submission queues provided below can be found in the limitations of the method for dynamically binding task flows and hardware submission queues described above, and will not be repeated here.
[0186] In one exemplary embodiment, such as Figure 11As shown, a dynamic binding device 1100 for task flow and hardware submission queue is provided, including: a first determination module 1102, a rebinding module 1104, and a first submission module 1106, wherein:
[0187] The first determining module 1102 is used to determine whether the rebinding conditions are met for the target task flow before the command of the target task flow is submitted to the first hardware submission queue. The first hardware submission queue is the hardware submission queue currently bound to the target task flow. The rebinding conditions include the target task flow being in an idle state and the first hardware submission queue being in a busy state.
[0188] The rebinding module 1104 is used to, when the rebinding condition is met, search for the second hardware submission queue that is currently idle from each hardware submission queue, and remove the binding relationship between the target task flow and the first hardware submission queue, and establish the binding relationship between the target task flow and the second hardware submission queue.
[0189] The first submission module 1106 is used to submit the commands of the target task flow to the second hardware submission queue.
[0190] The dynamic binding device for task flow and hardware submission queue provided in this application embodiment allows for dynamic adjustment of the binding relationship between task flow and hardware submission queue. When the hardware submission queue to which the task flow is bound is busy, the task flow that can be migrated can be rebound to an idle hardware submission queue. In this way, efficient utilization of hardware submission queue resources can be achieved, improving the overall resource utilization of the artificial intelligence chip and meeting the needs of high-concurrency computing power scheduling.
[0191] In one embodiment, determining whether the rebinding condition is currently met includes:
[0192] Determine the state of the target task flow and the state of the first hardware submission queue;
[0193] When the target task flow is in an idle state and the first hardware submission queue is in a busy state, it is determined that the rebinding condition is met.
[0194] In one embodiment, the device further includes:
[0195] The second determining module is used to determine that the rebinding condition is not currently met when the state of the target task flow is busy and / or the state of the first hardware submission queue is idle.
[0196] The second submission module is used to submit the commands of the target task flow to the first hardware submission queue.
[0197] In one embodiment, determining the state of the target task flow includes:
[0198] Obtain the task count value corresponding to the target task flow and the first count value corresponding to the first hardware submission queue. The task count value is the total number of commands in the first hardware submission queue when the target task flow last submitted a command to the first hardware submission queue, and the first count value is the total number of commands that have been completed in the first hardware submission queue.
[0199] When the task count value is less than or equal to the first count value, the target task flow is determined to be in an idle state; or, when the task count value is greater than the first count value, the target task flow is determined to be in a busy state.
[0200] In one embodiment, determining the state of the first hardware submission queue includes:
[0201] Obtain the first count value and the second count value corresponding to the first hardware submission queue. The first count value is the total number of commands completed in the first hardware submission queue, and the second count value is the total number of commands currently submitted in the first hardware submission queue.
[0202] When the first count value equals the second count value, the first hardware submission queue is determined to be in an idle state; or, when the first count value is less than the second count value, the first hardware submission queue is determined to be in a busy state.
[0203] In one embodiment, the step of searching for the second hardware submission queue that is currently idle from each hardware submission queue includes:
[0204] Traverse each hardware submission queue. When the currently traversed hardware submission queue is in an idle state, end the traversal operation and designate the currently traversed hardware submission queue as the second hardware submission queue.
[0205] In one embodiment, the device further includes:
[0206] The traversal module is used to traverse the reference count mapping table when creating the target task flow if multiple hardware submission queues have been created in the queue pool. The reference count mapping table is used to record the reference count corresponding to each hardware submission queue, and the reference count is used to represent the number of task flows bound to the hardware submission queue.
[0207] A binding module is used to determine the first hardware submission queue based on the reference count of each hardware submission queue, and to establish a binding relationship between the target task flow and the first hardware submission queue.
[0208] The first update module is used to update the reference count of the first hardware submission queue in the reference count mapping table based on an incrementing rule.
[0209] In one embodiment, determining the first hardware submission queue based on the reference count of each of the hardware submission queues includes:
[0210] The traversal operation ends when the reference count of the currently traversed hardware submission queue is 0, and the currently traversed hardware submission queue is designated as the first hardware submission queue; or,
[0211] If there is no hardware commit queue with a reference count of 0, the hardware commit queue with the smallest reference count among all the hardware commit queues shall be designated as the first hardware commit queue.
[0212] In one embodiment, the device further includes:
[0213] The second update module is used to update the reference count of the first hardware submission queue in the reference count mapping table based on a decrementing rule after the binding relationship between the target task flow and the first hardware submission queue is terminated; and,
[0214] After establishing the binding relationship between the target task flow and the second hardware submission queue, the reference count of the second hardware submission queue in the reference count mapping table is updated based on the increment rule.
[0215] Each module in the aforementioned dynamic binding device for task flow and hardware submission queue can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device in software form, so that the processor can invoke and execute the operations corresponding to each module.
[0216] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 12As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements a dynamic binding method between task flow and hardware submission queues. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.
[0217] Those skilled in the art will understand that Figure 12 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.
[0218] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.
[0219] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.
[0220] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.
[0221] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.
[0222] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.
[0223] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.
Claims
1. A method for dynamically binding task flows to hardware submission queues, characterized in that, The method includes: For a target task flow, before the command of the target task flow is submitted to the first hardware submission queue, it is determined whether the rebinding condition is met. The first hardware submission queue is the hardware submission queue currently bound to the target task flow. The rebinding condition includes the target task flow being in an idle state and the first hardware submission queue being in a busy state. When the rebinding condition is met, find the second hardware submission queue that is currently idle from each hardware submission queue, and remove the binding relationship between the target task flow and the first hardware submission queue, and establish the binding relationship between the target task flow and the second hardware submission queue. The commands of the target task flow are submitted to the second hardware submission queue.
2. The method according to claim 1, characterized in that, Determining whether the rebinding conditions are currently met includes: Determine the state of the target task flow and the state of the first hardware submission queue; When the target task flow is in an idle state and the first hardware submission queue is in a busy state, it is determined that the rebinding condition is met.
3. The method according to claim 2, characterized in that, The method further includes: When the target task flow is in a busy state and / or the first hardware submission queue is in an idle state, it is determined that the rebinding condition is not currently met. The commands of the target task flow are submitted to the first hardware submission queue.
4. The method according to any one of claims 1 to 3, characterized in that, Determining the state of the target task flow includes: Obtain the task count value corresponding to the target task flow and the first count value corresponding to the first hardware submission queue. The task count value is the total number of commands in the first hardware submission queue when the target task flow last submitted a command to the first hardware submission queue, and the first count value is the total number of commands that have been completed in the first hardware submission queue. When the task count value is less than or equal to the first count value, the target task flow is determined to be in an idle state; or, when the task count value is greater than the first count value, the target task flow is determined to be in a busy state.
5. The method according to any one of claims 1 to 3, characterized in that, Determining the state of the first hardware submission queue includes: Obtain the first count value and the second count value corresponding to the first hardware submission queue. The first count value is the total number of commands completed in the first hardware submission queue, and the second count value is the total number of commands currently submitted in the first hardware submission queue. When the first count value equals the second count value, the first hardware submission queue is determined to be in an idle state; or, when the first count value is less than the second count value, the first hardware submission queue is determined to be in a busy state.
6. The method according to any one of claims 1 to 3, characterized in that, The step of searching for the second hardware submission queue that is currently idle from each hardware submission queue includes: Traverse each hardware submission queue. When the currently traversed hardware submission queue is in an idle state, end the traversal operation and designate the currently traversed hardware submission queue as the second hardware submission queue.
7. The method according to any one of claims 1 to 3, characterized in that, The method further includes: When creating the target task flow, if multiple hardware submission queues have been created in the queue pool, the reference counting mapping table is traversed. The reference counting mapping table is used to record the reference count corresponding to each hardware submission queue. The reference count is used to represent the number of task flows bound to the hardware submission queue. The first hardware submission queue is determined based on the reference count of each hardware submission queue, and the binding relationship between the target task flow and the first hardware submission queue is established. The reference count of the first hardware submission queue in the reference count mapping table is updated based on the increment rule.
8. The method according to claim 7, characterized in that, Determining the first hardware submission queue based on the reference counts of each of the hardware submission queues includes: The traversal operation ends when the reference count of the currently traversed hardware submission queue is 0, and the currently traversed hardware submission queue is designated as the first hardware submission queue; or, If there is no hardware commit queue with a reference count of 0, the hardware commit queue with the smallest reference count among all the hardware commit queues shall be designated as the first hardware commit queue.
9. The method according to claim 7, characterized in that, The method further includes: After ending the binding relationship between the target task flow and the first hardware submission queue, the reference count of the first hardware submission queue in the reference count mapping table is updated based on the decrement rule; After establishing the binding relationship between the target task flow and the second hardware submission queue, the reference count of the second hardware submission queue in the reference count mapping table is updated based on the increment rule.
10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 9.
11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 9.
12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 9.
