Bandwidth-aware multi-core NPU task scheduling method, system and device
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HUNAN GREAT WALL GALAXY TECH CO LTD
- Filing Date
- 2026-04-23
- Publication Date
- 2026-05-26
Smart Images
Figure CN122086576A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of task scheduling technology for multi-core artificial intelligence chips, and in particular to a bandwidth-aware multi-core NPU task scheduling method, system, and device. Background Technology
[0002] With the computational demands of deep learning models (such as Transformer and Large Language Models LLM) growing exponentially, single-core NPUs can no longer meet real-time requirements. Modern AI chips generally adopt multi-core architectures, sharing off-chip memory (such as DRAM, LPDDR5, HBM) bandwidth through on-chip interconnect networks (NoC).
[0003] However, in actual operation, the performance of NPUs is often limited by the "memory wall" effect. Deep learning operators have obvious heterogeneous characteristics: operators such as convolution (Conv) and matrix multiplication (MatMul) are computationally intensive and have high arithmetic strength; while operators such as element-wise addition (Add), activation (ReLU), and pooling are bandwidth intensive and have low arithmetic strength. In traditional scheduling, if multiple cores execute bandwidth-intensive tasks simultaneously, bus requests will quickly reach saturation, causing each core to generate a large number of idle cycles (Stall) while waiting for data transmission, which seriously reduces the overall throughput and energy efficiency of the system.
[0004] In the current field of artificial intelligence chip design, multi-core neural network processor (NPU) architecture has become the mainstream solution to meet the ever-increasing computing power demands of deep learning models. These chips typically integrate multiple AI computing cores (AICores) and share the limited bus bandwidth of off-chip memory (such as LPDDR and HBM) through on-chip networks (NoC). However, operators in actual deep learning inference tasks exhibit significant heterogeneity: operators such as convolution (Conv) and matrix multiplication (MatMul) have high arithmetic strength and are computationally intensive; while operators such as element-wise addition (Add), activation functions (ReLU), softmax, and pooling have lower arithmetic strength and are bandwidth-intensive. Most existing multi-core NPU scheduling methods fall into two main technical categories: one is a static partitioning strategy based on FLOPs (Functional Load Balancers), which distributes different layers or operator blocks of the model roughly evenly across cores during the compilation phase according to theoretical computational load, aiming to make the computation time of each core similar; the other is a dynamic round-robin scheduling based on the busy / idle status of cores, which dispatches tasks indiscriminately to idle cores from a global queue. A common limitation of these existing technologies is that the scheduling decision only focuses on the "computing load" of each core, completely ignoring the significant differences in the "bandwidth load" of different tasks on the shared bus. When multiple cores are simultaneously assigned to bandwidth-intensive tasks, bus requests will reach saturation in a very short time, causing severe on-chip network congestion. At this point, although the cores are not fully loaded, they generate a large number of idle periods (Stall) while waiting for data to be transferred from off-chip memory. The overall system throughput is limited by bandwidth rather than computing power, i.e., it falls into the "memory wall" bottleneck. Furthermore, the randomness of bus contention leads to drastic fluctuations in the execution time of the same task under different system loads, significantly increasing long-tail latency and making it difficult to meet the stringent latency deterministic requirements of real-time scenarios such as autonomous driving and augmented reality. Simultaneously, the cores continuously consume dynamic power during idling, resulting in a decrease in energy efficiency. Therefore, there is an urgent need for a multi-core NPU scheduling method that can perceive the global bus bandwidth status in real time and proactively avoid congestion. Summary of the Invention
[0005] Therefore, it is necessary to provide a bandwidth-aware multi-core NPU task scheduling method, system, and device that can improve computing power utilization under the condition that multiple computing cores compete for limited bus and memory bandwidth, in order to address the above-mentioned technical problems.
[0006] A bandwidth-aware multi-core NPU task scheduling method, the method comprising: Obtain the bandwidth requirement characteristics of the task to be scheduled.
[0007] The real-time bandwidth status of the shared bus in the multi-core NPU is acquired through a hardware counter. The real-time bandwidth status includes: the current bandwidth usage and the remaining allowed bandwidth.
[0008] Based on bandwidth demand characteristics and real-time bandwidth status, a scheduling strategy is adopted to allocate tasks to idle cores of the requesting tasks. The specific steps of the scheduling strategy are as follows: when the remaining allowed bandwidth is higher than the first threshold, tasks with high bandwidth demand are given priority; when the remaining allowed bandwidth is lower than the second threshold, tasks with low bandwidth demand are given priority, so as to keep the total bandwidth usage within a safe range.
[0009] A bandwidth-aware multi-core NPU task scheduling system, the system comprising: The task profiling module is used to analyze tasks during the compilation phase and generate bandwidth requirement characteristics for each task.
[0010] A global status monitor is used to collect and calculate the real-time bandwidth status of the shared bus of a multi-core NPU through a hardware counter.
[0011] The bandwidth-aware scheduler is connected to both the task profiling module and the global status monitor, and is used to execute task scheduling strategies based on bandwidth demand characteristics and the real-time bandwidth status.
[0012] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program performing the following steps: Obtain the bandwidth requirement characteristics of the task to be scheduled.
[0013] The real-time bandwidth status of the shared bus in the multi-core NPU is acquired through a hardware counter. The real-time bandwidth status includes: the current bandwidth usage and the remaining allowed bandwidth.
[0014] Based on bandwidth demand characteristics and real-time bandwidth status, a scheduling strategy is adopted to allocate tasks to idle cores of the requesting tasks. The specific steps of the scheduling strategy are as follows: when the remaining allowed bandwidth is higher than the first threshold, tasks with high bandwidth demand are given priority; when the remaining allowed bandwidth is lower than the second threshold, tasks with low bandwidth demand are given priority, so as to keep the total bandwidth usage within a safe range.
[0015] The aforementioned bandwidth-aware multi-core NPU task scheduling method, system, and device significantly improve system throughput. First, it utilizes a hardware counter to monitor bus occupancy in real time and dynamically switches between high and low bandwidth task allocation based on remaining bandwidth, avoiding congestion and crashes caused by multiple cores simultaneously competing for the bus. Experiments show that under typical vision models (such as YOLOv8), the multi-core speedup is improved by 18%-25%. Second, the determinism of task execution latency is significantly enhanced. A dual-threshold hard constraint (prioritizing high-bandwidth tasks when above the first threshold and forcing low-bandwidth tasks when below the second threshold) actively confines the total bandwidth within a safe range, eliminating long-tail latency caused by random bus conflicts. This is particularly crucial for real-time scenarios such as autonomous driving. Third, energy efficiency is optimized. By avoiding excessive idling of cores while waiting for data (i.e., "busy waiting"), it reduces ineffective power consumption, resulting in higher computational output per unit power consumption. Finally, it combines the advantages of hardware compatibility and low overhead. Its core logic can be implemented in the scheduling layer through software or firmware, without modifying the NPU's internal computing array or on-chip network topology, nor requiring complex parallel strategy searches or offline multi-version storage, resulting in low deployment costs. In summary, this invention elevates the scheduling paradigm from "computing power load balancing" to "bandwidth-aware proactive congestion management," effectively unleashing the true parallel potential of multi-core NPUs in bandwidth-constrained scenarios and improving the scheduling efficiency and computing power utilization of parallel tasks. Attached Figure Description
[0016] Figure 1 This is an application scenario diagram of a bandwidth-aware multi-core NPU task scheduling method in one embodiment; Figure 2 This is a flowchart of a bandwidth-aware multi-core NPU task scheduling method in one embodiment; Figure 3 This is a schematic diagram illustrating the collaborative workflow between offline profiling and online scheduling in one embodiment; Figure 4 This is a block diagram of a bandwidth-aware multi-core NPU task scheduling system in one embodiment. Figure 5 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation
[0017] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0018] This application provides a bandwidth-aware multi-core NPU task scheduling method that can be applied to, for example... Figure 1The illustrated architecture is a bandwidth-aware multi-core NPU scheduling system. This architecture primarily includes: a task profiler module, multi-core NPU hardware, a global monitor, and an AwareScheduler.
[0019] The task profiling module is used during the model compilation phase to extract the input-output tensor size (bytes) and number of computational operands (FLOPs) of each operator through static analysis, and to calculate the arithmetic strength of the operators. ; And based on the peak bandwidth of the target hardware It is labeled as "computation-constrained" or "bandwidth-constrained".
[0020] Multi-core NPU hardware includes a bandwidth estimator, used to build a bandwidth consumption prediction model based on hardware topology (such as NoC topology, number of DRAM channels), and to calculate the expected bandwidth utilization for each scheduled task. ; in, The ideal execution time is required.
[0021] A global status monitor is used to collect real-time data on the bus transaction request frequency and latency of each core via hardware counters, as well as the current remaining available bandwidth of the dynamic computing system. BW margin .
[0022] A bandwidth-aware scheduler executes complementary scheduling algorithms based on task profiles and real-time available bandwidth. It also maintains multiple priority queues to store tasks with varying bandwidth demands.
[0023] In one embodiment, such as Figure 2 As shown, a bandwidth-aware multi-core NPU task scheduling method is provided, which is then applied to... Figure 1 Taking the system in the example, the following steps are included: Step 202: Obtain the bandwidth requirement characteristics of the task to be scheduled.
[0024] Specifically, during the model compilation phase, the task profiling module performs static analysis on the deep learning computation graph. First, it analyzes each operator in the computation graph and extracts its total number of computational operations. FLOPs (For example, for convolutional layers,) FLOPs= 2 × output feature map size × kernel size² × number of input channels × number of output channels) plus the total amount of data that needs to be moved from off-chip memory. Bytes (Including the total number of bytes in the input feature map, weights, and output feature map). Then, the arithmetic strength AI is calculated as FLOPs / Bytes. For example, for a 3×3 convolutional layer, FLOPs 1.2×10 9 , Bytes 3×10 7 ,but AI =40. Compare this arithmetic strength with the computational capability threshold at the target NPU's peak bandwidth (e.g., 64 GB / s): If AI If the value is ≥ a threshold (e.g., 30), it is marked as a "computation-intensive" task; otherwise, it is marked as a "bandwidth-intensive" task. Simultaneously, the expected bandwidth utilization of the task is recorded. ,in The ideal execution time is assumed to be under contention-free conditions (which can be pre-calibrated using a hardware counter). Finally, a task bandwidth requirement characteristic table is generated, containing the task ID, arithmetic strength label, expected bandwidth utilization, and required data volume.
[0025] Step 204: Collect the real-time bandwidth status of the shared bus in the multi-core NPU using a hardware counter.
[0026] Real-time bandwidth status includes: current bandwidth usage and remaining allowed bandwidth.
[0027] Specifically, the global status monitor periodically (e.g., every 10 microseconds) reads the hardware performance counters on the chip. These counters count the number of read / write transactions in bytes for each memory channel and on-chip network port. The monitor sums up the bus request data volume for all cores, divides it by the sampling period, and obtains the current total bandwidth usage (in GB / s). Simultaneously, it subtracts the current usage value from the system's preset peak bandwidth (e.g., 64 GB / s) to calculate the remaining allowed bandwidth. For example, if 1280 bytes are transferred on the bus in the current sampling period (10 μs), the current bandwidth usage is 1280B / 10μs = 128MB / s; if the peak bandwidth is 64 GB / s, the remaining bandwidth is approximately 63.872 GB / s. Furthermore, the monitor records the duration for which bandwidth usage exceeds a safety threshold (e.g., 85% of peak bandwidth) as a congestion warning signal. All real-time status data is cached in a register set directly accessible to the scheduler to ensure low-latency reads.
[0028] Step 206: Based on the bandwidth demand characteristics and real-time bandwidth status, a scheduling strategy is adopted to allocate tasks to the idle cores of the requesting tasks. The specific steps of the scheduling strategy are as follows: when the remaining allowed bandwidth is higher than the first threshold, tasks with high bandwidth demand are allocated first; when the remaining allowed bandwidth is lower than the second threshold, tasks with low bandwidth demand are allocated first, so as to keep the total bandwidth usage within a safe range.
[0029] Specifically, the bandwidth-aware scheduler maintains a multi-level priority queue, where high-priority queues store bandwidth-intensive tasks and low-priority queues store computationally intensive tasks. When the core... When a task request signal is sent, the scheduler performs the following sub-steps: First, read the current remaining bandwidth provided by the global status monitor. BW margin (For example, if the current bandwidth usage is 48GB / s, and the peak bandwidth is 64GB / s, then...) BW margin =16GB / s).
[0030] The first threshold (high watermark) is set at 50% of the peak bandwidth (i.e., 32GB / s), and the second threshold (low watermark) is set at 80% of the peak bandwidth (i.e., 51.2GB / s). Note: The first threshold is lower than the second threshold. The logic is: when the remaining bandwidth is higher than the first threshold, it means there is ample bandwidth, and high-bandwidth tasks can be allocated freely; when the remaining bandwidth is lower than the second threshold, it means there is bandwidth shortage, and low-bandwidth tasks must be allocated.
[0031] when BW margin When bandwidth exceeds 32GB / s (remaining bandwidth is above the first threshold), the scheduler retrieves a task from the head of the high-priority queue (bandwidth-intensive tasks) and checks its expected bandwidth utilization. R bw Is it less than BW margin If so, then allocate to the core. Otherwise, select a computationally intensive task from the low-priority queue for assignment.
[0032] when BW margin When the bandwidth is less than 51.2 GB / s (the remaining bandwidth is below the second threshold), the scheduler will forcibly allocate tasks from the low-priority queue (computation-intensive tasks). Even if there are waiting bandwidth-intensive tasks in the current queue, they will not be scheduled for the time being, thereby avoiding further congestion on the bus.
[0033] when BW margin When the situation falls between these two extremes, the scheduler uses a round-robin approach to alternately allocate tasks of both types in order to maintain a stable bandwidth usage.
[0034] Example: The current system has 4 cores. Core 1 and Core 2 are performing compute-intensive tasks (each using 8GB / s), while Core 3 is idle. Remaining bandwidth... BW margin =64-(8+8)=48GB / s, which is higher than the first threshold of 32GB / s but lower than the second threshold of 51.2GB / s. Since 48 < 51.2, it falls below the second threshold, and the scheduler should prioritize allocating tasks with lower bandwidth requirements. At this time, the task queue contains task A (Softmax, bandwidth requirement 40GB / s) and task B (residual block, bandwidth requirement 10GB / s). The scheduler selects task B and assigns it to core 3, making the total bandwidth 8+8+10=26GB / s, far below the safety limit, ensuring system stability. If core 1 completes its task and releases resources after a period of time... BW margin The speed increases to 56 GB / s, which is above the second threshold of 51.2 GB / s, at which point the scheduler can safely assign task A to core 1.
[0035] In the aforementioned bandwidth-aware multi-core NPU task scheduling method, firstly, the system's effective throughput is significantly improved. By utilizing a hardware counter to sense bus occupancy in real time and dynamically switching between high and low bandwidth task allocation based on remaining bandwidth, congestion and crashes caused by multiple cores simultaneously competing for the bus are avoided. Experiments show that under typical vision models (such as YOLOv8), the multi-core speedup is improved by 18%-25%. Secondly, the determinism of task execution latency is significantly enhanced. A dual-threshold hard constraint (prioritizing high-bandwidth tasks when above the first threshold and forcing low-bandwidth tasks when below the second threshold) actively clamps the total bandwidth within a safe range, eliminating long-tail latency caused by random bus conflicts. This is particularly crucial for real-time scenarios such as autonomous driving. Furthermore, energy efficiency is optimized. By avoiding excessive idling of cores while waiting for data (i.e., "busy waiting"), ineffective power consumption is reduced, resulting in higher computational output per unit power consumption. Finally, it combines the advantages of hardware compatibility and low overhead. Its core logic can be implemented in the scheduling layer through software or firmware, without modifying the NPU's internal computing array or on-chip network topology, nor requiring complex parallel strategy searches or offline multi-version storage, resulting in low deployment costs. In summary, this invention elevates the scheduling paradigm from "computing power load balancing" to "bandwidth-aware proactive congestion management," effectively unleashing the true parallel potential of multi-core NPUs in bandwidth-constrained scenarios and improving the scheduling efficiency and computing power utilization of parallel tasks.
[0036] In one embodiment, bandwidth requirement characteristics are generated based on the arithmetic intensity of the task to be scheduled when it is executed on the NPU.
[0037] In one embodiment, based on the model compilation stage of the task to be scheduled on the NPU, the deep learning computation graph is parsed, and the model is broken down into multiple operators or operator blocks that can be independently scheduled. For each operator or operator block, the total computational operations and the total amount of data to be transferred are extracted, and the arithmetic strength is calculated. Based on the arithmetic strength and the peak bandwidth of the target NPU, the operator or operator block is marked as a computationally intensive task or a bandwidth-intensive task, generating the bandwidth requirement characteristics of the task to be scheduled.
[0038] In one embodiment, when multiple cores are detected to be performing bandwidth-intensive tasks and the remaining allowable bandwidth is below a safety threshold, proactive congestion avoidance is performed.
[0039] In one embodiment, a hardware counter periodically collects the bus transaction request frequency and latency data of each core. Based on the bus transaction request frequency, latency data, and on-chip network topology, the total bandwidth usage of all currently running tasks awaiting scheduling is calculated. The remaining allowable bandwidth is dynamically calculated based on the peak bandwidth of the multi-core NPU and the total bandwidth usage.
[0040] In one embodiment, when it is predicted that allocating a new task will cause the total bandwidth usage to exceed a preset percentage of the peak bandwidth, the current new task is temporarily stored in the waiting queue, and a task to be scheduled with the greatest difference in arithmetic strength from the currently running task is allocated to the core of the requesting task.
[0041] In one embodiment, such as Figure 3 As shown, an offline profiling and online scheduling collaborative workflow is provided, with the following specific steps: 1. Offline feature modeling The compiler parses the deep learning computation graph and breaks the model down into independently schedulable subgraphs or operator blocks.
[0042] Generate a "Task-Bandwidth Requirement Table" to record each task. AI Value and predicted bandwidth utilization R bw .
[0043] 2. Real-time bandwidth margin assessment During operation, the scheduler periodically polls the global status monitor.
[0044] Set a secure bandwidth threshold TH safe (Typically 80%-85% of peak bandwidth) Calculate the upper limit of bandwidth that can be accepted for new tasks at present.
[0045] 3. Complementary task selection When the core When a new task is requested, the scheduler retrieves the queue of tasks to be executed.
[0046] Strategy 1 (Supplement Strategy): If the current total system bandwidth usage is low, prioritize dispatching bandwidth-intensive tasks (low bandwidth usage). AI ), rapidly consume data.
[0047] Strategy Two (Avoidance Strategy): If the current total bandwidth usage is close to TH safe Then, computationally intensive tasks (high-performance) will be forcibly assigned. AI Even if the core is better suited for other tasks, the bus request frequency is "diluted" by increasing the computation cycle.
[0048] 4. Adaptive window adjustment If all cores are found to be performing high-bandwidth tasks and experiencing excessive latency, the scheduler will proactively insert small latency periods (Stall Injection) or reduce the prefetch frequency of some non-critical path cores to alleviate momentary congestion.
[0049] In one embodiment, taking a multi-core NPU integrating 4 AI Cores as an example, the system peak bandwidth is 64GB / s: Scenario setting: Core 1 and Core 2 are performing high-load deep convolutions (computation-intensive, each using 8GB / s of bandwidth).
[0050] Scheduling decision: Core 3 is currently idle, and there are two tasks in the queue: Task A (Softmax, bandwidth requirement 40GB / s) and Task B (convolutional residual block, bandwidth requirement 10GB / s).
[0051] Traditional scheduling: If task A is scheduled sequentially, the total bandwidth usage becomes 8+8+40 = 56GB / s, which is close to saturation and may cause bus arbitration conflicts, resulting in slower computation of cores 1 and 2.
[0052] The scheduling mechanism of this invention is as follows: The scheduler identifies that the instantaneous bandwidth requirement of task A is too high, and that a core is already running. To ensure system stability, the scheduler prioritizes assigning task B to core 3, and reserves task A for execution in a window with more abundant total bandwidth (e.g., after core 1 completes its task).
[0053] Result: The system always runs in a highly efficient and uncongested range, avoiding a sudden drop in performance caused by "bandwidth collision".
[0054] It should be understood that, although Figures 2-3 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figures 2-3At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.
[0055] In one embodiment, such as Figure 4 As shown, a bandwidth-aware multi-core NPU task scheduling system is provided, including: a task profiling module 402, a global status monitor 404, and a bandwidth-aware scheduler 406, wherein: Task profiling module 402 is used to analyze tasks during the compilation phase and generate bandwidth requirement characteristics for each task.
[0056] The global status monitor 404 is used to collect and calculate the real-time bandwidth status of the multi-core NPU shared bus in real time through a hardware counter.
[0057] The bandwidth-aware scheduler 406 is connected to the task profiling module and the global status monitor, respectively, and is used to execute task scheduling strategies based on bandwidth demand characteristics and real-time bandwidth status.
[0058] In one embodiment, the bandwidth-aware scheduler 406 is further configured to prioritize the allocation of tasks with high bandwidth requirements when the remaining allowed bandwidth is higher than a first threshold, and prioritize the allocation of tasks with low bandwidth requirements when the remaining allowed bandwidth is lower than a second threshold, so as to keep the total bandwidth usage within a safe range.
[0059] In one embodiment, the congestion avoidance module 40602 is used to perform active congestion avoidance when it is detected that multiple cores are performing bandwidth-intensive tasks and the remaining allowable bandwidth is below a safety threshold.
[0060] Specific limitations regarding the bandwidth-aware multi-core NPU task scheduling system can be found in the limitations of the bandwidth-aware multi-core NPU task scheduling method described above, and will not be repeated here. Each module in the aforementioned bandwidth-aware multi-core NPU task scheduling system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device, or stored in the computer device's memory as software, so that the processor can invoke and execute the corresponding operations of each module.
[0061] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 5As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a bandwidth-aware multi-core NPU task scheduling method. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.
[0062] Those skilled in the art will understand that Figures 4-5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.
[0063] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to perform the following steps: Obtain the bandwidth requirement characteristics of the task to be scheduled.
[0064] The real-time bandwidth status of the shared bus in the multi-core NPU is acquired through a hardware counter. The real-time bandwidth status includes: the current bandwidth usage and the remaining allowed bandwidth.
[0065] Based on bandwidth demand characteristics and real-time bandwidth status, a scheduling strategy is adopted to allocate tasks to idle cores of the requesting tasks. The specific steps of the scheduling strategy are as follows: when the remaining allowed bandwidth is higher than the first threshold, tasks with high bandwidth demand are given priority; when the remaining allowed bandwidth is lower than the second threshold, tasks with low bandwidth demand are given priority, so as to keep the total bandwidth usage within a safe range.
[0066] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchlink, DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.
[0067] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0068] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.
Claims
1. A bandwidth-aware multi-core NPU task scheduling method, characterized in that, The method includes: Obtain the bandwidth requirement characteristics of the task to be scheduled; The real-time bandwidth status of the shared bus in the multi-core NPU is collected by a hardware counter; the real-time bandwidth status includes: the current bandwidth usage value and the remaining allowed bandwidth. Based on the bandwidth demand characteristics and the real-time bandwidth status, a scheduling strategy is adopted to allocate tasks to the idle cores of the requesting tasks. The specific steps of the scheduling strategy are as follows: when the remaining allowed bandwidth is higher than a first threshold, tasks with high bandwidth demand are allocated first; when the remaining allowed bandwidth is lower than a second threshold, tasks with low bandwidth demand are allocated first, so as to keep the total bandwidth occupancy within a safe range.
2. The method according to claim 1, characterized in that, Obtain the bandwidth requirement characteristics of the task to be scheduled, including: Bandwidth requirement characteristics are generated based on the arithmetic strength of the tasks to be scheduled when they are executed on the NPU.
3. The method according to claim 2, characterized in that, Bandwidth requirement characteristics are generated based on the arithmetic strength of the tasks to be scheduled when they are executed on the NPU, including: Based on the model compilation stage of the task to be scheduled on the NPU, the deep learning computation graph is parsed, and the model is split into multiple operators or operator blocks that can be independently scheduled. For each operator or operator block, after extracting the total computational operations and the total amount of data to be transferred, the arithmetic strength is calculated; Based on the arithmetic strength and the peak bandwidth of the target NPU, the operator or the operator block is marked as a computationally intensive task or a bandwidth-intensive task, and the bandwidth requirement characteristics of the task to be scheduled are generated.
4. The method according to any one of claims 1 to 3, characterized in that, When multiple cores are detected to be performing bandwidth-intensive tasks and the remaining allowable bandwidth is below the safety threshold, proactive congestion avoidance is performed.
5. The method according to claim 4, characterized in that, The real-time bandwidth status of the shared bus in a multi-core NPU is collected using a hardware counter, including: The frequency and latency of bus transaction requests for each core are collected periodically using a hardware counter. Based on the bus transaction request frequency, the latency data, and the topology of the on-chip network, calculate the total bandwidth usage of all currently running scheduled tasks. The remaining allowable bandwidth is dynamically calculated based on the peak bandwidth of the multi-core NPU and the total bandwidth usage.
6. The method according to claim 5, characterized in that, Implementing proactive congestion avoidance includes: When it is predicted that assigning a new task will cause the total bandwidth usage to exceed a preset percentage of the peak bandwidth, the current new task is temporarily stored in the waiting queue, and a task with the greatest difference in arithmetic strength from the currently running task is assigned to the core of the requesting task.
7. A bandwidth-aware multi-core NPU task scheduling system, characterized in that, The system includes: The task profiling module is used to analyze tasks during the compilation phase and generate bandwidth requirement characteristics for each task. A global status monitor is used to collect and calculate the real-time bandwidth status of the shared bus of a multi-core NPU in real time through a hardware counter. A bandwidth-aware scheduler is connected to the task profiling module and the global status monitor, respectively, and is used to execute task scheduling strategies based on the bandwidth demand characteristics and the real-time bandwidth status.
8. The system according to claim 7, characterized in that, The bandwidth-aware scheduler is also used to prioritize allocating tasks with high bandwidth requirements when the remaining allowed bandwidth is higher than a first threshold, and to prioritize allocating tasks with low bandwidth requirements when the remaining allowed bandwidth is lower than a second threshold, so as to keep the total bandwidth usage within a safe range.
9. The system according to claim 7, characterized in that, The system also includes a congestion avoidance module, which is used to perform active congestion avoidance when multiple cores are detected to be performing bandwidth-intensive tasks and the remaining allowable bandwidth is lower than a safety threshold.
10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.