SIMD architecture-oriented neural network processor intra-core scheduling method and system

Abstract

The invention discloses an SIMD architecture-oriented neural network processor intra-core scheduling method and system, and belongs to the technical field of processor scheduling. Through a three-level screening mechanism of the node type priority, the time window priority and the memory influence degree priority, in combination with dependency constraint processing and L0 cache mutual exclusion constraint management, a scheduling sequence which meets a topological sequence and has the minimum peak cache residence amount is generated, idle computing power in a chip is effectively utilized, and the resource utilization rate is improved. On the basis, cache allocation and swap-in and swap-out operation are optimized based on a cost-aware strategy, an optimized scheduling sequence containing swap-in and swap-out operation nodes is generated, swap-in and swap-out memory minimization is achieved, and the scheduling efficiency and the resource utilization rate are further improved. In addition, the method further optimizes the swap-in and swap-out position, and further reduces the total operation time of the operator on the premise of ensuring that the data carrying amount except the total amount is not obviously increased.

Description

Technical Field

[0001] This invention belongs to the field of processor scheduling technology, and more specifically, relates to a method and system for scheduling within the kernel of a neural network processor for SIMD architecture. Background Technology

[0002] With the deep application of artificial intelligence technology in edge computing, neural network accelerator processors (NPUs) have become the core hardware support for edge inference tasks due to their high computing efficiency. Among them, processors based on SIMD architecture are widely used in resource-constrained edge devices due to their significant advantages of simple hardware design and high area efficiency.

[0003] However, the contradiction between the hardware advantages of the SIMD architecture and the complexity of its software model adaptation has become a major bottleneck restricting its large-scale commercial application. During neural network inference, the execution efficiency of operators directly determines the inference latency and resource utilization of the model. The complete computation process of operators on the SIMD architecture hardware platform can essentially be decomposed into a fine-grained computation graph composed of hardware functional unit operations. The efficient execution of operators depends on the rational arrangement of the hardware unit operations within this fine-grained computation graph, forming a task flow that can run efficiently on the SIMD platform.

[0004] Existing neural network processors typically employ core-level exclusive scheduling algorithms for in-core scheduling. This ensures that when a thread is scheduled to execute on a specific computing core of the NPU, that core will not be preempted or shared by any other thread until the thread actively releases it or finishes execution. However, core-level exclusive scheduling allocates resources on a per-chip basis, resulting in coarse resource granularity and an inability to effectively utilize idle computing power within the chip, leading to low resource utilization. Summary of the Invention

[0005] In view of the above-mentioned defects or improvement needs of the prior art, the present invention provides a method and system for scheduling within the core of a neural network processor for SIMD architecture, so as to solve the technical problem of low resource utilization in the prior art.

[0006] To achieve the above objectives, in a first aspect, the present invention provides a method for scheduling within the kernel of a neural network processor for SIMD architecture, comprising:

[0007] S1. Receive a directed acyclic graph (DAG) representing the execution process within the kernel of a neural network operator; the DAG includes operation nodes, cache management nodes, and execution dependency edges between nodes; the cache management nodes include memory allocation nodes and memory release nodes; the cache management nodes carry the required cache type information and the required memory allocation or release capacity; S2. Add a dependency edge to the DAG, where the memory release node for releasing the L0 cache unit points to the memory allocation node for requesting the L0 cache unit, and ensure that the DAG is still a directed acyclic graph. S3. Prioritize all nodes in the DAG; the priorities of memory release nodes, operation nodes, and memory allocation nodes decrease sequentially; the priority of each memory allocation node increases with its latest start time; the priority of each memory release node decreases with its earliest start time; for memory allocation nodes with the same latest start time, the priority increases with the amount of memory they need to allocate; the priority of memory release nodes with the same earliest start time increases with the amount of memory they need to release. S4. Repeat the topology sorting operation until all nodes in the DAG are added to the scheduling sequence, and output the current scheduling sequence. The topology sorting operation includes: obtaining the node with an in-degree of 0 from the DAG and adding it to the ready queue; taking the node with the highest priority in the current ready queue as the current node to be executed and adding it to the scheduling sequence; and decrementing the in-degree of the adjacent nodes of the current node to be executed by one.

[0008] More preferably, the above-mentioned neural network processor core scheduling method further includes: Set up a corresponding list of free intervals for each cache type to record the starting address and size of all free space in each cache unit under the corresponding cache type; During the kernel scheduling process, the offset position of each currently pending node in the used cache unit is recorded, and the starting address and size of the free interval list of the corresponding cache type are updated. When the currently pending node is added to the scheduling sequence: When the current node to be executed is a memory allocation node, a free interval that meets preset conditions is searched from the corresponding free interval list. If a free interval is found, it is allocated to the current node to be executed. The preset conditions are: the difference between the size of the free interval and the memory capacity required by the current node to be executed is greater than a preset threshold, and it is adjacent to the allocated memory interval of the candidate memory usage cycle. The candidate memory usage cycle is a memory usage cycle whose interval with the memory usage cycle required by the current node to be executed is less than a preset interval. If no free interval is found, the allocated memory interval in the cache unit of the corresponding cache type is searched, and a swap-in / swap-out operation is performed on the memory interval. Dependency edges corresponding to each swap-in / swap-out operation are added to the DAG, including: a dependency edge from the memory allocation node of the cache unit used to request the swap-out operation to the operation node used to perform the corresponding swap-out operation; a dependency edge from the operation node used to perform the corresponding swap-out operation to the operation node used to perform the corresponding swap-in operation; and a dependency edge from the operation node used to perform the corresponding swap-in operation to the memory release node used to release the cache unit used to perform the corresponding swap-in operation. Edges are added to the DAG, with dependencies between executed nodes (corresponding to cache units requesting swap-out operations) and unexecuted nodes (corresponding to cache units corresponding to swap-in operations), thus updating the DAG. After prioritizing the newly added nodes, the process moves to S4. Executed nodes are those already in the scheduling sequence of the current DAG; unexecuted nodes are all nodes in the current DAG except for those already executed. The priority of nodes performing swap-in operations is the same as that of memory allocation nodes, increasing with their latest start time. The priority of nodes performing swap-out operations is the same as that of memory release nodes, decreasing with their earliest start time. For nodes with the same latest start time performing swap-in operations, the priority increases with the amount of memory they need to allocate. For nodes with the same earliest start time performing swap-out operations, the priority increases with the amount of memory they need to release. When the current node to be executed is a memory release node, merge the free intervals in the corresponding free interval list that are adjacent to the memory to be released by the current node to be executed, and update the starting address and size of all free spaces in each cache unit in the corresponding free interval list.

[0009] More preferably, the best-fit algorithm is used to search for the allocated memory range in the cache unit corresponding to the cache type.

[0010] More preferably, the above-described neural network processor core scheduling method further includes: step S5 executed after S4; wherein, S5 includes: S51. Using the scheduling sequence obtained in S4 as the initial scheduling sequence, adjust the initial scheduling sequence once or multiple times to obtain the adjusted scheduling sequence. The methods for adjusting the initial scheduling sequence include: For each cache unit whose memory usage cycle is greater than the preset cycle, the corresponding operation node for performing the swap-out operation is moved forward in the initial scheduling sequence, and the resulting scheduling sequence is used as the adjusted scheduling sequence. And / or, For each cache unit whose memory usage period is greater than the preset period, the corresponding operation node for performing the swap-in operation is moved backward in the initial scheduling sequence, and the resulting scheduling sequence is used as the adjusted scheduling sequence. S52. For the initial scheduling sequence and each adjusted scheduling sequence, the critical path analysis algorithm is used to calculate the total running time cycle and the total memory used for swap-in and swap-out operations during the execution of the scheduling sequence, and the weighted sum is used as the performance index of the scheduling sequence; the scheduling sequence corresponding to the maximum performance index is taken as the final scheduling sequence.

[0011] Secondly, the present invention provides a neural network processor, which uses the neural network processor kernel scheduling method provided in the first aspect of the present invention to realize the execution scheduling of neural network operators.

[0012] Thirdly, the present invention provides an edge computing device, including the neural network processor provided in the second aspect of the present invention.

[0013] Fourthly, the present invention provides an intra-core scheduling system for a neural network processor oriented to a SIMD architecture, comprising: a memory and a processor, wherein the memory stores a computer program, and the processor executes the intra-core scheduling method for a neural network processor provided in the first aspect of the present invention when executing the computer program.

[0014] Fifthly, the present invention also provides a computer-readable storage medium comprising a stored computer program, wherein the computer program, when executed by a processor, controls the device where the storage medium is located to execute the neural network processor kernel scheduling method provided in the first aspect of the present invention.

[0015] In a sixth aspect, the invention also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the neural network processor kernel scheduling method provided in the first aspect of the invention.

[0016] In summary, the above-described technical solutions conceived in this invention can achieve the following beneficial effects: 1. This invention provides a scheduling method for neural network processors with SIMD architecture. Through a three-level screening mechanism of node type priority, time window priority and memory impact priority, combined with dependency constraint processing and L0 cache mutual exclusion constraint management, a scheduling sequence that satisfies the topological order and minimizes the peak cache resident amount is generated, which effectively utilizes the idle computing power in the chip and improves resource utilization.

[0017] 2. Furthermore, considering the limited memory space of each cache unit, if cache unit allocation overflow occurs, additional swap-in and swap-out operations will occur, consuming significant time and computing power. The neural network processor core scheduling method provided in this invention optimizes cache allocation and swap-in / swap-out operations based on a cost-aware strategy during the core scheduling process, generating an optimized scheduling sequence containing swap-in / swap-out operation nodes. This minimizes swap-in / swap-out memory, further improving scheduling efficiency and resource utilization. Simultaneously, by maintaining a list of free intervals for each cache type, the free block with the closest size to the target buffer is selected for allocation, and adjacent free blocks are merged during release, reducing cache fragmentation and improving space utilization.

[0018] 3. Furthermore, the neural network processor kernel scheduling method provided by the present invention further optimizes the swap-in and swap-out positions, thereby further reducing the total operator running time without significantly increasing the total amount of additional data transport. Attached Figure Description

[0019] Figure 1 A flowchart of a neural network processor kernel scheduling method based on minimum cache residency provided in an embodiment of the present invention.

[0020] Figure 2 The flowchart illustrates the cache allocation and swap-in / swap-out optimization algorithm provided in this embodiment of the invention.

[0021] Figure 3 The flowchart illustrates a neural network processor kernel scheduling method based on minimum cache residency, cache allocation and swapping in / out, and performance optimization strategies, provided in an embodiment of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0023] Example 1 This embodiment provides a method for scheduling within the core of a neural network processor for SIMD architecture, such as... Figure 1 As shown, it includes: S1. Receive a directed acyclic graph (DAG) representing the execution process within the kernel of a neural network operator; the DAG includes operation nodes, cache management nodes, and execution dependency edges between nodes; the cache management nodes include memory allocation nodes and memory release nodes; the cache management nodes carry the required cache type information and the required memory allocation or release capacity; The received directed acyclic graph (DAG) represents the execution process within the kernel of a neural network operator, denoted as... The vertex set V contains operation nodes and cache management nodes; operation nodes represent computation or data transfer instructions executed on a specific execution unit; cache management nodes include operations for allocating contiguous cache space and operations for releasing contiguous cache space. The edge set... This indicates the execution dependency between nodes. An edge from node u to node v means that node v can only start executing after u has finished executing.

[0024] Each operation node in the DAG carries the following key attributes: ID number (a unique identifier for the node starting from 0), operation instruction name Op, pipeline used to specify which execution unit the operation runs on, number of clock cycles required to run the operation on that execution unit, required cache type information, etc.; for cache management nodes, it also carries the memory capacity to be allocated or released.

[0025] The operation instruction name Op can be any operation name except for memory allocation (ALLOC) and memory release (FREE) operations. For example, data moving operations such as data copying operations (COPY_IN operation) that copy data from other cache areas to the current cache area, and data copying operations (COPY_OUT operation) that copy data from the current cache area to other cache areas, and computation operations such as addition operations (ADD operation) and multiplication operations (MUL operation).

[0026] The cache types in this embodiment include: L0 cache (micro-operation cache), L1 cache (basic cache), and UB cache (unified cache); among which, L0 cache includes: L0A cache, L0B cache, and L0C cache; the capacity constraints of each cache type are as follows: the capacity of L1 cache is 4096kb, the capacity of UB cache is 1024kb, the capacity of L0A cache is 256kb, the capacity of L0B cache is 256kb, and the capacity of L0C cache is 512kb.

[0027] Different types of cache units are independent of each other, and allocation and scheduling are only performed within the same type of cache unit.

[0028] The DAG is preprocessed, including topology analysis and node classification, a dependency table between nodes is constructed, and attribute information of each node (such as operation type, cache requirement size, cache type, etc.) is extracted to lay the foundation for subsequent scheduling optimization.

[0029] S2. Add a dependency edge to the DAG, where the memory release node for releasing the L0 cache unit points to the memory allocation node for requesting the L0 cache unit, and ensure that the DAG is still a directed acyclic graph. Specifically, considering that an L0 cache unit can only have one buffer resident at a time, this embodiment implements L0 cache mutual exclusion constraint management. By abstracting each L0 cache type into an independent mutually exclusive resource, a constraint edge is added that points the memory release node to the next memory allocation node. Combined with a circular dependency detection mechanism, serialized execution is ensured.

[0030] Memory allocation nodes and memory deallocation nodes appear in pairs, forming multiple groups. Each time a node is added, it must be ensured that the DAG remains a directed acyclic graph. That is, the added dependency edge will not appear in the same group of memory allocation nodes and memory deallocation nodes. Instead, a dependency edge is added from a memory deallocation node in one group that deals with releasing L0 cache units to a memory allocation node in another group that requests L0 cache units. Each time, an edge is added, and then a circular dependency check is performed. When a circular dependency is detected, the added dependency edge is deleted, and the dependency edge is added again until the DAG after adding all the dependency edges is still a directed acyclic graph.

[0031] S3. Prioritize all nodes in the DAG; the priorities of memory release nodes, operation nodes, and memory allocation nodes decrease sequentially; the priority of each memory allocation node increases with its latest start time; the priority of each memory release node decreases with its earliest start time; for memory allocation nodes with the same latest start time, the priority increases with the amount of memory they need to allocate; the priority of memory release nodes with the same earliest start time increases with the amount of memory they need to release. During the scheduling process, the resident cache capacity can be calculated in real time. Initially, the resident cache capacity is 0. When the node to be executed is a memory allocation node, the cache size of that node is increased by the current resident cache capacity. When the node to be executed is a memory release node, the cache size of that node is decreased by the current resident cache capacity. When the node to be executed is an operation node, the current resident cache capacity does not change. The maximum value of the resident cache capacity during the scheduling process is recorded in real time.

[0032] To ensure that the maximum total memory required during the entire scheduling process is minimized, that is, to ensure that the maximum value of the aforementioned resident cache capacity is minimized, all nodes in the DAG are prioritized. This embodiment proposes a three-level screening mechanism based on node type priority, time window priority, and memory impact priority. The first level prioritizes nodes by type, with memory release nodes having the highest priority (executed first to release cached resources), operation nodes having a medium priority (ensuring continuous execution of computational tasks), and memory allocation nodes having the lowest priority (delayed execution to slow down memory usage).

[0033] The second level performs timing optimization based on the earliest start time (EST) and latest start time (LST). It calculates the earliest start time (EST) of a node through forward propagation, considering the execution time of all predecessor nodes; and calculates the latest start time (LST) of a node through backward propagation, taking into account the time constraints of successor nodes. The priority of memory allocation nodes increases with their latest start time, appropriately delaying memory allocation. The priority of memory release nodes decreases with their earliest start time, prioritizing nodes that can release memory as early as possible. Specifically, the earliest start time of a node is the maximum sum of the earliest start times of all its predecessor nodes in the current DAG and the number of execution cycles of that predecessor node; the latest start time of a node is the minimum difference between the latest start time of all its successor nodes in the current DAG and the number of execution cycles of that predecessor node.

[0034] The third level, based on memory impact, evaluates the effect of node scheduling on subsequent memory usage, prioritizing the scheduling of nodes with the greatest impact on memory usage. This means prioritizing memory allocation nodes that can quickly unblock more memory release nodes, further optimizing peak memory usage. Specifically, for memory allocation nodes with the same latest start time, the priority increases as the required memory allocation increases; similarly, for memory release nodes with the same earliest start time, the priority increases as the required memory release increases.

[0035] S4. Repeat the topology sorting operation until all nodes in the DAG are added to the scheduling sequence, and output the current scheduling sequence. The topology sorting operation includes: obtaining the node with an in-degree of 0 from the DAG and adding it to the ready queue; taking the node with the highest priority in the current ready queue as the current node to be executed and adding it to the scheduling sequence; and decrementing the in-degree of the adjacent nodes of the current node to be executed by one.

[0036] Under the above approach, this embodiment uses a three-level filtering mechanism of node type priority, time window priority, and memory impact priority, combined with dependency constraint processing and L0 cache mutual exclusion constraint management, to generate a scheduling sequence that satisfies the topological order and minimizes the peak cache resident amount.

[0037] Furthermore, considering the limited memory space of each cache unit, if cache unit allocation overflow occurs, additional swap-in and swap-out operations will occur, consuming significant time and computational resources. Therefore, preferably, in one optional implementation, a cost-aware swap-in and swap-out operation strategy and an optimal adaptive dynamic memory allocation algorithm are proposed. Under hardware cache capacity constraints, physical address offsets are allocated to each buffer, intelligently deciding the timing of data swap-in and swap-out operations, and generating an optimized scheduling sequence containing swap-in and swap-out operation nodes to minimize swap-in and swap-out memory, i.e., minimize overflow memory. Specifically, as follows... Figure 2 As shown, in this embodiment, the above-described neural network processor core scheduling method further includes: Set up a corresponding list of free intervals for each cache type to record the starting address and size of all free space in each cache unit under the corresponding cache type; During the kernel scheduling process, the offset position of each currently pending node in the used cache unit is recorded, and the starting address and size of the free interval list of the corresponding cache type are updated. When the currently pending node is added to the scheduling sequence: When the current node to be executed is a memory allocation node, a free interval that meets preset conditions is searched from the corresponding free interval list. If a free interval is found, it is allocated to the current node to be executed. The preset conditions are: the difference between the size of the free interval and the memory capacity required by the current node to be executed is greater than a preset threshold (in one optional implementation, this is the minimum allocated memory of all memory allocation nodes in the DAG), and it is adjacent to the allocated memory interval of the candidate memory usage cycle. The candidate memory usage cycle is a memory usage cycle whose interval with the memory usage cycle required by the current node to be executed is less than a preset interval (in one optional implementation, this is 5 clock cycles). If no free interval is found, the allocated memory interval in the cache unit of the corresponding cache type is searched, and a swap-in / swap-out operation is performed on the memory interval. A dependency edge corresponding to each swap-in / swap-out operation is added to the DAG, including: the memory allocation node of the cache unit used to request the swap-out operation (denoted as...). ) points to the operation node used to perform the corresponding swap-out operation (denoted as ) Dependency edges, and operation nodes used to perform the corresponding swap operations. Points to the operation node used to perform the corresponding swap-in operation (denoted as ). The dependency edges of the swap operation and the operation nodes used to perform the corresponding swap operations. A memory release node (denoted as ) that points to the cache unit used to release the corresponding swap-in operation. The DAG is updated by adding dependent edges to the DAG, such as: 1) the execution node corresponding to the cache unit requesting the swap-out operation pointing to the node performing the corresponding swap-out operation; 2) the non-executed node corresponding to the cache unit corresponding to the swap-in operation pointing to the node performing the corresponding swap-in operation; 3) the newly added nodes in the DAG are prioritized and then the process moves to S4; 4) the execution nodes are nodes already in the scheduling sequence of the current DAG; 5) the non-executed nodes are nodes in the current DAG other than the execution nodes; 6) the priority of the node performing the swap-in operation is the same as the priority of the memory allocation node, and its priority increases with its latest start time; 7) the priority of the node performing the swap-out operation is the same as the priority of the memory release node, and its priority decreases with its earliest start time; 8) the priority of each node performing the swap-in operation with the same latest start time increases with the amount of memory it needs to allocate; 9) the priority of each node performing the swap-out operation with the same earliest start time increases with the amount of memory it needs to release. When the current node to be executed is a memory release node, merge the free intervals in the corresponding free interval list that are adjacent to the memory to be released by the current node to be executed, and update the starting address and size of all free spaces in each cache unit in the corresponding free interval list.

[0038] Preferably, in an optional implementation, a boundary marking method is used to detect and merge the free intervals in the corresponding free interval list that are adjacent to the memory to be released by the current node to be executed.

[0039] It should be noted that the method for searching the allocated memory range in the cache unit of the corresponding cache type can be the best-fit algorithm, worst-fit algorithm, first-fit algorithm, optimal-fit algorithm, greedy algorithm, etc. There is no limitation here, but the best-fit algorithm is preferred.

[0040] In the above process, the cost-aware swap-in / swap-out operation strategy is mainly reflected in the following: when the current node to be executed is a memory allocation node and a free interval that meets the preset conditions cannot be searched from the corresponding free interval list, the cost-aware swap-in / swap-out operation strategy is triggered, and the data buffer is temporarily stored in DDR memory. When needed, it is swapped into the cache. The specific strategy is as follows: Swap-in / swap-out cost calculation: distinguish whether the buffer is used by the node performing the data copy operation (COPY_IN operation) that copies data from other cache areas to the current cache area. The used buffer only needs to perform the swap-in operation (i.e., SPILL_IN operation: data copy operation from memory to cache), and the amount of data moved is equal to the size of the buffer; the unused buffer needs to perform the swap-out operation (i.e., SPILL_OUT operation: data copy operation from cache to memory) and the swap-in operation, and the amount of data moved is twice the size of the buffer.

[0041] Swap-in / swap-out decision-making mechanism: Prioritize swapping out buffers with high unit cost and short lifecycle to minimize total additional data transfer. Swap-in / swap-out node management: Generate operation nodes (denoted as ) to perform corresponding swap-out operations. ) and the operation node used to perform the corresponding swap-in operation (denoted as ) Automatic insertion of dependent edges ( , , The buffer usage nodes are demarcated to ensure data consistency, and the scheduling sequence is updated to a complete sequence containing operation nodes for performing the corresponding swap-out operation and operation nodes for performing the corresponding swap-in operation.

[0042] The optimal adaptive dynamic memory allocation algorithm is mainly reflected in the following aspects: by maintaining a list of free intervals for each cache type, selecting the free block whose size is closest to the target buffer for allocation, and merging adjacent free blocks when releasing, so as to reduce cache fragmentation and improve space utilization.

[0043] This embodiment analyzes the temporal locality characteristics of the buffer, implements spatial isolation for buffers with overlapping lifecycles to avoid conflicts, implements space reuse for buffers with staggered lifecycles to improve cache utilization, and establishes a cache fragmentation defragmentation mechanism to periodically compress space and slow down the cache fragmentation process.

[0044] Preferably, based on the above method, an optional implementation further reduces the total operator running time by optimizing the swap-in and swap-out positions, while ensuring that the total additional data transfer volume does not increase significantly (in this implementation, the average increase does not exceed 30%).

[0045] The aforementioned neural network processor kernel scheduling method further includes: step S5 executed after S4; wherein, S5 includes: S51. Using the scheduling sequence obtained in S4 as the initial scheduling sequence, adjust the initial scheduling sequence once or multiple times to obtain the adjusted scheduling sequence. The methods for adjusting the initial scheduling sequence include: For each cache unit whose memory usage cycle is greater than a preset cycle (in one optional implementation, the value is the median of all memory usage cycles) corresponding to the operation node for performing the swap-out operation, move it forward in the initial scheduling sequence, and use the resulting scheduling sequence as the adjusted scheduling sequence. And / or, For each cache unit whose memory usage cycle is greater than a preset cycle (in one optional implementation, the value is the median of all memory usage cycles) corresponding to the operation node for performing the swap-in operation, move it backward in the initial scheduling sequence, and use the resulting scheduling sequence as the adjusted scheduling sequence. It should be noted that the step size in the above adjustment process can be a random step size or a preset step size (such as the number of 2 nodes), and there is no limitation here.

[0046] The memory usage cycle of the cache unit is calculated based on the earliest start time and the latest start time of each node in the current DAG.

[0047] In the above process, the timing optimization of swap-in and swap-out operations is based on buffer lifecycle analysis to optimize the execution timing of swap-in and swap-out operation nodes: for buffers with long lifecycles (where the scheduling positions of memory release nodes and memory allocation nodes differ significantly), swap-out operations are performed in advance to free up cache space for use by emergency buffers; for buffers that are about to be used, swap-in operations are delayed by analyzing the latest start time (LST) of subsequent nodes to minimize data residence time and reduce unnecessary cache occupation.

[0048] S52. For the initial scheduling sequence and each adjusted scheduling sequence, the critical path analysis algorithm is used to calculate the total running time cycle and the total memory used for swap-in and swap-out operations during the execution of the scheduling sequence, and the weighted sum is used as the performance index of the scheduling sequence; the scheduling sequence corresponding to the maximum performance index is taken as the final scheduling sequence.

[0049] It should be noted that the above weights can be dynamically adjusted according to the application scenario; in edge device scenarios, the weight corresponding to the total runtime cycle... The weight corresponding to the total amount of memory used for swap-in and swap-out operations. In a data center scenario, the weight corresponding to the total runtime cycle. The weight corresponding to the total amount of memory used for swap-in and swap-out operations. To achieve an approximation of the multi-objective Pareto front.

[0050] The flowchart of the neural network processor core scheduling method based on minimum cache residency, cache allocation and swapping, and performance optimization strategies is as follows: Figure 3 As shown, this illustrates the solutions offered by three technical approaches and their inherent connections.

[0051] The final optimization results output in this embodiment include: the optimized scheduling sequence of all example computation graphs DAG, the address offset mapping corresponding to each buffer, the new address offset mapping of the buffer caused by swap-in and swap-out operations, the peak cache resident amount, the total additional data transfer amount, and the total operator running time cycle.

[0052] In summary, this embodiment addresses the shortcomings or improvement needs of existing general-purpose neural network accelerators, such as complex model adaptation, low scheduling efficiency, high peak memory consumption, and large data transfer overhead. It provides a core scheduling method for SIMD-based neural network processors to solve the technical problems of existing scheduling methods, such as difficulty in adapting to heterogeneous computation graphs, insufficient resource utilization, and poor execution performance. By constructing a hierarchical scheduling optimization framework, integrating multi-level priority list scheduling algorithms, dynamic cache partitioning management mechanisms, and lifecycle-aware optimization strategies, it achieves multi-objective collaborative optimization under strict hardware constraints, minimizing peak memory consumption, reducing data transfer overhead, and shortening total execution time. First, a minimum cache-resident scheduling sequence is generated through three-level priority filtering. Then, cache allocation and swap-in / swap-out operations are optimized based on a cost-aware strategy. Finally, execution performance is optimized through a triple joint strategy, including critical path analysis. This invention possesses good versatility and scalability and can be widely applied to SIMD-based neural network processors in edge computing scenarios.

[0053] This embodiment is highly versatile and stable: it can be adapted to various computation graphs with different structures and sizes, such as Matmul, FlashAttention, and Conv, and can output stably from 1716 nodes to 36086 nodes. It is suitable for different types of neural network operators and SIMD architecture processors.

[0054] The peak memory optimization effect of this embodiment is significant: the average peak cache resident rate is only 10.23%, of which the peak rate for the computation graph of the Conv structure is as low as 1.83%, which greatly reduces memory requirements and reduces the probability of swapping in and out operations due to insufficient memory, making it possible to deploy larger-scale neural network models on edge devices.

[0055] This embodiment features low data transfer overhead: through a cost-aware swap-in / swap-out operation strategy and an optimal adaptive dynamic memory allocation algorithm, the total amount of additional data transfer is effectively controlled, with an average swap-in / swap-out ratio of 11.74%, an average L1 cache utilization rate of 95.2%, and an average UB cache utilization rate of 93.7%, significantly improving cache resource utilization.

[0056] The execution performance of this embodiment is excellent: under the premise of ensuring that the amount of data transport does not increase significantly (average increase of 22.5%), the total running time is significantly reduced, with a maximum reduction of 87.0% and an average reduction of more than 26%, effectively improving the real-time performance of neural network inference.

[0057] This embodiment has good scalability: the algorithm's time complexity is controlled at the polynomial level, making it suitable for large-scale computation graph scheduling scenarios. On the largest Conv structure computation graph (36086 nodes), the single scheduling time is less than 70ms, meeting the real-time requirements.

[0058] Example 2 A neural network processor uses the in-core scheduling method of the neural network processor provided in Embodiment 1 of the present invention to implement the execution scheduling of neural network operators.

[0059] The related technical solutions are the same as the neural network processor kernel scheduling method provided in Embodiment 1 of the present invention, and are not limited here.

[0060] Example 3 An edge computing device includes the neural network processor provided in Embodiment 2 of the present invention.

[0061] The related technical solutions are the same as those provided in Embodiment 2 of this invention, and are not limited here.

[0062] Example 4 A neural network processor kernel scheduling system for SIMD architecture includes: a memory and a processor, wherein the memory stores a computer program, and the processor executes the neural network processor kernel scheduling method provided in Embodiment 1 of the present invention when executing the computer program.

[0063] The related technical solutions are the same as the neural network processor kernel scheduling method provided in Embodiment 1 of the present invention, and are not limited here.

[0064] Example 5 A computer-readable storage medium includes a stored computer program, wherein the computer program, when executed by a processor, controls the device where the storage medium is located to execute the neural network processor kernel scheduling method provided in Embodiment 1 of the present invention.

[0065] The related technical solutions are the same as the neural network processor kernel scheduling method provided in Embodiment 1 of the present invention, and are not limited here.

[0066] Example 6 A computer program product includes a computer program / instructions that, when executed by a processor, implement the neural network processor kernel scheduling method provided in Embodiment 1 of the present invention.

[0067] The related technical solutions are the same as the neural network processor kernel scheduling method provided in Embodiment 1 of the present invention, and are not limited here.

[0068] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A kernel scheduling method for neural network processors with SIMD architecture, characterized in that, include: S1. Receive a directed acyclic graph (DAG) representing the execution process within the kernel of a neural network operator; the DAG includes operation nodes, cache management nodes, and execution dependency edges between nodes; Cache management nodes include memory allocation nodes and memory release nodes; The cache management node carries the required cache type information and the memory capacity to be allocated or released; S2. Add a dependency edge to the DAG, where the memory release node for releasing the L0 cache unit points to the memory allocation node for requesting the L0 cache unit, and ensure that the DAG is still a directed acyclic graph. S3. Prioritize all nodes in the DAG; the priorities of memory release nodes, operation nodes, and memory allocation nodes decrease sequentially; the priority of each memory allocation node increases with its latest start time; the priority of each memory release node decreases with its earliest start time; for memory allocation nodes with the same latest start time, the priority increases with the amount of memory they need to allocate; the priority of memory release nodes with the same earliest start time increases with the amount of memory they need to release. S4. Repeat the topology sorting operation until all nodes in the DAG are added to the scheduling sequence, and output the current scheduling sequence. The topology sorting operation includes: obtaining the node with an in-degree of 0 from the DAG and adding it to the ready queue; taking the node with the highest priority in the current ready queue as the current node to be executed and adding it to the scheduling sequence; and decrementing the in-degree of the adjacent nodes of the current node to be executed by one.

2. The neural network processor kernel scheduling method according to claim 1, characterized in that, Also includes: Set up a corresponding list of free intervals for each cache type to record the starting address and size of all free space in each cache unit under the corresponding cache type; During the kernel scheduling process, the offset position of each currently pending node in the used cache unit is recorded, and the starting address and size of the free interval list of the corresponding cache type are updated. When the currently pending node is added to the scheduling sequence: When the current node to be executed is a memory allocation node, a free interval that meets preset conditions is searched from the corresponding free interval list. If a free interval is found, it is allocated to the current node to be executed. The preset conditions are: the difference between the size of the free interval and the memory capacity required by the current node to be executed is greater than a preset threshold, and it is adjacent to the allocated memory interval of the candidate memory usage cycle. The candidate memory usage cycle is a memory usage cycle whose interval with the memory usage cycle required by the current node to be executed is less than a preset interval. If the cache cannot be found, search the allocated memory range in the cache unit of the corresponding cache type and perform swap-in and swap-out operations on that memory range. Add dependency edges to the DAG for each swap-in / swap-out operation, including: a dependency edge from the memory allocation node of the cache unit requesting the swap-out operation to the operation node executing the corresponding swap-out operation; a dependency edge from the operation node executing the corresponding swap-out operation to the operation node executing the corresponding swap-in operation; and a dependency edge from the operation node executing the corresponding swap-in operation to the memory release node releasing the cache unit of the corresponding swap-in operation. Update the DAG by adding a dependency edge from the executed node corresponding to the cache unit requesting the swap-out operation to the operation node executing the corresponding swap-out operation, and from the operation node executing the corresponding swap-in operation to the unexecuted node corresponding to the cache unit of the swap-in operation. Then, update the DAG by adding new nodes to the DAG. After prioritizing, proceed to S4; wherein, the executed nodes are nodes already in the scheduling sequence of the current DAG; the unexecuted nodes are nodes in the current DAG other than the executed nodes; the priority of the operation node used to perform the swap-in operation is the same as the priority of the memory allocation node, and its priority increases with the increase of its latest start time; the priority of the operation node used to perform the swap-out operation is the same as the priority of the memory release node, and its priority decreases with the increase of its earliest start time; the priority of each operation node used to perform the swap-in operation with the same latest start time increases with the increase of the memory capacity it needs to allocate; the priority of each operation node used to perform the swap-out operation with the same earliest start time increases with the increase of the memory capacity it needs to release; When the current node to be executed is a memory release node, merge the free intervals in the corresponding free interval list that are adjacent to the memory to be released by the current node to be executed, and update the starting address and size of all free spaces in each cache unit in the corresponding free interval list.

3. The neural network processor kernel scheduling method according to claim 2, characterized in that, The best-fit algorithm is used to search for the allocated memory range in the cache unit corresponding to the cache type.

4. The neural network processor kernel scheduling method according to claim 2 or 3, characterized in that, Also includes: Step S5 is performed after S4; wherein, S5 includes: S51. Using the scheduling sequence obtained in S4 as the initial scheduling sequence, the initial scheduling sequence is adjusted once or multiple times to obtain the adjusted scheduling sequence. The method for adjusting the initial scheduling sequence includes: For each cache unit whose memory usage cycle is greater than the preset cycle, the corresponding operation node for performing the swap-out operation is moved forward in the initial scheduling sequence, and the resulting scheduling sequence is used as the adjusted scheduling sequence. And / or, For each cache unit whose memory usage period is greater than the preset period, the corresponding operation node for performing the swap-in operation is moved backward in the initial scheduling sequence, and the resulting scheduling sequence is used as the adjusted scheduling sequence. S52. For the initial scheduling sequence and each adjusted scheduling sequence, the critical path analysis algorithm is used to calculate the total running time cycle and the total memory used for swap-in and swap-out operations during the execution of the scheduling sequence, and the weighted sum is used as the performance index of the scheduling sequence; the scheduling sequence corresponding to the maximum performance index is taken as the final scheduling sequence.

5. A neural network processor, characterized in that, The execution scheduling of neural network operators is implemented using the intra-kernel scheduling method of any one of claims 1-4.

6. An edge computing device, characterized in that, Includes the neural network processor as described in claim 5.

7. A scheduling system within the core of a neural network processor for SIMD architecture, characterized in that, include: A memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the neural network processor kernel scheduling method according to any one of claims 1-4.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein the computer program, when executed by a processor, controls the device where the storage medium is located to perform the neural network processor kernel scheduling method according to any one of claims 1-4.

9. A computer program product, characterized in that, It includes a computer program / instruction that, when executed by a processor, implements the neural network processor kernel scheduling method according to any one of claims 1-4.

Images