Neural network processing unit scheduling system and method

Abstract

The invention provides a neural network processing unit scheduling system and method, relates to the field of artificial intelligence accelerators, and can set a task progress register in a neural network processing unit for recording the task processing progress of the neural network processing unit for processing and calculating sub-tasks and exposing the task processing progress to a software layer. A scheduler and a global shared task queue can be arranged in the runtime module. The scheduler is used for scheduling the calculation subtasks to the neural network processing units, screening the slow processing units according to the task processing progress of the neural network processing units, determining the remaining subtasks which are not completed by the slow processing units, taking the remaining subtasks as the calculation subtasks again, and sending the calculation subtasks to the neural network processing units; and re-issuing to the idle neural network processing unit through the global shared task queue. Therefore, the remaining sub-tasks which are not completed by the low-speed processing unit can be re-issued to the idle neural network processing unit, so that idle computing resources can be avoided, and the processing efficiency is improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence accelerators, and in particular to a neural network processing unit scheduling system and method. Background Technology

[0002] Current multi-core NPU (Neural Processing Unit) architectures generally employ a static task partitioning strategy, distributing input tensors equally across cores according to spatial dimensions (such as height and width) or channel dimensions during model deployment. However, due to runtime perturbations (such as the unpredictable latency of off-chip memory access), even with the same workload, the actual completion time of each core can vary significantly. Fast cores must wait for slower cores to complete their tasks before moving to the next computation stage, resulting in idle computing resources and limited system throughput.

[0003] In related technologies, some solutions attempt to adjust the initial block ratio through offline performance modeling, but cannot cope with runtime disturbances during a single inference process; other solutions introduce hardware performance counters to monitor stall cycles, but lack closed-loop linkage with task scheduling. More importantly, existing neural network processing unit hardware typically does not expose task execution progress to the software layer, and the runtime module cannot know the real-time working status of each core, making it difficult to dynamically adjust the allocation strategy during task execution. Summary of the Invention

[0004] The purpose of this invention is to provide a neural network processing unit scheduling system and method, which can more effectively schedule tasks for multiple neural network processing units through software and hardware improvements, thereby reducing the waste of computing resources caused by inconsistent task progress of different neural network processing units.

[0005] To address the aforementioned technical problems, this invention provides a neural network processing unit scheduling system, comprising: The system includes a host computer and at least two neural network processing units. The host computer deploys a runtime module, which includes a globally shared task queue and a scheduler. The neural network processing units include task progress registers. The host computer is connected to the neural network processing units. The scheduler is configured to schedule computational subtasks to each neural network processing unit for processing; when a task completion interruption signal is received from any neural network processing unit, the scheduler reads the task progress register of each neural network processing unit to obtain the task processing progress; determines the slow processing unit and its unfinished remaining subtasks based on the task processing progress; sets the remaining subtasks as computational subtasks and adds them to the global shared task queue; and when a neural network processing unit requests to retrieve a computational subtask from the global shared task queue, the scheduler schedules the computational subtask to the neural network processing unit for processing. The neural network processing unit is configured to process the computational subtask and record the task processing progress through the task progress register; when the processing is completed, it sends the task completion interrupt signal, enters an idle state, and checks whether there is a computational subtask in the global shared task queue; if there is, it requests the scheduler to obtain the computational subtask.

[0006] Optionally, the neural network processing unit further includes a programmable address generation unit, and the neural network processing unit is connected to memory; The scheduler is further configured to configure address generation parameters for the programmable address generation unit of the neural network processing unit based on the tensor metadata, memory layout information, and spatial partitioning information corresponding to the tensor data to be processed in the computational subtask. The neural network processing unit is further configured to generate a first memory address and a second memory address using the programmable address generation unit, load each data unit in the tensor data in an orderly manner from the memory according to the first memory address, and write the processed data of the tensor data into the memory according to the second memory address.

[0007] Optionally, the scheduler is further configured to: Determine whether the task processing progress is greater than a preset threshold; If so, then the neural network processing unit to which the task processing progress belongs is determined to be the slow processing unit; The task processing progress of the slow processing unit is mapped to the spatial offset parameter of the programmable address generation unit of the slow processing unit, and the remaining sub-tasks that the slow processing unit has not completed are determined according to the spatial offset parameter.

[0008] Optionally, the runtime module further includes a task dependency manager; The task dependency manager is configured to record the dependencies and synchronization groups between computing tasks; The scheduler is also configured to: Obtain the current computing task, and determine whether there are subsequent computing tasks that depend on the current computing task based on the dependency relationship; If present, a synchronization group is created in the task dependency manager, the current computing task is divided into the computing subtasks, and all computing subtasks are added to the synchronization group; wherein, the scheduler only issues subsequent computing tasks that depend on the current computing task when all computing subtasks in the synchronization group are completed; If not, the current computing task is divided into the computing subtasks; The computational subtasks are distributed to each neural network processing unit or added to the global shared task queue.

[0009] Optionally, the scheduler is further configured to: When a task completion interruption signal is received from any neural network processing unit, it is determined whether the computational subtask executed by the neural network processing unit belongs to the synchronization group. If the computational subtask belongs to the synchronization group, then determine whether all computational subtasks in the synchronization group have been completed; If all the computational subtasks in the synchronization group have been completed, then the next computational task is obtained, and the process proceeds to the step of determining whether there are subsequent computational tasks that depend on the computational task based on the dependency relationship. If the computational subtask in the synchronization group is not completed, the process proceeds to the step of determining the slow processing unit based on the task processing progress, and determining whether the slow processing unit exists. If the slow processing unit does not exist, then the next computing task that does not depend on the currently executed computing task is obtained according to the dependency relationship, and the process proceeds to the step of determining whether there is a subsequent computing task that depends on the computing task according to the dependency relationship. If the slow processing unit exists, proceed to the step of determining the remaining sub-tasks that the slow processing unit has not completed based on the task processing progress; If the computational subtask does not belong to the synchronization group, then proceed to the step of determining the slow processing unit based on the task processing progress, and determine whether the slow processing unit exists. If the slow processing unit does not exist, the next computing task is obtained, and the process proceeds to the step of determining whether there are subsequent computing tasks that depend on the computing task based on the dependency relationship. If the slow processing unit exists, proceed to the step of determining the remaining sub-tasks that the slow processing unit has not completed based on the task processing progress.

[0010] Optionally, the scheduler is further configured to: Obtain the task identifier of the current computing task; wherein, the task identifier contains tensor metadata, memory layout information, and spatial partitioning information of the complete tensor data to be processed by the current computing task; The computational subtasks of each neural network processing unit are set according to the task identifiers to obtain the subtask identifiers of the computational subtasks, so as to evenly distribute the computational tasks to each neural network processing unit; wherein, the subtask identifiers include tensor metadata, memory layout information, spatial partitioning information and configuration templates corresponding to the tensor data to be processed by the computational subtasks. The subtask identifier is distributed to each neural network processing unit or added to the global shared task queue.

[0011] Optionally, the scheduler is further configured to: The hash table of the computational subtask is configured in the programmable address generation unit of the neural network processing unit; wherein the hash table is used to record the zero elements in the tensor data; The programmable address generation unit is configured to determine the zero element in the tensor data according to the hash table and skip the zero element in task processing.

[0012] Optionally, the scheduler is further configured to: The computational subtask is parsed to determine the weight tensor information and memory layout information of the weight tensor required to execute the computational subtask; Based on the weight tensor information and memory layout information of the weight tensor, a weight prefetch instruction is set, and the weight prefetch instruction is sent to the neural network processing unit so that the neural network processing unit loads the weight tensor according to the weight prefetch instruction.

[0013] Optionally, inter-core communication is provided between the neural network processing units; The scheduler is also configured to: When it is determined that multiple computational subtasks depend on the same intermediate data, a shared tag is added to the computational subtasks; The neural network processing unit is further configured as follows: When it is determined that the computational subtask has the shared tag, the intermediate data is loaded to other neural network processing units through the inter-core communication.

[0014] The present invention also provides a neural network processing unit scheduling method, applied to a neural network processing unit scheduling system, the neural network processing unit scheduling system including a host and at least two neural network processing units, the host deploying a runtime module, the runtime module including a globally shared task queue and a scheduler, the neural network processing unit including a task progress register, and the host being connected to the neural network processing unit; The method includes: The scheduler schedules computational subtasks to each neural network processing unit for processing. The neural network processing unit processes the computational subtask and records the task processing progress through the task progress register. When the neural network processing unit completes processing, it sends a task completion interrupt signal, enters an idle state, and checks whether there are any computational subtasks in the global shared task queue. When the scheduler receives a task completion interruption signal from any neural network processing unit, it reads the task progress register of each neural network processing unit to obtain the task processing progress. The scheduler determines the slow processing units and their unfinished remaining subtasks based on the task processing progress. The scheduler sets the remaining subtasks as computation subtasks and adds them to the global shared task queue; When the neural network processing unit detects that there is a computational subtask in the global shared task queue, it requests the scheduler to obtain the computational subtask. When the scheduler detects that the neural network processing unit requests to obtain a computational subtask from the global shared task queue, it schedules the computational subtask to the neural network processing unit for processing.

[0015] This invention provides a neural network processing unit scheduling system, comprising: a host and at least two neural network processing units. The host deploys a runtime module, which includes a globally shared task queue and a scheduler. Each neural network processing unit includes a task progress register. The host is connected to the neural network processing units. The scheduler is configured to schedule computational subtasks to each neural network processing unit for processing. When a task completion interruption signal is received from any neural network processing unit, the scheduler reads the task progress register of each neural network processing unit to obtain the task processing progress. Based on the task processing progress, it determines the slow processing unit and its unfinished remaining subtasks. The remaining subtasks are set as computational subtasks and added to the globally shared task queue. When a request is detected from a neural network processing unit to retrieve a computational subtask from the globally shared task queue, the scheduler schedules the computational subtask to the neural network processing unit for processing. Each neural network processing unit is configured to process the computational subtask and record the task processing progress through the task progress register. Upon completion, it sends a task completion interruption signal, enters an idle state, and checks whether a computational subtask exists in the globally shared task queue. If it exists, it requests the computational subtask from the scheduler.

[0016] The beneficial effects of this invention are as follows: Firstly, a task progress register can be set in the neural network processing unit to record the task processing progress of the neural network processing unit in processing computational subtasks, and this task processing progress is exposed to the software layer. Furthermore, this invention can also set a scheduler and a globally shared task queue in the runtime module. The scheduler is used to schedule computational subtasks to each neural network processing unit, and to filter slow processing units based on their task processing progress, identify the remaining subtasks that slow processing units have not completed, and then reassign the remaining subtasks as computational subtasks, and redistribute them to idle neural network processing units through the globally shared task queue. In actual operation, the scheduler first schedules the computational subtasks to each neural network processing unit for processing, and the neural network processing unit processes the computational subtasks and records the task processing progress using the task progress register. Subsequently, when the neural network processing unit completes task processing, it can send a task completion interrupt signal, enter an idle state, and check whether there are any computational subtasks in the globally shared task queue. When the scheduler receives a task completion interrupt signal from any neural network processing unit, it reads the task processing progress from the task progress register of each neural network processing unit and determines the slow processing unit and its unfinished subtasks based on the progress. The scheduler then sets the remaining subtasks as computational subtasks and adds them to the global shared task queue. When a neural network processing unit detects a computational subtask in the global shared task queue, it requests the scheduler to acquire and process it. Therefore, this invention, through hardware improvements to the neural network processing unit and software improvements to the runtime module, can filter out slow processing units based on the task processing progress reported by the neural network processing unit and reassign the unfinished subtasks of the slow processing units to idle neural network processing units. Faster cores can then assist slower cores in completing computational subtasks, thereby avoiding idle computing resources and improving system throughput efficiency.

[0017] The present invention also provides a method for scheduling neural network processing units, which has the above-mentioned beneficial effects. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0019] Figure 1 A structural block diagram of a neural network processing unit scheduling system provided in an embodiment of the present invention; Figure 2A schematic diagram of tensor data provided in an embodiment of the present invention; Figure 3 This is a structural block diagram of a neural network processing module provided in an embodiment of the present invention; Figure 4 A structural block diagram of another neural network processing module scheduling system provided in an embodiment of the present invention; Figure 5 A flowchart of a neural network processing unit scheduling method provided in an embodiment of the present invention; Figure 6 A flowchart of another neural network processing unit scheduling method provided in an embodiment of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Current multi-core NPU (Neural Processing Unit) architectures generally employ a static task partitioning strategy, distributing input tensors equally across cores according to spatial dimensions (such as height and width) or channel dimensions during model deployment. However, due to runtime perturbations (such as the unpredictable latency of off-chip memory access), even with the same workload, the actual completion time of each core can vary significantly. Fast cores must wait for slower cores to complete their tasks before moving to the next computation stage, resulting in idle computing resources and limited system throughput.

[0022] In related technologies, some solutions attempt to adjust the initial block ratio through offline performance modeling, but cannot cope with runtime disturbances during a single inference process; other solutions introduce hardware performance counters to monitor stall cycles, but lack closed-loop linkage with task scheduling. More importantly, existing neural network processing unit hardware typically does not expose task execution progress to the software layer, and the runtime module cannot know the real-time working status of each core, making it difficult to dynamically adjust the allocation strategy during task execution.

[0023] In view of this, in order to reduce the problem of idle computing resources caused by inconsistent task processing progress of different neural network processing units, the present invention provides a neural network processing unit scheduling system. Through hardware and software improvements, the system can perform more efficient task scheduling of multiple neural network processing units, thereby reducing the problem of wasted computing resources caused by inconsistent task progress of different neural network processing units.

[0024] For easier understanding, please refer to Figure 1 , Figure 1 This is a structural block diagram of a neural network processing unit scheduling system provided in an embodiment of the present invention. In one embodiment, the system may include: a host and at least two neural network processing units, with the host connected to each neural network processing unit. A runtime module is deployed in the host, and the runtime module may include a Global Shared Task Queue (GSTQ) and a scheduler. Each neural network processing unit may include a Task Progress Register (TPR).

[0025] The basic working mode between the runtime module and each neural network processing unit is as follows: the runtime module divides the current computation task into computational subtasks and distributes them to each neural network processing unit for processing. These computational tasks include convolution, matrix multiplication, pooling, normalization, attention mechanisms, and custom operators, generally involving different forms of computation on the input tensor data. To achieve parallel processing, the runtime module can divide the input tensor data of the computational task into multiple computational subtasks, allowing each neural network processing unit to process different regions of the input tensor data and merge the processing results into a complete output tensor data.

[0026] It should be noted that the runtime module is a software module that can run on various hardware components within the host machine. For example, the scheduler in the runtime module can be deployed on the host processor or a dedicated scheduling coprocessor, and the scheduling logic can be implemented purely in software, a hybrid of software and hardware (such as hardware-accelerated queue management), or in microcode. The globally shared task queue can be deployed in host memory or a dedicated scheduling coprocessor. Furthermore, the scheduler can also run on the control core of the neural network processing unit, and the globally shared task queue can also be deployed in the on-chip SRAM (Static Random-Access Memory) of the neural network processing unit.

[0027] In addition, the neural network processing unit may also include other hardware structures. For example, the neural network processing unit may also include a task control interface for receiving start instructions from the runtime module and triggering the cooperative execution pipeline. The neural network processing module may also include units for calculating multiply-accumulate operations, such as systolic arrays and MACs, as well as internal storage SRAM, controllers, schedulers, buffers, and other modules required to implement the corresponding computational tasks.

[0028] In addition, this embodiment does not limit the connection method between the host and the neural network processing unit. For example, it can be connected through a bus structure, and related technologies can be consulted.

[0029] The following describes the specific uses of the scheduler, the globally shared task queue, and the task progress register in task scheduling: The scheduler is configured to schedule computational subtasks to various neural network processing units for processing; when a task completion interrupt signal is received from any neural network processing unit, it reads the task progress register of each neural network processing unit to obtain the task processing progress; it determines the slow processing unit and its unfinished remaining subtasks based on the task processing progress; it sets the remaining subtasks as computational subtasks and adds them to the global shared task queue; when a neural network processing unit requests to retrieve computational subtasks from the global shared task queue, it schedules the computational subtasks to the neural network processing unit for processing. The neural network processing unit is configured to process computational subtasks and record the task processing progress through a task progress register. When processing is complete, it sends a task completion interrupt signal, enters an idle state, and checks whether there is a computational subtask in the global shared task queue. If there is, it requests the scheduler to obtain the computational subtask.

[0030] In this embodiment, to address the problem that the neural network processing unit cannot expose the task processing progress to the software layer and the runtime module cannot effectively schedule tasks based on the task processing progress, improvements can be made to the neural network processing unit and the runtime module respectively.

[0031] In this embodiment, a task progress register can be set in the neural network processing unit. This task progress register is automatically maintained by the neural network processing unit hardware and is used to record the current task processing progress in real time, and expose the progress to the outside world in the form of the number of data units completed or normalized percentage.

[0032] Furthermore, this embodiment can also improve the processing flow of the neural network processing unit. When the neural network processing unit completes the processing of a computational subtask, it can enter an idle state and begin detecting whether there are computational subtasks in the globally shared task queue. If it is determined that a computational subtask has been added to the queue, the neural network processing unit can request the runtime module to obtain and process the computational subtask.

[0033] In the runtime module, this embodiment can set up a scheduler and a globally shared task queue.

[0034] The scheduler, a task scheduling component within the runtime module, is primarily used to allocate and schedule computational subtasks to neural network processing units. Furthermore, based on the task processing progress recorded by each neural network processing unit, the scheduler can filter out slow-moving processing units and determine the remaining unfinished subtasks from these units. These remaining subtasks are then reassigned as computational subtasks and added to a global shared task queue, which is then used to redistribute the computational subtasks to idle neural network processing units.

[0035] It is worth noting that, in order to avoid wasting computing resources and ensure scheduling effectiveness, the scheduler does not periodically poll the progress registers of each task. Instead, it can adopt an event-driven, on-demand reading approach: when it receives a task completion interrupt signal from any neural network processing unit (i.e., the neural network processing unit has completed the assigned computational subtask), the scheduler reads the task progress registers of each neural network processing unit to obtain the task processing progress of each neural network processing unit.

[0036] The Global Shared Task Queue (GSTQ) is a task pool visible to all neural network processing units, used to store computational subtasks. A flag can be set in the GSTQ to indicate whether a new task is available. When a new computational subtask is added to the queue, this flag is broadcast to all idle neural network processing units, triggering them to retrieve the task from the queue from the scheduler. GSTQ can be a single global queue or multiple priority queues (e.g., separating high / low priority tasks).

[0037] As can be seen, in the architecture provided in this embodiment, each neural network processing unit continuously updates its Task Processing Progress Register (TPR) to reflect the completion status of the currently executing task. Based on this task processing progress information, the runtime module can determine the overall balance of system execution. If it is found that the remaining workload of a certain processing unit core is significantly greater than that of other cores (e.g., its TPR growth is slow or stagnant), the runtime module will dynamically split the computational subtasks handled by that core: retaining the currently loaded or processed portion, and extracting the sub-regions that have not yet started processing into an independent, structurally complete computational subtask. The newly formed computational subtask is then written into a globally shared task queue. Any idle neural network processing unit can "receive" a task to be processed from the queue without having to bind to a specific task source in advance. In this way, through hardware and software improvements, this invention can promptly filter out slow processing units and redistribute the remaining subtasks that slow processing units have not completed to idle neural network processing units, that is, fast cores help slow cores complete computational subtasks, thereby avoiding idle computing resources and improving system throughput efficiency.

[0038] In another embodiment, the neural network processing unit may further include a programmable address generation unit. Furthermore, the neural network processing unit may be connected to memory.

[0039] The scheduler can also be configured to configure address generation parameters for the programmable address generation unit of the neural network processing unit based on the tensor element information, memory layout information, and spatial partitioning information corresponding to the tensor data to be processed in the computational subtasks. The neural network processing unit can also be configured to generate a first memory address and a second memory address using a programmable address generation unit, load each data unit in the tensor data in an orderly manner from memory according to the first memory address, and write the processed data of the tensor data to memory according to the second memory address.

[0040] In this embodiment, the tensor data to be processed by the computational subtask is stored in memory, and the processing results of the computational subtask also need to be written to memory. Therefore, to facilitate data retrieval and result return, the neural network processing unit may also include a programmable address generation unit (AGU). This programmable address generation unit is used to configure address generation parameters and generate consecutive memory access addresses based on the address generation parameters. The address generation parameters may include a base address, tensor shape, memory step size, spatial offset, and block size. The base address represents the starting storage location of the tensor data in storage (such as memory), the tensor shape represents the geometric shape of the tensor data, the memory step size represents the sliding distance during computation within the tensor data, the spatial offset represents the offset address of different parts of the tensor data, and the block size is the partitioning information of the tensor data. The address generation parameters are generated based on the tensor metadata, memory layout information, and spatial partitioning information of the computational subtask. The tensor metadata includes metadata such as tensor dimension and shape; the memory layout information indicates the storage layout of the tensor data in memory; and the spatial partitioning information indicates the partitioning of the tensor data.

[0041] To facilitate understanding, the following example illustrates how to set and use the address generation parameters. Please refer to [link / reference]. Figure 2 , Figure 2 This is a schematic diagram of tensor data provided in an embodiment of the present invention. Figure 2 The two cubes shown are both feature maps, derived from the same complete feature map. Therefore, it is necessary to calculate all elements on each of these two feature maps separately, requiring data retrieval from external storage such as memory. Figure 1 If the base address in storage is 1, then the feature Figure 2 Its base address is 10. Features Figure 1 The address range of each element in the code is 1 to 9. Figure 2 The address range of each element in the array is 10~18. (Characteristics) Figure 2 and 1 The calculation method is similar, the difference being that they have different addresses in memory. Therefore, an offset address needs to be set to distinguish them. For example, in this example, the offset address is 9. The memory step size is the distance between the sliding points when each feature map is calculated internally; for example, if the step size is 2, the calculation order of the feature maps is 1, 3, 7, 9. The programmable address generation unit can generate memory access addresses regularly according to the size of this step size.

[0042] With a programmable address generation unit, the scheduler, when issuing tasks, can generate address generation parameters based on the tensor metadata, memory layout, and spatial partitioning information corresponding to the tensor data to be processed in the computational subtask. These parameters are then configured into the register group of the programmable address generation unit via a dedicated control bus. Upon task startup, the neural network processing unit can use the programmable address generation unit to generate a first memory address and a second memory address. Based on the first memory address, it sequentially loads each data unit from the tensor data from memory, and based on the second memory address, it writes the processed data of the tensor data to memory. This ensures that the neural network processing unit reads and processes data units in an orderly manner and writes back the processed data in an orderly fashion.

[0043] It's worth noting that during task partitioning, the scheduler ensures that newly generated computational subtasks belong to the same logical output tensor as their source tasks, and that their results can be aggregated by subsequent dependent tasks. In other words, all computational subtasks belonging to the same computational task share the same output memory layout and address space, allowing all neural network processing units to read and write back in an orderly manner. Furthermore, even if a computational subtask of a preceding task is moved to another core, subsequent computational tasks do not need to be aware of whether the preceding task has been dynamically moved.

[0044] Furthermore, the configuration of the programmable address generation unit can be issued through register writing, descriptor DMA loading, or instruction stream embedding. The layout formats supported by the programmable address generation unit include, but are not limited to: NCHW, NHWC, blocked compression, Winograd field, and sparse CSR / BCSR; The AGU can integrate address verification and out-of-bounds interrupt functions, which are captured by the Runtime for debugging or recovery.

[0045] In another implementation, the scheduler can also be configured as: Determine if the task processing progress exceeds a preset threshold; If so, then the neural network processing unit to which the task processing progress belongs is determined to be a slow processing unit; The task processing progress of the slow processing unit is mapped to the spatial offset parameter of the programmable address generation unit of the slow processing unit, and the remaining subtasks that the slow processing unit has not completed are determined based on the spatial offset parameter.

[0046] In this embodiment, the scheduler first identifies neural network processing units whose task processing progress exceeds a preset threshold as slow processing units based on the relationship between task processing progress and a preset threshold. It should be noted that this preset threshold can be set manually or dynamically based on the task processing progress of each non-idle neural network processing unit, such as setting the average task processing progress of each non-idle neural network processing unit as the preset threshold.

[0047] Furthermore, this embodiment can map the task processing progress to the spatial offset parameter of the programmable address generation unit. In other words, the processed portion of the tensor data by the neural network processing module can be determined based on the task processing progress, thereby identifying the unprocessed portion as the remaining subtasks. Thus, after identifying the slow processing unit, this embodiment can map the task processing progress of the slow processing unit to the spatial offset parameter of the programmable address generation unit of the slow processing unit, and determine the remaining subtasks that the slow processing unit has not completed based on the spatial offset parameter.

[0048] In another implementation, the scheduler is further configured as follows: The hash table for the computational subtask is configured into the programmable address generation unit of the neural network processing unit; wherein, the hash table is used to record the zero elements in the tensor data; A programmable address generation unit is configured to determine the zero element in the tensor data based on a hash table and skip the zero element during task processing.

[0049] In this embodiment, to improve processing efficiency, the scheduler can also configure a hash table for the programmable address generation unit. This hash table is used to record zero elements in the tensor data. When generating an address, the programmable address generation unit can determine whether the data unit currently read from the tensor data is a zero element based on the hash table. If so, it directly returns a zero value to the core of the neural network processing unit to skip zero elements in task processing.

[0050] In another implementation, the runtime module also includes a task dependency manager.

[0051] The task dependency manager is configured to record dependencies and synchronization groups between computation tasks. The scheduler is also configured as follows: Obtain the current computing task and determine whether there are subsequent computing tasks that depend on the current computing task based on the dependencies; If present, a synchronization group is created in the task dependency manager, the current computing task is divided into computing subtasks, and all computing subtasks are added to the synchronization group; the scheduler will only issue subsequent computing tasks that depend on the current computing task when all computing subtasks in the synchronization group are completed. If not, the current computation task will be divided into computation subtasks; The computational subtasks are distributed to each neural network processing unit or added to the global shared task queue.

[0052] In neural network computation graphs, many operators have strict data dependencies. For example, after a convolutional layer, batch normalization (BatchNorm) or element-wise addition (Add) occurs, subsequent computation tasks must wait for all output data from preceding tasks to be ready before starting. Therefore, when dividing computational subtasks, the dependencies between tasks must be considered. If it is determined that the current computational task is dependent on by subsequent computational tasks, the runtime module must ensure that all computational subtasks in the current task are completed before scheduling subsequent computational tasks that depend on the current task.

[0053] Therefore, the runtime module in this embodiment may further include a task dependency manager to record the dependencies and synchronization groups between computation tasks. When dividing the current computation task, the scheduler can determine whether there are subsequent computation tasks that depend on the current computation task based on the dependencies. If so, the scheduler can create a synchronization group in the task dependency manager, divide the computation task into computation subtasks, and add all computation subtasks to the synchronization group. The scheduler only issues subsequent computation tasks that depend on the current computation task after all computation subtasks in the synchronization group have been completed. If there are no subsequent computation tasks that depend on the current computation task, the scheduler does not need to create a synchronization group; it only needs to divide the current computation task into computation subtasks. Subsequently, the scheduler can issue the computation subtasks to each neural network processing unit or add them to the global shared task queue.

[0054] In another implementation, the scheduler is further configured as follows: Obtain the task identifier of the current computation task; where the task identifier contains tensor element information, memory layout information, and spatial partitioning information of the complete tensor data to be processed by the current computation task. The computational subtasks of each neural network processing unit are set according to the task identifiers, and the subtask identifiers of the computational subtasks are obtained so as to evenly distribute the computational tasks to each neural network processing unit. The subtask identifiers contain tensor element information, memory layout information, spatial partitioning information and configuration templates corresponding to the tensor data to be processed by the computational subtask. Subtask identifiers are distributed to each neural network processing unit or added to the global shared task queue.

[0055] In this embodiment, the global shared task queue can store subtask descriptors to be executed. Each descriptor contains tensor metadata, memory layout identifier, spatial partition coordinates, and AGU configuration template. The scheduler obtains the task identifier of the current computation task, extracts the tensor metadata, memory layout information, and spatial partition information of the complete tensor data to be processed by the current computation task from the task identifier, and sets the computation subtasks of each neural network processing unit to obtain the subtask identifiers of the computation subtasks, so as to evenly distribute the computation tasks to each neural network processing unit. Subsequently, the scheduler can send the subtask identifiers to each neural network processing unit. For example, based on the subtask identifier information, it can generate a set of configuration parameters adapted to the target core AGU interface, including base address, step size, block start offset, size, etc., and write the AGU configuration parameters into the AGU configuration register group of the target core through a dedicated control bus. Alternatively, the scheduler can also add the subtask identifiers to the global shared task queue.

[0056] In another implementation, the scheduler can also be configured as: When a task completion interrupt signal is received from any neural network processing unit, it is determined whether the computational subtask executed by the neural network processing unit belongs to the synchronization group. If the computational subtask belongs to the synchronization group, then determine whether all computational subtasks in the synchronization group have been completed. If all computational subtasks in the synchronization group have been completed, then the next computational task is obtained, and the process proceeds to the step of determining whether there are subsequent computational tasks that depend on the computational task based on the dependency relationship. If the computational subtask in the synchronization group is not completed, proceed to the step of determining the slow processing unit based on the task processing progress, and determine whether there is a slow processing unit. If there is no slow processing unit, then the next computing task that does not depend on the currently executed computing task is obtained according to the dependency relationship, and the process proceeds to the step of determining whether there are subsequent computing tasks that depend on the computing task according to the dependency relationship. If a slow processing unit exists, proceed to the step of determining the remaining subtasks that the slow processing unit has not completed based on the task processing progress. If the computational subtask does not belong to the synchronization group, proceed to the step of determining the slow processing unit based on the task processing progress, and determine whether there is a slow processing unit. If no slow processing unit exists, the next computation task is obtained, and the process proceeds to the step of determining whether there are subsequent computation tasks that depend on the computation task based on the dependency relationship. If a slow processing unit exists, proceed to the step of determining the remaining subtasks that the slow processing unit has not completed based on the task processing progress.

[0057] In this embodiment, due to the dependencies between computing tasks, there are multiple scenarios when the scheduler allocates the next computing subtask to an idle neural network processing unit.

[0058] First, if the computational subtasks completed by the neural network processing unit are determined to belong to a synchronization group, the scheduler needs to determine whether all computational subtasks in that synchronization group have been completed. If all computational subtasks in the synchronization group have been completed, it means that the current computational task has been completed, and the scheduler can schedule the next computational task that depends on the current computational task. If there are still unfinished computational subtasks in the synchronization group, the scheduler needs to check whether there are slow processing units. If so, the remaining subtasks of the slow processing units can be reassigned to idle neural network processing units.

[0059] It is worth noting that when it is determined that there are still unfinished computational subtasks in the synchronization group, and there are no slow processing units at this time, the scheduler can either wait for the synchronization group to complete its processing or schedule the next computational task that is independent of the current computational task for the idle neural network processing unit. For example, if a core completes its assigned task, but there are still other cores in its original synchronization group that have not completed their tasks, but there are other independent tasks to be processed in the system (such as those from different requests or different model branches), the scheduler can directly assign these external tasks to the idle core. In this way, this embodiment can perform task scheduling more flexibly and improve resource utilization.

[0060] Furthermore, if it is determined that the computational subtask completed by the neural network processing unit does not belong to the synchronization group, the scheduler needs to check whether there is a slow processing unit. If there is, the remaining subtasks of the slow processing unit can be reassigned to the idle neural network processing unit. If there is no slow processing unit, the next computational task can be scheduled.

[0061] As can be seen, when there is a data dependency between adjacent tasks, the runtime module in this embodiment can adopt a phased scheduling and dependency barrier management mechanism: Intra-stage parallelism: For multiple parallel branches with the same input dependency (such as multi-path convolution in the Inception module), the runtime module independently distributes the tasks of each branch to different cores and allows them to execute concurrently. The TPR of each core is only used for internal load balancing and does not affect cross-branch synchronization.

[0062] Inter-stage synchronization: When a group of tasks (such as all output tiles of a certain layer) serve as input for the next task, the runtime module marks these tasks as the same "synchronization group". Only when all subtasks in the group are completed (i.e., all relevant TPRs reach 100% and the corresponding interrupts have been received) will the runtime module add the task descriptor of the next stage to GSTQ or directly dispatch it.

[0063] When there is no data dependency between adjacent tasks (e.g., two independent detection heads, or different samples in multiple batch requests), the scheduling flexibility of the runtime module in this embodiment is significantly improved, allowing for more aggressive resource reuse strategies: Cross-task task acquisition: If a core completes its assigned task, but other cores in its original synchronization group have not yet completed their tasks, and there are other undependent pending tasks in the system (such as those from different requests or different model branches), the runtime module can directly assign these external tasks to the idle core.

[0064] Priority-driven scheduling: The runtime module can assign priority labels to different tasks (e.g., tasks with high real-time requirements are marked as high priority). When multiple tasks exist in GSTQ at the same time, idle cores will prioritize taking the high-priority tasks to ensure Quality of Service (QoS).

[0065] Heterogeneous task hybrid execution: In systems that support multi-model concurrency, different tasks may correspond to different AGU configuration modes (such as convolution vs. matrix multiplication). Since AGUs support runtime reconfiguration, runtime modules can alternately execute different types of tasks on the same core, as long as their memory regions do not conflict with each other.

[0066] It should be noted that, in order to achieve task scheduling in various scenarios, the following must be satisfied: 1. Semantic alignment between TPR and AGU: The progress reflected by TPR (such as the processed height range) can be directly mapped to the spatial offset parameters of AGU, enabling dynamic segmentation to have accuracy at the physical address level. 2. GSTQ's atomic operation guarantee: Task enqueueing and dequeueing are guaranteed to be consistent through hardware atomic instructions or locking mechanisms to prevent multi-core contention; 3. Context isolation in AGU configuration: Each time a task switches, the Runtime completely rewrites the AGU register set to ensure that the new task is not affected by the previous state; 4. Lifecycle management of dependency groups: Runtime maintains a reference count for each synchronization group and releases downstream tasks only when the count reaches zero.

[0067] The following examples illustrate scheduling in scenarios with dependent tasks and scenarios without dependent tasks.

[0068] Scheduling in scenarios with task dependencies: Consider a convolutional layer followed by a sequence of element-wise additions. The convolutional output tensor is divided into two spatial tiles, processed by Core0 and Core1 respectively. The runtime marks these two tiles as belonging to the same synchronization group.

[0069] During execution, Core0 completes its task first and issues an interrupt. The runtime reads Core1's TPR and finds that its progress is lagging. Since the Add operator depends on the complete output, the runtime does not immediately start Add. Instead, it divides the unprocessed portion of Core1 into new subtasks and adds them to the GSTQ. Core0 takes the task from the GSTQ, and the runtime configures new spatial offsets and sizes for its AGUs. The two cores continue to execute in parallel until all subtasks in the synchronization group are completed, at which point the runtime distributes the Add task to any available core.

[0070] Scheduling in dependency-free task scenarios: Consider a system that simultaneously processes inference requests for two independent images. Each request contains several convolutional layers. The runtime adds the tasks for each request to GSTQ separately.

[0071] After Core0 completes a certain layer of Request1, it finds that the next layer of Request1 is not yet ready (due to unmet dependencies), but an executable task for Request2 exists in GSTQ. The runtime directly assigns the task of Request2 to Core0 without waiting for the synchronization point of Request1. Since the two request data are completely independent, AGU can safely load the new base address and layout parameters, achieving cross-task resource reuse.

[0072] In one implementation, the scheduler can also be configured as: Analyze the computation subtasks to determine the weight tensor information and memory layout information of the weight tensors required to execute the computation subtasks; The weight prefetch instruction is set based on the weight tensor information and memory layout information of the weight tensor, and then sent to the neural network processing unit so that the neural network processing unit can load the weight tensor according to the weight prefetch instruction.

[0073] In this embodiment, the runtime module is not only responsible for scheduling activation data (such as input feature maps) tasks, but also has the ability to prefetch weight data. When the scheduler determines that a certain NPU core is about to execute a specific convolutional or fully connected task, the runtime module can parse the weight tensor information that the task depends on in advance, including its storage location, size, precision format and memory layout, and issue a weight prefetch instruction to the DMA controller of the core before the task is officially started.

[0074] This prefetch operation is independent of the main compute pipeline and occurs before the AGU is configured or during the execution of preceding tasks. It loads weight data from off-chip memory into core local SRAM or shared on-chip cache. Since weights typically remain unchanged and are reused multiple times during single-level inference, the prefetch mechanism ensures that compute units read weights directly from low-latency on-chip memory when needed, avoiding pipeline congestion caused by weight memory access.

[0075] In addition, the runtime module can merge weight prefetch requests for multiple cores that are about to execute the same operator according to the task scheduling plan, avoiding the repeated transfer of the same weight copy and further reducing off-chip bandwidth pressure.

[0076] In addition, weight prefetching can be explicitly triggered by the runtime module, or it can be automatically predicted and prefetched by the neural network processing unit hardware based on the task ID; Supports cross-layer weight prefetching (the weight of the next layer is preloaded before the current layer ends); Support for weight reuse detection: If multiple tasks use the same weight, it is prefetched only once and distributed through inter-core communication.

[0077] In one implementation, inter-core communication is provided between neural network processing units; The scheduler is also configured as follows: When it is determined that multiple computational subtasks depend on the same intermediate data, a shared tag is added to the computational subtasks; The neural network processing unit is also configured as follows: When it is determined that the computational subtasks have shared tags, intermediate data is loaded to other neural network processing units via inter-core communication.

[0078] In this embodiment, the multi-core NPU architecture integrates an on-chip interconnect network (NoC) to support direct data communication between cores. When multiple cores need to process tasks that rely on the same intermediate data (such as shared input feature maps, common weight copies, or broadcast tensors), the runtime module can utilize this interconnect capability to achieve data sharing instead of repeated loading.

[0079] Specifically, if CoreA has already loaded a segment of input data or intermediate results into its local SRAM, and CoreB's task also needs to access that data, the runtime module can mark the data as "shareable" during the task allocation phase. Subsequently, when CoreB executes, it initiates an on-chip read request to CoreA through NoC. CoreA's local memory controller responds and directly transfers the data to CoreB, without needing to read it from DDR again.

[0080] This mechanism is particularly suitable for the following scenarios: Common inputs in branching structures: such as multiple parallel convolutional paths sharing the same input feature map; Weight broadcasting: When multiple cores execute different spatial regions of the same convolution kernel, the weights can be prefetched by one core and broadcast to other cores.

[0081] In addition, inter-core data sharing can be achieved through NoC direct reading, on-chip broadcast bus, shared SRAM mapping, or message passing; Shared data types may include: input feature maps, weights, intermediate activation values, and reduction statistics (such as mean and variance). Data consistency can be guaranteed through explicit synchronization signals, producer-consumer tokens, or hardware cache consistency protocols.

[0082] Based on the above embodiments, the present invention first sets a task progress register in the neural network processing unit to record the task processing progress of the neural network processing unit in processing computational subtasks, and exposes this task processing progress to the software layer. Furthermore, the present invention can also set a scheduler and a globally shared task queue in the runtime module. The scheduler is used to schedule computational subtasks to each neural network processing unit, and to filter slow processing units according to the task processing progress of each neural network processing unit, identify the remaining subtasks that the slow processing units have not completed, and then reassign the remaining subtasks as computational subtasks, and redistribute them to idle neural network processing units through the globally shared task queue. In actual operation, the scheduler first schedules the computational subtasks to each neural network processing unit for processing, and the neural network processing unit processes the computational subtasks and records the task processing progress using the task progress register. Subsequently, when the neural network processing unit completes task processing, it can send a task completion interrupt signal, enter an idle state, and check whether there are any computational subtasks in the globally shared task queue. When the scheduler receives a task completion interrupt signal from any neural network processing unit, it reads the task processing progress from the task progress register of each neural network processing unit and determines the slow processing unit and its unfinished subtasks based on the progress. The scheduler then sets the remaining subtasks as computational subtasks and adds them to the global shared task queue. When a neural network processing unit detects a computational subtask in the global shared task queue, it requests the scheduler to acquire and process it. Therefore, this invention, through hardware improvements to the neural network processing unit and software improvements to the runtime module, can filter out slow processing units based on the task processing progress reported by the neural network processing unit and reassign the unfinished subtasks of the slow processing units to idle neural network processing units. Faster cores can then assist slower cores in completing computational subtasks, thereby avoiding idle computing resources and improving system throughput efficiency.

[0083] To facilitate understanding, the following section, with the aid of diagrams, introduces the complete structure and core operating mechanism of the neural network processing module and its scheduling system.

[0084] 1. Enhanced NPU core hardware architecture: Please refer to Figure 3 , Figure 3 This is a structural block diagram of a neural network processing module provided in an embodiment of the present invention. Each NPU core includes: Programmable Address Generation Unit (AGU): Supports receiving and loading new address generation parameters at runtime, including base address, tensor shape, memory step size, spatial offset and block size, and can autonomously generate continuous physical addresses according to the new task context; at the same time, the AGU can skip unnecessary calculations by identifying zero elements in the input data based on data structures such as hash tables, thereby improving processing speed and energy efficiency.

[0085] Task Progress Register (TPR): Automatically maintained by hardware, it records the current task's execution progress in real time and exposes it to the outside world in the form of the number of data units completed or normalized percentage; Task control interface: Receives startup instructions from the Runtime and triggers the collaborative execution pipeline of AGU, DMA and computing array.

[0086] Units used for multiplication and accumulation calculations, such as systolic arrays, MACs, and internal storage SRAM, as well as modules such as controllers, schedulers, and buffers required to implement the corresponding computing tasks.

[0087] 2. Runtime Scheduling System: Please refer to Figure 4 , Figure 4 A structural block diagram of another neural network processing module scheduling system provided in an embodiment of the present invention. The Runtime module is deployed on the Host processor and includes: Global Shared Task Queue (GSTQ): A task pool visible to all NPU cores that stores subtask descriptors to be executed. Each descriptor contains tensor metadata, memory layout identifier, spatial partition coordinates, and AGU configuration template. Lightweight scheduler: Employs an event-driven mechanism and listens for task completion interrupts from the NPU core; Task Dependency Manager: Maintains the sequential relationship between tasks and identifies synchronization group boundaries.

[0088] 1. Runtime's mechanism for publishing shared tasks based on Task Progress Registers (TPRs): In this invention's architecture, each NPU core continuously updates its Task Progress Register (TPR) to reflect the completion status of the currently executing task. The TPR can represent the number of data units processed, the range of completed spatial regions, or a normalized percentage of progress. Instead of periodically polling all TPRs, the runtime uses an event-driven, on-demand reading approach: when a core issues a "task completion interrupt," the runtime first confirms that the core has entered an idle state, and then actively reads the TPR values ​​of the remaining cores still executing tasks.

[0089] Based on this TPR information, the runtime can determine the overall balance of system execution. If it is found that the remaining workload of a certain core is significantly greater than that of other cores (e.g., its TPR growth is slow or stagnant), the runtime will dynamically split the original task region responsible for that core: retaining the part that is currently loaded or being processed, and extracting the sub-region that has not yet started processing into an independent, structurally complete subtask descriptor.

[0090] The subtask descriptor is then written to the Global Shared Task Queue (GSTQ). GSTQ is a task pool managed by the runtime and visible to all NPU cores. Each entry contains complete tensor information, memory layout identifiers, spatial offset coordinates, and an AGU configuration template. Any idle NPU core can "claim" a task from GSTQ without needing to bind to a specific task source beforehand.

[0091] Through this mechanism, the computing load originally allocated to the slow core is transformed into a new task that can be "picked up" by the fast core, thereby achieving the reallocation of computing resources without interrupting the original execution flow.

[0092] 2. The runtime's hardware scheduling process: The runtime schedules the NPU hardware through a low-latency control channel. When an idle core is ready to take on a new task (whether initially allocated or acquired from GSTQ), the runtime performs the following operations: Task descriptor parsing: Retrieve task descriptors from GSTQ or the initial task list, and parse their tensor dimensions, memory base addresses, layout formats, and spatial partitioning parameters; AGU Configuration Generation: Based on the above information, generate a set of configuration parameters adapted to the target core AGU interface, including base address, step size, block start offset, size, etc. Register writing: The AGU configuration parameters are written to the AGU configuration register group of the target core via a dedicated control bus; Task start trigger: Send a start signal to the core to activate the collaborative pipeline of its AGU, DMA and computing units.

[0093] The entire process is completed in microseconds by the Runtime's lightweight scheduler, ensuring that hardware resources can be quickly put into the execution of new tasks after they are idle, and avoiding computing units from being in a waiting state for a long time.

[0094] 3. Dynamic scheduling method flow: Initial task distribution: The runtime divides the input tensor into multiple subtasks, distributes them as evenly as possible to each NPU core according to the amount of computation, and writes them into its AGU configuration register; Execution and monitoring: During the execution of tasks by each core, the progress of TPR is continuously updated; after any core completes its task and issues an interrupt, the runtime reads the TPR of the remaining active cores; Dynamic task partitioning: If the remaining workload of a certain core is significantly greater than that of other cores, the runtime partitions its unprocessed area into independent subtasks and adds them to GSTQ; Task acquisition and reconfiguration: Idle cores acquire new tasks from GSTQ, the Runtime loads the new configuration for its AGU, and starts execution; Depends on synchronization control: (i) For dependent tasks, the Runtime will group all subtasks belonging to the same output tensor into a synchronization group, and release subsequent dependent tasks only when all tasks in the group are completed. (ii) For tasks without dependencies, Runtime allows idle cores to take any pending items in GSTQ across tasks, enabling elastic resource reuse.

[0095] 4. Runtime supports weighted data prefetching mechanism: In the system of this invention, the Runtime is not only responsible for scheduling tasks involving activation data (such as input feature maps), but also has the ability to prefetch and manage weight data. When the scheduler determines that a certain NPU core is about to execute a specific convolutional or fully connected task, the Runtime can parse the weight tensor information that the task depends on in advance, including its storage location, size, precision format and memory layout, and issue a weight prefetch instruction to the DMA controller of the core before the task is officially started.

[0096] This prefetch operation is independent of the main computation pipeline and occurs before AGU configuration is complete or during the execution of preceding tasks. It loads weight data from off-chip memory into core local SRAM or shared on-chip cache. Since weights typically remain unchanged and are reused multiple times during single-level inference, the prefetch mechanism ensures that computation units read weights directly from low-latency on-chip memory when needed, avoiding pipeline blockage due to weight memory access.

[0097] In addition, the runtime can merge weight prefetch requests for multiple cores that are about to execute the same operator according to the task scheduling plan, avoiding the repeated transfer of the same weight copy and further reducing off-chip bandwidth pressure.

[0098] 5. NPU multi-core supports inter-core communication and data sharing: The multi-core NPU architecture of this invention integrates an on-chip interconnect network (NoC) to support direct data communication between cores. When multiple cores need to process tasks that rely on the same intermediate data (such as shared input feature maps, common weight copies, or broadcast tensors), the runtime can utilize this interconnect capability to achieve data sharing instead of repeated loading.

[0099] Based on the above embodiments, the neural network processing unit scheduling method provided by the present invention will be described below. Please refer to... Figure 5 , Figure 5 This is a flowchart illustrating a neural network processing unit scheduling method provided in an embodiment of the present invention. The method is applied to a neural network processing unit scheduling system, which includes a host and at least two neural network processing units. The host deploys a runtime module, which includes a globally shared task queue and a scheduler. Each neural network processing unit includes a task progress register, and the host is connected to the neural network processing units. This method may include: S101, The scheduler schedules the computational subtasks to each neural network processing unit for processing.

[0100] In one embodiment, the neural network processing unit further includes a programmable address generation unit, and the neural network processing unit is connected to memory. The scheduler schedules computational subtasks to each neural network processing unit for processing, and may include: Step 11: The scheduler configures the address generation parameters for the programmable address generation unit of the neural network processing unit based on the tensor metadata, memory layout information, and spatial partitioning information corresponding to the tensor data to be processed in the computational subtask.

[0101] Step 12: The scheduler sends a task start command to the neural network processing unit to trigger the neural network processing unit to execute the computational subtask.

[0102] With a programmable address generation unit, the scheduler, when issuing tasks, can generate address generation parameters based on the tensor metadata, memory layout, and spatial partitioning information corresponding to the tensor data to be processed in the computational subtask. These parameters are then configured into the register group of the programmable address generation unit via a dedicated control bus. Upon task startup, the neural network processing unit can use the programmable address generation unit to generate a first memory address and a second memory address. Based on the first memory address, it sequentially loads each data unit from the tensor data from memory, and based on the second memory address, it writes the processed data of the tensor data to memory. This ensures that the neural network processing unit reads and processes data units in an orderly manner and writes back the processed data in an orderly fashion.

[0103] In another implementation, the runtime module further includes a task dependency manager for recording dependencies and synchronization groups between computational tasks. Before the scheduler schedules computational subtasks to the respective neural network processing units, it may also include: Step 21: The scheduler obtains the current computing task and determines whether there are subsequent computing tasks that depend on the current computing task based on the dependencies; if there are, proceed to step 22; if not, proceed to step 23. Step 22: Create a synchronization group in the Task Dependency Manager, divide the current computing task into computing subtasks, and add all computing subtasks to the synchronization group; the scheduler will only issue subsequent computing tasks that depend on the current computing task when all computing subtasks in the synchronization group are completed. Step 23: Divide the current computation task into computation subtasks; Step 24: Distribute the computational subtasks to each neural network processing unit or add them to the global shared task queue.

[0104] In this embodiment, the runtime module may further include a task dependency manager for recording dependencies and synchronization groups between computation tasks. When dividing the current computation task, the scheduler can determine whether there are subsequent computation tasks that depend on the current computation task based on the dependencies. If so, the scheduler can create a synchronization group in the task dependency manager, divide the computation task into computation subtasks, and add all computation subtasks to the synchronization group. The scheduler only issues subsequent computation tasks that depend on the current computation task after all computation subtasks in the synchronization group have been completed. If there are no subsequent computation tasks that depend on the current computation task, the scheduler does not need to create a synchronization group; it only needs to divide the current computation task into computation subtasks. Subsequently, the scheduler can issue the computation subtasks to each neural network processing unit or add them to a global shared task queue. This ensures that task scheduling is constrained by the dependencies between computation tasks.

[0105] In another implementation, dividing the current computing task into computing subtasks may include: Step 31: The scheduler obtains the task identifier of the current computation task; wherein, the task identifier contains tensor element information, memory layout information, and spatial partitioning information of the complete tensor data to be processed by the current computation task; Step 32: Set the computational subtasks of each neural network processing unit according to the task identifier, and obtain the subtask identifier of the computational subtask to evenly distribute the computational tasks to each neural network processing unit; wherein, the subtask identifier contains the tensor element information, memory layout information, spatial partitioning information and configuration template corresponding to the tensor data to be processed by the computational subtask. Accordingly, distributing computational subtasks to each neural network processing unit or adding them to a globally shared task queue can include: Step 41: The scheduler distributes the subtask identifiers to each neural network processing unit or adds them to the global shared task queue.

[0106] In this embodiment, the global shared task queue can store subtask descriptors to be executed. Each descriptor contains tensor metadata, memory layout identifier, spatial partition coordinates, and AGU configuration template. The scheduler obtains the task identifier of the current computation task, extracts the tensor metadata, memory layout information, and spatial partition information of the complete tensor data to be processed by the current computation task from the task identifier, and sets the computation subtasks of each neural network processing unit to obtain the subtask identifiers of the computation subtasks, so as to evenly distribute the computation tasks to each neural network processing unit. Subsequently, the scheduler can send the subtask identifiers to each neural network processing unit. For example, based on the subtask identifier information, it can generate a set of configuration parameters adapted to the target core AGU interface, including base address, step size, block start offset, size, etc., and write the AGU configuration parameters into the AGU configuration register group of the target core through a dedicated control bus. Alternatively, the scheduler can also add the subtask identifiers to the global shared task queue.

[0107] S102, the neural network processing unit processes the computational subtasks and records the task processing progress through the task progress register.

[0108] In one implementation, the neural network processing unit may process computational subtasks, including: Step 51: The neural network processing unit uses the programmable address generation unit to generate a first memory address and a second memory address, loads each data unit in the tensor data in order from the memory according to the first memory address, and writes the processed data of the tensor data into the memory according to the second memory address.

[0109] In one embodiment, the method may further include: Step 61: The scheduler configures the hash table of the computation subtask to the programmable address generation unit of the neural network processing unit; wherein, the hash table is used to record the zero elements in the tensor data.

[0110] Accordingly, the neural network processing unit may also include the following computational subtasks: Step 71: The programmable address generation unit determines the zero elements in the tensor data according to the hash table and skips the zero elements in the task processing.

[0111] In this embodiment, to improve processing efficiency, the scheduler can also configure a hash table for the programmable address generation unit. This hash table is used to record zero elements in the tensor data. When generating an address, the programmable address generation unit can determine whether the data unit currently read from the tensor data is a zero element based on the hash table. If so, it directly returns a zero value to the core of the neural network processing unit to skip zero elements in task processing.

[0112] S103. When the neural network processing unit completes processing, it sends a task completion interrupt signal, enters an idle state, and checks whether there is a computational subtask in the global shared task queue.

[0113] S104. When the scheduler receives a task completion interrupt signal from any neural network processing unit, it reads the task progress register of each neural network processing unit to obtain the task processing progress.

[0114] In one implementation, before reading the task progress register of each neural network processing unit to obtain the task processing progress, the following may be included: Step 81: When the scheduler receives a task completion interrupt signal from any neural network processing unit, it determines whether the computational subtask executed by the neural network processing unit belongs to the synchronization group; if the computational subtask belongs to the synchronization group, proceed to step 82; if the computational subtask does not belong to the synchronization group, proceed to step 87. Step 82: Determine whether all computational subtasks in the synchronization group have been completed; if all computational subtasks in the synchronization group have been completed, proceed to step 83; if the computational subtasks in the synchronization group have not been completed, proceed to step 84. Step 83: Obtain the next computation task and proceed to the step of determining whether there are subsequent computation tasks that depend on the computation task based on the dependency relationship; Step 84: Proceed to the step of determining slow processing units based on task processing progress, and determine whether there are slow processing units; if there are no slow processing units, proceed to step 85; if there are slow processing units, proceed to step 86. Step 85: Obtain the next computation task that does not depend on the currently executed computation task based on the dependency relationship, and proceed to the step of determining whether there are subsequent computation tasks that depend on the computation task based on the dependency relationship. Step 86: Proceed to the step of determining the remaining sub-tasks that have not been completed by the slow processing unit based on the task processing progress; Step 87: Proceed to the step of determining slow processing units based on task processing progress, and determine whether there are slow processing units; if there are no slow processing units, proceed to step 88; if there are slow processing units, proceed to step 89. Step 88: Obtain the next computation task and proceed to the step of determining whether there are subsequent computation tasks that depend on the computation task based on the dependency relationship; Step 89: Proceed to the step of determining the remaining subtasks that have not been completed by the slow processing unit based on the task processing progress.

[0115] In this embodiment, due to the dependencies between computing tasks, there are multiple scenarios when the scheduler allocates the next computing subtask to an idle neural network processing unit.

[0116] First, if the computational subtasks completed by the neural network processing unit are determined to belong to a synchronization group, the scheduler needs to determine whether all computational subtasks in that synchronization group have been completed. If all computational subtasks in the synchronization group have been completed, it means that the current computational task has been completed, and the scheduler can schedule the next computational task that depends on the current computational task. If there are still unfinished computational subtasks in the synchronization group, the scheduler needs to check whether there are slow processing units. If so, the remaining subtasks of the slow processing units can be reassigned to idle neural network processing units.

[0117] It is worth noting that when it is determined that there are still unfinished computational subtasks in the synchronization group, and there are no slow processing units at this time, the scheduler can either wait for the synchronization group to complete its processing or schedule the next computational task that is independent of the current computational task for the idle neural network processing unit. For example, if a core completes its assigned task, but there are still other cores in its original synchronization group that have not completed their tasks, but there are other independent tasks to be processed in the system (such as those from different requests or different model branches), the scheduler can directly assign these external tasks to the idle core. In this way, this embodiment can perform task scheduling more flexibly and improve resource utilization.

[0118] Furthermore, if it is determined that the computational subtask completed by the neural network processing unit does not belong to the synchronization group, the scheduler needs to check whether there is a slow processing unit. If there is, the remaining subtasks of the slow processing unit can be reassigned to the idle neural network processing unit. If there is no slow processing unit, the next computational task can be scheduled.

[0119] S105. The scheduler determines the slow processing unit and its unfinished remaining subtasks based on the task processing progress.

[0120] In one implementation, the scheduler determines slow processing units and their unfinished remaining subtasks based on task processing progress, including: Step 91: The scheduler determines whether the task processing progress is greater than a preset threshold; if it is, proceed to step 92. Step 92: Determine that the neural network processing unit to which the task processing progress belongs is a slow processing unit; Step 93: Map the task processing progress of the slow processing unit to the space offset parameter of the programmable address generation unit of the slow processing unit, and determine the remaining subtasks that the slow processing unit has not completed based on the space offset parameter.

[0121] In this embodiment, the scheduler first identifies neural network processing units whose task processing progress exceeds a preset threshold as slow processing units based on the relationship between task processing progress and a preset threshold. It should be noted that this preset threshold can be set manually or dynamically based on the task processing progress of each non-idle neural network processing unit, such as setting the average task processing progress of each non-idle neural network processing unit as the preset threshold.

[0122] Furthermore, this embodiment can map the task processing progress to the spatial offset parameter of the programmable address generation unit. In other words, the processed portion of the tensor data by the neural network processing module can be determined based on the task processing progress, thereby identifying the unprocessed portion as the remaining subtasks. Thus, after identifying the slow processing unit, this embodiment can map the task processing progress of the slow processing unit to the spatial offset parameter of the programmable address generation unit of the slow processing unit, and determine the remaining subtasks that the slow processing unit has not completed based on the spatial offset parameter.

[0123] S106. The scheduler sets the remaining subtasks as computation subtasks and adds them to the global shared task queue. S107. When the neural network processing unit detects that there is a computational subtask in the global shared task queue, it requests the scheduler to obtain the computational subtask. S108. When the scheduler detects that the neural network processing unit requests to obtain a computational subtask from the global shared task queue, it schedules the computational subtask to the neural network processing unit for processing.

[0124] Based on the above embodiments, the present invention improves the hardware of the neural network processing unit and the software of the runtime module. It can filter out slow processing units by the task processing progress reported by the neural network processing unit, and redistribute the remaining sub-tasks that the slow processing units have not completed to the idle neural network processing units. The fast core can help the slow core complete the computational sub-tasks, thereby avoiding idle computing resources and improving the system throughput efficiency.

[0125] For a complete understanding of the task scheduling process, please refer to [link / reference]. Figure 6 , Figure 6 A flowchart of another neural network processing unit scheduling method provided in an embodiment of the present invention.

[0126] In one embodiment, the method may further include: S201. The scheduler parses the computation subtask and determines the weight tensor information and memory layout information of the weight tensor required to execute the computation subtask. S202. Set the weight prefetch instruction according to the weight tensor information and memory layout information of the weight tensor, and send the weight prefetch instruction to the neural network processing unit so that the neural network processing unit loads the weight tensor according to the weight prefetch instruction.

[0127] In this embodiment, the runtime module is not only responsible for scheduling activation data (such as input feature maps) tasks, but also has the ability to prefetch weight data. When the scheduler determines that a certain NPU core is about to execute a specific convolutional or fully connected task, the runtime module can parse the weight tensor information that the task depends on in advance, including its storage location, size, precision format and memory layout, and issue a weight prefetch instruction to the DMA controller of the core before the task is officially started.

[0128] In one embodiment, inter-core communication is provided between neural network processing units. The method may further include: S301. When the scheduler determines that multiple computational subtasks depend on the same intermediate data, it adds a sharing flag to the computational subtasks. S302. When the neural network processing unit determines that the computational subtask has a shared label, it loads intermediate data to other neural network processing units through inter-core communication.

[0129] In this embodiment, the multi-core NPU architecture integrates an on-chip interconnect network (NoC) to support direct data communication between cores. When multiple cores need to process tasks that rely on the same intermediate data (such as shared input feature maps, common weight copies, or broadcast tensors), the runtime module can utilize this interconnect capability to achieve data sharing instead of repeated loading.

[0130] This invention also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the neural network processing unit scheduling method described in the above embodiments.

[0131] Since the embodiments of the computer program product portion correspond to the embodiments of the neural network processing unit scheduling method portion, please refer to the description of the embodiments of the neural network processing unit scheduling method portion for the embodiments of the computer program product portion, and will not be repeated here.

[0132] This invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the neural network processing unit scheduling method described in the above embodiments.

[0133] Since the embodiments of the computer-readable storage medium portion correspond to the embodiments of the neural network processing unit scheduling method portion, the embodiments of the storage medium portion are described in the description of the embodiments of the neural network processing unit scheduling method portion, and will not be repeated here.

[0134] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0135] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0136] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0137] The present invention provides a detailed description of a neural network processing unit scheduling system and method. Specific examples have been used to illustrate the principles and implementation methods of the invention. The descriptions of these embodiments are merely illustrative and are intended to help understand the method and core ideas of the present invention. It should be noted that those skilled in the art can make various improvements and modifications to the invention without departing from its principles, and these improvements and modifications also fall within the scope of protection of the present invention.

Claims

1. A neural network processing unit scheduling system, characterized in that, include: The system includes a host computer and at least two neural network processing units. The host computer deploys a runtime module, which includes a globally shared task queue and a scheduler. The neural network processing units include task progress registers. The host computer is connected to the neural network processing units. The scheduler is configured to schedule computational subtasks to each neural network processing unit for processing; when a task completion interruption signal is received from any neural network processing unit, the scheduler reads the task progress register of each neural network processing unit to obtain the task processing progress; determines the slow processing unit and its unfinished remaining subtasks based on the task processing progress; sets the remaining subtasks as computational subtasks and adds them to the global shared task queue; and when a neural network processing unit requests to retrieve a computational subtask from the global shared task queue, the scheduler schedules the computational subtask to the neural network processing unit for processing. The neural network processing unit is configured to process the computational subtask and record the task processing progress through the task progress register; when the processing is completed, it sends the task completion interrupt signal, enters an idle state, and checks whether there is a computational subtask in the global shared task queue. If it exists, request the scheduler to obtain the computational subtask.

2. The neural network processing unit scheduling system according to claim 1, characterized in that, The neural network processing unit further includes a programmable address generation unit, and the neural network processing unit is connected to memory. The scheduler is further configured to configure address generation parameters for the programmable address generation unit of the neural network processing unit based on the tensor metadata, memory layout information, and spatial partitioning information corresponding to the tensor data to be processed in the computational subtask. The neural network processing unit is further configured to generate a first memory address and a second memory address using the programmable address generation unit, load each data unit in the tensor data in an orderly manner from the memory according to the first memory address, and write the processed data of the tensor data into the memory according to the second memory address.

3. The neural network processing unit scheduling system according to claim 2, characterized in that, The scheduler is also configured to: Determine whether the task processing progress is greater than a preset threshold; If so, then the neural network processing unit to which the task processing progress belongs is determined to be the slow processing unit; The task processing progress of the slow processing unit is mapped to the spatial offset parameter of the programmable address generation unit of the slow processing unit, and the remaining sub-tasks that the slow processing unit has not completed are determined according to the spatial offset parameter.

4. The neural network processing unit scheduling system according to claim 2, characterized in that, The runtime module also includes a task dependency manager; The task dependency manager is configured to record the dependencies and synchronization groups between computing tasks; The scheduler is also configured to: Obtain the current computing task, and determine whether there are subsequent computing tasks that depend on the current computing task based on the dependency relationship; If present, a synchronization group is created in the task dependency manager, the current computing task is divided into the computing subtasks, and all computing subtasks are added to the synchronization group; wherein, the scheduler only issues subsequent computing tasks that depend on the current computing task when all computing subtasks in the synchronization group are completed; If not, the current computing task is divided into the computing subtasks; The computational subtasks are distributed to each neural network processing unit or added to the global shared task queue.

5. The neural network processing unit scheduling system according to claim 4, characterized in that, The scheduler is also configured to: When a task completion interruption signal is received from any neural network processing unit, it is determined whether the computational subtask executed by the neural network processing unit belongs to the synchronization group. If the computational subtask belongs to the synchronization group, then determine whether all computational subtasks in the synchronization group have been completed; If all the computational subtasks in the synchronization group have been completed, then the next computational task is obtained, and the process proceeds to the step of determining whether there are subsequent computational tasks that depend on the computational task based on the dependency relationship. If the computational subtask in the synchronization group is not completed, the process proceeds to the step of determining the slow processing unit based on the task processing progress, and determining whether the slow processing unit exists. If the slow processing unit does not exist, then the next computing task that does not depend on the currently executed computing task is obtained according to the dependency relationship, and the process proceeds to the step of determining whether there is a subsequent computing task that depends on the computing task according to the dependency relationship. If the slow processing unit exists, proceed to the step of determining the remaining sub-tasks that the slow processing unit has not completed based on the task processing progress; If the computational subtask does not belong to the synchronization group, then proceed to the step of determining the slow processing unit based on the task processing progress, and determine whether the slow processing unit exists. If the slow processing unit does not exist, the next computing task is obtained, and the process proceeds to the step of determining whether there are subsequent computing tasks that depend on the computing task based on the dependency relationship. If the slow processing unit exists, proceed to the step of determining the remaining sub-tasks that the slow processing unit has not completed based on the task processing progress.

6. The neural network processing unit scheduling system according to claim 4, characterized in that, The scheduler is also configured to: Obtain the task identifier of the current computing task; wherein, the task identifier contains tensor metadata, memory layout information, and spatial partitioning information of the complete tensor data to be processed by the current computing task; The computational subtasks of each neural network processing unit are set according to the task identifiers to obtain the subtask identifiers of the computational subtasks, so as to evenly distribute the computational tasks to each neural network processing unit; wherein, the subtask identifiers include tensor metadata, memory layout information, spatial partitioning information and configuration templates corresponding to the tensor data to be processed by the computational subtasks. The subtask identifier is distributed to each neural network processing unit or added to the global shared task queue.

7. The neural network processing unit scheduling system according to claim 2, characterized in that, The scheduler is also configured to: The hash table of the computational subtask is configured in the programmable address generation unit of the neural network processing unit; wherein the hash table is used to record the zero elements in the tensor data; The programmable address generation unit is configured to determine the zero element in the tensor data according to the hash table and skip the zero element in task processing.

8. The neural network processing unit scheduling system according to claim 1, characterized in that, The scheduler is also configured to: The computational subtask is parsed to determine the weight tensor information and memory layout information of the weight tensor required to execute the computational subtask; Based on the weight tensor information and memory layout information of the weight tensor, a weight prefetch instruction is set, and the weight prefetch instruction is sent to the neural network processing unit so that the neural network processing unit loads the weight tensor according to the weight prefetch instruction.

9. The neural network processing unit scheduling system according to claim 1, characterized in that, Inter-core communication is provided between the neural network processing units; The scheduler is also configured to: When it is determined that multiple computational subtasks depend on the same intermediate data, a shared tag is added to the computational subtasks; The neural network processing unit is further configured as follows: When it is determined that the computational subtask has the shared tag, the intermediate data is loaded to other neural network processing units through the inter-core communication.

10. A method for scheduling neural network processing units, characterized in that, An application is made in a neural network processing unit scheduling system, the neural network processing unit scheduling system including a host and at least two neural network processing units, the host deploying a runtime module, the runtime module including a globally shared task queue and a scheduler, the neural network processing unit including a task progress register, and the host being connected to the neural network processing unit; The method includes: The scheduler schedules computational subtasks to each neural network processing unit for processing. The neural network processing unit processes the computational subtask and records the task processing progress through the task progress register. When the neural network processing unit completes processing, it sends a task completion interrupt signal, enters an idle state, and checks whether there are any computational subtasks in the global shared task queue. When the scheduler receives a task completion interruption signal from any neural network processing unit, it reads the task progress register of each neural network processing unit to obtain the task processing progress. The scheduler determines the slow processing units and their unfinished remaining subtasks based on the task processing progress. The scheduler sets the remaining subtasks as computation subtasks and adds them to the global shared task queue; When the neural network processing unit detects that there is a computational subtask in the global shared task queue, it requests the scheduler to obtain the computational subtask. When the scheduler detects that the neural network processing unit requests to obtain a computational subtask from the global shared task queue, it schedules the computational subtask to the neural network processing unit for processing.

Images