Npu computing power dynamic allocation method and on-chip heterogeneous system

Abstract

The application discloses a kind of NPU computing power dynamic allocation method and on-chip heterogeneous system, method includes: the scheduling configuration parameter of the NPU is locked, wherein the scheduling configuration parameter includes the time slice allocation strategy of the NPU between the non-real-time operating system and the real-time operating system;In the non-real-time operating system, according to the scheduling configuration parameter initialization and start a periodic scheduling timer;In response to the time slice switching signal sent by the periodic scheduling timer, trigger first context switching operation;In response to the real-time preemption event generated by the real-time operating system side, trigger second context switching operation;After the computing task of the real-time operating system is executed, trigger third context switching operation, restore the hardware context of the non-real-time operating system previously preoccupied, so that interrupted computing task can continue to be executed.The application improves the security and efficiency of computing power dynamic allocation.

Description

Technical Field

[0001] This invention relates to the field of embedded industrial control technology, and in particular to a method for dynamic allocation of NPU computing power and an on-chip heterogeneous system. Background Technology

[0002] With the deepening development of Industry 4.0 and intelligent manufacturing, the computing power requirements of industrial edge computing devices are becoming increasingly complex. On the same hardware platform, two types of tasks often need to be processed in parallel: non-real-time artificial intelligence (AI) inference tasks and control and diagnostic tasks with hard real-time requirements. The RK3562 System-on-Chip (SoC) integrates a Neural Processing Unit (NPU), providing the necessary computing power foundation for these applications. However, its native software suite (SDK) is primarily designed for single-operating system scenarios, making it difficult to simultaneously meet the differentiated and highly deterministic scheduling requirements of non-real-time AI tasks and hard real-time control tasks for the same NPU computing resources. Currently, there are several technical solutions for sharing NPU computing power across multiple tasks, but all have their limitations: Single-system software scheduling scheme: Within a single operating system (usually Linux), multiple AI tasks access the NPU through software queues and schedulers. However, its scheduling is subject to the non-real-time scheduling strategy of the general operating system. High-priority real-time tasks may be blocked by low-priority non-real-time tasks or system activities, and microsecond-level deterministic latency cannot be guaranteed. It is not suitable for hard real-time control scenarios.

[0003] Multi-chip physical isolation solution: Equip real-time tasks and non-real-time tasks with independent NPUs or computing chips. Although this solution can achieve physical-level resource isolation and deterministic real-time performance, it significantly increases hardware cost, power consumption and physical size.

[0004] Limitations of the SoC's native SDK: Taking the RK3562's native SDK as an example, its NPU driver is typically designed as a single task queue, lacking native support for concurrent access from multiple operating systems, and further lacking hardware interrupt and context switching mechanisms for real-time tasks to preempt non-real-time tasks. If used directly, real-time tasks cannot obtain NPU computing power in a timely manner, and non-real-time tasks cannot effectively utilize NPU resources when real-time tasks are idle, resulting in wasted computing power.

[0005] Therefore, existing technologies cannot achieve hard isolation, dynamic allocation, and real-time preemption of NPU computing power across heterogeneous operating systems on low-cost, single-chip industrial hardware platforms. In other words, the security and efficiency of existing NPU computing power allocation are both low. Summary of the Invention

[0006] This invention provides a method for dynamic allocation of NPU computing power and an on-chip heterogeneous system, aiming to solve the problems of low security and efficiency in existing NPU computing power allocation.

[0007] In a first aspect, embodiments of the present invention provide a method for dynamically allocating NPU computing power, applied to an on-chip heterogeneous system, wherein the on-chip heterogeneous system includes a non-real-time operating system running on a first processor core and a real-time operating system running on a second processor core, and the NPU is configured to be shared and accessed by the first processor core and the second processor core, the method comprising: The scheduling configuration parameters of the NPU are locked, wherein the scheduling configuration parameters include the time slice allocation strategy of the NPU between the non-real-time operating system and the real-time operating system; In the non-real-time operating system, a periodic scheduling timer is initialized and started according to the scheduling configuration parameters; In response to the time slice switching signal issued by the periodic scheduling timer, a first context switching operation is triggered; In response to a real-time preemption event generated by the real-time operating system, a second context switch operation is triggered; After the computational task of the real-time operating system is completed, a third context switch operation is triggered to restore the hardware context of the non-real-time operating system that was previously preempted, so that the interrupted computational task can continue to be executed.

[0008] Secondly, embodiments of the present invention also provide an on-chip heterogeneous system, comprising: One of the primary processor cores is configured to run a non-real-time operating system; A second processor core is configured to run a real-time operating system; A shared NPU; The on-chip heterogeneous system is configured to dynamically schedule the computing power of the NPU between the non-real-time operating system and the real-time operating system according to the method described in the first aspect above.

[0009] This invention provides a method for dynamic allocation of NPU computing power and an on-chip heterogeneous system. The method includes: locking the scheduling configuration parameters of the NPU, wherein the scheduling configuration parameters include the time slice allocation strategy of the NPU between a non-real-time operating system and a real-time operating system; initializing and starting a periodic scheduling timer in the non-real-time operating system according to the scheduling configuration parameters; triggering a first context switch operation in response to a time slice switching signal issued by the periodic scheduling timer; triggering a second context switch operation in response to a real-time preemption event generated by the real-time operating system; and triggering a third context switch operation after the computing task in the real-time operating system is completed, restoring the previously preempted hardware context of the non-real-time operating system so that the interrupted computing task can continue to execute. The technical solution of this invention improves the security of dynamic computing power allocation by locking the scheduling configuration parameters to prevent malicious tampering; and ensures rapid response to urgent tasks through hard real-time preemption, improving the efficiency of real-time task execution while avoiding the blocking risks caused by traditional software scheduling, thereby improving the efficiency of dynamic computing power allocation. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a schematic diagram illustrating a scenario of a dynamic NPU computing power allocation method provided in an embodiment of the present invention. Figure 2 This is a flowchart illustrating a method for dynamically allocating NPU computing power according to an embodiment of the present invention. Figure 3 This is a schematic diagram of a sub-process of a dynamic NPU computing power allocation method provided in an embodiment of the present invention; Figure 4 A simplified flowchart of an NPU computing power dynamic allocation method provided in an embodiment of the present invention. Detailed Implementation

[0012] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0013] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0014] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0015] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0016] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0017] This invention proposes a method for dynamic allocation of NPU computing power and an on-chip heterogeneous system, which solves the problems of low security and efficiency in existing NPU computing power allocation. In this embodiment, by locking the scheduling configuration parameters to prevent malicious tampering, the security of dynamic computing power allocation is improved; by using hard real-time preemption to ensure rapid response to urgent tasks, the efficiency of real-time task execution is improved while avoiding the blocking risk caused by traditional software scheduling, thereby improving the efficiency of dynamic computing power allocation.

[0018] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0019] This invention proposes a method for dynamically allocating NPU computing power, which can be used in on-chip heterogeneous systems, such as... Figure 1As shown, the on-chip heterogeneous system 200 includes a first processor core 10, a second processor core 20, and a shared NPU 30. The first processor core 10 is configured to run a non-real-time operating system; the second processor core 20 is configured to run a real-time operating system; and the NPU 30 is configured to be shared and accessed by both the first processor core 10 and the second processor core 20. The NPU, short for Neural Processing Unit, is a dedicated hardware accelerator specifically designed for efficiently processing artificial intelligence neural network computation tasks. It should be noted that the on-chip heterogeneous system 200 is configured to execute a dynamic NPU computing power allocation method, dynamically scheduling the NPU's computing power between the non-real-time operating system and the real-time operating system. Please refer to... Figure 2 , Figure 2 A flowchart illustrating the NPU computing power dynamic allocation method according to an embodiment of the present invention is shown, as follows: Figure 2 As shown, the NPU computing power dynamic allocation method includes steps S110-S150.

[0020] S110. Lock the scheduling configuration parameters of the NPU, wherein the scheduling configuration parameters include the time slice allocation strategy of the NPU between the non-real-time operating system and the real-time operating system.

[0021] In this embodiment, the on-chip heterogeneous system includes a non-real-time operating system running on a first processor core and a real-time operating system running on a second processor core. The NPU is configured to be shared and accessed by both the first and second processor cores. Specifically, as shown... Figure 3As shown, step S110 includes steps S111-S112: S111, during the bootloader stage, the device tree memory area where the scheduling configuration parameters are located is set to read-only attribute to complete write protection in the initial startup phase; S112, when the second processor core starts, the scheduling configuration parameters are locked in the kernel data area through the kernel security mechanism. It should be noted that after the on-chip heterogeneous system is powered on, before the first processor core and the second processor core are initialized, the bootloader performs forced write protection on the device tree memory area storing the NPU scheduling configuration parameters. This operation is achieved by configuring the read-only attribute of the memory management unit, aiming to prevent the scheduling configuration parameters from being accidentally or maliciously modified during the startup process from the physical memory access level. When the boot process enters the non-real-time operating system kernel startup stage, that is, when the second processor core starts, the scheduling configuration parameters are further hardened through the security mechanism of the second processor core itself. Specifically, by utilizing the characteristics of the kernel's static key, read-only initialization data area, etc., the loaded scheduling configuration parameters (such as time slice length, period, etc.) are marked and locked in the read-only data segment. This dual locking mechanism, combining hardware and software, ensures the immutability of the core scheduling strategy throughout the entire operating cycle, thus laying a solid foundation for the secure and deterministic allocation of computing power.

[0022] Furthermore, the time-slice allocation strategy divides each fixed period of the periodic scheduling timer into a first duration and a second duration. The first duration is pre-allocated to the real-time operating system, and the second duration is pre-allocated to the non-real-time operating system. It should be noted that the first duration is shorter than the second duration. Understandably, if we assume the fixed period is T, the first duration is Tr, and the second duration is Ts, then T = Tr + Ts. In practical applications, when the period T is 1 millisecond, the first duration Tr is less than or equal to 250 microseconds. It should also be noted that in this embodiment, the time-slice allocation strategy fixes Tr to the real-time operating system, providing it with a deterministic dedicated time window to ensure microsecond-level hard real-time response; and fixes Ts to the non-real-time operating system, ensuring it receives a predictable and undisturbed share of computing power. The two are strictly isolated in time, eliminating interference between heterogeneous systems and achieving high utilization and secure sharing of NPU computing power.

[0023] S120. In the non-real-time operating system, a periodic scheduling timer is initialized and started according to the scheduling configuration parameters.

[0024] In this embodiment, within the first processor core, its NPU, based on the fixed period defined in the scheduling configuration parameters, calls the high-precision timer interface provided by the first processor core itself to create and initialize a periodic scheduling timer. This periodic scheduling timer is configured to periodically generate and emit time slice switching signals at the fixed interval, thereby driving the subsequent context switching process.

[0025] S130. In response to the time slice switching signal issued by the periodic scheduling timer, a first context switching operation is triggered.

[0026] In this embodiment, the first context switching operation includes: issuing a first control instruction to the NPU to suspend the computing tasks of the currently serving operating system and save the first hardware context corresponding to the computing tasks; and switching the access rights and hardware context of the NPU to another operating system to be served. It should be noted that the first control instruction causes the NPU to immediately suspend all currently executing computing pipelines and atomically saves the state, register values, and related memory address pointers of all its internal parallel computing units to a specific memory area allocated to the currently serving operating system, forming the first hardware context. Then, the access control of the NPU is switched to another operating system to be served, and its corresponding complete hardware context is loaded from a storage area preset for that operating system into the NPU, thereby completing a time-slice-based, controlled computing power and context switch.

[0027] S140. In response to a real-time preemption event generated by the real-time operating system, a second context switch operation is triggered.

[0028] In this embodiment, the second context switching operation includes: the second processor core directly issuing a second control instruction to the NPU to preemptively suspend the computing task currently occupied by the non-real-time operating system and saving the second hardware context corresponding to the computing task; loading the third hardware context corresponding to the computing task for the real-time operating system and executing it. It should be noted that the interrupt service routine for the real-time preemption event directly operates the NPU's registers via memory-mapped input / output. It should also be noted that both the first and second control instructions are implemented by writing a preset value to the same interrupt control register of the NPU to suspend the NPU's computing pipeline.

[0029] Specifically, when the real-time operating system running on the second processor core detects an external hard real-time event (such as a CAN bus error or sensor threshold alarm), it immediately triggers a real-time preemption event. The interrupt service routine on the second processor core responds to this event and bypasses any software scheduling layer of the non-real-time operating system, writing a preset specific value directly to the NPU's interrupt control register via memory-mapped input / output; this is the second control instruction. This instruction is received by the NPU with hardware-level priority and atomically and immediately suspends all computing tasks currently being used by the non-real-time operating system, halting their data processing pipeline. Immediately afterwards, the current state of all computing units, register contents, and related memory pointers within the NPU are completely saved to a protected memory area specifically allocated for the non-real-time operating system, forming the second hardware context. After this, the pre-stored or pre-compiled dedicated model parameters and execution state are quickly read from the secure memory area reserved for the real-time operating system and loaded as the third hardware context into the corresponding computing unit and memory of the NPU. Once loaded, the NPU immediately begins executing the corresponding real-time computing task (such as fault classification or vibration analysis) based on the newly loaded context, thereby achieving a hard real-time response with microsecond-level latency.

[0030] S150. After the computing task of the real-time operating system is completed, a third context switch operation is triggered to restore the hardware context of the non-real-time operating system that was previously preempted, so that the interrupted computing task can continue to be executed.

[0031] In this embodiment, after the real-time operating system's computational task completes or reaches its available time threshold, a third context switch operation is triggered. This operation first saves or clears the execution state of the real-time task. Then, it reads the previously fully saved non-real-time operating system (NPU) state data from the protected memory area storing the second hardware context. This data is precisely restored to the various computational units, control registers, and memory pointers within the NPU, thereby completely reverting the NPU's hardware execution context to the precise breakpoint at the time of preemption. Finally, the preemption lock on the NPU is released, and its computational pipeline is restarted, allowing the interrupted non-real-time computational task to seamlessly continue execution from the breakpoint, ensuring the integrity of the computational task.

[0032] It should be noted that the saving of the first and second hardware contexts, the loading of the third hardware context, and the restoration of the hardware context are all implemented by calling the NPU's context processing interface function. The following processing interface function encapsulates the underlying hardware access sequence associated with the NPU's power management state, enabling atomic execution of save and restore operations. This ensures the consistency, integrity, and isolation of the states of all computing units, registers, and memory pointers within the NPU when switching between different operating systems.

[0033] like Figure 4 As shown, the NPU computing power dynamic allocation method in this invention includes six steps: hardware-level locking during the U-Boot boot phase, software-level circuit breaking during the Linux kernel startup phase, scheduling timer initialization, periodic time slice switching, real-time event hard preemption, and preemption recovery. For ease of understanding, the specific implementation process of the above six steps is described below: Hardware-level locking during the U-Boot boot phase: Utilizing a pure open-source software stack, a secure, deterministic, and efficient heterogeneous resource-sharing system is built on a single chip. After power-on, the bootloader U-Boot executes first, using its standard memmap command to set the device tree memory region containing the NPU's scheduling configuration parameters to read-only. This operation establishes the first physical write protection barrier at the memory management unit level, ensuring the immutability of the core configuration during the initial startup of the operating system and laying a reliable hardware foundation for all subsequent scheduling operations.

[0034] Software-level circuit breaking during Linux kernel startup: As the boot process enters the non-real-time operating system kernel stage, a deeper level of "software circuit breaking" is implemented in the Linux kernel (the first processor kernel). In the initialization function loaded by the NPU driver module, the `static_key` mechanism and `__ro_after_init` attribute provided by the Linux kernel mainline are used to permanently lock the scheduling configuration parameters parsed from the device tree in the kernel's read-only data area. Any subsequent code path attempting to modify these parameters will be optimized by the compiler into a static jump, thus becoming logically unreachable. This mechanism, combined with kernel module mandatory signatures and integrity measurement architecture, forms a robust software defense against malicious runtime tampering, achieving configuration protection without relying on dedicated security firmware.

[0035] Scheduler initialization: After configuration locking, the driver immediately calls the Linux kernel's standard high-precision timer interface to create and start a periodic scheduler based on the fixed parameters. Once the system enters a stable operating state, the dynamic scheduling mechanism begins to work. Its main function is based on the periodic time slice switching of the high-precision timer.

[0036] Periodic time-slice switching: The periodic scheduling timer periodically triggers an interrupt. In its callback function, a specific value is first written to the NPU's interrupt control register, atomically pausing the current computation pipeline. Then, the NPU's native power management context saving function is called to save the complete hardware execution state to the private memory area of ​​the currently serving operating system. Next, NPU access is switched to another operating system, and the corresponding context is loaded to allow execution to continue. This process achieves predictable computational power rotation between the non-real-time operating system and the real-time operating system at a fixed ratio (e.g., 250μs:750μs), ensuring fairness for non-real-time tasks and overall system throughput.

[0037] Real-time event hard preemption: Clock-driven polling alone is insufficient for hard real-time requirements; therefore, hardware interrupt-based real-time preemption is a key enhancement. When the real-time operating system running on the second processor core captures an emergency event such as a CAN error, its interrupt service routine completely bypasses the Linux kernel scheduler, directly writing a preemption instruction to the same interrupt control register of the NPU via memory-mapped I / O. This hardware-level operation can forcibly suspend any NPU tasks being executed by Linux within microseconds. Subsequently, the context of the interrupted task is saved, and its dedicated lightweight AI model context is quickly loaded for the real-time operating system and executed immediately. This path ensures that the latency from the occurrence of the event to the NPU starting to process the real-time task is strictly controlled within 50μs, meeting the deterministic limits of industrial control.

[0038] Preemptive Recovery: After a real-time task completes, a preemptive recovery process is automatically triggered. The real-time operating system notifies the non-real-time operating system to reload the previously fully saved task context into the NPU, enabling the interrupted non-real-time computing task to seamlessly resume execution from the breakpoint. This ensures the integrity of the computing task and avoids wasting computing power.

[0039] In summary, the NPU computing power dynamic allocation method in this invention establishes a secure foundation for the scheduling strategy through "dual locking during startup," achieves efficient reuse of computing power through "timer periodic rotation," and ensures real-time response through "hardware direct preemption." The entire solution cleverly reuses the NPU for context switching, utilizes standard Linux kernel mechanisms for security solidification, and ultimately solves the three core challenges of isolation, efficiency, and real-time performance when sharing dedicated computing power between heterogeneous systems on a low-cost single-chip platform without adding any dedicated hardware. This provides a reliable and verifiable dynamic computing power allocation solution for industrial edge intelligent devices.

[0040] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for dynamically allocating NPU computing power, applied to on-chip heterogeneous systems, characterized in that, The on-chip heterogeneous system includes a non-real-time operating system running on a first processor core and a real-time operating system running on a second processor core. The NPU is configured to be shared by the first processor core and the second processor core. The method includes: The scheduling configuration parameters of the NPU are locked, wherein the scheduling configuration parameters include the time slice allocation strategy of the NPU between the non-real-time operating system and the real-time operating system; In the non-real-time operating system, a periodic scheduling timer is initialized and started according to the scheduling configuration parameters; In response to the time slice switching signal issued by the periodic scheduling timer, a first context switching operation is triggered; In response to a real-time preemption event generated by the real-time operating system, a second context switch operation is triggered; After the computing task of the real-time operating system is completed, a third context switch operation is triggered to restore the hardware context of the non-real-time operating system that was previously preempted, so that the interrupted computing task can continue to be executed. The first context switching operation includes: A first control command is issued to the NPU to suspend the computing task currently serving the operating system and save the first hardware context corresponding to the computing task; Switch the NPU's usage rights and hardware context to another operating system to be served; The second context switching operation includes: The second processor core directly issues a second control instruction to the NPU to preemptively pause the computing task currently occupied by the non-real-time operating system and save the second hardware context corresponding to the computing task. The real-time operating system loads and executes the third hardware context corresponding to the computing task.

2. The method according to claim 1, characterized in that, Locking the scheduling configuration parameters of the NPU includes: During the bootloader stage, the device tree memory area containing the scheduling configuration parameters is set to read-only to complete write protection during the initial startup phase. When the second processor core starts up, the scheduling configuration parameters are locked in the kernel data area through the kernel security mechanism.

3. The method according to claim 1, characterized in that, Both the first control instruction and the second control instruction are implemented by writing a preset value to the same interrupt control register of the NPU to pause the NPU's computing pipeline.

4. The method according to claim 1, characterized in that, The saving of the first and second hardware contexts, the loading of the third hardware context, and the restoration of the hardware context are all achieved by calling the NPU's context processing interface function.

5. The method according to claim 1, characterized in that, The time slice allocation strategy divides each fixed period of the periodic scheduling timer into a first duration and a second duration, wherein the first duration is pre-allocated to the real-time operating system and the second duration is pre-allocated to the non-real-time operating system.

6. The method according to claim 5, characterized in that, The first duration is shorter than the second duration.

7. The method according to claim 1, characterized in that, The second processor core responds to the interrupt service routine of the real-time preemption event and directly operates the NPU registers through memory-mapped input / output.

8. An on-chip heterogeneous system, characterized in that, include: One of the primary processor cores is configured to run a non-real-time operating system; A second processor core is configured to run a real-time operating system; A shared NPU; The on-chip heterogeneous system is configured to dynamically schedule the computing power of the NPU between the non-real-time operating system and the real-time operating system according to any one of claims 1 to 7.

Images