A neural network processor acceleration engine for an industrial real-time operating system and electronic device

Abstract

The application discloses an industrial real-time operating system-oriented neural network processor acceleration engine and electronic equipment, belongs to the field of industrial embedded operating systems, and solves the problem of high system call overhead in an existing user mode driving scheme. The application discloses the following technical scheme: a kernel mode driving module adopts an interrupt upper and lower half separation mechanism to process a neural network processor interrupt, and allocates an independent interrupt processing thread for each calculation core; a shared memory management module opens a continuous physical memory area in a kernel space, maps to a user mode address space, and transmits data between the continuous physical memory area and the neural network processor through a direct memory access controller; a kernel mode encryption module is used for performing an encryption operation before data is written into the continuous physical memory area, and performing a decryption operation before the data is transmitted from the continuous physical memory area to the neural network processor; and a task scheduling module dynamically allocates the computing power resources of the neural network processor based on a task priority. The application is used in the field of oilfield inspection.

Description

Technical Field

[0001] This invention belongs to the field of industrial embedded operating systems, and in particular relates to a neural network processor acceleration engine for industrial real-time operating systems. Background Technology

[0002] As a critical national energy infrastructure sector, the petroleum industry's production data and equipment control system information security directly impact energy supply stability. Currently, handheld terminals for oilfield inspections generally use foreign commercial mobile operating systems, which suffer from opaque system permissions, uncontrollable data transmission, and potential backdoor risks. These systems fail to meet the Level 3 or higher independent controllability requirements of the "Petroleum Industry Information Security Level Protection Specification." Therefore, the industry is gradually adopting OpenEuler Industrial Edition as the core choice for domestic terminal operating system replacement. This operating system boasts high real-time performance, high reliability, and customizable features, adapting to the stringent operational requirements of industrial scenarios. The hardware selection for these inspection terminals often uses the RK3588 chip, which integrates a neural network processor with 6 TOPS integer processing capability, supporting the computing power requirements of multimodal edge detection tasks. However, the current OpenEuler Industrial Edition has significant technical deficiencies in its support for this chip's neural network processor. Rockchip only provides the closed-source user-space RKNNSDK development package, and only exposes a minimal device control interface in the kernel space. It does not release the complete kernel-space driver source code for operator scheduling, interrupt management, memory management, etc., preventing community developers from performing deep optimizations at the kernel layer. Furthermore, the OpenEuler 22.03 LTS mainline kernel has not yet incorporated the official native driver for the RK3588 neural network processor. Existing third-party adaptation solutions are all wrappers and calls to the official user-space library. Moreover, the PREEMPT_RT real-time preemption patch included in the OpenEuler industrial version imposes strict constraints on the kernel driver's concurrent timing and interrupt priority. User-space solutions can avoid the complex real-time kernel adaptation work; therefore, a native kernel-space driver solution has never emerged in the industry.

[0003] The primary problem with existing user-mode driver solutions is the excessively high system call overhead. Since all hardware interaction logic is implemented in user mode, each neural network computation task requires frequent context switching between user mode and kernel mode. In actual measurements, system call overhead for a single computation task accounts for more than 35% of the total time, and the computing power utilization of the neural network processor is less than 40%. The performance of mainstream inspection and detection tasks is only 53% of that under the Android system, failing to fully utilize hardware computing power and making it difficult to support the computing power requirements of multi-task concurrency in field inspections. Secondly, data transmission efficiency is low. Existing drivers use a multi-level memory copy data transmission mechanism, requiring computational data to be copied three times sequentially through user application memory, kernel page cache, and neural network processor video memory. The transmission latency of a single frame of 1080P visible light image exceeds 20ms, and data transfer overhead accounts for more than 25% of the overall computation time, becoming a performance bottleneck for edge computing. The zero-copy transmission scheme of general-purpose Linux systems cannot be directly migrated to this platform. The DMA controller built into the RK3588 neural network processor only supports 4KB aligned contiguous physical memory, which is incompatible with the distributed-aggregate DMA mode of general-purpose Linux. Unaligned transmission will trigger hardware address errors. Meanwhile, the platform's CPU and neural network processor belong to different bus domains, lacking hardware cache consistency. The general zero-copy solution lacks software cache refresh and failure maintenance, leading to dirty reads and distorted calculation results. Furthermore, the PREEMPT_RT real-time preemption patch alters the scheduling timing of DMA completion interrupts, and the general driver lacks interrupt thread isolation, making it prone to DMA synchronization timeouts and transmission interruptions, failing to meet the reliability requirements of industrial scenarios.

[0004] Secondly, there is insufficient adaptability to industrial scenarios. Existing general-purpose drivers are not customized for oilfield inspection scenarios and lack industrial-grade task scheduling and resource isolation capabilities. When multiple computing tasks are concurrent, issues such as computing power contention and memory out-of-bounds access can easily occur. The system's continuous fault-free operation time is less than 72 hours, making it unable to adapt to the harsh operating environment of oilfields with high temperatures, high humidity, and strong electromagnetic interference. Finally, there is a lack of security and compliance capabilities. The existing driver system does not have built-in domestic cryptographic security capabilities. Neural network model weights, inspection data, and fault detection results are all stored and transmitted in plaintext, which does not comply with the compliance requirements for full lifecycle encryption of data in the oil industry and poses a risk of data leakage. The industry has attempted to introduce the SM4 algorithm for end-to-end encryption, but irreconcilable technical contradictions exist. On one hand, introducing encryption / decryption logic into user space significantly increases system overhead and disrupts real-time computation. On the other hand, the RK3588 neural network processor does not support direct ciphertext computation, and user-space encryption schemes require an additional memory copy, resulting in a performance loss exceeding 15% in actual tests. Furthermore, existing drivers do not pipeline encryption / decryption with DMA transfer and kernel scheduling, failing to offset the performance loss caused by encryption. Therefore, a feasible low-loss national cryptographic acceleration solution has yet to be developed for industrial applications. To date, the industry has not yet developed a complete RK3588 neural network processor kernel driver and acceleration solution based on the OpenEuler industrial version, adapted to oilfield inspection scenarios. The inability to simultaneously achieve edge computing power release, data security protection, and industrial-grade reliability under a domestically developed operating system has become a core technical bottleneck for the full-stack domestic deployment of oilfield inspection terminals. Summary of the Invention

[0005] In view of this, the present invention aims to propose a neural network processor acceleration engine and electronic device for industrial real-time operating systems, so as to solve the problem of excessive system call overhead in existing user-mode driven solutions.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a neural network processor acceleration engine for industrial real-time operating systems, the engine comprising a kernel-mode driver module, a shared memory management module, a kernel-mode encryption module, and a task scheduling module; The kernel-mode driver module provides a hardware acceleration interface in the form of a kernel character device, uses an interrupt top-and-bottom separation mechanism to handle neural network processor interrupts, and allocates an independent interrupt handling thread to each computing core. The shared memory management module allocates a contiguous physical memory region in the kernel space and maps it to the user space address space. It then transmits data between the contiguous physical memory region and the neural network processor through the direct memory access controller. The kernel-mode encryption module is integrated into the kernel-mode driver module and is used to perform encryption operations before data is written to the contiguous physical memory region and to perform decryption operations before data is transmitted from the contiguous physical memory region to the neural network processor. The task scheduling module runs in kernel mode and dynamically allocates the computing resources of the neural network processor based on task priority.

[0007] Furthermore, a preferred approach is proposed, in which the upper and lower halves of the interrupt separation mechanism only perform interrupt status register reading and interrupt flag clearing operations, while the lower half of the interrupt asynchronously processes the calculation result verification and status update through the kernel work queue; the independent interrupt handling thread is bound to a preset isolated CPU core for real-time scheduling strategy.

[0008] Furthermore, a preferred approach is proposed, wherein the shared memory management module performs consistency maintenance operations for the non-consistent cache architecture: before initiating direct memory access transfer, a cache flush operation is performed on the contiguous physical memory region; after the neural network processor completes the calculation and writes back the data, a cache invalidation operation is performed on the contiguous physical memory region.

[0009] Furthermore, a preferred approach is proposed, wherein the direct memory access controller adopts a linked list-based block transfer mechanism, and the block size is aligned with the page size of the memory management unit; the shared memory management module is also configured with a pipelined prefetch mechanism, which triggers direct memory access transfer of the next frame of input data when the progress of the neural network processor in processing the current frame of data reaches a preset threshold.

[0010] Furthermore, a preferred method is proposed, wherein the kernel-mode encryption module adopts a block symmetric encryption algorithm, and the encryption scope covers the weights of the artificial intelligence model, the input data, and the inference results; the model weights are decrypted in kernel mode and directly loaded into the dedicated video memory of the neural network processor during loading, and the intermediate data in the inference process exists in plaintext form inside the neural network processor.

[0011] Furthermore, a preferred embodiment is proposed, wherein the engine further includes a hardware key storage unit for storing encryption keys; the kernel-mode encryption module reads the key into the kernel encryption register through the chip's internal security bus, and the plaintext of the key is not exposed to the system main memory.

[0012] Furthermore, a preferred approach is proposed, wherein the task scheduling module includes three levels of task priority, and high-priority tasks have the right to preempt low-priority tasks; the scheduler calculates task weights with a fixed time slice period, and the weights are determined by the task priority coefficient, the remaining execution time coefficient, and the task load coefficient.

[0013] Furthermore, a preferred embodiment is proposed, wherein the task scheduling module also includes a load balancing unit, which monitors the load status of each computing core of the neural network processor in real time, dynamically allocates tasks to idle computing cores, and controls the load difference between cores within a preset range.

[0014] Furthermore, a preferred approach is proposed, wherein the engine also includes a hardware abstraction layer, which encapsulates the details of underlying hardware register operations and instruction sets, and provides standardized interfaces for model loading, operator submission, memory management, and status query; the hardware abstraction layer has a built-in operator mapping unit for industrial inspection scenarios, which directly maps computational operators to combinations of native instructions from the neural network processor hardware.

[0015] Based on the same inventive concept, the present invention also proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor is coupled to a neural network processor, and the computer program, when executed by the processor, implements the function of an acceleration engine as described in any one of the above.

[0016] Compared with the prior art, the beneficial effects of the present invention are: The acceleration engine constructed in this invention is based on the core principle of completely sinking the hardware control logic, which was originally detached from the kernel, into the operating system kernel mode, thus reconstructing the hardware resource call path and data flow mechanism. At the kernel-mode driver level, the traditional user-mode encapsulated intermediate layer is abandoned, and the register access and instruction issuance of the RK3588 neural network processor are directly taken over based on the character device interface. To address the specific constraints of industrial real-time operating systems, a fine-grained interrupt splitting mechanism is designed, strictly compressing the interrupt upper half into atomic operations containing only status reading and flag clearing, thereby freeing up interrupt bus resources and ensuring microsecond-level response of critical peripherals such as infrared sampling and gas detection. The lower half relies on the kernel work queue to asynchronously process the computation logic and binds an independent real-time interrupt thread to each computing core. Through CPU affinity configuration, it is isolated to a specific physical core to avoid preemption by ordinary business processes, thus achieving physical isolation between computing resources and safety-critical tasks at the hardware interrupt level.

[0017] At the data transfer level, considering the hardware characteristics of the RK3588 chip's non-uniform cache architecture, the common distributed-aggregate DMA mode was abandoned, and instead a 4KB aligned contiguous physical memory region was forcibly allocated in the kernel space. By establishing a shared address mapping between user space and kernel space, the redundant copying process from user memory to kernel cache in traditional solutions was eliminated. More importantly, a software-led cache consistency maintenance strategy was introduced at critical nodes in data flow: a cache flush was forcibly performed before DMA transfer to ensure that the data in physical memory was the latest version; and a cache invalidation was forcibly performed after the neural network processor wrote back data to ensure that the CPU read the updated hardware results. This hardware-software combined synchronization mechanism solved the problem of dirty reads in heterogeneous multi-core architectures. Simultaneously, based on dynamic monitoring of bus bandwidth, pipeline prefetch timing control logic was designed to utilize the gaps in the processing of the current frame by the computing units to initiate the DMA transfer of the next frame of data in advance, achieving parallel overlap of computation and transmission.

[0018] At the security mechanism level, it breaks through the traditional design concept of separating encryption and computation, embedding the SM4 national cryptographic algorithm logic into the kernel-mode driven data transmission path. By pipelined integration of encryption / decryption operations with DMA transfer, it achieves real-time encryption before data is written to shared memory and real-time decryption before it is read, ensuring that plaintext data exists only momentarily within the neural network processor and does not end up in main memory accessible to user mode. Combined with the chip's built-in efuse security fuse area storing the key, it avoids the risk of key exposure in DDR memory. At the task scheduling level, it constructs a dedicated kernel scheduling domain independent of the operating system's native scheduler. Task priorities are divided according to the urgency of oil inspection operations, and a dynamic weight calculation model including priority coefficients, remaining time coefficients, and load coefficients is introduced. This achieves microsecond-level preemption of high-priority security tasks and load balancing among multiple computing cores, and blocks unauthorized access between tasks through memory address space isolation technology. Attached Figure Description

[0019] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings: Figure 1 This is a schematic diagram of the overall engine architecture hierarchy described in this invention; Figure 2 This is a schematic diagram of the entire AI inference process described in this invention. Figure 3 This is a comparison chart of the inference frame rates of the three types of oil inspection models described in this invention; Figure 4 This is a schematic diagram illustrating the performance contribution of each module described in this invention and its impact on SM4 encryption performance; Figure 5 This is a schematic diagram comparing the emergency task response time and NPU core load balancing described in this invention, wherein (a) is a schematic diagram of emergency task response time and (b) is a heat map of NPU core load balancing. Figure 6 This is a schematic diagram of the 720h long-term operation stability curve described in this invention; Figure 7 This is a schematic diagram comparing the inference frame rate before and after end-to-end encryption as described in this invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of the present invention can be combined with each other, and the described embodiments are only some embodiments of the present invention, not all embodiments.

[0021] Implementation Method 1: This implementation method proposes a neural network processor acceleration engine for industrial real-time operating systems. The engine includes a kernel-mode driver module, a shared memory management module, a kernel-mode encryption module, and a task scheduling module. The kernel-mode driver module provides a hardware acceleration interface in the form of a kernel character device, uses an interrupt top-and-bottom separation mechanism to handle neural network processor interrupts, and allocates an independent interrupt handling thread to each computing core. The shared memory management module allocates a contiguous physical memory region in the kernel space and maps it to the user space address space. It then transmits data between the contiguous physical memory region and the neural network processor through the direct memory access controller. The kernel-mode encryption module is integrated into the kernel-mode driver module and is used to perform encryption operations before data is written to the contiguous physical memory region and to perform decryption operations before data is transmitted from the contiguous physical memory region to the neural network processor. The task scheduling module runs in kernel mode and dynamically allocates the computing resources of the neural network processor based on task priority.

[0022] In the interrupt upper and lower halves separation mechanism described in this embodiment, the upper half of the interrupt only performs interrupt status register reading and interrupt flag clearing operations, while the lower half of the interrupt asynchronously processes the calculation result verification and status update through the kernel work queue; the independent interrupt handling thread is bound to a preset isolated CPU core for real-time scheduling strategy.

[0023] The shared memory management module described in this embodiment performs consistency maintenance operations for non-consistent cache architectures: before initiating direct memory access transfer, it performs cache flushing operations on the contiguous physical memory region; after the neural network processor completes calculations and writes back data, it performs cache invalidation operations on the contiguous physical memory region.

[0024] In this embodiment, the direct memory access controller adopts a linked list-based block transfer mechanism, and the block size is aligned with the page size of the memory management unit; the shared memory management module is also configured with a pipelined prefetch mechanism, which triggers direct memory access transfer of the next frame of input data when the progress of the neural network processor in processing the current frame of data reaches a preset threshold.

[0025] In this embodiment, the kernel-mode encryption module adopts a block symmetric encryption algorithm, and the encryption scope covers the weights of the artificial intelligence model, input data and inference results. When loading, the model weights are decrypted in kernel mode and directly loaded into the dedicated video memory of the neural network processor. The intermediate data in the inference process exists in plaintext form inside the neural network processor.

[0026] Furthermore, the engine also includes a hardware key storage unit for storing encryption keys; the kernel-mode encryption module reads the key into the kernel encryption register through the chip's internal security bus, and the plaintext key is not exposed to the system main memory.

[0027] In this embodiment, the task scheduling module includes three levels of task priority, with high-priority tasks having the right to preempt low-priority tasks; the scheduler calculates task weights using a fixed time slice as a period, and the weights are determined by the task priority coefficient, the remaining execution time coefficient, and the task load coefficient.

[0028] Furthermore, the task scheduling module also includes a load balancing unit, which monitors the load status of each computing core of the neural network processor in real time, dynamically allocates tasks to idle computing cores, and controls the load difference between cores within a preset range.

[0029] In this embodiment, the engine further includes a hardware abstraction layer, which encapsulates the details of underlying hardware register operations and instruction sets, and provides standardized interfaces for model loading, operator submission, memory management, and status query. The hardware abstraction layer has a built-in operator mapping unit for industrial inspection scenarios, which directly maps computational operators to combinations of native instructions from the neural network processor hardware.

[0030] Implementation Method Two: This implementation method is a complete embodiment of the neural network processor acceleration engine for industrial real-time operating systems described in Implementation Method One, including: This implementation constructs a five-layer integrated acceleration architecture comprising a hardware layer, a kernel driver layer, a hardware abstraction layer, an inference framework layer, and a business application layer. Through the collaborative work of five core modules—kernel-mode native drivers, hardware abstraction adaptation, zero-copy data transmission, domestic cryptographic encryption, and priority task scheduling—it achieves end-to-end performance optimization and security hardening for AI inference. The technical principles, core designs, and implementation logic of each module are as follows: 1. Kernel-mode NPU driver module This module implements a full-featured native driver for the RK3588 NPU hardware at the kernel level, providing a unified hardware acceleration call entry point to user space in the form of a character device. It fully supports all NPU hardware primitives such as convolution, pooling, activation, and tensor operations, and completes the entire process of operator submission, hardware scheduling, and result return directly in kernel mode, eliminating the system overhead caused by frequent switching between user mode and kernel mode from the root of the architecture.

[0031] A two-level optimization is implemented for the NPU hardware interrupt mechanism, employing a separate upper and lower half interrupt design: the upper half only performs interrupt status register reading and interrupt flag clearing operations, quickly releasing the hardware interrupt line; the lower half asynchronously processes the subsequent logic of operator execution through the kernel work queue, avoiding prolonged system interrupt blocking. An independent interrupt handling thread is allocated to each computing core of the NPU, enabling independent interrupt response for multi-core parallel computing.

[0032] The simplified design of the upper half of the interrupt is based on the fact that the oilfield inspection handheld terminal simultaneously carries multiple key industrial peripherals, such as infrared thermal imaging sensors, combustible gas sensors, and explosion-proof safety modules, all of which rely on hardware interrupts to achieve microsecond-level real-time sampling. Adding time-consuming operations such as operator verification and result copying to the upper half of the NPU interrupt would consume interrupt bus resources, causing interrupt response delays for other peripherals to exceed 10ms. This could lead to problems such as frame loss in infrared sampling and missed gas concentration detection, directly affecting the integrity of inspection data and explosion-proof safety compliance. Therefore, this solution compresses the upper half operations to only status reading and flag clearing, controlling the blocking time to within 2μs. All non-real-time logic is devolved to the lower half work queue to ensure the interrupt priority of core industrial peripherals.

[0033] Interrupt thread affinity and real-time scheduling adaptation: Independent interrupt threads are allocated to the three computing cores of the NPU and bound to two pre-isolated real-time CPU cores (CPU2 and CPU3) in the OpenEuler industrial version through CPU affinity. These threads are physically isolated from the cores running ordinary business processes, preventing business scheduling from interfering with interrupt response. The interrupt thread scheduling strategy is configured as SCHED_FIFO real-time scheduling, with a priority level of 90. This is lower than the 98 priority level for infrared and gas sensor acquisition interrupts but higher than ordinary system processes. This ensures the real-time performance of NPU interrupts without preempting interrupt responses from safety peripherals, matching the safety priority design of the oilfield scenario.

[0034] The calculation logic for the total interrupt response delay is as follows:

[0035] in, To interrupt the processing time of the first half, The time consumed by the asynchronous processing in the second half This reduces thread scheduling latency. By using a split design (top and bottom halves), the blocking time for interrupt response can be reduced to less than 2 microseconds, improving overall interrupt handling efficiency by more than 60%.

[0036] This implementation increases NPU computing power utilization from less than 40% in traditional solutions to over 68%. The remaining 32% overhead consists of three parts: approximately 12% is the pipeline-inherent overhead of NPU hardware operator switching, which is a fixed overhead at the chip architecture level and cannot be eliminated through driver optimization; approximately 10% is the context saving and switching overhead of multi-task scheduling; and approximately 10% is the overhead of cache coherence maintenance and memory synchronization. This solution has reduced the software-side optimizable overhead to less than 20% through operator fusion and scheduling optimization, and the 68% utilization rate has reached the optimal level of the driver layer in industrial scenarios of the RK3588 platform.

[0037] 2. Hardware Abstraction Layer Adaptation Module This module encapsulates the low-level hardware register operations and instruction set details of the RK3588 NPU, providing a unified and standardized calling interface for the inference framework. It includes four main categories of interface functions: model loading, operator submission, memory management, and status query, thereby decoupling the inference framework from the underlying hardware driver. Upper-level algorithms can be adapted to this acceleration engine without modification.

[0038] The hardware abstraction layer has a built-in operator mapping unit specifically for oilfield scenarios. For high-frequency operators in oilfield inspection, such as non-uniform correction of infrared thermal imaging, weighted fusion of multimodal features, and non-maximum suppression of small targets, it directly maps them to native NPU hardware instruction combinations, avoiding the additional scheduling overhead caused by splitting general operators. At the same time, it provides model format adaptation capabilities, automatically completes operator verification, graph optimization and hardware adaptation of the model, and significantly shortens the deployment cycle of oilfield inspection AI algorithms.

[0039] 3. Zero-copy data transmission optimization module This module employs DMA (Direct Memory Access) in conjunction with kernel shared memory technology to construct a unified three-level zero-copy data transfer path across user space, kernel space, and the NPU. A fixed-size contiguous physical memory region is allocated in kernel space and exposed to both user-space applications and NPU hardware through address mapping, eliminating the copying step from user memory to kernel cache in traditional drivers. The DMA controller directly reads inference input data from the shared memory region and transfers it to the NPU's video memory. After inference, the result is directly written back to a specified address in the shared memory, simplifying the traditional three-level transfer path—from user memory to kernel cache, and then to NPU video memory—to a single level.

[0040] The decision to allocate 512MB of shared memory capacity is based on the peak demand of supporting a maximum of three concurrent AI tasks (oil and gas leak detection, infrared fault detection, and weld defect identification). A single task includes a single frame of 1080P visible light input (approximately 6MB), a 640×512 16-bit infrared input (approximately 0.6MB), and a model intermediate feature cache (approximately 150MB), with a peak memory usage of approximately 160MB per task. The total demand for three concurrent tasks is approximately 480MB, with an 8% safety margin, ultimately resulting in 512MB of contiguous physical memory. This capacity accounts for 6.25% of the terminal's total memory (8GB), ensuring it does not encroach on the memory quota for industrial real-time applications while simultaneously meeting the peak demands of multi-task concurrency.

[0041] The logic for calculating the total transmission delay of a single frame of data is as follows:

[0042] in, The time required for shared memory address mapping (executed only once during initialization). For DMA hardware transfer time, This reduces the time required for hardware synchronization. Compared to traditional three-level copy schemes, the data transmission path is shortened by 67%, and transmission latency is reduced by more than 85%.

[0043] DMA-based block transfer uses a 4KB basic block size, based on platform hardware alignment constraints and performance testing: the RK3588's MMU page size is 4KB, and the NPU cache line is 64 bytes. Strict alignment between the 4KB transfer block and the memory page size avoids address remapping overhead caused by misaligned transfers. Comparison of 2KB, 4KB, 8KB, and 16KB block sizes in testing revealed: 2KB blocks resulted in excessively high transfer interrupt frequency and a 30% increase in CPU overhead; 8KB / 16KB blocks exhibited increased latency fluctuations, with jitter exceeding 2ms, failing to meet industrial real-time requirements; 4KB blocks achieved 92% bandwidth utilization with latency jitter ≤0.5ms, balancing throughput and determinism, and meeting the stable operation requirements of oilfield inspection scenarios.

[0044] Simultaneously, a pipelined prefetching mechanism is designed to start the DMA transfer of the next frame's input data in advance during the inference computation of the current frame, realizing pipelined parallelism of data transfer and inference computation; the DMA transfer adopts a linked list-based block scheduling, which supports continuous non-blocking transmission of large-size inspection images, further improving the overall throughput of the system.

[0045] In this embodiment, the pipeline prefetch timing control strategy includes: setting the prefetch trigger time to start the DMA transfer of the next frame when the NPU completes 80% of the operator scheduling for the current frame and enters the last computation stage. At this time, the NPU's main computing power is used for computation, and the system bus bandwidth utilization drops to below 30% of the peak value, so DMA prefetch will not compete with inference computation for bus bandwidth. At the same time, a bus bandwidth monitoring unit is designed to automatically suspend the prefetch operation when the real-time bandwidth utilization exceeds the 70% threshold, ensuring the bandwidth priority of the current frame's inference computation and avoiding inference latency fluctuations caused by prefetch.

[0046] This implementation of RK3588 platform-specific cache consistency handling includes: For the non-consistent bus architecture of the RK3588 CPU and NPU, a software consistency maintenance strategy of "flush before write, invalidate after read" is adopted: Before initiating DMA transfer, the CPU performs a `dma_map_single` operation to flush the cache, ensuring consistency between memory data and the cache; after the NPU completes inference and writes the result back to shared memory, the driver performs a `dma_unmap_single` operation to invalidate the cache, ensuring that the CPU reads the latest hardware-written data and avoiding inference result errors caused by dirty data. This solution is specifically adapted for the non-consistent DMA domain of the RK3588 platform, solving the frequent data corruption problems of general zero-copy solutions on this platform.

[0047] 4. Domestically produced full-link cryptographic encryption module This module integrates the SM4 domestic commercial cryptographic algorithm into the kernel-mode driver, employing a 128-bit key length to construct an end-to-end encryption mechanism for the entire lifecycle of AI data. The encryption scope covers three core data categories: locally stored AI model weights, multi-source input data collected during inspections, and fault detection results output during inference. Encryption and decryption operations are completed entirely within the kernel-mode driver, making them completely transparent to upper-layer applications. Security protection can be achieved without modifying existing AI algorithm code.

[0048] Encryption / Decryption Boundaries and Implementation: This implementation strictly defines the encryption / decryption boundaries, adhering to the principles of storage encryption, plaintext computation, and transmission encryption. During loading, decryption is completed once in kernel mode. The decrypted plaintext weights are directly sent to the NPU's dedicated video memory and do not enter the system main memory. No repeated decryption is required during inference. This operation is only performed during model initialization and does not consume inference time.

[0049] Each inference iteration performs end-to-end processing. Inspection data is encrypted before being written to shared memory, decrypted in kernel space using hardware acceleration via the ARMv8.2 Crypto extension instruction set before being read by the NPU, and encrypted immediately after the inference result is written back to shared memory. Plaintext data exists only briefly within the NPU computing unit and is never written to memory regions accessible to user space.

[0050] The performance overhead caused by encryption operations is calculated as follows:

[0051] in, The time required to encrypt a single frame of data Decryption takes time. This represents the total inference time per frame. By accelerating encryption computation through hardware instruction sets, the encryption time for a single 1080P image frame is controlled within 1ms, with an overall performance loss of less than 2%, minimizing the impact on inference real-time performance while ensuring data security.

[0052] The encryption key is stored in the efuse security fuse area built into the RK3588 chip, which is a chip-level secure storage and cannot be read by software. During driver startup, the key is read into a kernel-specific encryption register via the chip's internal security bus; the plaintext key is not exposed to DDR memory. To prevent side-channel attacks, randomized instruction insertion is used during SM4 operations to scramble the timing of calculations. The encrypted memory area uses dynamic address mapping, randomizing the physical address mapping relationship each time the model is loaded, reducing the identifiability of electromagnetic side-channel attacks.

[0053] SM4 is a 128-bit block cipher algorithm. For data less than 128 bits aligned, it uses the PKCS#7 standard for padding. The encrypted data length is aligned upwards to 16 bytes. The data size increase for a single 1080P image frame is ≤0.1%, negligible to overall performance. Encrypted data is not directly input to the NPU computing unit. The hardware abstraction layer decrypts and removes padding before DMA transfer, restoring the original tensor format before sending it to the NPU. This process is completely transparent to upper-layer operators, requiring no modification to the AI ​​model structure or operator definitions.

[0054] 5. Kernel-mode priority task scheduling module This module implements a dedicated AI task scheduler in the kernel, employing a priority-driven preemptive scheduling algorithm. Based on the oilfield inspection business scenario, AI tasks are divided into three levels: urgent, important, and general, each corresponding to different computing resource quotas. The scheduler makes scheduling decisions in 10ms time slices, allowing high-priority tasks to directly preempt the NPU computing resources of low-priority tasks, ensuring real-time response for safety-related tasks such as oil and gas leak detection and fire alarms.

[0055] The task scheduling weight calculation logic is as follows:

[0056] in, This is the task priority coefficient (3 for urgent level, 2 for important level, and 1 for general level). The coefficient for the remaining execution time of the task. This represents the task load factor. The scheduler dynamically allocates NPU computing core resources based on the weight value, with tasks having higher weights receiving priority in computing power allocation.

[0057] The scheduler has a built-in load balancing unit that monitors the load status of each computing core of the NPU in real time and dynamically adjusts the allocation of tasks to idle cores, keeping the load difference between cores within 10%. At the same time, through memory address space isolation technology, it enables independent allocation of memory resources for different tasks, prohibits unauthorized access between tasks, avoids overall system crashes caused by single task anomalies, and improves the operational stability in industrial scenarios.

[0058] Implementation Method 3: Participation Figures 1 to 7 This embodiment describes a specific application example of the neural network processor acceleration engine for an industrial real-time operating system described in Embodiment 2, including: The hardware carrier of this embodiment is a handheld oilfield explosion-proof inspection terminal, equipped with an RK3588 octa-core processor, an integrated NPU with 6 TOPS integer computing power, 8GB LPDDR4X RAM and 128GB encrypted storage; the operating system adopts OpenEuler 22.03 LTS SP3 industrial version, with a real-time kernel version 5.10; the business scenarios cover three core inspection businesses: infrared thermal imaging fault detection of oil equipment, wellhead oil and gas leak detection, and pipeline weld defect detection, and are suitable for field operation environments with no network, high temperature difference, and strong electromagnetic interference.

[0059] 1. Specific implementation of kernel-mode NPU driver This is a character device driver for the RK3588 NPU, developed based on the OpenEuler kernel architecture. The driver fully adapts to all hardware computing units of the RK3588 NPU, including 3D convolution, pooling, nonlinear activation, and general tensor operations, supporting hardware acceleration for all mainstream AI operators. The driver provides a unified entry point to user space through a device file, supporting four core operations: model loading, operator submission, status query, and result reading. All hardware interaction logic is completed in kernel space, requiring no user-space intervention.

[0060] Interrupt handling employs a two-tiered architecture: the upper half only reads the interrupt status register and clears the interrupt flag, with the time taken to within 2 microseconds, quickly releasing the hardware interrupt line and avoiding impact on the interrupt response of other industrial peripherals such as infrared and gas sensors; the lower half asynchronously processes the logic of result verification and status update after operator execution through the kernel work queue, without blocking system interrupt scheduling. For the three independent computing cores of the NPU, the driver allocates an independent interrupt handling thread and task queue to each core. The thread is bound to an isolated real-time CPU core, and the SCHED_FIFO real-time scheduling strategy is adopted to achieve independent interrupt response and task scheduling for multi-core parallel computing, improving multi-core scheduling efficiency by more than 40%.

[0061] 2. Specific implementation of the hardware abstraction layer The hardware abstraction layer runs in user space, interfacing with the kernel-level NPU driver via standard system calls, and providing standardized operator calls, memory management, and task submission interfaces for the inference framework, completely shielding the implementation differences of underlying hardware registers and instruction sets. AI models developed based on this standard format can run directly on this acceleration engine without code modification, reducing adaptation costs by over 80%.

[0062] The hardware abstraction layer incorporates a dedicated operator mapping table for oilfield inspection. For high-frequency operators in oilfield scenarios, such as non-uniform correction of infrared thermal imaging, weighted fusion of multimodal features, non-maximum suppression of small targets, and vibration signal filtering, these are directly mapped to native NPU hardware instruction combinations. This avoids the additional scheduling overhead caused by splitting general operators, improving operator execution efficiency by over 30%. Simultaneously, a built-in model graph optimization unit automatically performs operator fusion, redundant node removal, and branch simplification, streamlining the model computation graph and reducing hardware scheduling overhead during inference.

[0063] 3. Specific implementation of zero-copy data transmission A contiguous 512MB physical shared memory region is allocated in the kernel space. This region is simultaneously exposed to the user-mode address space through page table mapping, allowing both user-mode applications and kernel-mode drivers to directly read and write to this region without data copying. The DMA controller directly reads inference input data from the shared memory region and transfers it to the NPU's video memory via a high-speed bus. After inference is complete, the NPU directly writes the result back to the designated address in the shared memory. Only one DMA hardware transfer occurs throughout the entire process, achieving a true zero-copy data path. For the RK3588's non-consistent cache architecture, cache flush and invalidate operations are performed before and after the transfer to ensure data consistency.

[0064] DMA transfer uses 4KB as the basic transfer block and supports linked-list block transfer, enabling continuous, non-blocking transfer of large-size data such as 1080P visible light images and 640×512 16-bit infrared thermal imaging images. It also implements a pipelined prefetch mechanism, initiating DMA transfer of the next frame's data in advance when the current frame's inference computation is nearing completion, achieving parallel execution of data transfer and inference computation, thus improving the overall system throughput by more than 35%.

[0065] 4. Specific implementation of the security encryption module The SM4 symmetric encryption algorithm is integrated into the kernel-mode driver, using a 128-bit key length, fully complying with national commercial cryptographic algorithm standards. Encryption covers the entire data lifecycle: AI model weight files stored locally are encrypted using SM4. During model loading, decryption is performed in kernel mode and the data is directly sent to the NPU memory; the decrypted plaintext is not written to the storage medium. Visible light, infrared, and sensor data collected during inspection are encrypted before being written to shared memory, and automatically decrypted when read by the NPU. Inference results are encrypted before being output, and decrypted for use by upper-layer applications.

[0066] Encryption and decryption operations are accelerated in parallel using the ARMv8.2 Crypto hardware instruction set. The encryption time for a single 1080P image frame is controlled within 1ms, with an impact of less than 2% on overall inference performance. The key is stored in the built-in efuse security fuse area of ​​the RK3588 chip and read through the chip's internal security bus during driver operation. The plaintext key is not exposed in the memory space to prevent key leakage and ensure the security of the encryption system.

[0067] 5. Specific implementation of kernel-mode task scheduling A dedicated AI task scheduler is implemented in the kernel, employing a priority-based preemptive scheduling algorithm. Based on the oil inspection business, tasks are divided into three levels: the emergency level corresponds to safety-related tasks such as oil and gas leak detection and fire alarms, which are allocated 100% of the computing power quota and can preempt all low-priority tasks; the important level corresponds to maintenance-related tasks such as equipment fault diagnosis and pipeline defect detection, which are allocated 50% of the basic computing power quota; and the general level corresponds to background tasks such as data uploading and report generation, which are allocated 20% of the basic computing power quota.

[0068] The scheduler makes scheduling decisions in 10ms time slices. When a high-priority task arrives, the current low-priority task is immediately suspended, and the complete task context (including register state, memory state, and execution progress) is saved before switching to the high-priority task. After the high-priority task completes, the execution of the low-priority task is automatically resumed, with context switching time controlled within 1ms. The scheduler monitors the load status of each NPU computing core in real time and dynamically adjusts task allocation to achieve load balancing among cores, improving the overall NPU computing power utilization to over 68%. Simultaneously, through memory address space isolation technology, an independent memory region is allocated to each task, access permission boundaries are set, and illegal memory access between tasks is prohibited, preventing system crashes caused by single-task memory overflows.

[0069] 6. The complete execution flow of a single AI inference is as follows: (1) The application layer initiates an AI inference request, carrying input data and task priority identifier; (2) The kernel-mode scheduler receives the request, determines whether to preempt the currently running task based on the priority, and allocates NPU computing resources; (3) The SM4 encryption module encrypts the input data and writes it to a specified address in the shared memory; (4) Perform a cache flush operation to maintain data consistency. The DMA controller starts the transfer and directly transfers the data in the shared memory to the NPU video memory. (5) The NPU performs hardware inference calculations and writes the results back to the specified area of ​​shared memory after completion; (6) Execute the cache invalidate operation, and the SM4 encryption module encrypts the inference result; (7) The scheduler notifies the application layer that inference is complete, and the application layer reads the result from shared memory and decrypts it; (8) The scheduler schedules the next task to be executed and releases the computing resources occupied by the current task.

[0070] 7. Experimental verification of the effect 7.1. Comparison Test Baseline Description The baseline scheme for this test uses the user-space RKNN driver (version 1.6.0) of the OpenEuler community edition released by Rockchip. The kernel version is the same as this scheme, namely OpenEuler 22.03 LTS SP3 5.10.0-rt real-time kernel. The compiler uses GCC 10.3.1-O2 optimization level. The hardware platform is completely identical (RK3588, 8GB LPDDR4X, hardware from the same batch) to ensure that the test variables are unique and the results are comparable.

[0071] For details, see Figure 3 This embodiment addresses the specific challenges of limited computing power in oilfield handheld terminals and the simultaneous deployment of multiple inspection models on the same terminal. It employs a kernel-mode direct-drive architecture as a differentiated technical approach, replacing the traditional user-mode drive scheme. This eliminates the redundant overhead of frequent context switching between user and kernel modes, adapting to the parallel operation requirements of three core inspection models: well site infrared fault detection, oil and gas leak detection, and weld defect detection. Quantitative results show that the inference frame rate of the three models is increased by an average of over 80% compared to the official baseline solution, single-frame transmission latency is significantly reduced, and CPU utilization is significantly decreased. This allows for the reservation of more computing resources for high-priority oilfield-specific tasks such as gas sensing and safety alarms, comprehensively improving the efficiency of single-terminal inspection operations.

[0072] 7.2 Ablation Experiment Verification See Figure 4 In this embodiment, five groups of controlled variables were set up to verify the independent performance contribution of each module. The test results are as follows:

[0073] Experiments demonstrate that the kernel-mode driver adapts to industrial-grade interrupt isolation requirements, zero-copy DMA matches the high-speed data throughput scenario of multiple sensors in oilfields, SM4 national cryptographic encryption meets the Level 3 compliance requirements of the petroleum industry, scheduling optimization adapts to the priority requirements of multi-task inspections, and each module is independently decoupled and can be tailored as needed. Quantitative results show that the kernel-mode driver brings a 33.9% increase in frame rate, zero-copy DMA further improves by 35.2%, and end-to-end SM4 encryption only incurs a 1.8% performance loss. The single-frame transmission latency is reduced from 22.0ms to 2.8ms, fully ensuring the real-time transmission of inspection data while meeting petroleum data compliance requirements.

[0074] 7.3 Comprehensive Performance Test Results Under the aforementioned hardware and system environment, three mainstream AI models for oilfield inspection were tested. (See [link / reference]). Figure 5 The core performance indicators for this embodiment are as follows: Single-model inference performance: The inference frame rate of the infrared fault detection model was improved from 5.3fps to 9.6fps, reaching 96% of the performance of the Android system; the inference frame rate of the weld seam small target detection model was improved from 4.8fps to 9.2fps, meeting the requirements of real-time inspection.

[0075] Data transmission latency: The transmission latency of a single 1080P image frame was reduced from 22ms to 2.8ms, a reduction of 87.3%, and the data transfer overhead ratio was reduced from 26% to 4.2%.

[0076] Encryption performance loss: After enabling full-link SM4 encryption, the overall inference performance loss is 1.8%, which is less than the design target of 2% and has no significant impact on real-time performance.

[0077] Multi-task scheduling performance: When the three types of tasks run concurrently, the response time of the emergency-level leak detection task is reduced from 2.3 seconds to 0.35 seconds, meeting the real-time response requirements of security tasks.

[0078] 7.4 Industrial Stability Test Test environment conditions: Temperature range -20℃ / 60℃ cycle (simulating the day and night temperature difference in an oil field), applied electromagnetic radiation interference of 10V / m, 1kHz / 1GHz (compliant with GB / T 17626 Industrial Electromagnetic Compatibility Level 3 Standard), relative humidity 90%RH; load conditions are 3 types of AI tasks running continuously concurrently, and 5 background industrial tasks such as infrared sampling, gas detection, and data storage running simultaneously.

[0079] See Figure 6 This embodiment demonstrates the following test results: it operated continuously under pressure for 720 hours in extreme environments without failure, increasing the mean time between failures (MTBF) from 72 hours in the traditional solution to 2150 hours. Memory leakage was controlled at the KB level, and latency fluctuation was less than 2%, fully meeting the stringent requirements for long-term stable operation of oilfield field terminals.

[0080] 7.5 Security Compliance Verification See Figure 7This embodiment addresses the specific compliance requirements of Level 3 Information Security Protection for petroleum production data. It employs a differentiated technical approach: kernel-level SM4 national cryptographic full-link encryption with keys embedded in the chip fuse area. Encryption and decryption of inspection data and model weights are completed at the kernel layer, mitigating the risk of plaintext leakage in user space and adapting to the high-security data transmission needs of oilfields. Quantitative results show that the inference performance loss under full-link national cryptographic encryption is only 1.8%, far lower than the industry's conventional encryption schemes with losses exceeding 10%. The key cannot be read by the application layer, eliminating the risk of plaintext data leakage. While ensuring the security and compliance of petroleum data, it has almost no impact on the real-time performance and efficiency of field inspection operations.

[0081] Implementation Method 4: This implementation method proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor is coupled to a neural network processor, and when the computer program is executed by the processor, it implements the function of an acceleration engine as described in any one of Implementation Methods 1 to 3.

[0082] The specific embodiments of the present invention disclosed above are merely illustrative of the invention. These embodiments do not exhaustively describe all details, nor do they limit the invention to the specific embodiments described. Many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention.

Claims

1. A neural network processor acceleration engine for industrial real-time operating systems, characterized in that, The engine includes a kernel-mode driver module, a shared memory management module, a kernel-mode encryption module, and a task scheduling module; The kernel-mode driver module provides a hardware acceleration interface in the form of a kernel character device, uses an interrupt top-and-bottom separation mechanism to handle neural network processor interrupts, and allocates an independent interrupt handling thread to each computing core. The shared memory management module allocates a contiguous physical memory region in the kernel space and maps it to the user space address space. It then transmits data between the contiguous physical memory region and the neural network processor through the direct memory access controller. The kernel-mode encryption module is integrated into the kernel-mode driver module and is used to perform encryption operations before data is written to the contiguous physical memory region and to perform decryption operations before data is transmitted from the contiguous physical memory region to the neural network processor. The task scheduling module runs in kernel mode and dynamically allocates the computing resources of the neural network processor based on task priority.

2. The neural network processor acceleration engine for industrial real-time operating systems according to claim 1, characterized in that, In the interrupt upper and lower half separation mechanism, the upper half of the interrupt only performs interrupt status register reading and interrupt flag clearing operations, while the lower half of the interrupt asynchronously processes the calculation result verification and status update through the kernel work queue; the independent interrupt handling thread is bound to a preset isolated CPU core for real-time scheduling strategy.

3. The neural network processor acceleration engine for industrial real-time operating systems according to claim 1, characterized in that, The shared memory management module performs consistency maintenance operations for the non-consistent cache architecture: before initiating direct memory access transfer, it performs cache flushing operations on the contiguous physical memory region; After the neural network processor completes the calculation and writes back the data, a cache invalidation operation is performed on the contiguous physical memory region.

4. The neural network processor acceleration engine for industrial real-time operating systems according to claim 1, characterized in that, The direct memory access controller adopts a linked list-based block transfer mechanism, with the block size aligned with the page size of the memory management unit; the shared memory management module is also configured with a pipelined prefetch mechanism, which triggers direct memory access transfer of the next frame of input data when the neural network processor reaches a preset threshold in processing the current frame of data.

5. A neural network processor acceleration engine for industrial real-time operating systems according to claim 1, characterized in that, The kernel-mode encryption module uses a block symmetric encryption algorithm, which covers the weights of the artificial intelligence model, input data and inference results. The model weights are decrypted in kernel mode and directly loaded into the dedicated video memory of the neural network processor when loaded. The intermediate data in the inference process exists in plaintext form inside the neural network processor.

6. A neural network processor acceleration engine for industrial real-time operating systems according to claim 5, characterized in that, The engine also includes a hardware key storage unit for storing encryption keys; the kernel-mode encryption module reads the key into the kernel encryption register through the chip's internal security bus, and the plaintext key is not exposed to the system main memory.

7. A neural network processor acceleration engine for industrial real-time operating systems according to claim 1, characterized in that, The task scheduling module includes three levels of task priority, with high-priority tasks having the right to preempt low-priority tasks; the scheduler calculates task weights using a fixed time slice period, and the weights are determined by the task priority coefficient, the remaining execution time coefficient, and the task load coefficient.

8. A neural network processor acceleration engine for industrial real-time operating systems according to claim 7, characterized in that, The task scheduling module also includes a load balancing unit, which monitors the load status of each computing core of the neural network processor in real time, dynamically allocates tasks to idle computing cores, and controls the load difference between cores within a preset range.

9. A neural network processor acceleration engine for industrial real-time operating systems according to claim 1, characterized in that, The engine also includes a hardware abstraction layer, which encapsulates the details of underlying hardware register operations and instruction sets, and provides standardized interfaces for model loading, operator submission, memory management, and status query. The hardware abstraction layer has a built-in operator mapping unit for industrial inspection scenarios, which directly maps computational operators to combinations of native instructions from the neural network processor hardware.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor is coupled to a neural network processor, and the computer program, when executed by the processor, implements the function of the acceleration engine as described in any one of claims 1 to 9.

Images