Edge computing video stream processing method and system based on NPU

Abstract

The invention discloses an NPU-based edge computing video stream processing method and system. The method comprises the steps of video stream data acquisition and writing-in, zero-copy hardware preprocessing, model loading and asynchronous reasoning and result acquisition and post-processing. According to the method, a'control-calculation 'hardware-level decoupling architecture is constructed, and a special NPU is started to execute reasoning, so that the load and the power consumption of a CPU are greatly reduced, and the problems of frequency reduction and dead halt of a system caused by overheating of the CPU are thoroughly solved. A full-link zero-copy data channel based on DMA and RGA hardware is established, so that the video stream processing delay is remarkably reduced, and stable real-time analysis under a high frame rate is realized. By adopting the operator fusion and INT8 hybrid quantization technology, the calculation and memory access efficiency of the model on the NPU is multiplied, and higher energy efficiency ratio and throughput are obtained while the precision is ensured. Finally, the edge equipment can stably run at low temperature for a long time in a closed environment without fan heat dissipation.

Description

Technical Field

[0001] This invention relates to the field of embedded artificial intelligence and edge computing technology, and more specifically, to an edge computing video stream processing method and system for a neural processing unit (NPU). Background Technology

[0002] With the deep integration of artificial intelligence and Internet of Things (IoT) technologies, intelligent video surveillance systems are increasingly being used in fields such as smart transportation, public safety, and industrial inspection. These applications typically require real-time execution of computationally intensive tasks such as target detection and feature extraction within massive video streams, placing stringent demands on system response speed, power consumption, and stability. Traditional centralized cloud computing models, limited by network bandwidth and transmission latency, struggle to meet real-time requirements. Therefore, edge computing architectures, which offload computing tasks to network edge devices (such as smart cameras and embedded industrial PCs), have become a crucial development direction for the industry.

[0003] Currently, most mainstream embedded edge vision systems employ a computing architecture centered on a general-purpose processor (CPU), or supplemented by a graphics processing unit (GPU) for acceleration. However, when processing continuous high-resolution video streams and running modern deep neural network models such as YOLO, this type of general-purpose computing architecture exposes the following inherent defects, becoming the main bottleneck restricting system performance and reliability: (1) Mismatch in computing architecture and preemption of computing power: As a general-purpose processor, the CPU is well-suited for handling complex control logic and serial tasks, but it is not suitable for executing highly parallel convolutional operations in deep learning models. In actual deployments, the CPU often needs to simultaneously handle multiple tasks such as video stream acquisition, image preprocessing, neural network inference, and upper-layer application logic (such as alarms and data uploads). This deep coupling between "control" and "computation" tasks leads to a large amount of CPU resources being occupied by high-load inference tasks, which in turn causes sluggish response of upper-layer business logic, a decrease in overall system throughput, and even stuttering, making it impossible to meet the real-time requirements of highly dynamic scenarios.

[0004] (2) Data flow latency and bandwidth bottleneck caused by frequent memory copying: In the traditional processing flow based on general-purpose operating systems such as Linux, video frame data usually needs to be copied multiple times from the camera driver buffer in kernel mode to the application memory space in user mode for CPU preprocessing and postprocessing. This frequent memory copying operation not only consumes a lot of CPU cycles, but also seriously occupies the limited external memory (DDR) bandwidth, introducing considerable end-to-end processing latency. For high frame rate video streams, this problem is particularly prominent and becomes the main obstacle to achieving deterministic low-latency processing.

[0005] (3) Risks of thermal failure and system stability caused by high power consumption: The energy efficiency ratio (TOPS / W) of the CPU for performing floating-point intensive neural network inference is low. Under long-term, high-load operation, the chip will generate a lot of heat, causing the core temperature to rise rapidly. When the temperature exceeds the hardware protection threshold (usually 75°C-85°C), the system will trigger a temperature-controlled frequency reduction mechanism, forcibly reducing the CPU operating frequency to reduce heat generation, which will directly lead to a precipitous drop in the processing frame rate. In extreme cases, the system may restart or crash due to overheating, making it impossible to guarantee 24 / 7 continuous and stable operation in unattended, closed industrial environments.

[0006] To address these challenges, existing technologies have proposed several improvement schemes. For example, some patents, such as CN116977826A, "A Reconfigurable Neural Network Object Detection System and Method under an Edge Computing Architecture," propose an edge object detection system that utilizes a reconfigurable neural network accelerator working in conjunction with a processor. Other patents, such as CN113240101A, "A Heterogeneous SoC Implementation Method for Hardware-Software Co-acceleration of Convolutional Neural Networks," focus on accelerating convolutional neural networks through hardware-software co-design. However, these solutions either fail to completely decouple control and computation tasks at the hardware level, or do not systematically solve the end-to-end zero-copy problem from data acquisition to inference output, or lack fine-grained scheduling strategies for the thermal power consumption of edge devices. Therefore, there is still considerable room for improvement in overall energy efficiency, real-time performance, and long-term thermal stability.

[0007] In summary, existing edge video analytics solutions based on general-purpose processors generally suffer from three major pain points when dealing with continuous, high-load real-time video stream processing tasks: conflicting computing resources, large data processing latency, and poor system thermal stability. Therefore, there is an urgent need for an edge computing video stream processing method and system that can fundamentally restructure the computing architecture, optimize data flow paths, and significantly improve energy efficiency. Summary of the Invention

[0008] To address the aforementioned technical problems in related technologies, this invention proposes an edge computing video stream processing method and system for a neural processing unit (NPU), which can overcome the above-mentioned shortcomings of the prior art.

[0009] To achieve the above-mentioned technical objectives, the technical solution of the present invention is implemented as follows: An edge computing video stream processing method based on NPU includes the following steps: S1 Video Stream Data Acquisition and Writing: Controls the image acquisition device to acquire video stream data, and writes the acquired data directly to a pre-created physical contiguous shared memory pool via Direct Memory Access (DMA). S2 Zero-copy hardware preprocessing: Call the dedicated hardware preprocessing unit integrated in the SoC to directly preprocess the video stream data stored in the physical contiguous shared memory pool to generate tensor data that meets the input requirements of the target neural network model, and write the processed data directly into the input memory area of ​​the NPU; S3 Model Loading and Asynchronous Inference: The target neural network model, after computation graph optimization, operator fusion, and low-precision quantization, is loaded onto the onboard NPU; the central processing unit (CPU) asynchronously sends inference instructions for the tensor data to the NPU; the NPU independently executes the inference computation task. S4 Result Acquisition and Post-processing: After the NPU completes the inference calculation, it notifies the CPU to acquire the structured inference result through an interrupt mechanism; the CPU then performs post-processing on the acquired inference result.

[0010] Furthermore, in step S2, the dedicated hardware preprocessing unit is a 2D graphics acceleration engine RGA, and the preprocessing includes at least color space conversion and image scaling; the physical contiguous shared memory pool is requested and managed through the Direct Render Manager (DRM) and DMA-BUF mechanism of the Linux operating system.

[0011] Furthermore, in step S3, the computation graph optimization and operator fusion of the target neural network model includes: identifying continuous computation nodes in the model, merging the convolutional layer, batch normalization layer and activation layer into a single composite operator, so that the intermediate layer feature data flows in the NPU internal cache and avoids writing to the external dynamic random access memory DDR.

[0012] Furthermore, in step S3, the low-precision quantization adopts INT8 full integer quantization based on KL divergence calibration, or adopts a hybrid quantization strategy; the hybrid quantization strategy is to keep the precision-sensitive network layers in the model in FP16 format, while quantizing the feature extraction layers in the model in INT8 format.

[0013] Furthermore, in step S3, the asynchronous issuance of inference instructions from the CPU to the NPU specifically means that after the CPU calls the NPU runtime interface to issue the instructions, it immediately returns and executes other control tasks or enters a suspended state, without blocking and waiting for the NPU inference to complete.

[0014] Furthermore, during system operation, a Dynamic Voltage and Frequency Adjustment (DVFS) step is also included: real-time monitoring of the load of the NPU and / or the temperature of the SoC, and dynamic adjustment of the operating voltage and frequency of the NPU based on the monitoring results.

[0015] An NPU-based edge computing video stream processing system, running on a system-on-a-chip (SoC) integrating an NPU and a dedicated hardware preprocessing unit, includes: The zero-copy data channel management module is used to create and manage a physically contiguous shared memory pool, control image acquisition data to be directly written to the shared memory pool via DMA, and control the dedicated hardware preprocessing unit to directly read and write data in the shared memory pool. The model optimization and loading module is used to perform computation graph optimization, operator fusion and low-precision quantization on the target neural network model for the NPU, and load the optimized model into the NPU; An asynchronous inference scheduling module is used to enable the CPU to asynchronously issue inference tasks to the NPU and receive the results returned by the NPU after it completes the calculation. The system tuning module is used to dynamically adjust the operating state of the NPU based on the load of the NPU and / or the temperature of the SoC.

[0016] Furthermore, the dedicated hardware preprocessing unit is the 2D graphics acceleration engine RGA; the zero-copy data channel management module uses the Linux kernel's Direct Render Manager (DRM) subsystem and the Contiguous Memory Allocator (CMA) mechanism to reserve and manage the physical contiguous shared memory pool.

[0017] Furthermore, the asynchronous inference scheduling module includes an asynchronous callback unit and a dynamic frequency adjustment unit; the asynchronous callback unit is used to implement non-blocking calls of the CPU and responses to NPU completion interrupts; the dynamic frequency adjustment unit is used to execute the decisions of the system tuning module and adjust the voltage and frequency of the NPU in real time.

[0018] The beneficial effects of this invention are as follows: By constructing a hardware-level decoupled architecture for "control-computation" and enabling a dedicated NPU for inference, this invention significantly reduces CPU load and power consumption, completely resolving the system throttling and crashing issues caused by CPU overheating. By establishing a full-link zero-copy data channel based on DMA and RGA hardware, video stream processing latency is significantly reduced, enabling stable real-time analysis at high frame rates. By employing operator fusion and INT8 hybrid quantization techniques, the computational and memory access efficiency of the model on the NPU is doubled, achieving higher energy efficiency and throughput while maintaining accuracy. Finally, the entire method enables edge devices to operate stably at low temperatures for extended periods in a fanless, enclosed environment, meeting industrial-grade reliability requirements. Detailed Implementation

[0019] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention are within the scope of protection of the present invention.

[0020] An edge computing video stream processing method based on an NPU according to an embodiment of the present invention includes the following steps: S1 Video Stream Data Acquisition and Writing: Controls the image acquisition device to acquire video stream data, and writes the acquired data directly to a pre-created physical contiguous shared memory pool via Direct Memory Access (DMA). S2 Zero-copy hardware preprocessing: Call the dedicated hardware preprocessing unit integrated in the SoC to directly preprocess the video stream data stored in the physical contiguous shared memory pool to generate tensor data that meets the input requirements of the target neural network model, and write the processed data directly into the input memory area of ​​the NPU; S3 Model Loading and Asynchronous Inference: The target neural network model, after computation graph optimization, operator fusion, and low-precision quantization, is loaded onto the onboard NPU; the central processing unit (CPU) asynchronously sends inference instructions for the tensor data to the NPU; the NPU independently executes the inference computation task. S4 Result Acquisition and Post-processing: After the NPU completes the inference calculation, it notifies the CPU to acquire the structured inference result through an interrupt mechanism; the CPU then performs post-processing on the acquired inference result.

[0021] Preferably, in step S2, the dedicated hardware preprocessing unit is a 2D graphics acceleration engine RGA, and the preprocessing includes at least color space conversion and image scaling; the physical contiguous shared memory pool is requested and managed through the Direct Render Manager (DRM) and DMA-BUF mechanism of the Linux operating system.

[0022] Preferably, in step S3, the computation graph optimization and operator fusion of the target neural network model includes: identifying continuous computation nodes in the model, merging the convolutional layer, batch normalization layer and activation layer into a single composite operator, so that the intermediate layer feature data flows in the NPU internal cache and avoids writing to the external dynamic random access memory DDR.

[0023] Preferably, in step S3, the low-precision quantization adopts INT8 full integer quantization based on KL divergence calibration, or adopts a hybrid quantization strategy; the hybrid quantization strategy is: to keep the precision-sensitive network layers in the model in FP16 format, while quantizing the feature extraction layers in the model in INT8 format.

[0024] Preferably, in step S3, the asynchronous issuance of inference instructions from the CPU to the NPU specifically means that after the CPU calls the NPU runtime interface to issue the instructions, it immediately returns and executes other control tasks or enters a suspended state, without blocking and waiting for the NPU inference to complete.

[0025] Preferably, during system operation, a Dynamic Voltage and Frequency Adjustment (DVFS) step is also included: real-time monitoring of the load of the NPU and / or the temperature of the SoC, and dynamic adjustment of the operating voltage and frequency of the NPU based on the monitoring results.

[0026] An NPU-based edge computing video stream processing system, running on a system-on-a-chip (SoC) integrating an NPU and a dedicated hardware preprocessing unit, includes: The zero-copy data channel management module is used to create and manage a physically contiguous shared memory pool, control image acquisition data to be directly written to the shared memory pool via DMA, and control the dedicated hardware preprocessing unit to directly read and write data in the shared memory pool. The model optimization and loading module is used to perform computation graph optimization, operator fusion and low-precision quantization on the target neural network model for the NPU, and load the optimized model into the NPU; An asynchronous inference scheduling module is used to enable the CPU to asynchronously issue inference tasks to the NPU and receive the results returned by the NPU after it completes the calculation. The system tuning module is used to dynamically adjust the operating state of the NPU based on the load of the NPU and / or the temperature of the SoC.

[0027] Preferably, the dedicated hardware preprocessing unit is a 2D graphics acceleration engine RGA; the zero-copy data channel management module uses the Linux kernel's Direct Render Manager (DRM) subsystem and the Contiguous Memory Allocator (CMA) mechanism to reserve and manage the physical contiguous shared memory pool.

[0028] Preferably, the asynchronous inference scheduling module includes an asynchronous callback unit and a dynamic frequency adjustment unit; the asynchronous callback unit is used to implement non-blocking calls of the CPU and responses to NPU completion interrupts; the dynamic frequency adjustment unit is used to execute the decisions of the system tuning module and adjust the voltage and frequency of the NPU in real time.

[0029] To facilitate understanding of the above technical solutions of the present invention, the following detailed description of the above technical solutions of the present invention will be provided through specific usage methods.

[0030] This invention proposes an edge computing video stream processing architecture based on the Rockchip SoC platform. This architecture, through a hardware-software co-design, restructures the video acquisition, preprocessing, and inference tasks originally handled by the CPU into a heterogeneous pipeline centered on the NPU. The specific implementation steps are as follows: Step 1: Build an edge computing inference engine dedicated to NPU. This step aims to solve the problem of low efficiency of general models running on edge devices by improving inference performance through underlying operator optimization and specific quantization strategies.

[0031] (1) Model reconstruction and operator fusion: Based on the characteristics of the underlying instruction set of Rockchip NPU, the original YOLOv8 model is optimized by using the RKNN-Toolkit toolchain.

[0032] The specific fusion method is to identify continuous computation nodes in the model structure and merge the convolutional layer (Conv), batch normalization layer (Batch Norm), and activation layer (SiLU / ReLU) into a single super operator (SuperLayer).

[0033] Eliminating read / write operations: Before fusion, the computation results of each layer need to be written to off-chip memory (DDR) and then read back for the next layer's computation. After fusion, intermediate layer data (such as feature maps after convolution) are directly stored in the NPU's high-speed temporary storage (SRAM / Cache) for the next level of computation, without needing to be written back to DDR. This "on-chip computation" mechanism eliminates repeated read / write operations of intermediate data in DDR memory, significantly reducing memory bandwidth usage and improving computation speed.

[0034] (2) Full Integer Quantization: A calibration method based on Kullback-Leibler Divergence is used to map the original 32-bit floating-point (FP32) model weights to 8-bit integers (INT8). Specific calibration method: 1) Prepare a calibration dataset containing 200-500 typical scene images; 2) Input the calibration images into the FP32 model for forward inference and calculate the probability distribution histogram of activation values ​​for each layer; 3) Calculate the minimum information loss between the FP32 distribution and the quantized INT8 distribution using the KL divergence algorithm, thereby determining the optimal quantization cutoff threshold (Scale Factor and Zero Point) for each layer. 4) Generate a quantization table based on the threshold, and convert the weights and activation values ​​to INT8 format.

[0035] (3) Technical effect: Through the above operator fusion and KL divergence quantization, the theoretical peak computing power (TOPS) of INT8 operation on Rockchip NPU is more than twice that of FP16. At the same time, due to the reduction of memory access operations and floating-point operations, the chip power consumption is reduced by about 50%, which is the core foundation for realizing the system's "40°C low temperature operation".

[0036] Step 2: Establish a "zero-copy" video streaming channel based on DRM / DMA This step aims to eliminate the CPU load and latency during data transfer, thus resolving the lag issue.

[0037] (1) Physical address pass-through: This invention abandons the mechanism of copying video frames from kernel space to user space in traditional Linux systems. It utilizes Linux's DRM (Direct Rendering Manager) subsystem and DMA-BUF mechanism to request a physically contiguous shared memory pool.

[0038] (2) Zero copy across the entire link: After the camera captures an image, the data is directly written to the shared memory physical address via DMA through the MIPI interface; the Rockchip RGA (Raster Graphic Acceleration, 2D graphics acceleration engine) hardware is called to directly perform resize and padding operations on the image at the physical address to generate tensor data that conforms to the model input; the NPU driver directly reads the data through the physical address pointer for inference.

[0039] (3) Technical effect: The CPU is involved in the whole process without any memcpy operation, and the CPU utilization rate drops sharply from 60-80% of the existing technology to 5-10%.

[0040] Step 3: Asynchronous Scheduling Strategy for Heterogeneous Computing Power This step achieves a complete separation of "control" and "computation".

[0041] (1) Asynchronous Inference Callback: After the CPU issues an inference instruction, it does not block and wait, but immediately suspends or processes other lightweight business (such as heartbeat packet sending). When the NPU completes inference, it wakes up the CPU through a hardware interrupt and returns structured results (such as coordinate box [x,y,w,h] and class confidence).

[0042] (2) Dynamic Voltage Frequency Adjustment (DVFS) Strategy: The system monitors the NPU load and SoC temperature in real time. When the screen is static or there is no target, the NPU clock frequency is automatically reduced to power saving mode; when a high-frequency dynamic target is detected, the NPU frequency is instantly increased.

[0043] (3) Technical effect: This "on-demand power supply" strategy ensures that the heat of the chip will not continue to accumulate and maintain thermal balance.

[0044] Example 1: Industrial Edge Computing Inspection System Based on Orange Pi 5 plus (RK3588) This embodiment demonstrates the specific construction and operation method of an industrial-grade pedestrian detection and anomaly alarm system based on a domestically produced high-performance SoC platform.

[0045] 1. Hardware platform and environment configuration Core computing unit: The Orange Pi 5 plus development board is used, equipped with a Rockchip RK3588 octa-core 64-bit processor (4×Cortex-A76 + 4×Cortex-A55). This SoC integrates a tri-core NPU (Neural Processing Unit) with a total computing power of 6.0 TOPS, supporting INT4 / INT8 / INT16 / FP16 multi-precision mixed operations.

[0046] Memory configuration: 8GB LPDDR4x memory, using a shared memory architecture (UMA), with 1GB of contiguous physical memory reserved as a shared buffer between the NPU and the camera through Linux's CMA (Contiguous Memory Allocator) mechanism.

[0047] Image acquisition device: Connect an IMX415 camera via the MIPI-CSI interface and input an NV12 format video stream with a resolution of 1920×1080@60fps.

[0048] 2. Edge-end migration and hybrid quantization of the model To balance detection accuracy and inference speed, this embodiment does not employ a crude global INT8 quantization, but instead implements a refined "hybrid quantization" strategy: Step A - Model Conversion: Set up an Ubuntu 20.04 development environment on a PC and use the rknn-toolkit2 toolchain to load the weights of the trained YOLOv8 (PyTorch version) model.

[0049] Step B - Sensitive Layer Analysis and Protection: Using the toolchain's "Quantization Precision Analysis" function, the detection head and the first convolutional layer in the model are identified as being extremely sensitive to precision. When configuring the quantization parameters, this layer is forced to be specified as FP16 (half-precision floating-point) format, while the feature extraction layers (Backbone & Neck) in the middle of the model are specified as INT8 (8-bit integer) format.

[0050] Step C - Calibration and Export: Use 200 real-world scene images as a calibration dataset to perform KL divergence calibration, and finally generate a .rknn model file specifically for the edge.

[0051] Results: Compared to the original FP32 model, the hybrid quantization model size is compressed to 1 / 4 (approximately 12MB), and the measured mAP (mean accuracy) only decreases by 0.5%, which is almost lossless.

[0052] 3. Heterogeneous pipeline operation process based on RGA and NPU After the system powers on, the following high-efficiency pipeline is executed through a C++ application (based on the RKNN Runtime SDK): Phase 1: Zero-copy video capture (0ms CPU consumption) The camera driver directly writes each frame of image data it captures to a pre-allocated DRM (Direct Rendering Manager) video memory buffer (physical address) via DMA (Direct Memory Access). At this time, the CPU only holds a handle (file descriptor) to the buffer and does not perform any data copying.

[0053] Phase Two: RGA Hardware Preprocessing (2ms / frame) Traditional methods require the CPU to perform YUV to RGB conversion and resize operations, which takes about 40ms.

[0054] This embodiment utilizes the RGA (2D raster graphics acceleration engine) hardware built into the RK3588 to directly perform color space conversion (to RGB888) and scaling (resize to 640×640) on the 1080P NV12 image in the aforementioned DRM video memory, and writes the result directly to the NPU's input physical address. The entire process takes only 2ms and does not consume any CPU resources.

[0055] Phase 3: Asynchronous NPU Inference (15ms / frame) The NPU inference is triggered by calling the rknn_run interface. The three cores inside the NPU are automatically scheduled according to the load, and convolution operations are processed in parallel. The single-frame inference time is stable at around 15ms (equivalent throughput > 60fps).

[0056] An asynchronous callback mechanism is adopted: the CPU suspends or processes the heartbeat packet immediately after triggering inference, and wakes up the CPU through a hardware interrupt after the NPU completes the calculation.

[0057] Phase Four: Minimalist Post-Processing The NPU returns the three feature layer tensors to the CPU. The CPU only needs to perform NMS (non-maximum suppression) and thresholding on these extremely small amounts of data (at the KB level) to generate the final bounding boxes.

[0058] 4. Measured Data and Effect Verification A 2-hour continuous stress test was conducted in a closed chassis environment with a 25°C room temperature, no fan, and only passive aluminum heatsinks:

[0059] Compared with traditional embedded vision solutions that rely on general-purpose processors (CPUs) for software inference and are widely used in existing technologies, this invention achieves the following significant advantages through the reconstruction of the underlying computing architecture and the optimization of hardware and software collaboration: (1) Significant improvement in energy efficiency and breakthrough in thermal stability. This invention completely solves the problem of heat buildup caused by full CPU load in embedded devices by enabling the dedicated neural processing unit (NPU) on the SoC to share the high-load convolution operation. As shown in the table below, under the same ambient temperature and operating load, this invention reduces the operating temperature of the core chip by about 46%. This breakthrough in temperature control means that the device does not need to rely on expensive active cooling modules (fans), and can maintain a low temperature steady state of about 40°C in an industrial sealed case with only passive heat sinks. This fundamentally eliminates the risk of system crashes caused by high temperature-triggered hardware forced frequency reduction (thermal throttling) and meets the industrial standard of 24 / 7 continuous and stable operation.

[0060] (2) Real-time processing capability with extremely low latency: By establishing a DMA-based zero-copy video stream channel, this invention eliminates redundant data transfer operations between kernel mode and user mode by the CPU. Compared with the memory bandwidth blocking caused by frequent memcpy operations of the CPU in the prior art, this invention achieves "significantly improved throughput" and "zero stuttering" when processing high frame rate video streams. Actual tests show that the system can still maintain a stable frame rate after running continuously for 2 hours, without the cumulative latency common in the prior art.

[0061] (3) Specific performance comparison data The detailed performance comparison between the present invention and the conventional technical solutions based on general-purpose CPU architecture is shown in the table below:

[0062] In summary, by employing the technical solutions described above in this invention, and through the construction of a hardware-level decoupled architecture for "control-computation" and the activation of a dedicated NPU for inference, CPU load and power consumption are significantly reduced, completely resolving the system throttling and crashing issues caused by CPU overheating. By establishing a full-link zero-copy data channel based on DMA and RGA hardware, video stream processing latency is significantly reduced, enabling stable real-time analysis at high frame rates. The use of operator fusion and INT8 hybrid quantization techniques doubles the computational and memory access efficiency of the model on the NPU, achieving higher energy efficiency and throughput while maintaining accuracy. Finally, the entire method enables edge devices to operate stably at low temperatures for extended periods in a fanless, enclosed environment, meeting industrial-grade reliability requirements.

[0063] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for edge computing video stream processing based on NPU, characterized in that, Includes the following steps: S1 Video Stream Data Acquisition and Writing: Controls the image acquisition device to acquire video stream data, and writes the acquired data directly to a pre-created physical contiguous shared memory pool via Direct Memory Access (DMA). S2 Zero-copy hardware preprocessing: Call the dedicated hardware preprocessing unit integrated in the SoC to directly preprocess the video stream data stored in the physical contiguous shared memory pool to generate tensor data that meets the input requirements of the target neural network model, and write the processed data directly into the input memory area of ​​the NPU; S3 Model Loading and Asynchronous Inference: The target neural network model, after computation graph optimization, operator fusion, and low-precision quantization, is loaded onto the onboard NPU; the central processing unit (CPU) asynchronously sends inference instructions for the tensor data to the NPU; the NPU independently executes the inference computation task. S4 Result Acquisition and Post-processing: After the NPU completes the inference calculation, it notifies the CPU to acquire the structured inference result through an interrupt mechanism; the CPU then performs post-processing on the acquired inference result.

2. The edge computing video stream processing method based on NPU according to claim 1, characterized in that, In step S2, the dedicated hardware preprocessing unit is a 2D graphics acceleration engine RGA, and the preprocessing includes at least color space conversion and image scaling; the physical contiguous shared memory pool is requested and managed through the Direct Render Manager (DRM) and DMA-BUF mechanism of the Linux operating system.

3. The edge computing video stream processing method based on NPU according to claim 1, characterized in that, In step S3, the computation graph optimization and operator fusion of the target neural network model includes: identifying continuous computation nodes in the model, merging the convolutional layer, batch normalization layer and activation layer into a single composite operator, so that the intermediate layer feature data flows in the NPU internal cache and avoids writing to the external dynamic random access memory DDR.

4. The NPU-based edge computing video stream processing method according to claim 1 or 3, characterized in that, In step S3, the low-precision quantization adopts INT8 full integer quantization based on KL divergence calibration, or a hybrid quantization strategy. The hybrid quantization strategy is to keep the precision-sensitive network layers in the model in FP16 format, while quantizing the feature extraction layers in the model in INT8 format.

5. The edge computing video stream processing method based on NPU according to claim 1, characterized in that, In step S3, the asynchronous issuance of inference instructions from the CPU to the NPU specifically means that after the CPU calls the NPU runtime interface to issue the instructions, it immediately returns and executes other control tasks or enters a suspended state, without blocking and waiting for the NPU inference to complete.

6. The NPU-based edge computing video stream processing method according to claim 1 or 5, characterized in that, During system operation, a Dynamic Voltage and Frequency Adjustment (DVFS) step is also included: real-time monitoring of the load of the NPU and / or the temperature of the SoC, and dynamic adjustment of the operating voltage and frequency of the NPU based on the monitoring results.

7. An edge computing video stream processing system based on an NPU, running on a system-on-a-chip (SoC) integrating an NPU and a dedicated hardware preprocessing unit, characterized in that, include: The zero-copy data channel management module is used to create and manage a physically contiguous shared memory pool, control image acquisition data to be directly written to the shared memory pool via DMA, and control the dedicated hardware preprocessing unit to directly read and write data in the shared memory pool. The model optimization and loading module is used to perform computation graph optimization, operator fusion and low-precision quantization on the target neural network model for the NPU, and load the optimized model into the NPU; An asynchronous inference scheduling module is used to enable the CPU to asynchronously issue inference tasks to the NPU and receive the results returned by the NPU after it completes the calculation. The system tuning module is used to dynamically adjust the operating state of the NPU based on the load of the NPU and / or the temperature of the SoC.

8. The system according to claim 7, characterized in that, The dedicated hardware preprocessing unit is the 2D graphics acceleration engine RGA; the zero-copy data channel management module uses the Linux kernel's Direct Render Manager (DRM) subsystem and the Contiguous Memory Allocator (CMA) mechanism to reserve and manage the physical contiguous shared memory pool.

9. The system according to claim 7, characterized in that, The asynchronous inference scheduling module includes an asynchronous callback unit and a dynamic frequency adjustment unit; the asynchronous callback unit is used to implement non-blocking calls of the CPU and responses to NPU completion interrupts; the dynamic frequency adjustment unit is used to execute the decisions of the system tuning module and adjust the voltage and frequency of the NPU in real time.

Images