Video content intelligent analysis method based on edge computing platform

Abstract

The invention provides a video content intelligent analysis method based on an edge computing platform, and the method comprises the steps: collecting a video stream through an image collection interface of the edge computing platform, and carrying out the preprocessing of a video frame at a CPU end, and obtaining standardized image data; the method comprises the following steps: converting a pre-trained neural network model into a special format model adaptive to an NPU (Network Processing Unit), and adopting a hierarchical quantization strategy in the conversion process: quantizing at least one network layer responsible for basic feature extraction in the model by adopting first numerical value precision, quantizing other network layers responsible for classification and regression in the model by adopting second numerical value precision lower than the first numerical value precision; dynamically scheduling the number of processing cores in an activated state in the NPU based on a processing load determined by analyzing the current video frame content represented by the standardized image data, and utilizing the processing cores in the activated state to load a special format model to infer the standardized image data; and post-processing the reasoning output result of the NPU to generate a structured analysis result.

Description

Technical Field

[0001] This invention belongs to the fields of computer vision, embedded edge computing and intelligent analysis technology, and specifically relates to a video content intelligent analysis method based on an edge computing platform. Background Technology

[0002] With the deep integration of artificial intelligence and Internet of Things (IoT) technologies, intelligent video content analysis is increasingly being applied in fields such as security monitoring, industrial quality inspection, and smart transportation, becoming a key enabling technology for achieving digital and intelligent transformation of industries. However, when moving complex intelligent analysis tasks from the cloud to resource-constrained edge devices, a series of intertwined and coupled technical challenges are generally faced, and existing solutions often struggle to achieve a good balance between analysis accuracy, real-time response, energy efficiency, and cost.

[0003] Firstly, regarding model deployment and computational accuracy, edge devices typically employ dedicated neural network processing units to accelerate inference. This necessitates converting pre-trained floating-point models to a device-supported specific format and performing quantization compression. However, while general post-training quantization or uniform precision quantization strategies significantly reduce model size and improve inference speed, they are highly susceptible to introducing substantial accuracy losses in complex scenarios. Particularly for tasks emphasizing fine-grained feature extraction (such as industrial micro-defect detection), the basic feature maps extracted by the shallow layers of the model's backbone network are extremely sensitive to quantization errors. Uniform low-precision quantization severely impairs their feature representation capabilities, leading to a significant drop in final analysis accuracy and failing to meet the reliability requirements of practical applications. Therefore, effectively suppressing the accuracy loss of models deployed on edge devices without significantly increasing computational overhead is a pressing issue.

[0004] Secondly, regarding computing resource scheduling and energy efficiency, while edge SoC chips with multi-core NPUs offer considerable peak computing power, traditional static or simple polling scheduling strategies are ill-suited to the dynamically changing computing load in video stream analysis. For example, the number, size, and density of targets in a video scene fluctuate in real time. Using a fixed NPU core activation strategy (such as full-time full-core operation) leads to idle computing power and wasted energy during low-load periods, while insufficient computing power during high-load periods can cause processing queue backlogs and latency spikes. This mismatch between computing resource supply and real-time analysis requirements results in low overall system computing power utilization, making it impossible to achieve optimal energy efficiency while meeting real-time constraints (such as high frame rate video stream processing).

[0005] Furthermore, regarding system-level collaboration and task pipeline optimization, edge computing platforms integrate multiple processing units such as CPUs and NPUs. Existing solutions often employ a serial execution mode, where the CPU completes all preprocessing before handing the data over to the NPU for inference, and finally returns it to the CPU for post-processing. This mode fails to fully utilize the heterogeneous characteristics of the CPU's big.LITTLE cores and the parallel processing capabilities between the CPU and NPU, resulting in significant idle waiting time for each processing unit, limiting the overall system throughput, and making it difficult to further reduce end-to-end processing latency. Designing an efficient hardware and software collaboration mechanism to build a refined task pipeline is key to unlocking the overall performance potential of the edge platform.

[0006] Furthermore, when faced with diverse application scenarios, existing edge analytics devices often rely on fixed analysis models and processing strategies, lacking the ability to adapt and adjust at runtime. Different scenarios (such as perimeter security and production line inspection) have significantly different requirements for model types and computing resources. Fixed systems cannot flexibly switch analysis models or adjust resource allocation strategies based on real-time scenario content or task instructions, resulting in narrow applicability and high deployment and maintenance costs.

[0007] Therefore, the industry urgently needs a new method for intelligent video content analysis for edge computing platforms. This method can systematically solve the core problems mentioned above, such as model deployment accuracy loss, dynamic computing load adaptation, heterogeneous computing resource collaboration, and multi-scenario adaptation, without relying on the powerful computing power of the cloud. This will enable high-precision, low-latency, and high-efficiency intelligent video analysis at resource-constrained edge devices. Summary of the Invention

[0008] To address the shortcomings and deficiencies of existing technologies, this invention provides a video content intelligent analysis method and system based on an edge computing platform, aiming to achieve high-precision, low-latency, and high-energy-efficiency edge video analysis. This method constructs a pipelined parallel architecture of preprocessing-inference-postprocessing, allocating preprocessing tasks to low-power CPU cores, inference tasks to the NPU, and postprocessing tasks to high-performance CPU cores. Cooperative scheduling between tasks is achieved through standardized communication interfaces and an event-driven framework, significantly improving overall processing throughput. Regarding model deployment, a hierarchical quantization strategy based on network layer functions is adopted. Higher numerical precision quantization is used for network layers responsible for basic feature extraction, while lower numerical precision quantization is used for network layers responsible for classification and regression, improving inference efficiency while ensuring model accuracy. Simultaneously, the system can dynamically schedule the number of active processing cores in the NPU based on the processing load determined by the current video frame content, achieving adaptive allocation of computing resources. Furthermore, the system supports local storage and network uploading of structured analysis results, with network uploading achieved through a high-speed Ethernet interface and upload latency controlled to not exceed a preset threshold. This invention effectively solves the problems of accuracy loss, excessive latency, and insufficient computing power utilization faced by edge devices when processing high frame rate video streams under resource-constrained conditions.

[0009] The specific technical solution adopted by this invention to solve its technical problem is as follows:

[0010] A video content intelligent analysis method based on an edge computing platform, wherein the edge computing platform includes a central processing unit (CPU) and a neural network processing unit (NPU) with multiple processing cores, the method comprising:

[0011] The video stream is acquired through the image acquisition interface of the edge computing platform, and the video frames are preprocessed on the CPU to obtain standardized image data.

[0012] The pre-trained neural network model is converted into a dedicated format model adapted to the NPU. During the conversion process, a hierarchical quantization strategy based on network layer function is adopted: a first numerical precision quantization is adopted for at least one network layer in the model that is responsible for basic feature extraction, and a second numerical precision quantization lower than the first numerical precision is adopted for the remaining network layers in the model that are responsible for classification and regression.

[0013] Based on the processing load determined by analyzing the current video frame content represented by the standardized image data, the number of active processing cores in the NPU is dynamically scheduled, and the dedicated format model is loaded using the active processing cores to perform inference on the standardized image data.

[0014] The inference output of the NPU is post-processed to generate structured analysis results.

[0015] Furthermore, the first numerical precision is floating-point precision, and the second numerical precision is integer precision; the network layer responsible for basic feature extraction is the initial layer responsible for core feature extraction in the backbone network of the model, and the network layer responsible for classification and regression is the classification head layer and regression head layer of the model.

[0016] Furthermore, the processing load is determined by the number of targets to be detected in the current video frame; the rule for dynamically scheduling the number of active processing cores in the NPU is as follows: when the number of targets to be detected is less than a first threshold, the NPU is activated to run on a single core; when it is between the first threshold and a second threshold, the NPU is activated to run on a dual-core; when it is greater than the second threshold, the NPU is activated to run on a triple-core.

[0017] Furthermore, the preprocessing on the CPU side includes frame extraction, noise reduction, size normalization, color gamut conversion, and numerical normalization. Among them, noise reduction adopts Gaussian filtering or other noise reduction algorithms that adapt to edge computing; size normalization is to scale the video frame to a preset size adapted to the model through interpolation; color gamut conversion is to convert the color gamut format of the original video frame to a color gamut format compatible with the model; and numerical normalization is to map the pixel value to the numerical range adapted to the model inference. Moreover, the preprocessing tasks are assigned to the low-power core of the CPU for execution, while the high-performance core of the CPU is reserved for core task processing.

[0018] Furthermore, the model conversion is achieved through a model conversion tool adapted to the NPU, converting pre-trained neural network models in ONNX format or other general formats into NPU-compatible dedicated format models; during the conversion process, an adapted sample set covering different application scenarios is imported for calibration to control the loss of quantization accuracy to not exceed a preset threshold.

[0019] It supports dynamic switching between at least two dedicated format models with different functions, and shortens the model switching time by caching at least one loaded model in the memory of the edge computing platform; the neural network model includes at least two of the following: target detection model, industrial defect detection model and behavior recognition model.

[0020] Furthermore, the post-processing includes non-maximum suppression and inverse coordinate transformation; wherein, non-maximum suppression uses a preset intersection-union ratio threshold to filter overlapping detection boxes, and inverse coordinate transformation is based on the scaling ratio of standardized image data and original video frames to restore the normalized coordinates to the pixel scale of the original video frames; the structured analysis results include timestamp, target type, target location and confidence information.

[0021] Furthermore, it also includes an exception handling mechanism: when video acquisition is interrupted, a reconnection mechanism is triggered at multiple intervals of preset time. If the reconnection fails, an alarm is output; when NPU inference times out or fails, the current task is first added to the retry queue, with a maximum of preset number of retries. If the retry fails, it is automatically downgraded to CPU-based lightweight model inference; when the available space of the edge computing platform's storage medium is lower than a preset threshold, the structured analysis results before the preset time are automatically deleted to avoid storage overflow.

[0022] Furthermore, the preprocessing, inference, and postprocessing tasks are executed in a pipelined parallel manner. The preprocessing tasks are assigned to the low-power cores of the CPU, the inference tasks are executed by the NPU, and the postprocessing tasks are assigned to the high-performance cores of the CPU. Cooperative scheduling between tasks is achieved through a standardized communication interface and an event-driven framework to improve the overall processing throughput.

[0023] Furthermore, the structured analysis results support two output methods: local storage and network upload. When stored locally, files are divided according to a preset period and stored in a dedicated storage area of ​​the storage medium. When uploaded via network, the upload is achieved through a high-speed Ethernet interface based on a preset communication protocol, and the upload latency is controlled to not exceed a preset threshold.

[0024] Furthermore, a video content intelligent analysis system based on an edge computing platform, wherein the edge computing platform includes a central processing unit (CPU), a neural network processing unit (NPU) with multiple processing cores, an image acquisition interface, memory, and storage media, and the system includes:

[0025] Video acquisition and preprocessing module: Acquires video streams through the image acquisition interface, performs preprocessing on video frames on the low-power core of the CPU, and outputs standardized image data;

[0026] Model conversion module: Converts the pre-trained neural network model into a dedicated format model adapted to the NPU. During the conversion process, a hierarchical quantization strategy based on network layer functions is adopted: the first numerical precision quantization is adopted for at least one network layer in the model that is responsible for basic feature extraction, and the second numerical precision quantization, which is lower than the first numerical precision, is adopted for the remaining network layers in the model that are responsible for classification and regression.

[0027] Dynamic scheduling and inference module: Based on the processing load determined by analyzing the current video frame content represented by the standardized image data, dynamically schedule the number of active processing cores in the NPU, load the dedicated format model using the active processing cores, perform inference on the standardized image data, and output the inference results;

[0028] Post-processing module: Performs post-processing on the inference results on the high-performance core of the CPU, generates structured analysis results, and outputs them;

[0029] The video acquisition and preprocessing module, dynamic scheduling and inference module, and post-processing module constitute a pipeline, which achieves collaborative scheduling through a standardized communication interface and an event-driven framework.

[0030] And a computer device including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the method described above.

[0031] A non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described above.

[0032] Compared to existing technologies, this invention and its preferred solution effectively balance model accuracy and deployment efficiency. Through a targeted hierarchical quantization strategy, it significantly compresses model size and reduces storage footprint while minimizing accuracy loss caused by quantization, solving the problem of balancing accuracy and resource consumption in existing edge model deployments. It significantly improves the real-time performance and computing efficiency of video analytics by dynamically scheduling NPU core computing power based on video frame content and constructing a CPU-NPU pipelined parallel processing architecture, avoiding wasted computing power and significantly reducing end-to-end processing latency, meeting the high real-time requirements of edge scenarios. It also possesses good scenario adaptability and flexible scalability, supporting the rapid deployment of various functional models. Rapid dynamic switching allows for adjustments to processing strategies based on different application scenarios, adapting to diverse needs in fields such as intelligent security, industrial monitoring, and smart transportation. Enhanced system stability and reliability, a robust anomaly handling mechanism enables timely responses and retry / degradation measures to address anomalies in video acquisition, model loading, inference execution, and storage usage, ensuring the continuity of the analysis process. Deep synergistic optimization of hardware characteristics with the operating system and algorithm models fully leverages the hardware potential of the edge computing platform, improving the overall efficiency of data reading / writing, model loading, and task scheduling, while reducing system power consumption, providing strong support for the large-scale application of intelligent video analysis at the edge. Attached Figure Description

[0033] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0034] Figure 1 This is an overall architecture interaction diagram of an embodiment of the present invention;

[0035] Figure 2 This is a block diagram of the server module structure according to an embodiment of the present invention;

[0036] Figure 3 This is a schematic diagram of the entire process of intelligent video analysis according to an embodiment of the present invention;

[0037] Figure 4 This is a diagram of the uloop event scheduling logic in an embodiment of the present invention. Detailed Implementation

[0038] To make the features and advantages of the present invention more apparent and understandable, specific embodiments are described below in detail:

[0039] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used in this specification have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0040] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0041] To address the problems of high bandwidth / latency in existing video intelligent analysis solutions, low accuracy / insufficient frame rate in traditional embedded systems, and wasted computing power / serious accuracy loss in edge computing platforms such as RK3588, this invention provides a video content intelligent analysis method based on RK3588, which fully adapts to hardware characteristics and balances real-time performance and accuracy.

[0042] It should be noted that the RK3588 is the preferred hardware carrier for implementing the solution in this embodiment, but it is not limited to this. Those skilled in the art can replace it with an edge gateway of the same type or with corresponding functions according to actual needs.

[0043] This solution acquires 1080P@60fps video streams via the RK3588's MIPI CSI interface. After lightweight preprocessing on the CPU, it uses RKNN-Toolkit2 to convert intelligent analysis models such as YOLOv8 into an NPU-compatible RKNN format (emphasizing hierarchical quantization to control accuracy loss). Based on the dbus communication interface and the uloop event-driven framework, it achieves CPU-NPU collaborative scheduling, utilizing the NPU's three cores for real-time inference, and generating structured analysis results through post-processing. It achieves end-to-end analysis latency ≤60ms / frame, target detection accuracy ≥95%, and NPU computing power utilization ≥90%, addressing the pain points of poor real-time performance, insufficient accuracy, and wasted computing power in existing edge video analysis solutions. It is suitable for edge scenarios such as intelligent security, industrial monitoring, and smart transportation.

[0044] The implementation of this invention embodiment is based on an RK3588 hardware platform equipped with 8GB LPDDR4X memory and 64GB eMMC storage, and an Ubuntu 20.04 operating system, and includes the following steps:

[0045] (1) Video data acquisition: The IMX477 camera is connected through the MIPI CSI 2.0 interface of RK3588 to acquire a video stream with a resolution of 1920×1080 and a frame rate of 60fps. The H.265 Main Profile format is used for decoding, and the decoding bit rate is controlled at 8Mbps.

[0046] (2) Video preprocessing: Frame extraction, 5×5 Gaussian filtering for noise reduction, bilinear interpolation for size normalization (to 640×640 pixels), and YUV420 to RGB color gamut conversion are performed on the A76 big core of the CPU. The preprocessing time for a single frame is ≤10ms.

[0047] (3) Model adaptation and conversion: The pre-trained YOLOv8n model was converted to RK3588 compatible RKNN format using the RKNN-Toolkit2 tool, and INT8 hybrid quantization was enabled (the first two layers of the backbone retained FP16 accuracy, and the remaining layers were quantized with INT8). 1200 scene adaptation samples were imported for calibration, and the quantization accuracy loss was ≤3%.

[0048] (4) NPU computing power scheduling: Based on the dbus interface, a "inference request / status feedback" communication protocol is defined. Tasks are allocated through the uloop event-driven framework. The NPU dynamically switches between single-core / dual-core / triple-core running modes according to the number of targets in the video frame (triple-core is enabled when the number of targets is >10).

[0049] (5) Intelligent analysis and inference: The model is loaded by calling the rknn_inference interface of RKNN-Lite2. The input is a normalized RGB image, and the output is the target category (supports 20 common objects), normalized coordinates, and confidence score. The inference time is ≤40ms / frame.

[0050] (6) Post-processing and output of results: The CPU performs non-maximum suppression with an IOU threshold of 0.45 and inverse coordinate transformation (restored to the original video frame scale) to generate a JSON format report containing "timestamp-target type-location-confidence". It supports local 64GB eMMC storage or 1000Mbps Ethernet real-time upload.

[0051] As a preferred option, the hybrid quantization strategy in step (3) is implemented through the “layer_group” parameter of RKNN-Toolkit2. The Conv2d layer responsible for feature extraction in the model retains FP16 precision, and the classification / regression head layer is quantized using INT8. The quantized model is stored in the / data / models directory of the 64GB eMMC of RK3588, and the loading time is ≤500ms.

[0052] As a preferred option, the interface name of the dbus communication protocol in step (4) is “com.rockchip.npu.video.infer”, which includes two methods: “SendFrame” (transmit preprocessed image) and “GetResult” (get inference result). It is implemented based on dbus-daemon in Ubuntu 20.04 and the communication delay is ≤5ms.

[0053] As a preferred option, the intelligent analysis model in step (5) supports dynamic switching. Different RKNN models (such as YOLOv8n for object detection and PP-Human for behavior recognition) can be loaded through uloop timer events. During the switching process, 8GB of memory is used to implement model caching, and the switching time is ≤200ms.

[0054] As a preferred option, it also includes an exception handling module that triggers a three-reconnection mechanism when video acquisition is interrupted, automatically downgrades to CPU lightweight model inference when NPU inference fails, ensuring the continuity of analysis; exception logs are written to the / var / log / video_analysis / directory of the 64GB eMMC in real time for easy troubleshooting.

[0055] like Figure 1 As shown, the interaction method of the overall architecture in this embodiment is as follows: the video acquisition device establishes a data connection with the RK3588 platform through a dedicated interface. The preprocessing module, model conversion module, and NPU scheduling module built into the RK3588 platform complete the data processing in sequence. The process data and the final result are synchronously stored in 64GB eMMC local storage, and simultaneously pushed to the output terminal through the data interface, realizing the full-link interaction of "acquisition-processing-storage-output".

[0056] like Figure 2 As shown, the server-side module structure and data flow method are as follows: data is first input to the preprocessing module to complete noise reduction and frame extraction, then passed to the model conversion module for format adaptation, and then the NPU scheduling module allocates computing power and pushes it to the inference computing module to perform AI analysis; each module achieves seamless communication based on the Ubuntu 20.04 system environment to ensure the continuity and stability of data flow.

[0057] As shown in Figure 3, the entire video intelligent analysis process is as follows: After the video acquisition device obtains the raw video stream, it first temporarily stores keyframes in an 8GB memory cache before entering the preprocessing stage; after model conversion and NPU inference, normal results are directly output; if any abnormal data is detected in any stage, the corresponding branch processing mechanism (including retry, degradation, and alarm) is triggered:

[0058] Video capture process error: After processing, return to the video capture process to reacquire the video stream;

[0059] Anomalies in the preprocessing and model transformation stages: The starting node of the corresponding abnormal stage in the regression is re-executed after processing;

[0060] NPU inference stage anomaly: After processing, the regression inference stage reloads the model and executes inference;

[0061] Storage failure: After processing, continue outputting results (if local storage fails, prioritize switching to network upload);

[0062] Once an anomaly is resolved, it is reintroduced into the main process. If the anomaly remains unresolved, the processing of the current frame is terminated and logged to ensure the integrity and fault tolerance of the analysis process.

[0063] In a more specific embodiment, the implementation process includes the following steps:

[0064] Step 1: Video Data Acquisition

[0065] Hardware configuration: It adopts two MIPI CSI 2.0 interfaces of RK3588 to connect to the IMX477 8-megapixel camera (supports global shutter), and the camera output format is RAW12; the hardware platform is equipped with 8GB LPDDR4X memory (2GB of memory is reserved for video frame buffer) and 64GB eMMC storage (10GB of dedicated partition is allocated for temporary storage of raw video frames).

[0066] System compatibility: Compile and install the V4L2 driver (version 1.20.0) on Ubuntu 20.04, and enable memory mapping (mmap) mode to improve frame reading efficiency.

[0067] Decoding Processing: The video stream is decoded using the built-in H.265 hardware decoder of the RK3588. The "Constant Bitrate (CBR)" mode is configured, and the bitrate is locked at 8Mbps. After decoding, the original video frames in YUV420 format are output, and the decoding time per frame is ≤5ms.

[0068] Core data collection code snippet (C language):

[0069] #include<stdio.h>

[0070] #include<stdlib.h>

[0071] #include<linux / videodev2.h>

[0072] #include<fcntl.h>

[0073] #include<unistd.h>

[0074] #include<sys / mman.h>

[0075] #define DEVICE " / dev / video0

[0076] #define WIDTH 1920

[0077] #define HEIGHT 1080

[0078] #define FPS 60

[0079] int main() {

[0080] int fd = open(DEVICE, O_RDWR);

[0081] struct v4l2_format fmt = {0};

[0082] fmt.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;

[0083] fmt.fmt.pix.width = WIDTH;

[0084] fmt.fmt.pix.height = HEIGHT;

[0085] fmt.fmt.pix.pixelformat = V4L2_PIX_FMT_YUV420;

[0086] fmt.fmt.pix.field = V4L2_FIELD_NONE;

[0087] ioctl(fd, VIDIOC_S_FMT, &fmt); / / Configure the acquisition format

[0088] / / Allocate a frame buffer (use 8GB of memory to buffer 3 frames)

[0089] struct v4l2_requestbuffers req = {0};

[0090] req.count = 3;

[0091] req.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;

[0092] req.memory = V4L2_MEMORY_MMAP;

[0093] ioctl(fd, VIDIOC_REQBUFS, &req);

[0094] / / Start video stream capture

[0095] enum v4l2_buf_type type = V4L2_BUF_TYPE_VIDEO_CAPTURE;

[0096] ioctl(fd, VIDIOC_STREAMON, &type);

[0097] / / Loop through and process frames

[0098] while (1) {

[0099] struct v4l2_buffer buf = {0};

[0100] buf.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;

[0101] buf.memory = V4L2_MEMORY_MMAP;

[0102] ioctl(fd, VIDIOC_DQBUF, &buf); / / Retrieve frame data

[0103] process_frame((unsigned char*)buf.m.userptr, buf.length); / / Pass in the preprocessing module

[0104] ioctl(fd, VIDIOC_QBUF, &buf); / / Frame buffer reclamation

[0105] }

[0106] close(fd);

[0107] return 0;

[0108] }

[0109] Step 2: Video Preprocessing

[0110] The preprocessed data is sent to the NPU scheduling module via the dbus interface (see step 4), including:

[0111] Frame extraction: Based on the buffer queue mechanism of V4L2 driver, keyframes are extracted from the decoded video stream (supports "frame-by-frame extraction" or "extraction at intervals of 1-5 frames", which can be dynamically adjusted through the configuration file / etc / video_analysis / config.ini in Ubuntu 20.04).

[0112] Noise reduction: A 5×5 Gaussian filter algorithm (σ=0.8) is used to reduce noise in video frames and eliminate noise interference in low-light scenes.

[0113] Normalization and color gamut conversion: The frame size is scaled to 640×640 pixels using the cv2.resize function (bilinear interpolation) in OpenCV 4.5.5 (compiled and adapted for Ubuntu 20.04), and then the YUV420 format is converted to RGB format using the cv2.cvtColor function, and a normalization operation of [0,255]→[0,1] is performed.

[0114] Core preprocessing code snippet (Python):

[0115] import cv2

[0116] import numpy as np

[0117] def preprocess_frame(yuv_data, width=1920, height=1080):

[0118] # YUV420 to RGB color gamut

[0119] yuv_frame = np.frombuffer(yuv_data, dtype=np.uint8).reshape((height*3 / / 2, width))

[0120] rgb_frame = cv2.cvtColor(yuv_frame, cv2.COLOR_YUV2RGB_I420)

[0121] # 5×5 Gaussian filter noise reduction (σ=0.8)

[0122] denoised_frame = cv2.GaussianBlur(rgb_frame, (5, 5), 0.8)

[0123] # Bilinear interpolation normalized to 640×640 pixels

[0124] resized_frame = cv2.resize(denoised_frame, (640, 640),interpolation=cv2.INTER_LINEAR)

[0125] # Normalize values ​​to [0,1]

[0126] normalized_frame = resized_frame / 255.0

[0127] return normalized_frame.astype(np.float32)

[0128] Performance optimization: Utilizing the multi-threaded scheduling mechanism of Ubuntu 20.04, preprocessing tasks are assigned to the A55 small cores of the CPU for execution, freeing up the resources of the A76 large cores, and the total time for preprocessing a single frame is ≤10ms.

[0129] Step 3: Model Adaptation and Conversion

[0130] Model selection: YOLOv8n is selected as the base model (balancing accuracy and speed), and can be replaced with PP-YOLOE-lite (industrial defect detection), PP-Human (behavior recognition) and other models for different scenarios.

[0131] Environment configuration: Install RKNN-Toolkit2 (version 1.4.0, compatible with RK3588) on Ubuntu 20.04 system. The dependent libraries include Python 3.8, numpy 1.21.6, and onnx 1.12.0.

[0132] RKNN format conversion: Convert the ONNX format model to RKNN format using the rknn.build() interface of RKNN-Toolkit2, and set "mean_values=[[0,0,0]]" and "scale_values=[[255,255,255]]" to match the range of the preprocessed data.

[0133] Hybrid quantization optimization: The model is divided into two groups using the “layer_group” parameter. The first group (Conv2d_0 / Conv2d_1 layers of the backbone) uses FP16 quantization, and the second group (the remaining layers) uses INT8 quantization. 1200 samples covering “indoor / outdoor / low light” scenes are imported for calibration.

[0134] Core conversion code snippet (Python):

[0135] import RKNN from rknn.api

[0136] def convert_yolov8_to_rknn(onnx_path, rknn_path):

[0137] rknn = RKNN()

[0138] # Initialize the RK3588 runtime environment

[0139] rknn.init_runtime(target_platform='rk3588', device_id='12345678')

[0140] # Load pre-trained models in ONNX format

[0141] if rknn.load_onnx(model=onnx_path) != 0:

[0142] raise Exception("Model loading failed")

[0143] # Configure a layered quantization strategy (core innovation): The first two layers of the backbone are FP16, and the rest are INT8.

[0144] calib_dataset = '. / calib_dataset.txt' # 1200 scene adaptation calibration samples

[0145] rknn.build(

[0146] model=onnx_path,

[0147] dataset=calib_dataset,

[0148] mean_values=[[0, 0, 0]],

[0149] scale_values=[[255, 255, 255]],

[0150] quantized_dtype='mixed', # Mixed quantization mode

[0151] layer_group=[[0, 1], [2, -1]] # Layered quantization configuration )

[0153] # Export RKNN model to 64GB eMMC storage

[0154] rknn.export_rknn(rknn_path) # Storage path: / data / models / yolov8n.rknn

[0155] rknn.release()

[0156] if __name__ == "__main__":

[0157] convert_yolov8_to_rknn('. / yolov8n.onnx', ' / data / models / yolov8n.rknn')

[0158] Storage optimization: After quantization, the model size is compressed to 1 / 4 of the original FP32 model (approximately 8MB) and stored in a dedicated 64GB eMMC partition. The ext4 file system of Ubuntu 20.04 is used to improve the model loading speed, with an accuracy loss of ≤3%.

[0159] Step 4: NPU computing power scheduling

[0160] dbus service configuration: On the Ubuntu 20.04 system, configure dbus permissions and register the "com.rockchip.npu.video.infer" service through / etc / dbus-1 / system.d / com.rockchip.npu.video.infer.conf.

[0161] Core service registration code snippet (Python):

[0162] import dbus

[0163] import dbus.service

[0164] import numpy as np

[0165] from dbus.mainloop.glib import DBusGMainLoop

[0166] from rknnlite.api import RKNNLite

[0167] class NPUInferService(dbus.service.Object):

[0168] def __init__(self):

[0169] # Register the dbus system service (interface name: com.rockchip.npu.video.infer)

[0170] bus_name = dbus.service.BusName('com.rockchip.npu.video.infer', bus=dbus.SystemBus())

[0171] dbus.service.Object.__init__(self, bus_name, ' / com / rockchip / npu / video / infer')

[0172] self.model_cache = {} # 8GB memory cache model, switching time ≤200ms

[0173] self.load_model(0, ' / data / models / yolov8n.rknn') # Load the default detection model

[0174] # Receive preprocessed frame data and trigger NPU inference.

[0175] @dbus.service.method('com.rockchip.npu.video.infer', in_signature='iy', out_signature='s')

[0176] def SendFrame(self, model_id, frame_data):

[0177] frame = np.frombuffer(frame_data, dtype=np.float32).reshape((640, 640, 3))

[0178] return self.infer_frame(model_id, frame)

[0179] # Obtain the reasoning results

[0180] @dbus.service.method('com.rockchip.npu.video.infer', in_signature='s', out_signature='(ay,i)')

[0181] def GetResult(self, request_id):

[0182] return (b'result_data', 0)

[0183] # Load the RKNN model to support three-core NPU operation

[0184] def load_model(self, model_id, model_path):

[0185] rknn_lite = RKNNLite()

[0186] rknn_lite.load_rknn(model_path)

[0187] rknn_lite.init_runtime(runtime_core=3) # Enable triple-core mode (6 TOPS computing power)

[0188] self.model_cache[model_id] = rknn_lite

[0189] if __name__ == "__main__":

[0190] DBusGMainLoop(set_as_default=True)

[0191] service = NPUInferService()

[0192] import gobject

[0193] gobject.MainLoop().run() # Start the uloop event loop

[0194] uloop event scheduling: Based on the libuloop library in Ubuntu 20.04 (compiled to adapt for arm64 architecture), a "preprocessing-inference-postprocessing" pipeline is built. It listens for dbus requests via uloop file descriptor events and monitors NPU load via timer events (10ms intervals), dynamically adjusting the number of NPU cores based on the number of targets in the video frames.

[0195] Target number < 3: Enable NPU single-core operation (power consumption ≤ 2W);

[0196] 3 ≤ target number ≤ 10: Enable NPU dual-core operation;

[0197] Target number > 10: Enable NPU triple-core operation (computing power up to 6 TOPS).

[0198] like Figure 4 As shown, the server-side NPU scheduling method based on uloop is as follows: The system monitors the task queue status through uloop events. When a new task is received, the core computing power is dynamically switched according to the real-time load of the three-core NPU, and the task is allocated to the idle core. At the same time, the queue priority sorting ensures that high-priority tasks are executed first, so as to achieve efficient utilization of NPU computing power.

[0199] As a preferred embodiment, the smooth transition of core switching is achieved by synchronizing the task queue buffer with the core state: before switching, the current frame inference task is stored in a temporary queue. After the target core is initialized (initialization time ≤ 5ms), the queue task is migrated to the newly activated core for execution. During the switching process, the NPU computing power output is kept continuous to avoid latency fluctuations caused by task interruption. At the same time, the load status of each core is synchronized in real time through the uloop timer to ensure that the switching time avoids the execution period of high-priority tasks.

[0200] Step 5: Intelligent Analysis and Reasoning

[0201] The scheduling module sends preprocessed data and core count instructions to the inference module via the dbus interface:

[0202] Model loading: RKNN models are loaded via the rknn_init() interface of RKNN-Lite2 (adapted for Ubuntu 20.04 arm64), and core switching is achieved by setting "runtime_core=RKNN_NPU_CORE_AUTO"; two models are cached using 8GB of memory, and the switching time is ≤200ms.

[0203] Inference execution: The preprocessed RGB image is passed to the rknn_inputs_set() interface, and the NPU inference is triggered by calling rknn_run(). After the inference is completed, the result is obtained through rknn_outputs_get(). The result is a floating-point array in the format of "[Category ID, Confidence, x1, y1, x2, y2]".

[0204] Core inference code snippet (Python):

[0205] from rknnlite.api import RKNNLite

[0206] class NPUInfer:

[0207] def __init__(self, model_path, model_cache=None):

[0208] self.rknn_lite = RKNNLite()

[0209] # Prioritize loading models from the 8GB memory cache (the original material mentions "caching 2 models, switching time ≤200ms")

[0210] if model_cache and model_path in model_cache:

[0211] self.rknn_lite = model_cache[model_path]

[0212] else:

[0213] # Read and load the model from a 64GB eMMC file (original code annotations)

[0214] self.rknn_lite.load_rknn(model_path)

[0215] # Initialize the NPU runtime (Enabling three cores on Ubuntu 20.04, original code parameters)

[0216] ret = self.rknn_lite.init_runtime(runtime_core=RKNNLite.RKNN_NPU_CORE_0_1_2)

[0217] if ret != 0:

[0218] raise Exception("NPU initialization failed")

[0219] # Load and store in memory cache

[0220] if model_cache:

[0221] model_cache[model_path] = self.rknn_lite

[0222] def infer(self, frame):

[0223] # Perform inference (original code method name + logic)

[0224] outputs = self.rknn_lite.inference(inputs=[frame])

[0225] # Analysis Results

[0226] result = self.parse_output(outputs)

[0227] return result

[0228] def parse_output(self, outputs):

[0229] # YOLOv8 result parsing logic (original code structure)

[0230] boxes = outputs[0][0]

[0231] scores = outputs[1][0]

[0232] classes = outputs[2][0]

[0233] result = []

[0234] for i in range(len(scores)):

[0235] if scores[i] > 0.5: # Confidence threshold for the original material

[0236] result.append({

[0237] "class": int(classes[i]),

[0238] "score": float(scores[i]),

[0239] "box": [float(x) for x in boxes[i]]

[0240] })

[0241] return result

[0242] Multi-task support: Supports loading two RKNN models simultaneously (such as YOLOv8n+PP-Human), and switching inference tasks via the model_id parameter, with a switching time of ≤200ms.

[0243] As a preferred embodiment, the model cache adopts an LRU (Least Recently Used) replacement strategy: when a new model needs to be loaded and the cache is full, the cached model that has been used the longest is replaced, ensuring that frequently used models are retained first; at the same time, the parameter data of the cached model is locked in physical memory through a memory locking mechanism (calling the mlock function) to prevent it from being swapped out to disk by the system page replacement mechanism, thus ensuring the memory read and write efficiency when loading the model and further reducing the switching time.

[0244] Step 6: Result Post-processing and Output

[0245] Non-maximum suppression (NMS): Performs NMS on the inference results with an IOU threshold of 0.45 to filter overlapping detection boxes.

[0246] Inverse coordinate transformation: Based on the scaling ratio of the original video frame and the normalized frame, the normalized coordinates are restored to pixel coordinates at a scale of 1920×1080.

[0247] Results encapsulation and output: Generates a JSON-formatted report containing "timestamp (milliseconds), target type (e.g., 'person', 'car'), location (x1, y1, x2, y2), and confidence score", supporting two output methods:

[0248] 1. Local storage: Write to the / data / analysis_reports directory on a 64GB eMMC, divide the files by day (naming format: YYYYMMDD_report.json), and archive them periodically using the logrotate tool in Ubuntu 20.04;

[0249] 2. Network upload: Push to the remote monitoring platform via a 1000Mbps Ethernet interface, implemented using socket, with an upload latency of ≤10ms.

[0250] Core output code snippet (Python):

[0251] import json

[0252] import time

[0253] import socket

[0254] import os

[0255] class ResultProcessor:

[0256] def __init__(self, raw_frame_shape=(1920, 1080), iou_thresh=0.45):

[0257] self.raw_h, self.raw_w = raw_frame_shape

[0258] self.iou_thresh = iou_thresh

[0259] def postprocess_result(self, infer_result):

[0260] # Inverse coordinate transformation (640×640 → 1920×1080 original resolution)

[0261] scale_x = self.raw_w / 640.0

[0262] scale_y = self.raw_h / 640.0

[0263] for res in infer_result:

[0264] # Adjust coordinates in the order of [Category ID, Confidence, x1, y1, x2, y2].

[0265] res[2] *= scale_x

[0266] res[3] *= scale_y

[0267] res[4] *= scale_x

[0268] res[5] *= scale_y

[0269] Nonmaximum suppression with IOU=0.45

[0270] filtered_result = self.nms(infer_result)

[0271] return filtered_result

[0272] def nms(self, boxes):

[0273] # Non-maximum suppression implementation (filtering overlapping detection boxes)

[0274] if not boxes:

[0275] return []

[0276] boxes = sorted(boxes, key=lambda x: x[1], reverse=True) # Sort by confidence in descending order

[0277] keep = []

[0278] while boxes:

[0279] current = boxes.pop(0)

[0280] keep.append(current)

[0281] boxes = [box for box in boxes if self.calculate_iou(current[2:6], box[2:6]) < self.iou_thresh]

[0282] return keep

[0283] def calculate_iou(self, box1, box2):

[0284] # Calculate the Intersection over Union (IOU) of two boxes.

[0285] x1, y1, x2, y2 = box1

[0286] x1_, y1_, x2_, y2_ = box2

[0287] intersection_x1 = max(x1, x1_)

[0288] intersection_y1 = max(y1, y1_)

[0289] intersection_x2 = min(x2, x2_)

[0290] intersection_y2 = min(y2, y2_)

[0291] intersection_area = max(0, intersection_x2 - intersection_x1)* max(0, intersection_y2 - intersection_y1)

[0292] area1 = (x2 - x1) * (y2 - y1)

[0293] area2 = (x2_ - x1_) * (y2_ - y1_)

[0294] return intersection_area / (area1 + area2 - intersection_area)

[0295] def save_result(self, result):

[0296] # Store to 64GB eMMC (partitioned by day)

[0297] save_path = f" / data / analysis_reports / {time.strftime('%Y%m%d')}_report.json"

[0298] # Encapsulate JSON formatted data (including millisecond-level timestamps)

[0299] data = {

[0300] "timestamp": int(time.time() * 1000),

[0301] "results": [

[0302] {

[0303] "target_type": self.get_target_name(res[0]), # Convert category ID to name

[0304] "confidence": res[1],

[0305] "position": [res[2], res[3], res[4], res[5]]

[0306] } for res in result ]

[0308] }

[0309] # Append to file

[0310] with open(save_path, 'a+') as f:

[0311] json.dump(data, f)

[0312] f.write('\n')

[0313] def upload_result(self, result):

[0314] # Upload to remote monitoring platform via 1000Mbps Ethernet

[0315] client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

[0316] client.connect(('192.168.1.100', 8080)) # Remote platform address (configurable)

[0317] # Encapsulate uploaded data

[0318] upload_data = json.dumps({

[0319] "timestamp": int(time.time() * 1000),

[0320] "results": result

[0321] }).encode('utf-8')

[0322] client.send(upload_data)

[0323] client.close()

[0324] def get_target_name(self, class_id):

[0325] # Category ID to Name Mapping (Supports 20 common object categories)

[0326] class_map = {

[0327] 0: "person", 1: "car", 2: "bicycle", 3: "motorcycle", 4: "airplane",

[0328] 5: "bus", 6: "train", 7: "truck", 8: "boat", 9: "traffic_light",

[0329] 10: "fire_hydrant", 11: "stop_sign", 12: "parking_meter",13: "bench",

[0330] 14: "bird", 15: "cat", 16: "dog", 17: "horse", 18: "sheep", 19: "cow"

[0331] }

[0332] return class_map.get(class_id, "unknown")

[0333] Step 7: Exception Handling Mechanism

[0334] Video capture error: When the camera disconnects, it triggers 3 reconnection attempts (500ms interval); if the reconnection is successful, it returns to the video capture stage to reacquire the video stream; if the reconnection fails, it outputs a "capture interrupted" alarm to the syslog of Ubuntu 20.04 and terminates the processing of the current frame.

[0335] Model loading failure: If the RKNN model fails to load, it will automatically be downgraded to the CPU-based YOLOv8ntiny lightweight model inference (accuracy reduced to 90%, but basic analytical capabilities are guaranteed). After downgrading, it will directly return to the output node of the model conversion stage and continue to execute the NPU inference stage.

[0336] NPU Inference Error: If NPU inference times out (>100ms), the current task will be added to the retry queue (maximum 2 retries). If the retry is successful, the task will return to the output node of the inference stage to perform post-processing. If the retry fails, the task will be diverted to the CPU to perform inference. After the inference is completed, the post-processing stage will continue to be executed.

[0337] Storage anomaly: Monitor the remaining 64GB eMMC space. When the available space is less than 10GB, automatically delete analysis reports older than 7 days. After cleaning, return to the storage stage to continue writing results. If the space is still insufficient after cleaning, prioritize switching to network upload mode to output results.

[0338] The main innovative designs of the above solutions in the embodiments of the present invention include:

[0339] 1. Layered quantization and model adaptation technology: Targeting the accuracy characteristics of the RK3588 NPU, a hybrid quantization strategy of "critical layer FP16 + non-critical layer INT8" is proposed. Under the premise of 75% model volume compression, the accuracy loss is controlled within 3%, which is 8% higher than the full INT8 quantization scheme.

[0340] 2. Dynamic computing power scheduling architecture: Based on uloop event-driven and dbus standardized interface (adapted to Ubuntu 20.04), it realizes dynamic switching of the number of NPU cores, reduces power consumption in scenarios with a small number of targets (single-core mode power consumption ≤2W), and ensures real-time performance in high-load scenarios (three-core mode computing power reaches 6TOPS).

[0341] 3. Hardware-System Co-optimization: Optimizing video frame caching, model caching, and storage partitioning under Ubuntu 20.04 system, taking advantage of the RK3588's 8GB memory and 64GB eMMC hardware features, to improve data read / write and model loading efficiency;

[0342] 4. CPU-NPU Pipeline Processing: The preprocessing-inference-postprocessing tasks are split and executed in parallel by the CPU's A55 small cores, the NPU, and the CPU's A76 large cores, forming a pipelined operation, which improves throughput by 60% compared to the serial processing solution;

[0343] 5. Multi-scenario adaptive mechanism: Supports dynamic model switching, frame extraction interval adjustment, and output mode configuration, and can adapt to the needs of different scenarios such as intelligent security (frame-by-frame analysis), industrial monitoring (analysis at 2-frame interval), and intelligent transportation (analysis at 1-frame interval).

[0344] Performance comparison of this invention with existing solutions (based on 1080P@60fps video stream, RK3588 8GB / 64GB, Ubuntu 20.04 tests):

[0345]

[0346] The following provides detailed information about a test instance:

[0347] 1: Hardware Environment Configuration

[0348] Core hardware: RK3588 industrial-grade development board (4-core A76 + 4-core A55 + 3-core NPU), equipped with 8GB LPDDR4X memory, 64GB eMMC storage, and 2 MIPI CSI 2.0 interfaces;

[0349] Peripheral configuration: Connects to an IMX477 8-megapixel camera (MIPI CSI interface), and connects to a 1000Mbps Ethernet network via an RJ45 interface;

[0350] Software environment: Install Ubuntu 20.04 LTS arm64 operating system (Linux kernel 5.4.0), deploy RKNN-Lite2 SDK (version 1.4.0, compiled and adapted for Ubuntu 20.04), OpenCV 4.5.5 (with NEON optimization enabled), V4L2 driver toolkit (version 1.20.0), and libuloop (version 1.0.0).

[0351] 2: Model Conversion and Deployment

[0352] Model acquisition: Download the YOLOv8n pre-trained model in ONNX format (input size 640×640) from the Ultralytics official repository.

[0353] Preparation of Quantization Calibration Dataset: Collect 1200 scene images covering indoor security, industrial production lines, and urban roads, label them, and save them as a COCO format dataset, stored in the / data / dataset directory of a 64GB eMMC.

[0354] RKNN format conversion: Start RKNN-Toolkit2 on the x86 host (Ubuntu 20.04), execute the above model conversion code, and generate the "yolov8n.rknn" model file;

[0355] Model deployment: Upload the generated RKNN model file to the / data / models directory on the RK3588 development board using the SCP tool, and modify the file permissions to 755 (to adapt to the permission mechanism of Ubuntu 20.04).

[0356] 3: Implementation of Video Analysis Process

[0357] System service configuration: Register the dbus service code as a systemd service in Ubuntu 20.04 (file path: / etc / systemd / system / npu-infer.service), and configure it to start automatically at boot:

[0358] [Unit]

[0359] Description=NPU Video Infer Service

[0360] After=network.target

[0361] [Service]

[0362] Type=simple

[0363] User=root

[0364] ExecStart= / usr / bin / python3 / opt / video_analysis / npu_infer_service.py

[0365] Restart=on-failure

[0366] [Install]

[0367] WantedBy=multi-user.target

[0368] To start the service: Execute `systemctl daemon-reload && systemctl start npu-infer && systemctl enable npu-infer`;

[0369] Video capture and preprocessing: Compile the V4L2 capture code (gcc capture.c -o capture -lv4l2), execute . / capture to start capture, and the preprocessing module obtains the captured frames through inter-process communication (shared memory);

[0370] Inference and Post-processing: Run the Python client program, call the NPU inference service via dbus, receive the results, perform post-processing, and output the results.

[0371] 4: Performance Testing and Verification

[0372] Latency test: Using the time command in Ubuntu 20.04 combined with loop inference, the end-to-end processing time of 1000 frames of video was recorded, and the average latency was calculated to be 50ms / frame.

[0373] Accuracy test: The target detection results were verified using a labeled test set (200 frames), and the statistical accuracy was 95.3%.

[0374] Computing power utilization monitoring: The NPU load was monitored in real time using the "rknpu_monitor" tool (compatible with Ubuntu 20.04) that comes with the RK3588, with an average utilization of 91.2%;

[0375] Stability test: After running continuously for 72 hours (monitoring the service status using the watchdog daemon process in Ubuntu 20.04), no crashes or memory leaks were recorded, and the reconnection mechanism successfully recovered when the camera was disconnected.

[0376] Storage test: 64GB eMMC continuously wrote to an analysis report (approximately 5GB of data) for 7 days, with read and write speeds stable at 80MB / s (sequential write) and 120MB / s (sequential read), without any lag.

[0377] Under the aforementioned hardware environment (RK3588 8GB / 64GB, Ubuntu 20.04), a continuous 72-hour test was conducted on a 1080P@60fps surveillance video stream. The test results showed that the average preprocessing time per frame was 8ms, the NPU inference time was 35ms, the post-processing time was 7ms, and the total end-to-end latency was 50ms. The target detection accuracy was 95.3%, and the abnormal behavior recognition accuracy was 92.7%. The average NPU computing power utilization was 91.2%, and the overall power consumption of the development board was 10.8W. The 64GB eMMC storage utilization was stable, with no memory leaks or crashes, fully meeting the performance and stability requirements of edge video intelligent analysis.

[0378] The intelligent video content analysis method based on RK3588 (8GB / 64GB, Ubuntu 20.04) provided in this invention can be directly deployed in devices such as smart security cameras, industrial defect detection terminals, and smart transportation edge boxes. In smart security scenarios, it can provide real-time early warnings for abnormal events such as climbing over walls and crowds gathering. In industrial scenarios, it can detect dimensional deviations and surface defects of production line parts. In smart transportation scenarios, it can count traffic flow and identify illegal driving behaviors. This method fully leverages the hardware potential of the RK3588 and, combined with adaptation and optimization for the Ubuntu 20.04 system, achieves low-latency, high-precision intelligent video content analysis while balancing power consumption and computing power utilization, demonstrating significant industrial application value and prospects for large-scale promotion.

[0379] Based on the same inventive concept, this invention also provides a computer device, comprising: one or more processors, and a memory for storing one or more computer programs; the programs include program instructions, and the processor executes the program instructions stored in the memory. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, used to implement one or more instructions, specifically for loading and executing one or more instructions stored in a computer storage medium to implement the above-described method.

[0380] It should be further explained that, based on the same inventive concept, the present invention also provides a computer storage medium storing a computer program, which, when executed by a processor, performs the above-described method. This storage medium can be any combination of one or more computer-readable media. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0381] It should be noted that, unless otherwise defined, the technical or scientific terms used in this invention should have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0382] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

[0383] This invention is not limited to the above-described preferred embodiments. Anyone inspired by this invention can derive other forms of intelligent video content analysis methods based on edge computing platforms. All equivalent variations and modifications made within the scope of the claims of this invention shall fall within the scope of this invention.

Claims

1. A video content intelligent analysis method based on an edge computing platform, characterized in that, The edge computing platform includes a central processing unit (CPU) and a neural network processing unit (NPU) with multiple processing cores. The method includes: The video stream is acquired through the image acquisition interface of the edge computing platform, and the video frames are preprocessed on the CPU to obtain standardized image data. The pre-trained neural network model is converted into a dedicated format model adapted to the NPU. During the conversion process, a hierarchical quantization strategy based on network layer function is adopted: a first numerical precision quantization is adopted for at least one network layer in the model that is responsible for basic feature extraction, and a second numerical precision quantization lower than the first numerical precision is adopted for the remaining network layers in the model that are responsible for classification and regression. Based on the processing load determined by analyzing the current video frame content represented by the standardized image data, the number of active processing cores in the NPU is dynamically scheduled, and the dedicated format model is loaded using the active processing cores to perform inference on the standardized image data. The inference output of the NPU is post-processed to generate structured analysis results.

2. The intelligent video content analysis method based on an edge computing platform according to claim 1, characterized in that: The first numerical precision is floating-point precision, and the second numerical precision is integer precision; the network layer responsible for basic feature extraction is the initial layer responsible for core feature extraction in the backbone network of the model, and the network layer responsible for classification and regression is the classification head layer and regression head layer of the model.

3. The intelligent video content analysis method based on an edge computing platform according to claim 1, characterized in that: The processing load is determined by the number of targets to be detected in the current video frame; the rule for dynamically scheduling the number of active processing cores in the NPU is as follows: when the number of targets to be detected is less than a first threshold, the NPU is activated to run on a single core; when it is between the first threshold and the second threshold, the NPU is activated to run on a dual-core; when it is greater than the second threshold, the NPU is activated to run on a triple-core.

4. The intelligent video content analysis method based on an edge computing platform according to claim 1, characterized in that: The CPU-side preprocessing includes frame extraction, noise reduction, size normalization, color gamut conversion, and numerical normalization. Noise reduction employs Gaussian filtering or other noise reduction algorithms adapted to edge computing. Size normalization uses interpolation to scale video frames to a preset size suitable for the model. Color gamut conversion converts the original video frame's color gamut format to a model-compatible format. Numerical normalization maps pixel values ​​to the numerical range suitable for model inference. The preprocessing tasks are assigned to the low-power cores of the CPU for execution, while reserving high-performance cores for core tasks.

5. The intelligent video content analysis method based on an edge computing platform according to claim 1, characterized in that: The model conversion is achieved through a model conversion tool adapted to the NPU, which converts pre-trained neural network models in ONNX format or other general formats into NPU-compatible dedicated format models. During the conversion process, an adapted sample set covering different application scenarios is imported for calibration to control the loss of quantization accuracy to not exceed a preset threshold. It supports dynamic switching between at least two dedicated format models with different functions, and shortens the model switching time by caching at least one loaded model in the memory of the edge computing platform; the neural network model includes at least two of the following: target detection model, industrial defect detection model and behavior recognition model.

6. The intelligent video content analysis method based on an edge computing platform according to claim 1, characterized in that: The post-processing includes non-maximum suppression and inverse coordinate transformation; wherein, non-maximum suppression uses a preset intersection-union ratio threshold to filter overlapping detection boxes, and inverse coordinate transformation is based on the scaling ratio of standardized image data and original video frames to restore the normalized coordinates to the pixel scale of the original video frames; the structured analysis results include timestamp, target type, target location and confidence information.

7. The intelligent video content analysis method based on an edge computing platform according to claim 1, characterized in that: It also includes an exception handling mechanism: when video acquisition is interrupted, a reconnection mechanism is triggered at multiple intervals of preset time. If the reconnection fails, an alarm is output; when NPU inference times out or fails, the current task is first added to the retry queue, and retry is performed up to a preset number of times. If the retry fails, it is automatically downgraded to CPU-based lightweight model inference; when the available space of the storage medium of the edge computing platform is lower than a preset threshold, the structured analysis results before the preset time are automatically deleted to avoid storage overflow.

8. The intelligent video content analysis method based on an edge computing platform according to claim 1, characterized in that: The preprocessing, inference, and postprocessing tasks are executed in a pipelined parallel manner. The preprocessing tasks are assigned to the low-power cores of the CPU, the inference tasks are executed by the NPU, and the postprocessing tasks are assigned to the high-performance cores of the CPU. The coordinated scheduling between the tasks is achieved through a standardized communication interface and an event-driven framework to improve the overall processing throughput.

9. The intelligent video content analysis method based on an edge computing platform according to claim 1, characterized in that: The structured analysis results support two output methods: local storage and network upload. When stored locally, files are divided according to a preset period and stored in a dedicated storage area of ​​the storage medium. Network uploads are achieved via a high-speed Ethernet interface based on a preset communication protocol, with upload latency controlled to not exceed a preset threshold.

10. A video content intelligent analysis system based on an edge computing platform, characterized in that, The edge computing platform includes a central processing unit (CPU), a neural network processing unit (NPU) with multiple processing cores, an image acquisition interface, memory, and storage media. The system includes: Video acquisition and preprocessing module: Acquires video streams through the image acquisition interface, performs preprocessing on video frames on the low-power core of the CPU, and outputs standardized image data; Model conversion module: Converts the pre-trained neural network model into a dedicated format model adapted to the NPU. During the conversion process, a hierarchical quantization strategy based on network layer functions is adopted: the first numerical precision quantization is adopted for at least one network layer in the model that is responsible for basic feature extraction, and the second numerical precision quantization, which is lower than the first numerical precision, is adopted for the remaining network layers in the model that are responsible for classification and regression. Dynamic scheduling and inference module: Based on the processing load determined by analyzing the current video frame content represented by the standardized image data, dynamically schedule the number of active processing cores in the NPU, load the dedicated format model using the active processing cores, perform inference on the standardized image data, and output the inference results; Post-processing module: Performs post-processing on the inference results on the high-performance core of the CPU, generates structured analysis results, and outputs them; The video acquisition and preprocessing module, dynamic scheduling and inference module, and post-processing module constitute a pipeline, which achieves collaborative scheduling through a standardized communication interface and an event-driven framework.

Images