Lightweight multi-model collaborative edge intelligent target detection system and method

Abstract

The invention provides a lightweight multi-model collaborative edge intelligent target detection system and a lightweight multi-model collaborative edge intelligent target detection method, belongs to the technical field of edge target detection, and aims to solve the problem that a traditional target monitoring scheme depends on a cloud server to carry out deep learning model reasoning; in order to solve the problem that high-precision and low-delay real-time target monitoring cannot be realized on a resource-limited edge device, the detection system comprises an edge calculation module, an image acquisition module and a storage module. The detection method comprises the following steps: acquiring a to-be-detected image, and calling an IPS module through a performance core and an energy efficiency core to pre-process the acquired image; optimizing the YOLO model, and inputting the preprocessed image into a deep learning model; performing target detection on the preprocessed image data through a deep learning model; and a detection result is input into the storage module for storage and is uploaded to equipment connected with the peripheral interface for visual display and a remote monitoring center.

Description

Technical Field

[0001] This invention relates to a lightweight multi-model collaborative edge intelligent target detection system and method, belonging to the field of edge target detection technology. Background Technology

[0002] Traditional target detection solutions rely on cloud servers for deep learning model inference, requiring data collected by edge devices to be transmitted to the cloud for processing. This results in high latency, high bandwidth consumption, and privacy risks. With the development of edge computing, local deployment of lightweight models has become a trend, but achieving high-precision, low-latency real-time target detection on resource-constrained edge devices remains a technical challenge.

[0003] Existing edge target detection solutions are mostly based on YOLO series models. For example: Solution A: YOLOv5 is deployed to edge devices through model pruning and quantization compression, but excessive compression leads to a significant decrease in the detection accuracy of small targets; Solution B: Multiple models are dynamically switched, selecting different scale models according to scene complexity, but the switching logic is complex and lacks real-time performance; Solution C: YOLO inference is accelerated using FPGA, which improves computing speed, but the hardware cost is high and the scalability is poor. Current ultra-low-cost edge devices (such as the ESP32-S3 development board) achieve local deployment through extreme model pruning (such as Tiny YOLO v3), but the detection accuracy is greatly reduced due to limitations in computing power (CPU computing only) and memory bandwidth. These solutions are mostly used for single target detection in static scenes and cannot meet the needs of real-time multi-target monitoring in dynamic environments.

[0004] The contradiction between model accuracy and resource consumption: Existing lightweight methods (such as Scheme A) oversimplify the model structure to adapt to the computing power of edge devices, resulting in a sharp drop in detection accuracy in small targets and occluded scenes (in contrast to the "dynamic resolution input + attention mechanism optimization" of this invention); 1. Poor adaptability to dynamic scenes: Existing solutions (such as Scheme B) rely on fixed threshold switching models, making it difficult to respond in real time to complex environments such as changes in lighting and sudden changes in target movement speed (this invention adaptively adjusts model parameters through an environment perception module); 2. Strong hardware dependence: Dedicated acceleration solutions (such as Scheme C) require customized hardware, resulting in high deployment costs and difficulty in compatibility with heterogeneous edge devices (this invention proposes a general computing resource scheduling strategy that supports GPU / CPU hybrid architecture); 3. Imbalance between energy consumption and real-time performance: Existing methods do not optimize the power consumption of edge devices, and continuous high-load operation leads to device overheating and shortened lifespan (this invention introduces a task grading mechanism, with low-priority tasks being processed at reduced frequencies). 4. Performance ceiling of ultra-low-cost hardware: Existing low-cost solutions (such as 40 RMB-level development boards) rely on CPU software inference, which cannot balance accuracy and real-time performance, and lack hardware acceleration units (such as NPU), resulting in a failure rate of up to 60% for deploying complex models. 5. Balance between low cost and high performance: With a hardware cost of ≤60 RMB, an improved YOLO model and heterogeneous computing scheduling achieve detection accuracy ≥90% (compared to 75% for the ESP32 solution) and a frame rate ≥25 FPS (compared to 8 FPS for ESP32), breaking through the performance bottleneck of ultra-low-cost devices. Summary of the Invention

[0005] To address the problem that traditional target detection schemes rely on cloud servers for deep learning model inference, which cannot achieve high-precision, low-latency real-time target detection on resource-constrained edge devices, this invention proposes a lightweight, multi-model collaborative edge intelligent target detection system and method.

[0006] The technical solution adopted by this invention to solve the above problems is: a lightweight multi-model collaborative edge intelligent target detection system proposed in this invention, comprising: The edge computing module adopts a multi-core heterogeneous processing architecture and is equipped with a deep learning model. It is used to receive image data from the image acquisition module and perform target detection through the deep learning model. The image acquisition module is used to acquire image data and transmit the acquired image data to the edge computing module; The storage module is used to store the target detection results and acquired image data in an encrypted form.

[0007] Furthermore, the edge computing module includes n performance cores, n energy efficiency cores, and peripheral interfaces, and is equipped with an integrated GPU and a dedicated NPU, and is deployed with a deep learning model; The performance core and energy efficiency core are used to preprocess the received image data and schedule the various modules. Peripheral interfaces are used to connect external devices; A separate NPU is used to invoke a deep learning model to perform object detection on preprocessed image data; Integrated GPUs are used to accelerate the object detection process in deep learning models.

[0008] Furthermore, deep learning models include the YOLO model and optimized YOLO models; The YOLO model includes a data preprocessing submodule and a feature extraction submodule; The data preprocessing submodule is used to preprocess the received image data; The feature extraction submodule is used to receive preprocessed image data and perform image detection, and then transmit the detection results to the storage module. The optimized YOLO model includes a data preprocessing submodule and an optimized feature extraction submodule; The data preprocessing submodule is used to preprocess the received image data; The optimized feature extraction submodule includes several depthwise separable convolutional layers and a neck network. Each depthwise separable convolutional layer includes a 3×3 depthwise convolutional layer and a 1×1 pointwise convolutional layer. The neck network includes a spatial pyramid pooling-Fast module, a pooling layer with a pooling window of 3×3, and a fully connected layer.

[0009] Furthermore, the image acquisition module includes a vision acquisition module and an IPS module; The visual acquisition module is used to acquire images and automatically turns on the infrared fill light in low-light scenes. The image acquisition supports MJPEG / H.264 encoding. Under the call of the performance core and energy efficiency core, the IPS module preprocesses the acquired images, encodes the processed images, and uploads them to the edge computing module.

[0010] Furthermore, the storage module includes an edge cache module and a central device module; The edge cache module uses an LRU strategy to manage storage space and stores detected images within a preset time period using the AES-256 encryption algorithm. The central equipment module is used to aggregate image detection results and remotely upload them to the remote monitoring center.

[0011] A lightweight, multi-model collaborative edge intelligent object detection method includes: Step 1: Acquire the image to be detected based on the visual acquisition module, and call the IPS module to preprocess the acquired image through the performance core and energy efficiency core in the edge computing module; Step 2: Optimize the YOLO model to obtain the optimized YOLO model, and input the preprocessed image into the deep learning model; Step 3: The deep learning model is invoked by an independent NPU to perform object detection on the preprocessed image data, and the detection process is accelerated by an integrated GPU; Step 4: Input the test results into the storage module for storage and upload them to the device connected to the peripheral interface for visualization and remote monitoring.

[0012] Furthermore, the preprocessing steps for the acquired images in step 1 include: Step 1.1: Scale the acquired image to the input size required by the YOLO model and the optimized YOLO model; Step 1.2: Normalize the resized image, converting the pixel value range from [0,255] to [0,1]; Step 1.3: Perform enhancement operations on the normalized image to complete the preprocessing of the acquired image, and then transmit the preprocessed image to the deep learning model.

[0013] Furthermore, step 2, which involves inputting the preprocessed image into the deep learning model, specifically includes: An adaptive data transmission protocol is constructed. When network conditions are good, the MQTTS protocol is used for reliable transmission, employing a sliding window mechanism to control the data transmission rate. The window size is dynamically adjusted based on the network's RTT (Real-Time To Day). In scenarios with unstable networks or limited bandwidth, the protocol automatically switches to UDP and incorporates FEC (Flexible Encoding) technology for data transmission. FEC encoding divides the original data into multiple blocks, adds redundant check blocks, and then transmits the data. A deep learning model uses the Reed-Solomon algorithm to reconstruct the original data based on the received data blocks and check blocks, ensuring the accuracy of data transmission. Simultaneously, the transmitted data is compressed in blocks using the Zstandard compression algorithm, dynamically adjusting the compression level according to the data type. For image data, a medium compression level is selected while ensuring real-time performance. Furthermore, step 2 optimizes the YOLO model, resulting in the optimized YOLO model, which specifically includes: Based on the YOLO model, the ordinary convolutional layers in the CSPDarknet structure of the YOLO model are replaced with depthwise separable convolutional layers, and the Spatial Pyramid Pooling-Fast module is introduced to replace the SPP module, and the pooling window is set to a 3×3 pooling layer. For depthwise separable convolutional layers, 8-bit integer quantization is used to quantize weights and activation values ​​into 8-bit integers. During quantization, quantization parameters are calibrated using a calibration dataset, and quantization error is calculated using KL divergence to dynamically adjust the quantization threshold. For fully connected layers, 16-bit half-precision floating-point quantization is used to balance computational accuracy and speed. CUDA graph technology is used to optimize the inference process of the quantized model, merging multiple inference operations into a single CUDA graph. The training mechanism was optimized by employing a designed incremental training mechanism based on transfer learning. In the initial training phase, the backbone network parameters of the YOLO model were frozen, and only the detection head part was trained to learn general target features. During the custom dataset training phase, the shallow parameters of the backbone network were gradually unfrozen, and a cosine annealing learning rate strategy was adopted, dynamically adjusting the learning rate according to the training epochs. The initial learning rate was set to 0.001, and during training, the learning rate gradually decreased to 0.00001 according to the cosine function. A knowledge distillation mechanism was introduced during training. In the knowledge distillation training process, the optimized YOLO model was used as the teacher model, and the YOLO model was used as the student model, constructing a teacher-student model collaborative training framework. The student model learned hard labels from the original training data, and a distillation loss function was introduced. The system guides student models to learn the soft knowledge output by the teacher model, dynamically adjusts temperature parameters, and incorporates the standard cross-entropy loss of the student model. and distillation loss function Constructing a joint loss function After distillation training, a dynamic fusion strategy for data augmentation is introduced. The probability of using data augmentation methods is dynamically adjusted according to the data distribution. For target categories with fewer samples, augmentation probability operation is adopted, and the positive and negative sample matching thresholds are adaptively adjusted according to the target size, occlusion degree and scene complexity. The expression for the joint loss function is: (1); In formula (1), The standard cross-entropy loss for the student model, Distillation loss as a measure of Kullback-Leibler (KL) divergence. and This is a dynamically adjusted coefficient.

[0014] Furthermore, step 3 involves calling a deep learning model to perform object detection on the preprocessed image data, specifically including: The system adaptively selects between the YOLO model and the optimized YOLO model based on the performance requirements of the scenario. The YOLO model is selected in low-performance scenarios, while the optimized YOLO model is selected in high-performance scenarios. Target detection is performed using GPU acceleration. The detection results are transmitted to the connected device for visualization through the peripheral interface and remotely transmitted to the remote monitoring center through the central device module. Furthermore, when the CPU utilization exceeds 80% or the GPU memory usage exceeds 90%, the dynamic model switching mechanism is automatically triggered to automatically switch between the YOLO model and the optimized YOLO model.

[0015] The beneficial effects of this invention are: 1. This invention employs an improved YOLOv8+ dynamic label allocation method and knowledge distillation. The dynamic label allocation strategy dynamically adjusts the matching thresholds of positive and negative samples based on the target scale and occlusion degree (e.g., the IoU threshold for small targets is reduced to 0.3, while that for large targets is increased to 0.5), reducing missed detections caused by fixed thresholds (experimental comparison: AP for small target detection is improved by 12.7%). The YOLOv8 network structure is improved by introducing a lightweight channel attention module (ECA-Net) in the neck layer, which only increases the number of parameters by 0.8%, but improves the multi-scale feature fusion efficiency by 19%, and the model mAP reaches 90.2% (compared to 83.5% for Tiny YOLOv8). Furthermore, on Rockchip chips (cost ≤ 60 RMB), through the collaborative design of dynamic label allocation, structural attention optimization, and knowledge distillation mechanisms, the accuracy loss is controlled within 5% (compared to an average accuracy loss of 15%-30% for similar low-cost solutions). This achieves a core breakthrough of "lightweight without sacrificing accuracy," overcoming the contradiction between "model accuracy and resource consumption" and solving the problem of increased false negative rates for small targets caused by excessive compression in traditional lightweight models (such as Tiny YOLO). Knowledge distillation: By introducing a distillation loss function (e.g., cross-entropy + KL divergence), the lightweight student model is guided to learn "soft knowledge" such as the category distribution and feature representation output by the teacher model, significantly improving detection accuracy and robustness while maintaining a lightweight model structure. Experimental results show that, while keeping the number of student model parameters unchanged, the average mAP is improved by approximately 4.6%, and the ability to recognize small targets and occluded targets in complex environments is significantly enhanced.

[0016] 2. This invention proposes a sensor fusion strategy driven by environment perception and dynamic resolution input. This strategy integrates a lighting sensor and a motion detection algorithm, dynamically adjusting the input resolution (e.g., enabling high-resolution detection of 640×640 in well-lit conditions and switching to 416×416 in low-light environments to reduce noise interference). Furthermore, this invention performs adaptive inference frequency, adjusting the detection frame rate based on the target's motion speed (estimated using optical flow) (5 FPS for static scenes, 25 FPS for high-speed motion scenes), reducing computational resource consumption by 37%. Dynamic resolution and frame rate adjustment ensure FPS fluctuation ≤ ±5% in complex scenes (compared to ±20% in scheme B of reference 5), and achieves "on-demand computation" through environment perception, avoiding resource waste. This solves the problem of "poor adaptability to dynamic scenes."

[0017] 3. This invention employs an RKNN+heterogeneous computing resource scheduling scheme. RKNN model optimization: Utilizing the INT8 quantization capability of the Rockchip chip's built-in NPU, the improved YOLOv8 model is compressed to 1.2MB (original model 4.3MB), reducing memory usage by 72%. CPU / NPU collaborative scheduling algorithm: A task priority queue is designed, allocating object detection preprocessing (such as image normalization) to the CPU, while model inference is executed by the NPU, improving parallelization efficiency to 89% (compared to 45% for a pure CPU solution). This invention, based on the hardware-software co-optimization of domestically produced chips, reduces hardware costs to below 60 RMB (compared to 80 RMB for the Actions triple-core solution), and supports cross-platform deployment (compatibility with Rockchip RK3566 / RK3588 has been verified), breaking down the ecosystem barriers of dedicated hardware and eliminating the problem of strong software dependency.

[0018] 4. This invention employs a task-level classification and low-power mode, dividing detection tasks into high-priority (moving targets) and low-priority (stationary backgrounds), with the latter being downclocked to 1 FPS. Furthermore, it utilizes Rockchip's DVFS (Dynamic Voltage and Frequency Scaling) technology to shut down the NPU during idle periods, retaining only the CPU's low-power threads. This task-level classification strategy extends the device's battery life to 8.5 hours (compared to 4 hours for the ESP32 solution) and reduces peak temperature by 14°C (actual measured data), meeting industrial-grade stability requirements.

[0019] 5. This invention reconstructs model operators for Rockchip's NPU instruction set (e.g., replacing Conv2D with DepthwiseConv natively supported by the NPU), increasing inference speed to 25 FPS (compared to ESP32's 8 FPS), achieving deep adaptation to domestic chips. Furthermore, this invention processes data entirely locally, eliminating the need for cloud transmission, reducing latency to 35ms (compared to 200ms+ for cloud solutions), and complies with GDPR / CCPA privacy compliance requirements. Achieving mAP of 90.2%+25 FPS under a cost constraint of 60 yuan, its overall performance reaches the level of mid-range devices (100-200 yuan level), and it builds a triple barrier of "security-performance-cost" through fully localized deployment. Attached Figure Description

[0020] Figure 1 A schematic diagram of the structure of a lightweight multi-model collaborative edge intelligent target detection system provided by the present invention; Figure 2 A schematic diagram of a locally deployed edge caching module; Figure 3 A schematic diagram of the SC3336 COM V2 visual acquisition module camera deployed locally; Figure 4 This is a schematic diagram of data transmission for the entire system.

[0021] Figure 5 A flowchart for software simulator model verification; Figure 6 A schematic diagram of the application process for model inference deployed locally; Figure 7 A schematic diagram of a locally deployed ONNX model; Figure 8 Flowchart of an edge target detection algorithm for local deployment; Figure 9 A schematic diagram illustrating the allocation of core elements to dynamic tags; Figure 10 The teacher-student model structure and distillation process diagram are shown. Detailed Implementation

[0022] Specific implementation method one: Combining Figure 1-4 This implementation method is described as follows: Figure 1 As shown, the structure of the lightweight multi-model collaborative edge intelligent target detection system described in this embodiment includes: Edge computing module, the edge computing module adopts such as Figure 2The Rockchip LuckFox Pico Pro / Max edge computing platform shown integrates the RK3588 series SoC, featuring multi-core heterogeneous processing capabilities. It includes four Cortex-A76 performance cores and four Cortex-A55 energy-efficient cores, paired with an integrated ARM Mali-G610 GPU and a dedicated NPU (with a maximum computing power of 6 TOPS). A shared memory architecture improves data access efficiency between the CPU, GPU, and NPU. System scheduling and data preprocessing tasks are handled by the A76 / A55 cores, while inference for deep learning models such as YOLO is executed by the NPU, significantly improving processing efficiency and reducing power consumption.

[0023] The device features multiple high-speed interfaces for expanding external resources: a USB 3.1 interface for connecting vision modules, external storage devices, or communication modules; an M.2 interface supporting NVMe SSDs for high-performance data storage; and a Gigabit Ethernet interface supporting high-throughput data upload and remote management. The system supports both Android and Linux operating systems and can deploy TensorFlow Lite / ONNX Runtime / RKNN Toolkit for model inference, accelerating the local response speed of edge intelligence tasks.

[0024] Image acquisition module, using such Figure 3 The SC3336 COM V2 camera shown serves as a visual acquisition module, equipped with a 1 / 2.9-inch 3MP CMOS image sensor, supporting a maximum output resolution of 2304x1296. It features low-light imaging capabilities and wide dynamic range (WDR) support, making it suitable for monitoring and analysis tasks in complex outdoor environments. Image data is transmitted at high speed to the uckFox Pico Pro / Max edge device via a MIPI CSI-2 interface.

[0025] To adapt to diverse image acquisition needs, the module supports electronic shutter control and ISP functions such as automatic exposure (AE) and automatic white balance (AWB). In nighttime or low-light scenarios, the infrared filter automatically switches and activates the infrared fill light (if integrated) to improve image clarity. The built-in ISP module can perform real-time image noise reduction, gamma correction, and other preprocessing operations to ensure the quality of subsequent algorithm inputs. Image acquisition supports MJPEG / H.264 encoding, and encoded images can be uploaded to a local system for further processing via Ethernet or USB.

[0026] The storage module employs a distributed storage architecture that combines edge caching with central processing. Each edge node in the LuckFox Pico Pro / Max integrates an eMMC storage module as a local high-speed cache for temporarily storing detection images, intermediate model results, and status logs. The cache layer dynamically manages storage space using an LRU (Least Recently Used) strategy, defaulting to retaining images and analysis data from the last 10 minutes to improve the efficiency of reading short-term data and the speed of algorithm feedback.

[0027] The central device is an ESP32-S3 module, responsible for data aggregation and remote uploading. luckFox periodically sends detection results, image summary information, or alarm status to the ESP32-S3 via serial port (UART) or LAN UDP protocol. The central device can connect to a Wi-Fi network to achieve data synchronization with a cloud platform or remote server.

[0028] To enhance data security, LuckFox employs hardware-accelerated AES-256 encryption in its local storage process to encrypt and store critical images and logs. The keys are dynamically generated and managed by the system's TEE (Trusted Execution Environment). Simultaneously, CRC-32 verification is performed before data is transmitted to the ESP32-S3 via serial port or network to ensure data integrity during transmission. The ESP32-S3 can be optionally equipped with an external SPI Flash or microSD card as a central archiving storage medium, enabling long-term storage of event data and supporting local backtracking queries.

[0029] Hardware connections and data transmission, such as Figure 4 As shown: The ESP32S3 is cross-connected to the UART4_TX_M1 pin of the LuckFox Pico Pro / Max via the RX pin, and the GEN pin is grounded. The LuckFox Pico Pro / Max is connected to the SC3336 COM V2.

[0030] Specific Implementation Method Two: Combining Figure 5-10 This embodiment will be described as follows: Figure 8 As shown, the steps of the lightweight multi-model collaborative edge intelligent target detection method described in this embodiment include: S1: Hardware platform selection and structural design; like Figure 6As shown, this implementation uses an edge computing module with high computing power and low power consumption as the deployment carrier, supporting heterogeneous computing resources such as ARM CPUs, GPUs, and NPUs to meet the real-time inference requirements of deep learning models such as YOLO. The device should provide multiple standard interfaces such as MIPI / USB / M.2 / CSI to support connections to peripherals such as cameras, sensors, and SSD storage, facilitating the construction of an integrated edge acquisition and intelligent analysis system. Cameras are configured with different resolution and frame rate combinations according to scene requirements to ensure data acquisition clarity and dynamic response. To meet the needs of temporary caching and historical recording of on-site data, the system is equipped with local eMMC storage and expandable high-capacity storage media (such as USB SSDs).

[0031] S2: Deep learning model selection and optimized deployment; The steps for locally deploying an ONNX model are as follows: Figure 7 As shown, it includes: S201: Model structure improvement; This implementation improves the feature extraction network based on YOLOv5. The ordinary convolutional layers in the original CSPDarknet structure are replaced with depthwise separable convolutional layers, reducing the number of parameters and computational cost. Specifically, each depthwise separable convolutional module contains a 3×3 depthwise convolutional layer and a 1×1 point convolutional layer. The depthwise convolutional layer performs convolution operations independently on each input channel, while the point convolutional layer achieves information fusion between channels. The improved network structure reduces computational cost by 40% while maintaining detection accuracy. In the neck network, a Spatial Pyramid Pooling-Fast (SPPF) module is introduced to replace the traditional SPP module. Through continuous 3×3 max pooling operations, the network's feature extraction capability for targets at different scales is enhanced without significantly increasing computational cost, improving detection speed by 15%. S202: Quantization and Acceleration Technology; This implementation employs mixed-precision quantization technology to differentiate between convolutional and fully connected layers of the model. For convolutional layers, 8-bit integer quantization is used. TensorRT's INT8 quantization mode quantizes weights and activation values ​​into 8-bit integers. During quantization, a calibration dataset is used to calibrate quantization parameters, and the quantization error is calculated using KL divergence. The quantization threshold is dynamically adjusted to ensure that the accuracy loss of the quantized model is controlled within 1%. For fully connected layers, 16-bit half-precision floating-point quantization is used to balance computational accuracy and speed. Simultaneously, CUDA Graphs technology is used to optimize the inference process of the quantized model, merging multiple inference operations into a single CUDA graph, reducing communication overhead between the CPU and GPU, and improving inference efficiency by 30%. S203: Custom dataset training optimization; This implementation design employs an incremental training mechanism based on transfer learning. In the initial training phase, the backbone network parameters of the YOLO model are frozen, and training is performed only on the detection head portion. A dataset of over 36,000 fire incident images and a self-built fall dataset are used for training to learn general target features. During the custom dataset training phase, the shallow parameters of the backbone network are gradually unfrozen, and a cosine annealing learning rate strategy is adopted. The learning rate is dynamically adjusted according to the training epochs, with an initial learning rate of 0.001. During training, the learning rate gradually decreases to 0.00001 according to a cosine function. A knowledge distillation mechanism is introduced during training. In the knowledge distillation training process, the optimized YOLO model is used as the teacher model, and the YOLO model is used as the student model, constructing a teacher-student model collaborative training framework. The student model learns hard labels from the original training data, and a distillation loss function is introduced. (For example, cross-entropy + KL divergence) guides the student model to learn the soft knowledge output by the teacher model, dynamically adjusts the temperature parameters, and combines the student model's standard cross-entropy loss. and distillation loss function Constructing a joint loss function After distillation training, a dynamic fusion strategy for data augmentation is introduced. The probability of using data augmentation methods is dynamically adjusted based on the data distribution. For target categories with fewer samples, probability augmentation operations are used. Furthermore, the positive and negative sample matching thresholds are adaptively adjusted based on target size, occlusion level, and scene complexity. Figure 10 As shown, for example, a lower IoU threshold (e.g., 0.3) can be set for small targets, while increasing it to 0.5 for large targets, thereby reducing the false negatives and false positives caused by a fixed threshold. Experiments show that this strategy can improve the AP value of small target detection by an average of 12.7%, significantly improve the model's adaptability to complex edge scenarios while maintaining overall mAP stability, and further enhance the reliability of the system in real-world deployment environments. The expression for the joint loss function is: (1); In formula (1), The standard cross-entropy loss for the student model, Distillation loss as a measure of Kullback-Leibler (KL) divergence. and This is a dynamically adjusted coefficient.

[0032] S204: Optimization of Multi-Model Construction and Dynamic Label Allocation; To enhance the system's adaptability to various targets and complex dynamic scenarios, this implementation constructs a multi-prediction model architecture, including the basic YOLOv5, the improved YOLOv8, and a custom deep convolutional neural network. Each model is optimized in terms of detection accuracy, inference speed, and resource consumption, allowing for flexible switching and scheduling based on task requirements and edge device resource availability, achieving "on-demand computation." In inference task allocation, the YOLOv8 main model is prioritized for high-performance scenarios, while resource-constrained or simple target scenarios automatically switch to lightweight sub-models, improving overall system stability and response speed.

[0033] This invention employs a task-level classification and low-power mode, dividing detection tasks into high-priority (moving targets) and low-priority (stationary backgrounds), with the latter being downclocked to 1 FPS. Furthermore, it utilizes Rockchip's DVFS (Dynamic Voltage and Frequency Scaling) technology to shut down the NPU during idle periods, retaining only the CPU's low-power threads. This task-level classification strategy extends the device's battery life to 8.5 hours (compared to 4 hours for the ESP32 solution) and reduces peak temperature by 14°C (actual measured data), meeting industrial-grade stability requirements.

[0034] This invention employs an improved YOLOv8 + knowledge distillation + dynamic label allocation method. The dynamic label allocation strategy dynamically adjusts the matching thresholds for positive and negative samples based on target scale and occlusion level (e.g., reducing the IoU threshold for small targets to 0.3 and increasing it to 0.5 for large targets), reducing missed detections caused by fixed thresholds (experimental comparison: AP for small target detection improved by 12.7%). The knowledge distillation mechanism constructs a teacher-student model collaborative learning framework, using the improved YOLOv8 model as the teacher model to guide a custom lightweight network as the student model for learning. The improved YOLOv8 network structure introduces a lightweight channel attention module (ECA-Net) in the neck layer, increasing the number of parameters by only 0.8%, but improving multi-scale feature fusion efficiency by 19%, achieving a model mAP of 90.2% (compared to Tiny YOLOv8's 83.5%). Furthermore, on Rockchip chips (cost ≤ 60 yuan), through the collaborative optimization of dynamic label allocation and attention mechanism, the accuracy loss is controlled within 5% (compared to the average accuracy loss of 15%-30% for similar low-cost solutions), achieving the core breakthrough of "lightweight without sacrificing accuracy", breaking through the contradiction of "model accuracy-resource consumption", and solving the problem of increased false negative rate of small targets caused by excessive compression in traditional lightweight models (such as Tiny YOLO).

[0035] S3: Data processing and transmission; S301: Data Preprocessing Pipeline A multi-stage parallel data preprocessing pipeline is constructed. The first stage is an image acquisition buffer queue, employing a circular buffer structure with a size of 16 frames of image data. Double buffering technology enables parallel operations of image acquisition and preprocessing. The second stage is a parallel preprocessing unit containing three parallel processing threads: a scaling thread uses bilinear interpolation to scale the image to 640×640; a normalization thread processes pixel values ​​in parallel using SIMD instructions, normalizing them to the [0,1] range; and an enhancement thread performs data enhancement operations such as random elastic deformation and Gaussian noise addition. The third stage is a preprocessing result buffer queue, also employing a circular buffer, which interfaces with the YOLO model inference queue. Synchronization between threads is achieved through a semaphore mechanism to ensure the efficiency and stability of the data processing flow. This invention proposes a sensor fusion strategy driven by environment perception and dynamic resolution input. This strategy integrates a lighting sensor and a motion detection algorithm, dynamically adjusting the input resolution (e.g., enabling high-resolution detection of 640×640 in well-lit conditions and switching to 416×416 in low-light environments to reduce noise interference). Furthermore, this invention performs adaptive inference frequency, adjusting the detection frame rate based on the target's motion speed (estimated using optical flow) (5 FPS for static scenes and 25 FPS for high-speed motion scenes), reducing computational resource consumption by 37%. Dynamic resolution and frame rate adjustment ensure FPS fluctuation ≤ ±5% in complex scenes (compared to ±20% in Scheme B of Reference 5), and achieves "on-demand computation" through environment perception, avoiding resource waste. This solves the problem of "poor adaptability to dynamic scenes."

[0036] S302: Data transmission optimization An adaptive data transmission protocol is designed. Under good network conditions, the MQTTS protocol is used for reliable transmission, employing a sliding window mechanism to control the data transmission rate. The window size is dynamically adjusted based on the network's RTT (Round-Trip Time). In scenarios with unstable networks or limited bandwidth, the protocol automatically switches to UDP and incorporates FEC (Forward Error Correction) technology for data transmission. FEC encoding divides the original data into multiple blocks, adds redundant check blocks, and transmits them. The receiving end uses the Reed-Solomon algorithm to reconstruct the original data based on the received data blocks and check blocks, ensuring the accuracy of data transmission. Simultaneously, the transmitted data is compressed in blocks using the Zstandard compression algorithm. The compression level is dynamically adjusted according to the data type. For image data, a medium compression level is selected while ensuring real-time performance, achieving a compression ratio of 1:5 to reduce the amount of data transmitted. S303: The edge computing module interacts with hardware devices through the device driver layer. The device driver layer adopts a layered architecture design, consisting of a hardware abstraction layer, a device control layer, and an interface adaptation layer. The hardware abstraction layer encapsulates the register operations of the hardware devices, providing a unified access interface; the device control layer generates control commands according to application requirements, such as camera focus adjustment commands and storage device read / write commands; the interface adaptation layer is responsible for interfacing with the operating system's system call interface. In the target detection process, after the image acquisition module acquires image data, it undergoes data preprocessing through a data preprocessing pipeline and is then sent to the edge computing module via the data transmission module. The edge computing module loads the optimized YOLO model and utilizes GPU acceleration for target detection inference. The inference results are processed by the post-processing module (non-maximum suppression, class confidence filtering) and then output to a local display screen via HDMI for visualization, while also being sent to a remote monitoring center via a network transmission module. Simultaneously, the system periodically monitors the resource usage of the edge computing module. When CPU utilization exceeds 80% or GPU memory usage exceeds 90%, a dynamic model switching mechanism is automatically triggered, switching the improved YOLOv8 model to the lightweight YOLOv5-tiny model to ensure the system's real-time performance and stability.

[0037] This invention employs an RKNN+heterogeneous computing resource scheduling scheme. RKNN model optimization: Utilizing the INT8 quantization capability of the Rockchip chip's built-in NPU, the improved YOLOv8 model is compressed to 1.2MB (original model 4.3MB), reducing memory usage by 72%. CPU / NPU collaborative scheduling algorithm: A task priority queue is designed, allocating object detection preprocessing (such as image normalization) to the CPU, while model inference is executed by the NPU, improving parallelization efficiency to 89% (compared to 45% for a pure CPU solution). This invention, based on the hardware-software co-optimization of domestically produced chips, reduces hardware costs to below 60 RMB (compared to 80 RMB for the Actions triple-core solution), and supports cross-platform deployment (compatibility with Rockchip RK3566 / RK3588 has been verified), breaking down the ecosystem barriers of dedicated hardware and eliminating the problem of strong software dependency.

[0038] S4: System Testing and Optimization; like Figure 5As shown, this implementation method performs performance tests on the deployed edge target monitoring system, including detection accuracy, detection speed, and resource consumption. Multiple detections are performed on the system using a test dataset, and metrics such as target detection accuracy and recall are calculated to evaluate detection accuracy. Detection speed is measured by recording the time it takes for the model to process a single image and calculating the number of frames per second (FPS). Resource consumption, such as CPU usage, GPU usage, and memory usage, of the edge computing module during operation is monitored to comprehensively understand system performance. Based on the performance test results, the system is optimized and adjusted. If the detection accuracy does not meet requirements, the model structure can be further optimized, training data can be increased, or training parameters can be adjusted. If the detection speed is slow, further operations such as quantization and pruning of the model can be performed, or the data processing and transmission process can be optimized. If resource consumption is too high, the software environment can be optimized by closing unnecessary background processes and rationally allocating system resources to ensure stable and efficient operation of the system on edge devices.

[0039] In summary, this invention reconstructs the model operator for Rockchip's NPU instruction set (e.g., replacing Conv2D with DepthwiseConv natively supported by the NPU), improving inference speed to 25 FPS (compared to ESP32's 8 FPS), achieving deep adaptation to domestic chips. Furthermore, this invention processes data entirely locally, eliminating the need for cloud transmission, reducing latency to 35ms (compared to 200ms+ for cloud solutions), and complies with GDPR / CCPA privacy compliance requirements. Achieving mAP of 90.2%+ 25 FPS under a cost constraint of 60 yuan, its overall performance reaches the level of mid-range devices (100-200 yuan level), and it builds a triple barrier of "security-performance-cost" through fully localized deployment.

[0040] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent substitutions, and improvements made to the above embodiments without departing from the scope of the present invention, based on the technical essence of the present invention and within the spirit and principles of the present invention, shall still fall within the protection scope of the present invention.

Claims

1. A lightweight, multi-model collaborative edge intelligent target detection system, characterized in that, include: The edge computing module adopts a multi-core heterogeneous processing architecture and is equipped with a deep learning model. It is used to receive image data from the image acquisition module and perform target detection through the deep learning model. The image acquisition module is used to acquire image data and transmit the acquired image data to the edge computing module; The storage module is used to store the target detection results and acquired image data in an encrypted form.

2. The lightweight multi-model collaborative edge intelligent target detection system according to claim 1, characterized in that, The edge computing module includes n performance cores, n energy efficiency cores, and peripheral interfaces. It is equipped with an integrated GPU and an independent NPU, and is deployed with a deep learning model. The performance core and energy efficiency core are used to preprocess the received image data and schedule the various modules. The peripheral interface is used to connect external devices; The independent NPU is used to invoke a deep learning model to perform object detection on the preprocessed image data; The integrated GPU is used to accelerate the object detection process of the deep learning model.

3. The lightweight multi-model collaborative edge intelligent target detection system according to claim 2, characterized in that, The deep learning model includes the YOLO model and an optimized YOLO model; The YOLO model includes a data preprocessing submodule and a feature extraction submodule; The data preprocessing submodule is used to preprocess the received image data; The feature extraction submodule is used to receive preprocessed image data and perform image detection, and transmit the detection results to the storage module. The optimized YOLO model includes a data preprocessing submodule and an optimized feature extraction submodule; The data preprocessing submodule is used to preprocess the received image data; The optimized feature extraction submodule includes several depthwise separable convolutional layers and a neck network. Each depthwise separable convolutional layer includes a 3×3 depthwise convolutional layer and a 1×1 pointwise convolutional layer. The neck network includes a spatial pyramid pooling-Fast module, a pooling layer with a pooling window of 3×3, and a fully connected layer.

4. The lightweight multi-model collaborative edge intelligent target detection system according to claim 1, characterized in that, The image acquisition module includes a vision acquisition module and an IPS module; The visual acquisition module is used to acquire images and automatically turn on the infrared fill light in low-light scenes. The image acquisition supports MJPEG / H.264 encoding. The IPS module, under the call of the performance core and energy efficiency core, preprocesses the acquired images, encodes the processed images, and uploads them to the edge computing module.

5. The lightweight multi-model collaborative edge intelligent target detection system according to claim 1, characterized in that, The storage module includes an edge cache module and a central device module; The edge cache module uses an LRU strategy to manage storage space and is used to store detected images within a preset time period using the AES-256 encryption algorithm. The central equipment module is used to aggregate image detection results and remotely upload them to the remote monitoring center.

6. A lightweight multi-model collaborative edge intelligent target detection method, applied to the lightweight multi-model collaborative edge intelligent target detection system described in any one of claims 1-5, characterized in that, include: Step 1: Acquire the image to be detected based on the visual acquisition module, and call the IPS module to preprocess the acquired image through the performance core and energy efficiency core in the edge computing module; Step 2: Optimize the YOLO model to obtain the optimized YOLO model, and input the preprocessed image into the deep learning model; Step 3: The independent NPU calls the deep learning model to perform object detection on the preprocessed image data, and the integrated GPU is used to accelerate the detection process; Step 4: Input the test results into the storage module for storage and upload them to the device connected to the peripheral interface for visualization and remote monitoring.

7. The lightweight multi-model collaborative edge intelligent target detection method according to claim 6, characterized in that, Step 1, which involves preprocessing the acquired images, includes: Step 1.1: Scale the acquired image to the input size required by the YOLO model and the optimized YOLO model; Step 1.2: Normalize the resized image, converting the pixel value range from [0,255] to [0,1]; Step 1.3: Perform enhancement operations on the normalized image to complete the preprocessing of the acquired image, and then transmit the preprocessed image to the deep learning model.

8. The lightweight multi-model collaborative edge intelligent target detection method according to claim 6, characterized in that, Step 2, which involves inputting the preprocessed image into the deep learning model, specifically includes: An adaptive data transmission protocol is constructed. When network conditions are good, the MQTTS protocol is used for reliable transmission, employing a sliding window mechanism to control the data transmission rate. The window size is dynamically adjusted based on the network's RTT (Real-Time To Day). In scenarios with unstable networks or limited bandwidth, the protocol automatically switches to UDP and incorporates FEC (Flexible Encoding) technology for data transmission. FEC encoding divides the original data into multiple blocks, adds redundant check blocks, and then transmits the data. A deep learning model uses the Reed-Solomon algorithm to reconstruct the original data based on the received data blocks and check blocks, ensuring the accuracy of data transmission. Simultaneously, the transmitted data is compressed in blocks using the Zstandard compression algorithm, dynamically adjusting the compression level according to the data type. For image data, a medium compression level is selected while ensuring real-time performance.

9. A lightweight multi-model collaborative edge intelligent target detection method according to claim 6, characterized in that, Step 2 involves optimizing the YOLO model, resulting in the optimized YOLO model, which specifically includes: Based on the YOLO model, the ordinary convolutional layers in the CSPDarknet structure of the YOLO model are replaced with depthwise separable convolutional layers, and the Spatial Pyramid Pooling-Fast module is introduced to replace the SPP module, and the pooling window is set to a 3×3 pooling layer. For depthwise separable convolutional layers, 8-bit integer quantization is used to quantize weights and activation values ​​into 8-bit integers. During quantization, quantization parameters are calibrated using a calibration dataset, and quantization error is calculated using KL divergence to dynamically adjust the quantization threshold. For fully connected layers, 16-bit half-precision floating-point quantization is used to balance computational accuracy and speed. CUDA graph technology is used to optimize the inference process of the quantized model, merging multiple inference operations into a single CUDA graph. The training mechanism was optimized by employing a designed incremental training mechanism based on transfer learning. In the initial training phase, the backbone network parameters of the YOLO model were frozen, and only the detection head part was trained to learn general target features. During the custom dataset training phase, the shallow parameters of the backbone network were gradually unfrozen, and a cosine annealing learning rate strategy was adopted, dynamically adjusting the learning rate according to the training epochs. The initial learning rate was set to 0.001, and during training, the learning rate gradually decreased to 0.00001 according to the cosine function. A knowledge distillation mechanism was introduced during training. In the knowledge distillation training process, the optimized YOLO model was used as the teacher model, and the YOLO model was used as the student model, constructing a teacher-student model collaborative training framework. The student model learned hard labels from the original training data, and a distillation loss function was introduced. The system guides student models to learn the soft knowledge output by the teacher model, dynamically adjusts temperature parameters, and incorporates the standard cross-entropy loss of the student model. and distillation loss function Constructing a joint loss function After distillation training, a dynamic fusion strategy for data augmentation is introduced. The probability of using data augmentation methods is dynamically adjusted according to the data distribution. For target categories with fewer samples, augmentation probability operation is adopted, and the positive and negative sample matching thresholds are adaptively adjusted according to the target size, occlusion degree and scene complexity. The expression for the joint loss function is: （1）； In formula (1), The standard cross-entropy loss for the student model, Distillation loss as a measure of Kullback-Leibler (KL) divergence. and This is a dynamically adjusted coefficient.

10. A lightweight multi-model collaborative edge intelligent target detection method according to claim 6, characterized in that, Step 3 involves calling a deep learning model to perform object detection on the preprocessed image data, specifically including: The system adaptively selects between the YOLO model and the optimized YOLO model based on the performance requirements of the scenario. The YOLO model is selected in low-performance scenarios, while the optimized YOLO model is selected in high-performance scenarios. Target detection is performed using GPU acceleration. The detection results are transmitted to the connected device for visualization through the peripheral interface and remotely transmitted to the remote monitoring center through the central device module. Furthermore, when the CPU utilization exceeds 80% or the GPU memory usage exceeds 90%, the dynamic model switching mechanism is automatically triggered to automatically switch between the YOLO model and the optimized YOLO model.

Images