Intelligent internet of things edge computing terminal and dynamic feature recognition method

Abstract

The application relates to the technical field of Internet of Things edge computing and discloses an intelligent Internet of Things edge computing terminal and a dynamic feature recognition method. The terminal comprises a main control board, an NPU, a sensor interface, a dynamic feature recognition module, a self-adaptive learning module, a security encryption module and a cloud-edge cooperation module. Through the cooperative design of hardware and software, low-delay and high-precision dynamic feature recognition is realized at the edge side, and the terminal has the capability of locally updating the model according to environmental changes. The application solves the problems of insufficient dynamic feature recognition capability, high delay, great bandwidth pressure and high privacy security risk in the existing edge computing, and can be widely applied to the fields of intelligent security, industrial Internet of Things and smart city.

Description

Technical Field

[0001] This application relates to the field of IoT edge computing technology, and in particular to an intelligent IoT edge computing terminal and a dynamic feature recognition method. Background Technology

[0002] In recent years, the Internet of Things (IoT), 5G communication and artificial intelligence technologies have developed rapidly. The efficient processing of massive amounts of data has become the core challenge for the digital upgrading of industries. The traditional "cloud computing center" model has many technical bottlenecks and can no longer meet the application needs of the IoT era.

[0003] The high latency issue is particularly prominent. In key areas such as industrial quality control, autonomous driving, and public safety monitoring, taking 1080P images as an example, the process from terminal transmission to cloud processing and return of instructions takes hundreds of milliseconds to several seconds even in an ideal network environment. This cannot meet the millisecond-level response requirements and may lead to defective products or safety accidents. Furthermore, the raw data such as high-definition video streams consume extremely high network bandwidth. Hundreds of cameras in a medium-sized factory can consume the dedicated bandwidth of a company's dedicated line, and a large amount of redundant data consumes cloud storage and computing resources, causing communication and operating costs to soar, resulting in enormous bandwidth and cost pressures. Secondly, data involving personal privacy, corporate production secrets, or national geographic information faces the risk of theft, tampering, and leakage during transmission over public networks and centralized cloud storage. Moreover, cross-border data flows raise legal and sovereignty issues, leading to… Increased privacy and security risks, coupled with the fact that most existing edge computing devices are equipped with static, fixed models that cannot adapt to environmental changes such as lighting variations, target appearance evolution, and dynamic scene adjustments, lead to a decrease in recognition accuracy (i.e., "model drift"). Relying on cloud retraining would lead back to high latency and high bandwidth consumption, resulting in insufficient dynamic recognition capabilities of edge devices. For example, patent CN201810001234A only targets static image recognition and lacks real-time streaming media parsing and model optimization mechanisms; patent US20190056789B1 focuses on data preprocessing and lacks complex deep learning models and advanced feature recognition capabilities; and patent JP2020105678C's edge terminal is only responsible for inference, with model updates entirely dependent on the cloud and lacking autonomous adaptive adjustment capabilities. To address these issues, an intelligent IoT edge computing terminal and a dynamic feature recognition method are proposed. Summary of the Invention

[0004] This application proposes an intelligent IoT edge computing terminal and a dynamic feature recognition method, which has the advantages of extreme real-time performance, ultra-high broadband efficiency, inherent privacy and security, strong environmental adaptability and high deployment flexibility, in order to solve the technical problems mentioned in the background art.

[0005] To achieve the above objectives, this application adopts the following technical solution: an intelligent IoT edge computing terminal, comprising a terminal body, a main control board, and functional modules. The terminal body consists of a core computing architecture, a perception and interaction layer, and software functional modules. The main control board includes an integrated main processor and a neural network processing unit (NPU), forming a heterogeneous computing core. The main processor adopts an ARM Cortex-A76 architecture or a Rockchip RK3588 chip and is responsible for system scheduling and complex logic processing. The NPU has a computing power of not less than 6 TOPS and is dedicated to executing parallel computing of deep learning models. The perception and interaction layer includes a sensor interface and a network communication interface. The sensor interface is a standardized multi-protocol interface used to realize multimodal environmental data acquisition. The network communication interface supports Wi-Fi 6 and 5G communication protocols. The software functional modules include a dynamic feature recognition module, an adaptive learning module, a security encryption module, and a cloud-edge collaboration module. The core of the dynamic feature recognition module is a lightweight deep learning model deployed on the NPU. The adaptive learning module is the key to realizing the intelligent evolution of the terminal. The adaptive learning module has a built-in incremental learning algorithm. The physical carrier of the security encryption module is the trusted platform module security chip. The cloud-edge collaboration module adopts an asynchronous communication mode and a message queuing mechanism.

[0006] Furthermore, a dynamic feature recognition method for an intelligent IoT edge computing terminal is provided. This method tightly integrates hardware capabilities with software algorithms to form a closed-loop intelligent processing flow, including the following steps: S1: Multi-source data acquisition, synchronously or asynchronously acquiring multimodal data from the environment in real time through the sensor interface; S2: Data preprocessing and enhancement, which involves standardizing and preprocessing the collected raw data; S3: Real-time dynamic inference, calling a lightweight deep learning model deployed on the NPU to perform forward propagation inference on the preprocessed data; S4: Edge-side intelligent decision-making and execution. Based on the recognition results generated by S3, the preset decision logic is immediately executed at the edge. S5: Online model adaptation and fine-tuning, through the adaptive learning module, the online learning process is started periodically (e.g., during daily off-peak hours); S6: Secure and efficient cloud-edge collaboration. Through the cloud-edge collaboration module, the model weight update amount calculated in S5 or the key event statistical summary after de-identification is encrypted and asynchronously uploaded to the cloud.

[0007] Furthermore, the main processor of the main control board includes four Cortex-A76 large cores (maximum frequency 2.4GHz) and four Cortex-A55 small cores (maximum frequency 1.8GHz), supports dual-channel LPDDR4 / LPDDR5 memory (maximum capacity 32GB) and eMMC 5.1, SPI NAND Flash, and the NPU supports INT8 / INT16 / FP16 mixed quantization calculation, with a dedicated memory bandwidth of 20GB / s.

[0008] Furthermore, the visual sensor interface of the perception and interaction layer provides 4 MIPI CSI-2 interfaces.

[0009] Furthermore, in step 3, the model inference is optimized using INT8 symmetric quantization, operator fusion, and zero-copy memory sharing techniques, with a single frame image or single audio inference time of 20-30 milliseconds. In step 5, the fine-tuning process combines knowledge distillation technology, using soft labels generated by the teacher model to guide student model training.

[0010] Furthermore, the cloud-edge collaboration in step 6 adopts a federated learning framework, allowing the terminal to participate in global model training without leaking its local raw data.

[0011] Furthermore, the dynamic feature recognition includes at least one of target detection, behavior recognition, and abnormal event detection, with an end-to-end latency of less than 100ms.

[0012] This application provides an intelligent IoT edge computing terminal and a dynamic feature recognition method. Through the hardware and software collaborative technical solution designed in this application, it achieves extreme real-time performance, with end-to-end latency below 100 milliseconds, meeting industrial-grade control requirements. By co-optimizing a high-performance NPU and a lightweight deep learning model, complex feature recognition calculations are migrated from the cloud to the edge where data is generated. The NPU undergoes hardware-level optimization for the parallel computing characteristics of model inference, reducing the inference time for a single frame of image or a single audio recording to 20-30 milliseconds. Combined with an efficient data preprocessing pipeline, the end-to-end latency of the entire "acquisition-processing-recognition-decision" process is strictly controlled within 100 milliseconds. This represents an order-of-magnitude improvement compared to the latency of traditional cloud center solutions, which can easily reach hundreds of milliseconds to seconds. It can be directly applied to scenarios with stringent real-time requirements, such as real-time quality inspection of industrial robotic arms, foreign object recognition on high-speed conveyor belts, and sensor fusion for autonomous driving.

[0013] Secondly, the edge terminal of this application no longer uploads raw, highly redundant streaming media data (such as unprocessed video streams). Instead, it transmits highly compressed structured data to the cloud only through the cloud-edge collaboration module. This mainly includes two types: feature summaries and model increments. The feature summaries are used to provide a structured description of the recognition results (such as timestamps, locations, event types, confidence levels, etc.), and their data volume is only one percent or even one-thousandth of the original video stream. In federated learning mode, the model increments are the encrypted and compressed model parameter updates (gradients), rather than the original data itself. Through the above design, the uplink network traffic generated by a continuously operating 1080P camera using this solution can be reduced by 90% to 95% compared to the traditional solution of uploading video streams throughout the entire process, which greatly alleviates network bandwidth pressure and reduces communication costs.

[0014] Meanwhile, this application improves security performance, providing end-to-end protection from transmission to storage. By constructing a dual security system of localized computing and hardware encryption, sensitive raw data (such as faces, license plates, and production process videos) is processed and analyzed within the terminal, avoiding the risk of theft or tampering during transmission over public networks. Furthermore, the hardware device uses the TPM2.0 security chip, providing a unique identifier for the device and hardware encryption for the model, user configuration, and log data stored locally on the terminal. Combined with a secure boot mechanism, this ensures the integrity and trustworthiness of the system software stack and effectively defends against underlying firmware attacks.

[0015] Meanwhile, this application breaks through the limitations of static models by introducing an online incremental learning mechanism on the edge side. When the deployment environment changes (such as seasonal changes leading to changes in lighting, the emergence of new categories of monitored targets, or new failure modes of equipment), the adaptive learning module can use newly generated local data to fine-tune the model in a targeted manner. This mechanism enables the model to continuously learn and adapt to the dynamic characteristics of its specific environment, thereby effectively combating the "model drift" problem. In long-term testing in actual deployment, compared with static models that are not updated after deployment, the model with adaptive capabilities can maintain a stable average recognition accuracy (mAP) and improve by 15% to 30% after running for several weeks, significantly reducing false positives and false negatives, thus enhancing the environmental adaptability.

[0016] Secondly, the terminal in this application adopts a modular, standardized interface and containerized software architecture, providing multiple standard sensor interfaces and network interfaces, enabling rapid connection to different types of sensing devices and network environments. Core algorithms and applications are encapsulated in containers, supporting one-click deployment and remote updates. Combined with the aforementioned cloud-edge collaboration module, it enables centralized management of large-scale terminal groups and algorithm canary releases. This design allows the terminal to be seamlessly and quickly deployed in diverse scenarios such as traffic intersections in smart cities, production lines in factories, security systems in industrial parks, and monitoring points in farmland. It achieves flexible deployment capabilities of "develop once, deploy anywhere," significantly reducing the implementation and maintenance costs of large-scale applications, and possesses high deployment flexibility, supporting rapid adaptation to multiple IoT scenarios. Attached Figure Description

[0017] The accompanying drawings, which form part of this specification, illustrate embodiments disclosed in this application and, together with the specification, serve to explain the principles disclosed in this application.

[0018] This disclosure will become clearer with reference to the accompanying drawings and the following detailed description, wherein: Figure 1 This is a flowchart of the image and audio collection algorithm of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0020] Please see Figure 1 A smart IoT edge computing terminal and dynamic feature recognition method are disclosed, comprising a terminal body, a main control board, and functional modules. The terminal body consists of a core computing architecture, a perception and interaction layer, and software functional modules. The main control board includes an integrated main processor and a neural network processing unit (NPU), forming a heterogeneous computing core. The main processor adopts an ARM Cortex-A76 architecture or a Rockchip RK3588 chip, which is responsible for system scheduling and complex logic processing. The NPU has a computing power of no less than 6 TOPS and is an AI-dedicated acceleration engine, which is used to execute parallel computing of deep learning models. It can complete model inference with power consumption and latency far lower than that of the CPU, and is the computing power cornerstone for realizing real-time dynamic feature recognition.

[0021] An intelligent IoT edge computing terminal and dynamic feature recognition method are disclosed. This method tightly integrates hardware capabilities with software algorithms to form a closed-loop intelligent processing flow, including the following steps: S1: Multi-source data acquisition, synchronously or asynchronously acquiring multimodal data from the environment in real time through sensor interfaces, such as high-definition video streams, ambient audio streams, and physical sensor readings; S2: Data preprocessing and augmentation. The raw data is standardized and preprocessed. For video frames, this includes color space conversion (such as YUV2RGB), image normalization, and random cropping / flipping (for online data augmentation). For audio data, this involves pre-emphasis, frame segmentation, windowing, and conversion to Mel spectrograms. This step aims to improve data quality and enhance the model's generalization ability. S3: Real-time dynamic inference, which calls a lightweight deep learning model deployed on the NPU to perform forward propagation inference on the preprocessed data. Thanks to the parallel computing architecture of the NPU and the lightweight design of the model, this step can be completed in tens of milliseconds, realizing real-time feature extraction and recognition of dynamic scenes, and outputting structured recognition results (such as target category, location coordinates, confidence score). S4: Edge-side intelligent decision-making and execution. Based on the recognition results generated by S3, the preset decision logic is immediately executed at the edge. For example, when "safety helmet wearing violation" is detected, the on-site broadcast alarm is directly triggered. When abnormal equipment vibration is detected, control signals are immediately sent to adjust the equipment operating parameters, realizing closed-loop autonomy of "perception, decision-making and execution", completely eliminating the real-time dependence on the cloud. S5: Online model adaptation and fine-tuning. Through the adaptive learning module, the online learning process is started regularly (such as during daily off-peak hours). Using recently collected local data with (automatic or manual) labels, the model is iterated and fine-tuned for one or several rounds with a small learning rate, so that the model parameters can better fit the dynamic characteristics of the current environment, and realize the personalization of the model and performance self-optimization. S6: Secure and efficient cloud-edge collaboration. Through the cloud-edge collaboration module, the model weight update amount calculated in S5 or the key event statistical summary after de-identification is encrypted and asynchronously uploaded to the cloud. The cloud aggregates information from numerous edge terminals, performs global model optimization, and then distributes the optimized incremental update package. After verifying its security and integrity, the terminal applies it to its local model, thereby achieving global knowledge sharing and continuous co-evolution of edge intelligence.

[0022] The perception and interaction layer includes sensor interfaces and network communication interfaces. The sensor interfaces are standardized multi-protocol interfaces (such as MIPI-CSI for cameras and I2C / SPI for other sensors) used to connect devices such as high-definition cameras, microphone arrays, and temperature and humidity sensors to form the terminal's sensory system. They are mainly used to realize multimodal environmental data acquisition. The network communication interface supports Wi-Fi 6 and 5G communication protocols to ensure that the terminal can obtain high-speed, low-latency network connections in various deployment environments, providing communication guarantees for efficient cloud-edge collaboration.

[0023] The software functional modules include a dynamic feature recognition module, an adaptive learning module, a security encryption module, and a cloud-edge collaboration module. The core of the dynamic feature recognition module is a lightweight deep learning model deployed on the NPU, selected from MobileNetV3 or EfficientNet-Lite, used for real-time parsing of streaming media data and feature recognition. These models, through advanced neural network structures such as depthwise separable convolutions and channel attention mechanisms, significantly reduce the number of parameters and computational load while maintaining high accuracy. This makes them ideal for real-time parsing of streaming media data such as video and audio streams on resource-constrained edge devices, performing tasks such as object detection, behavior recognition, and abnormal event detection. The adaptive learning module is key to achieving intelligent evolution of the terminal. It incorporates an incremental learning algorithm that can periodically or triggerively fine-tune the lightweight deep learning model using locally collected new data. This process typically involves freezing some lower network layers and updating only the top layer. The strategy of classifiers or specific task layers aims to efficiently utilize new knowledge while effectively preventing "catastrophic forgetting" of learned knowledge. The physical carrier of the security encryption module is the trusted platform module security chip, which provides hardware-level security functions, including: providing a unique hardware identity for the device; generating and storing encryption keys in a secure area; transparently encrypting locally stored model files and sensitive user data; and ensuring the integrity and trustworthiness of the system software chain through a secure boot mechanism. The cloud-edge collaboration module adopts an asynchronous communication mode and message queuing mechanism. The cloud-edge collaboration module is responsible for managing all communication with the cloud server, uploading the terminal's local model parameter increments, model performance indicators, and non-privacy feature summaries of key events to the cloud. At the same time, it receives the aggregated and optimized incremental model update package from the cloud and securely completes the model update during the terminal's idle time. The cloud-edge collaboration module preferably supports the federated learning framework, enabling the terminal to participate in the collaborative training of the global model without leaking any local raw data.

[0024] The main processor of the control board includes four Cortex-A76 large cores (maximum frequency 2.4GHz) and four Cortex-A55 small cores (maximum frequency 1.8GHz). Each A76 core is equipped with 512KB L2 cache, each A55 core is equipped with 256KB L2 cache, and they share 3MB L3 cache. It supports dual-channel LPDDR4 / LPDDR5 memory (maximum capacity 32GB) and eMMC 5.1, SPI NAND Flash, and supports NVMe SSD expansion through PCIe 3.0 interface. The NPU supports INT8 / INT16 / FP16 mixed quantization calculation, integrates an independent dual-core NPU, fully supports more than 120 common AI operators, has a dedicated memory bandwidth of 20GB / s, and a peak computing power of 6 TOPS (INT8). It supports the simultaneous execution of multiple neural network models.

[0025] The visual sensor interface of the perception and interaction layer provides four MIPI CSI-2 interfaces for connecting cameras, each of which supports up to 4K@60fps data acquisition.

[0026] In step 3, the model inference is optimized using INT8 symmetric quantization, operator fusion, and zero-copy memory sharing techniques. The inference time for a single frame image or a single audio is 20-30 milliseconds. In step 5, the fine-tuning process combines knowledge distillation techniques, using soft labels generated by the teacher model to guide student model training.

[0027] The cloud-edge collaboration in step 6 adopts a federated learning framework, allowing the terminal to participate in global model training without leaking its local raw data.

[0028] Dynamic feature recognition includes at least one of target detection, behavior recognition, and abnormal event detection, with an end-to-end latency of less than 100ms.

[0029] This application first acquires video using the V4L2 framework, acquires audio using ALSA, and reads sensor data through the IIO subsystem. Next, the acquired data is preprocessed: video frames undergo YUV→RGB conversion, normalization, and resizing; audio data is processed by extracting Mel spectrograms. The RKNN-Toolkit2 is used to call the NPU to perform model inference. Post-processing employs non-maximum suppression to filter the results, and a rule engine is used to match and identify the results, triggering GPIO control alarms or device control signals for local decision-making. The underlying network of the model is frozen, and the top layer is fine-tuned with a learning rate of 1e-4, combined with knowledge distillation techniques for incremental learning. The incremental model is compressed, encrypted, and uploaded. After receiving and decrypting the cloud update package, the local model is updated. This supports federated learning for local training and weight update calculation, enabling cloud-edge collaboration. Furthermore, real-time data collection of CPU / memory usage, NPU temperature, and other status data enables daily monitoring.

Claims

1. An intelligent internet of things edge computing terminal, comprising a terminal body, a main control board and a function module, characterized in that: The terminal body consists of a core computing architecture, a perception and interaction layer, and software functional modules. The main control board includes an integrated main processor and a neural network processing unit (NPU) to form a heterogeneous computing core. The main processor adopts an ARM Cortex-A76 architecture or a Rockchip RK3588 chip and is responsible for system scheduling and complex logic processing. The NPU has a computing power of no less than 6 TOPS and is dedicated to parallel computing of deep learning models. The perception and interaction layer includes a sensor interface and a network communication interface. The sensor interface is a standardized multi-protocol interface used to realize multimodal environmental data acquisition. The network communication interface supports Wi-Fi 6 and 5G communication protocols. The software functional modules include a dynamic feature recognition module, an adaptive learning module, a security encryption module, and a cloud-edge collaboration module. The core of the dynamic feature recognition module is a lightweight deep learning model deployed on the NPU. The adaptive learning module is the key to realizing the intelligent evolution of the terminal. The adaptive learning module has a built-in incremental learning algorithm. The physical carrier of the security encryption module is the trusted platform module security chip. The cloud-edge collaboration module adopts an asynchronous communication mode and a message queuing mechanism. 2.The dynamic feature recognition method of the intelligent Internet of Things edge computing terminal according to claim 1, characterized in that, This method tightly integrates hardware capabilities with software algorithms to form a closed-loop intelligent processing flow, including the following steps: S1: Multi-source data acquisition, synchronously or asynchronously acquiring multimodal data from the environment in real time through the sensor interface; S2: Data preprocessing and enhancement, which involves standardizing and preprocessing the collected raw data; S3: Real-time dynamic inference, calling a lightweight deep learning model deployed on the NPU to perform forward propagation inference on the preprocessed data; S4: Edge-side intelligent decision-making and execution. Based on the recognition results generated by S3, the preset decision logic is immediately executed at the edge. S5: Online model adaptation and fine-tuning, through the adaptive learning module, the online learning process is started periodically (e.g., during daily off-peak hours); S6: Secure and efficient cloud-edge collaboration. Through the cloud-edge collaboration module, the model weight update amount calculated in S5 or the key event statistical summary after de-identification is encrypted and asynchronously uploaded to the cloud. 3.The dynamic feature recognition method of the intelligent Internet of Things edge computing terminal according to claim 2, characterized in that, The main processor of the main control board includes four Cortex-A76 large cores (maximum frequency 2.4GHz) and four Cortex-A55 small cores (maximum frequency 1.8GHz), supports dual-channel LPDDR4 / LPDDR5 memory (maximum capacity 32GB) and eMMC 5.1, SPINAND Flash, and the NPU supports INT8 / INT16 / FP16 mixed quantization calculation, with a dedicated memory bandwidth of 20GB / s. 4.The dynamic feature recognition method of the intelligent Internet of Things edge computing terminal according to claim 3, characterized in that, The visual sensor interface of the perception and interaction layer provides 4 MIPI CSI-2 interfaces. 5.The dynamic feature recognition method of the intelligent internet-of-things edge computing terminal according to claim 4, characterized in that, In step 3, the model inference is optimized using INT8 symmetric quantization, operator fusion, and zero-copy memory sharing techniques. The inference time for a single frame image or a single audio recording is 20-30 milliseconds. In step 5, the fine-tuning process combines knowledge distillation techniques, using soft labels generated by the teacher model to guide student model training. 6.The dynamic feature recognition method of the intelligent internet-of-things edge computing terminal according to claim 5, characterized in that, The cloud-edge collaboration in step 6 adopts a federated learning framework, in which the terminal participates in global model training without leaking local raw data. 7.The dynamic feature recognition method of the intelligent internet-of-things edge computing terminal according to claim 6, characterized in that, The dynamic feature recognition includes at least one of target detection, behavior recognition, and abnormal event detection, with an end-to-end latency of less than 100ms.

Images