NPU-based lightweight neural network target detection, positioning and tracking system
By using lightweight neural networks and reinforcement learning techniques based on NPU, the problems of large hardware size, high power consumption, and slow response of traditional systems in portable devices are solved, achieving miniaturization, low power consumption, and real-time target detection, localization, and tracking, thus improving the system's adaptability in complex scenarios.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-25
- Publication Date
- 2026-03-03
AI Technical Summary
Traditional target detection, localization and tracking systems suffer from problems such as large hardware size, high power consumption, slow response and poor adaptability to complex scenarios in portable and miniaturized devices. Existing technologies have failed to fully utilize the characteristics of dedicated hardware, resulting in the model performance not being fully realized and the data sample coverage being limited.
By employing a lightweight neural network based on an NPU, combined with reinforcement learning and data augmentation techniques, and through hardware and software co-design, the model architecture is optimized to achieve system miniaturization, low power consumption, real-time performance, and adaptive capabilities.
The system achieves miniaturization, low power consumption, and real-time performance, improving detection and tracking stability in complex scenarios and adapting to diverse scenario requirements.
Smart Images
Figure CN121600379A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer vision and artificial intelligence application technology, specifically a lightweight neural network target detection, localization and tracking system based on NPU (Neural Processing Unit). Background Technology
[0002] In practical applications of computer vision technology, target detection, localization, and tracking are key components for achieving scene perception and intelligent decision-making, and have been widely used in fields such as security, consumer electronics, and transportation. However, traditional target detection, localization, and tracking systems still face many technical bottlenecks in their implementation, making it difficult to meet the needs of diverse scenarios.
[0003] Traditional systems primarily rely on general-purpose processors (CPUs) or graphics processing units (GPUs) for computation. While CPU-based systems offer strong compatibility, they lack sufficient support for parallel computing in neural networks. When handling complex detection tasks, they often fail to achieve real-time response and consume significant power over extended periods, making them unsuitable for portable and miniaturized devices. GPU-based systems offer improved processing speeds, but their large size and high cost, coupled with unresolved power consumption issues, make them unsuitable for size- and power-sensitive applications such as miniature surveillance and wearable devices.
[0004] With the development of lightweight neural network technology, some systems have attempted to reduce hardware dependence by compressing model parameters. However, existing technologies mostly focus on software-level model optimization, failing to fully leverage the computational characteristics of dedicated hardware, resulting in underperformance of the models. Furthermore, traditional systems exhibit poor stability in complex scenarios (such as sudden changes in lighting, target occlusion, and background interference), lacking adaptive adjustment mechanisms. Data samples largely rely on offline collection and annotation, resulting in limited coverage and insufficient model generalization ability. In terms of energy consumption control, a hardware-software co-optimization scheme has not been developed, failing to meet the requirements for low-power operation.
[0005] Therefore, there is an urgent need for a target detection, localization and tracking system that integrates the advantages of dedicated hardware, optimizes the model architecture, has adaptive capabilities, and meets the requirements of miniaturization, low power consumption and real-time performance, so as to break through the limitations of existing technologies and promote the practical application of target detection technology in more scenarios. Summary of the Invention
[0006] This invention aims to provide a lightweight neural network target detection, localization and tracking system based on an NPU. Through the collaborative design of hardware selection and software architecture, it integrates reinforcement learning functions and data augmentation technology to solve the problems of limited deployment, high power consumption, slow response and poor adaptability to complex scenarios in traditional systems. It achieves system miniaturization, intelligence, low power consumption operation and meets real-time requirements.
[0007] The technical solution adopted by the present invention to achieve the above objectives is as follows:
[0008] A lightweight neural network-based target detection, localization, and tracking method based on an NPU includes the following steps:
[0009] 1) Acquire scene images and preprocess them, converting them into a format recognizable by the NPU;
[0010] 2) Perform data augmentation on the preprocessed image;
[0011] 3) Use a lightweight neural network deployed on the NPU to classify and locate targets in the enhanced image to obtain target location and category information;
[0012] 4) Use historical tracking data to assess the stability of the neural network's detection results, and adjust the neural network's parameters based on the assessment results to optimize the neural network.
[0013] Step 2) specifically refers to:
[0014] The image is enhanced by randomly selecting one of the following methods: horizontal flipping, rotation, and brightness adjustment.
[0015] Before detection, the lightweight neural network needs to be optimized, including pruning and layer fusion.
[0016] The lightweight neural network sequentially extracts target features, classifies and locates the input image to obtain preliminary target location information and category information.
[0017] Step 4) specifically refers to:
[0018] Based on historical tracking data, the magnitude of change in the target bounding box and the category confidence in the current detection result are judged. If both the magnitude of change in the target bounding box and the category confidence meet the threshold conditions, it is considered a valid result; otherwise, the model detection threshold and tracking box update strategy in the neural network are adjusted through reinforcement learning so that the output detection result meets the threshold conditions for the magnitude of change in the target bounding box and the category confidence, thereby optimizing the detection result.
[0019] A lightweight neural network-based target detection, localization, and tracking system based on an NPU includes:
[0020] The image acquisition and preprocessing module is used to acquire scene images, preprocess them, and convert them into an NPU-recognizable format;
[0021] The data augmentation module is used to augment the data of the preprocessed image;
[0022] The NPU core unit is used to classify and locate targets in the enhanced image using a lightweight neural network deployed on the NPU, and obtain target location information and category information.
[0023] The reinforcement learning module is used to assess the stability of the neural network's detection results using historical tracking data, and to adjust the neural network's parameters and optimize the neural network based on the assessment results.
[0024] It also includes a result output unit consisting of a communication module and a display module, used for the presentation and interaction of detection results.
[0025] It also includes storage units for storing programs, model parameters, and historical data.
[0026] The present invention has the following beneficial effects and advantages:
[0027] 1. Miniaturization: Relying on the compact hardware design of the NPU and the low resource requirements of the lightweight neural network, the overall hardware size of the system is reduced by more than 60% compared with the traditional GPU solution, and it can be deployed in miniaturized scenarios such as portable devices and micro monitoring devices;
[0028] 2. Low power consumption: The NPU's high efficiency in neural network operations, combined with software-level computational optimizations, reduces system power consumption by more than 50% compared to traditional CPU solutions, extending device battery life (e.g., portable devices can run continuously for more than 8 hours on a single charge).
[0029] 3. Real-time performance: The combination of hardware computing acceleration and software process simplification keeps the system's end-to-end response latency within an acceptable range for practical applications, meeting the needs of real-time interaction (such as when tracking dynamic targets, the screen updates without noticeable lag).
[0030] 4. Intelligence: The reinforcement learning module enables the system to autonomously adapt to complex scenarios, while the data augmentation module improves the model's generalization ability. The combination of the two allows the system to maintain stable detection and tracking performance in scenarios such as strong light, low light, and target occlusion.
[0031] 5. Versatility: The system hardware and software modules are compatible, and the model parameters and hardware configuration can be adjusted according to different application scenarios (such as monitoring, vehicle-mounted, and terminal equipment) (such as replacing image acquisition modules with different resolutions), making it widely applicable. Attached Figure Description
[0032] Figure 1 A block diagram of a lightweight neural network target detection, localization, and tracking system based on an NPU;
[0033] Figure 2A flowchart of a lightweight neural network target detection, localization and tracking system based on NPU. Detailed Implementation
[0034] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.
[0035] A lightweight neural network-based target detection, localization, and tracking system based on an NPU is described in detail from three dimensions: hardware architecture, software model, and functional modules.
[0036] (1) Hardware architecture design
[0037] The NPU serves as the core computing unit of the system, replacing the traditional CPU or GPU. Designed specifically for neural network computation, the NPU possesses parallel processing capabilities, efficiently handling core operations such as convolution and pooling in lightweight neural networks. This significantly reduces system power consumption while improving computational speed. It is paired with miniaturized peripheral hardware: employing micro-image acquisition modules (such as miniature cameras) to capture scene images, using low-power storage modules to store programs and data, and configuring compact power supply modules (such as small lithium batteries), thus controlling the overall hardware size and meeting the requirements for miniaturized deployment.
[0038] (2) Lightweight Neural Network Optimization
[0039] A lightweight neural network architecture adapted to the computing characteristics of the NPU (such as the MobileNet series and YOLO-Lite series) is selected as the base model. Optimization techniques such as model pruning (removing redundant convolutional kernels) and layer fusion (merging adjacent computational layers) are used to reduce redundant parameters and computational complexity. The optimized model, while maintaining detection and localization accuracy, can quickly adapt to the NPU's computational logic, reducing data transmission and computation time, thus supporting the system's real-time performance.
[0040] (3) Integration of core functional modules
[0041] Data Augmentation Module: Sets up online data augmentation functions. When the system is running, it automatically performs operations such as image flipping, angle rotation, brightness and contrast adjustment on real-time acquired image data, dynamically expands the sample types, improves the model's adaptability to different lighting and angle scenes, and solves the problem of insufficient coverage of traditional offline samples.
[0042] Reinforcement Learning Module: This module introduces a reinforcement learning mechanism, using the accuracy of target detection and localization (e.g., class confidence) and the continuity of tracking (e.g., the magnitude of bounding box changes) as reward signals to adjust the model's detection threshold (e.g., confidence threshold) and tracking box update strategy (e.g., interpolation update, prediction update) in real time. When encountering complex situations such as target occlusion or sudden changes in lighting, the system autonomously optimizes its decisions through reinforcement learning to ensure tracking stability and improve its intelligence level.
[0043] Real-time optimization module: Reduce computation and data processing latency by optimizing NPU computing task scheduling (such as prioritizing core detection layers) and simplifying data transmission paths (such as reducing intermediate caching steps); at the same time, streamline the model inference process and prioritize key detection areas in the image (such as areas where targets may appear) to further shorten response time and ensure that the system meets real-time requirements.
[0044] like Figure 1 As shown in the structural block diagram, the present invention provides a lightweight neural network target detection, localization and tracking system based on NPU, which mainly includes an image acquisition unit, a preprocessing module, a data augmentation module, an NPU core unit (lightweight neural network), a reinforcement learning module, a result output unit, a storage unit and a power supply module.
[0045] The functions of each unit module are as follows:
[0046] Image acquisition unit: Acquires scene images in real time to provide raw data for the system; the core component is a miniature image sensor.
[0047] Preprocessing module: Normalizes and standardizes the original image, converts it into an NPU-recognizable format, and eliminates the impact of data differences on detection;
[0048] Data augmentation module: Expands sample types through dynamic augmentation operations, improves model generalization ability, and can be flexibly started and stopped according to the scenario;
[0049] NPU Core Unit: The core computing unit of the system, which runs a lightweight neural network to complete target detection and localization, and improves efficiency and reduces power consumption by relying on parallel computing;
[0050] Reinforcement learning module: Optimizes strategies based on detection results and historical data to solve tracking stability issues in complex scenarios and improve the level of intelligence;
[0051] The result output unit consists of a communication module (wireless transmission) and a display module (external device), which realizes the presentation and interaction of the detection results.
[0052] Storage unit: Stores programs, model parameters, and historical data, providing data support for system operation and optimization;
[0053] Power module: Provides stable power to each module, reduces energy consumption and extends device battery life through dynamic power adjustment.
[0054] like Figure 2 As shown in the system flowchart, the overall process of this system is as follows: system startup → hardware self-test and initialization → image acquisition → image preprocessing → online data augmentation → NPU model inference → reinforcement learning strategy optimization → output detection and tracking results → record result data → loop image acquisition.
[0055] The specific process is as follows: After the system starts, it first performs a hardware self-test to ensure that each module is normal, and then loads parameters to complete the initialization, laying the foundation for subsequent operation; in the image acquisition and preprocessing stage, the original image is converted into standardized data to adapt to the NPU computing requirements and ensure data quality; in the online data augmentation stage, the sample types are dynamically expanded to balance efficiency and diversity and improve the model's adaptability to different scenarios.
[0056] After image information processing is complete, NPU model inference is the core computational stage. It directly outputs the target location and category through a lightweight neural network, which is crucial for detection and localization. In the reinforcement learning strategy optimization stage, the strategy is adjusted based on historical data to solve tracking problems in complex scenarios and ensure stable results. The calculation results are fed back to the NPU, and parameters are updated based on the gradient descent algorithm to achieve model iteration. Finally, the result output and recording stage enables data interaction and reference for subsequent optimization. At the same time, the system enters a loop to continuously complete the detection and tracking task.
[0057] Example
[0058] 1. System Hardware Composition and Selection
[0059] The system hardware consists of five core units, and the composition and selection of each unit are as follows:
[0060] (1) NPU core unit: a dedicated low-power NPU chip is selected to support the acceleration of common lightweight neural networks (such as MobileNetV3 and YOLOv5-Lite), and has the ability to process data in parallel through multiple channels. The hardware size is controlled within 20mm×20mm to meet the needs of miniaturized deployment.
[0061] (2) Image acquisition unit: It adopts a miniature image sensor (size ≤10mm×10mm), which has automatic focusing and light adjustment functions. It can acquire scene images with a resolution of 1080P in real time and transmit the data to the NPU core unit through a high-speed MIPI interface.
[0062] (3) Storage unit: Low-power flash memory modules are selected, and the storage capacity is configured according to the application requirements (8GB~32GB) to store system programs, model parameters and temporary image data. The data read and write speed is ≥100MB / s to meet the real-time processing requirements.
[0063] (4) Power supply unit: It adopts a rechargeable lithium battery or external DC power supply (5V / 2A) mode, and is equipped with a low power management module. It can dynamically adjust the power supply according to the system operating status (standby, detection, tracking) to further reduce energy consumption;
[0064] (5) Communication unit: Integrates a miniaturized wireless communication module (Bluetooth 5.0, Wi-Fi 6), with a size ≤8mm×8mm, used to transmit the detection, positioning and tracking results to external display devices (mobile phone screen, terminal display, etc.) or back-end management system to realize data interaction.
[0065] 2. Software System Implementation Process
[0066] (1) System initialization
[0067] After the system starts up, it first completes a hardware self-test: checking the working status of the NPU, image acquisition unit, and communication unit (such as whether they are powered on normally and whether the interfaces are connected normally); if there are no hardware abnormalities, it loads the lightweight neural network model weight file parameters and the initial parameters of the reinforcement learning strategy (initial reward threshold, detection threshold), completes the initialization configuration, and enters the standby state.
[0068] (2) Image acquisition and preprocessing
[0069] The image acquisition unit acquires scene images in real time at a set frame rate (30FPS) and transmits the acquired RGB format image data to the preprocessing module. The preprocessing module normalizes the image size and standardizes the pixel values (mapping to the range of 0 to 1) to eliminate the influence of differences in image size and brightness on the detection results; at the same time, it converts the processed image data into a format that the NPU can recognize and transmits it to the next module.
[0070] (3) Data augmentation and model inference
[0071] The preprocessed image data enters the data augmentation module, which randomly selects one of three augmentation operations (horizontal flip, rotation, brightness adjustment) to generate augmented image samples. If the current scene lighting is stable and the target has no obvious angle change, the augmentation operation can be skipped and the original preprocessed image can be directly output, balancing processing efficiency and sample diversity.
[0072] The enhanced sample (or the original preprocessed image) is input into the NPU core unit. The NPU, following the computational logic of the lightweight neural network, sequentially completes target feature extraction (through convolutional layers and pooling layers), classification (through fully connected layers), and localization calculation (through regression layers), and outputs preliminary target location information (x and y coordinates of the bounding box and its width and height) and category information ("pedestrian", "vehicle", etc.).
[0073] (4) Reinforcement learning optimization and tracking
[0074] The reinforcement learning module receives the detection results output by the NPU and combines them with historical tracking data to determine the stability of the current detection results: if the change in the target bounding box is ≤10% and the class confidence is ≥0.7, it is considered a valid result; if the target is occluded or there is a sudden change in lighting, the module uses "tracking continuity" and "localization error" as indicators to adjust the model detection threshold and tracking box update strategy (such as using historical position to predict the current position) to optimize the detection results and ensure that target tracking is not interrupted.
[0075] (5) Results output and feedback
[0076] The optimized target detection, localization, and tracking results (including target location, category, and tracking status, such as "normal tracking" and "occlusion tracking") are transmitted to an external display device via the communication unit and presented in graphic form (the target bounding box is marked in the image, and the category is labeled); at the same time, they are transmitted to the back-end system for subsequent analysis and statistics on the number of targets, trajectories, etc.
[0077] The system records the results of each detection and tracking (detection time, confidence level, tracking status) and stores them in the storage unit as a reference for subsequent reinforcement learning strategy optimization (adjusting the weight of the reward function).
[0078] This invention proposes a lightweight neural network-based target detection, localization, and tracking system. Through collaborative hardware and software design, it effectively solves the problems of limited deployment, high power consumption, slow response, and poor adaptability to complex scenarios inherent in traditional target detection systems. The system uses the NPU as its core computing unit, combined with lightweight neural networks and reinforcement learning and data augmentation functions, achieving the core requirements of miniaturization, intelligence, low power consumption, and real-time performance. It can be widely applied in intelligent monitoring, portable devices, vehicle assistance, and other scenarios.
[0079] From a technical perspective, the system overcomes the challenge of balancing performance, power consumption, and size by combining hardware selection (NPU + miniaturized peripheral modules) with software optimization (model pruning and dynamic strategy adjustment). From an application perspective, the system is adaptable and versatile, and its configuration can be adjusted according to different needs to meet the detection and tracking requirements of diverse scenarios.
[0080] Further optimization directions include: first, exploring more efficient lightweight neural network architectures to further reduce model size and computational load while ensuring accuracy; second, optimizing reinforcement learning algorithms to shorten policy adjustment time and improve response speed in complex scenarios; and third, expanding hardware compatibility to adapt to more types of NPU chips and peripheral modules, thereby broadening the system's application scope.
Claims
1. A lightweight neural network-based target detection, localization, and tracking method based on an NPU, characterized in that, Includes the following steps: 1) Acquire scene images and preprocess them, converting them into a format recognizable by the NPU; 2) Perform data augmentation on the preprocessed image; 3) Use a lightweight neural network deployed on the NPU to classify and locate targets in the enhanced image to obtain target location and category information; 4) Use historical tracking data to assess the stability of the neural network's detection results, and adjust the neural network's parameters based on the assessment results to optimize the neural network.
2. The lightweight neural network target detection, localization, and tracking method based on NPU according to claim 1, characterized in that, Step 2) specifically refers to: The image is enhanced by randomly selecting one of the following methods: horizontal flipping, rotation, and brightness adjustment.
3. The lightweight neural network target detection, localization, and tracking method based on NPU according to claim 1, characterized in that, Before detection, the lightweight neural network needs to be optimized, including pruning and layer fusion.
4. The lightweight neural network target detection, localization, and tracking method based on NPU according to claim 1, characterized in that, The lightweight neural network sequentially extracts target features, classifies and locates the input image to obtain preliminary target location information and category information.
5. The lightweight neural network target detection, localization, and tracking method based on NPU according to claim 1, characterized in that, Step 4) specifically refers to: Based on historical tracking data, the magnitude of change in the target bounding box and the category confidence in the current detection result are judged. If both the magnitude of change in the target bounding box and the category confidence meet the threshold conditions, it is considered a valid result; otherwise, the model detection threshold and tracking box update strategy in the neural network are adjusted through reinforcement learning so that the output detection result meets the threshold conditions for the magnitude of change in the target bounding box and the category confidence, thereby optimizing the detection result.
6. A lightweight neural network target detection, localization, and tracking system based on an NPU, characterized in that, include: The image acquisition and preprocessing module is used to acquire scene images, preprocess them, and convert them into an NPU-recognizable format; The data augmentation module is used to augment the data of the preprocessed image; The NPU core unit is used to classify and locate targets in the enhanced image using a lightweight neural network deployed on the NPU, and obtain target location information and category information. The reinforcement learning module is used to assess the stability of the neural network's detection results using historical tracking data, and to adjust the neural network's parameters and optimize the neural network based on the assessment results.
7. A lightweight neural network target detection, localization, and tracking system based on an NPU according to claim 6, characterized in that, It also includes a result output unit consisting of a communication module and a display module, used for the presentation and interaction of detection results.
8. A lightweight neural network target detection, localization, and tracking system based on an NPU according to claim 6, characterized in that, It also includes storage units for storing programs, model parameters, and historical data.