Real-time target detection method and system based on RTSP flow and NPU collaborative optimization

Abstract

A real-time target detection method and system based on RTSP flow and NPU collaborative optimization belong to the technical field of computer vision and artificial intelligence, and are characterized by comprising the following steps: adopting dynamic memory optimization management of a hybrid pipeline architecture, and performing single-time continuous copying through a CPU (Central Processing Unit); through deep integration of innovative technologies such as DMA direct transmission NPU continuous memory pool management, dynamic batch processing scheduling, hybrid assembly line processing, intelligent equipment load balancing, parallel preprocessing optimization and vectorization post-processing, RTSP flow collaborative optimization and the like, the NPU utilization rate is improved to 85% or above, the overall average FPS is improved by 200% or above, the assembly line parallelism degree achieves three times of performance gain, and the production efficiency is greatly improved. The data transmission delay is reduced by 80%, the system stability is remarkably improved, performance degradation is avoided after long-time operation, and the method is suitable for various scenes such as edge calculation and cloud reasoning.

Description

Technical Field

[0001] This invention belongs to the fields of computer vision and artificial intelligence technology, and in particular relates to a real-time target detection method and system based on RTSP stream and NPU collaborative optimization. Background Technology

[0002] With the rapid development of artificial intelligence technology, real-time object detection has been widely used in video surveillance, autonomous driving, and intelligent security. Traditional object detection systems still face the following technical problems when processing high-resolution video streams: 1) Low batch processing efficiency: Existing systems generally use serial pipelines in the video decoding and preprocessing stages, resulting in limited throughput improvement from batch processing (e.g., Batchsize=8) compared to single-frame processing (Batchsize=1), failing to unleash the parallel computing potential of the NPU. 2) High memory management overhead: Frequent memory allocation and deallocation operations lead to system performance degradation, especially in high-concurrency scenarios, where memory fragmentation severely impacts system stability. 3) Insufficient pipeline parallelism: The lack of effective parallel mechanisms in the preprocessing, inference, and post-processing stages results in low hardware resource utilization. 4) Static and rigid load balancing strategies: Existing solutions only allocate tasks based on round-robin or fixed hashing, failing to perceive the real-time load of the NPU (e.g., allocating tasks even when utilization is >90%), leading to large load differences between devices. 5) Latency jitter: System latency is unstable, with significant differences between P95 and P99 latency, affecting real-time requirements.

[0003] In addition, existing open-source solutions such as NVIDIA TRTIS and Clipper support adaptive batching, but they only make fixed threshold judgments based on queue depth and do not combine LSTM latency prediction models and frame importance classification mechanisms, resulting in significant latency jitter in high dynamic RTSP scenarios. Summary of the Invention

[0004] The present invention aims to solve the above problems and provide a real-time target detection method and system based on RTSP stream and NPU collaborative optimization.

[0005] In a first aspect, this invention discloses a real-time target detection method based on RTSP stream and NPU collaborative optimization, comprising: The system employs a hybrid pipeline architecture for dynamic memory optimization management. It uses a single CPU copy to generate contiguous memory, which is then directly transferred to NPU contiguous memory via DMA. Efficient memory reuse is achieved through the NPU memory pool API, significantly reducing memory allocation overhead and data copy latency. We employ an LSTM-based latency prediction model combined with a priority queue latency jitter control algorithm. Through time window aggregation and dynamic batch processing scheduling techniques involving partial batch processing, we achieve latency stability optimization. Compared to single-frame processing, we achieve effective batch processing performance gains through a dynamic frame aggregation mechanism, and combine this with an intelligent frame discarding strategy to ensure low latency. An asynchronous-synchronous collaborative processing mechanism with a hybrid pipeline architecture is adopted to achieve hybrid pipeline processing with asynchronous preprocessing, synchronous inference, and asynchronous postprocessing. This architecture achieves backpressure control and load balancing through cross-stage queue deep linkage control, supporting the operation of a "three-stage hybrid pipeline of asynchronous preprocessing, synchronous inference, and asynchronous postprocessing". Each stage has an independent thread pool, and parallel pipeline processing is achieved through efficient queue connections to maximize hardware resource utilization. An intelligent load balancing algorithm based on NPU device status awareness monitors multi-dimensional metrics of each NPU device in real time, including load, inference latency, memory utilization, and temperature status. By combining device performance profiles and historical load patterns, the algorithm dynamically adjusts task allocation strategies to achieve precise load balancing with device awareness, ensuring system stability and optimal performance. RTSP stream collaborative optimization employs hardware-accelerated decoding and NPU DMA transfer optimization technology, achieving efficient data stream processing and NPU collaborative optimization through a three-layer collaborative architecture.

[0006] Furthermore, the real-time target detection method based on RTSP stream and NPU collaborative optimization described in this invention includes the following steps in its dynamic memory optimization management: 1) Hybrid pipeline memory pool pre-allocation, creating a dedicated memory pool that supports asynchronous-synchronous collaboration; 2) The CPU performs a single continuous copy, which is then directly transferred to the NPU's continuous memory via DMA. Data can be operated directly in the NPU's memory space, supporting pipeline-level inter-transfer. 3) Dynamic memory reuse mechanism, supporting frame-level and batch-level memory reuse to avoid duplicate allocation; 4) Intelligent zero-fill strategy, which optimizes the number of fill frames based on the LSTM latency prediction model; 5) Asynchronous memory management: Filling operations are executed in parallel with NPU computation, hiding memory operation latency; 6) The memory pool dynamically expands and shrinks, with priority dynamically calculated using a weighting formula: Priority = 2 * (α × QueueLoad - β × NPUUtilization) - 0.2, where priority > 0.6 for expansion and priority < -0.4 for shrinking; where queue weight α has an initial value of 1.2; NPU weight β has an initial value of 1.0; QueueLoad represents queue load rate; and NPUUtilization represents NPU utilization. 7) Cross-stage memory optimization: collaborative memory management across the preprocessing, inference, and postprocessing stages.

[0007] Furthermore, the real-time target detection method based on RTSP stream and NPU collaborative optimization described in this invention, wherein the LSTM-based latency prediction model combined with the priority queue latency jitter control algorithm, specifically includes the following steps: 1) Construct a latency prediction model based on LSTM, and train it using historical latency data to predict future latency trends; 2) Priority queue design: Establish multi-level priority queues based on frame importance and timestamps; 3) Delay jitter control algorithm, which dynamically adjusts the batch processing strategy by predicting the delay; 4) Dynamic batch processing scheduling optimizes batch size and scheduling timing based on latency prediction results; 5) Dynamic frame aggregation strategy mechanism, intelligently selects filling strategy to reduce computational overhead; 6) Time window aggregation: Collect multiple frames within a fixed time window to construct a real batch; 7) Partial batch processing, supporting efficient processing of incomplete batches; 8) Intelligent frame discarding strategy: When the post-processing queue depth is >10 frames, old frames are discarded first to maintain real-time performance.

[0008] Furthermore, the real-time target detection method based on RTSP stream and NPU collaborative optimization described in this invention, specifically includes the following steps in its asynchronous-synchronous collaborative processing mechanism within the hybrid pipeline architecture: 1) Hybrid pipeline architecture design, establishing a hybrid pipeline that combines asynchronous and synchronous collaboration; 2) Cross-stage queue deep linkage control to achieve back pressure control and flow regulation; 3) Asynchronous preprocessing thread pool, supporting multi-threaded parallel image processing; 4) Synchronous inference scheduling engine to ensure determinism and low latency during the inference phase; 5) Asynchronous post-processing thread pool, supporting parallel vectorized NMS and coordinate transformation; 6) A pipeline coordination mechanism enables efficient data flow between stages; 7) Dynamic load balancing: The thread pool size is dynamically adjusted based on the load at each stage. 8) Abnormal isolation and recovery: a single-stage failure does not affect the overall pipeline operation.

[0009] Furthermore, the real-time target detection method based on RTSP stream and NPU collaborative optimization described in this invention, specifically includes the following steps in its intelligent load balancing algorithm based on NPU device status awareness: 1) Equipment perception and monitoring system: Establish multi-dimensional NPU device status monitoring; 2) Equipment performance profile construction: establishing equipment feature models based on historical data; 3) Intelligent load balancing algorithm, which combines real-time status and performance profile for dynamic allocation; 4) Equipment health assessment: comprehensively evaluate equipment availability using multiple indicators; 5) Adaptive weight adjustment: The weight allocation is dynamically adjusted based on the real-time NPU utilization of the device. 6) Fault prediction and switching, preventative equipment switching based on trend analysis; 7) Load forecasting model, which predicts future load distribution based on historical patterns; 8) Multi-level task queues, supporting priority and device affinity scheduling; 9) Cross-stage queue linkage control: When the queue depth in the post-processing stage is >10, after receiving the back pressure signal in the inference stage, the Batchsize will be reduced from 8 to 1, and the queue depth will be checked in each subsequent scheduling cycle. If the queue depth is ≤2 and lasts for 2 cycles, the Batchsize will be restored to 8. If the queue depth is still >2, the Batchsize will be kept at 1. 10) Device affinity scheduling: The video stream sequence number stream_id is modulo the number of available NPUs n, i.e., target_npu=stream_id mod (n), to obtain the target NPU number, realizing fixed binding between stream and device. When the NPU fails or the load is abnormal, the corresponding video stream is automatically remapped to the available NPU with the lowest current load, balancing binding stability and load balancing.

[0010] Furthermore, the real-time target detection method based on RTSP stream and NPU collaborative optimization described in this invention specifically includes the following process: 1) Decoding layer: FFmpeg hardware decoding, utilizing the hardware acceleration interface of the NPU device to process RTSP streams; 2) Transport layer: The CPU generates contiguous memory in a single copy, and the contiguous memory is directly transferred to the NPU contiguous memory via DMA; 3) Processing layer: SIMD vectorization preprocessing and batch NMS postprocessing, supporting parallel processing.

[0011] Secondly, the present invention provides a real-time target detection system based on RTSP stream and NPU collaborative optimization, including a dynamic memory optimization management module, a latency jitter control algorithm and a priority queue scheduling module, an asynchronous-synchronous collaborative processing module with a hybrid pipeline architecture, a device-aware intelligent load balancing module and an RTSP stream collaborative optimization module. The dynamic memory optimization management module, based on a hybrid pipeline architecture, uses a single CPU copy to generate contiguous memory, which is then directly transferred to NPU contiguous memory via DMA. Memory reuse is achieved through the NPU memory pool API. This module is the core foundation of the hybrid pipeline architecture, providing efficient memory support for asynchronous-synchronous collaborative mechanisms. The latency jitter control algorithm and priority queue scheduling module, based on the LSTM latency prediction model and the priority queue latency jitter control algorithm, achieve latency stability optimization through time window aggregation and dynamic batch processing scheduling technology with partial batch processing. This module integrates dynamic batch processing scheduling and dynamic frame aggregation strategy mechanism, which achieves batch processing performance gain through dynamic frame aggregation mechanism and ensures low latency by combining intelligent frame discarding strategy.

[0012] The asynchronous-synchronous collaborative processing module of the hybrid pipeline architecture adopts the asynchronous-synchronous collaborative processing mechanism of the hybrid pipeline architecture to realize hybrid pipeline processing with asynchronous preprocessing, synchronous inference, and asynchronous postprocessing. The device-aware intelligent load balancing module, based on the NPU device status awareness intelligent load balancing algorithm, monitors the multi-dimensional indicators of each NPU device in real time, and dynamically adjusts the task allocation strategy by combining device performance profiles and historical load patterns to achieve accurate load balancing based on device awareness. The RTSP stream collaborative optimization module employs hardware-accelerated decoding and NPU DMA transmission optimization technology, achieving efficient data stream processing and NPU collaborative optimization through a three-layer collaborative architecture.

[0013] Furthermore, in the real-time target detection system based on RTSP stream and NPU co-optimization described in this invention, the delay jitter control algorithm adopts an LSTM-based delay prediction model. It trains a neural network using historical delay data, predicts future delay trends in real time, and dynamically adjusts the processing strategy to achieve delay jitter control and priority queue scheduling.

[0014] Furthermore, in the real-time target detection system based on RTSP stream and NPU co-optimization described in this invention, the LSTM-based delay prediction model employs a time series neural network, achieving millisecond-level delay prediction accuracy and real-time delay jitter control through the LSTM delay prediction model.

[0015] Furthermore, the real-time target detection system based on RTSP stream and NPU collaborative optimization described in this invention includes several NPU devices.

[0016] The real-time target detection method and system based on RTSP stream and NPU collaborative optimization described in this invention achieves a significant improvement in NPU utilization to over 85%, an overall average FPS increase of over 200%, a 3x performance gain in pipeline parallelism, an 80% reduction in data transmission latency, and a significant improvement in system stability. This is achieved through the deep integration of innovative technologies such as single continuous copying by the CPU, direct DMA transfer to NPU continuous memory pool management, dynamic batch processing scheduling, hybrid pipeline processing, intelligent device load balancing, parallel preprocessing optimization and vectorized post-processing, and RTSP stream collaborative optimization. These improvements result in a deep integration of innovative technologies, including CPU single continuous copying, DMA direct transfer to NPU continuous memory pool management, dynamic batch processing scheduling, hybrid pipeline processing, intelligent device load balancing, parallel preprocessing optimization and vectorized post-processing, and RTSP stream collaborative optimization. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the structure of the real-time target detection system based on RTSP stream and NPU co-optimization as described in an embodiment of the present invention. Detailed Implementation

[0018] The following detailed description of the real-time target detection method and system based on RTSP stream and NPU collaborative optimization of the present invention is provided with reference to the accompanying drawings and embodiments.

[0019] Example 1 This embodiment discloses a real-time target detection method based on RTSP stream and NPU collaborative optimization, including: The system employs a hybrid pipeline architecture for dynamic memory optimization management. It generates contiguous memory using a single CPU copy, and then directly transfers the contiguous memory to the NPU contiguous memory via DMA. It achieves efficient memory reuse through the NPU memory pool API, reducing memory allocation overhead by 95% and significantly reducing data copy latency (<0.1ms / frame).

[0020] In this embodiment of the disclosure, the dynamic memory optimization management specifically includes the following steps: 1) Hybrid pipeline memory pool pre-allocation, creating a dedicated memory pool that supports asynchronous-synchronous collaboration; 2) The CPU performs a single continuous copy, which is then directly transferred to the NPU's continuous memory via DMA. Data can be operated directly in the NPU's memory space, supporting pipeline-level inter-transfer. 3) Dynamic memory reuse mechanism, supporting frame-level and batch-level memory reuse to avoid duplicate allocation; 4) Intelligent zero-fill strategy, which optimizes the number of fill frames based on the LSTM latency prediction model; 5) Asynchronous memory management: Filling operations are executed in parallel with NPU computation, hiding memory operation latency; 6) The memory pool dynamically expands and shrinks, with priority dynamically calculated using a weighting formula: Priority = 2 * (α × QueueLoad - β × NPUUtilization) - 0.2, where priority > 0.6 for expansion and priority < -0.4 for shrinking; where queue weight α has an initial value of 1.2; NPU weight β has an initial value of 1.0; QueueLoad represents queue load rate; and NPUUtilization represents NPU utilization. 7) Cross-stage memory optimization: collaborative memory management across the preprocessing, inference, and postprocessing stages.

[0021] A latency prediction model based on LSTM is adopted, combined with a priority queue latency jitter control algorithm. Through time window aggregation and dynamic batch processing scheduling technology of partial batch processing, latency stability is optimized. A dynamic frame aggregation mechanism is used to achieve batch processing performance gain, and a smart frame dropping strategy is combined to ensure low latency.

[0022] In this embodiment of the disclosure, the LSTM-based latency prediction model combined with the priority queue latency jitter control algorithm specifically includes the following steps: 1) Construct a latency prediction model based on LSTM, and train it using historical latency data to predict future latency trends; 2) Priority queue design: Establish multi-level priority queues based on frame importance and timestamps; 3) Delay jitter control algorithm, which dynamically adjusts the batch processing strategy by predicting the delay; 4) Dynamic batch processing scheduling optimizes batch size and scheduling timing based on latency prediction results; 5) Dynamic frame aggregation strategy mechanism, intelligently selects filling strategy to reduce computational overhead; 6) Time window aggregation: Collect multiple frames within a fixed time window to construct a real batch; 7) Partial batch processing, supporting efficient processing of incomplete batches; 8) Intelligent frame discarding strategy: When the post-processing queue depth is >10 frames, old frames are discarded first to maintain real-time performance.

[0023] An asynchronous-synchronous collaborative processing mechanism with a hybrid pipeline architecture is adopted to achieve hybrid pipeline processing with asynchronous preprocessing, synchronous inference, and asynchronous postprocessing. This architecture achieves backpressure control and load balancing through cross-stage queue deep linkage control, supporting the operation of a "three-stage hybrid pipeline of asynchronous preprocessing, synchronous inference, and asynchronous postprocessing". Each stage has an independent thread pool, and parallel pipeline processing is achieved through efficient queue connections to maximize hardware resource utilization.

[0024] In this embodiment of the disclosure, the asynchronous-synchronous collaborative processing mechanism of the hybrid pipeline architecture specifically includes the following steps: 1) Hybrid pipeline architecture design, establishing a hybrid pipeline that combines asynchronous and synchronous collaboration; 2) Cross-stage queue deep linkage control to achieve back pressure control and flow regulation; 3) Asynchronous preprocessing thread pool, supporting multi-threaded parallel image processing; 4) Synchronous inference scheduling engine to ensure determinism and low latency during the inference phase; 5) Asynchronous post-processing thread pool, supporting parallel vectorized NMS and coordinate transformation; 6) A pipeline coordination mechanism enables efficient data flow between stages; 7) Dynamic load balancing: The thread pool size is dynamically adjusted based on the load at each stage. 8) Abnormal isolation and recovery: a single-stage failure does not affect the overall pipeline operation.

[0025] An intelligent load balancing algorithm based on NPU device status awareness monitors multi-dimensional metrics of each NPU device in real time, including load, inference latency, memory utilization, and temperature status. By combining device performance profiles and historical load patterns, the algorithm dynamically adjusts task allocation strategies to achieve precise load balancing with device awareness, ensuring system stability and optimal performance.

[0026] In this embodiment of the disclosure, the intelligent load balancing algorithm based on NPU device status awareness specifically includes the following steps: 1) Equipment perception and monitoring system: Establish multi-dimensional NPU device status monitoring; 2) Equipment performance profile construction: establishing equipment feature models based on historical data; 3) Intelligent load balancing algorithm, which combines real-time status and performance profile for dynamic allocation; 4) Equipment health assessment: comprehensively evaluate equipment availability using multiple indicators; 5) Adaptive weight adjustment: The weight allocation is dynamically adjusted based on the real-time NPU utilization of the device. 6) Fault prediction and switching, preventative equipment switching based on trend analysis; 7) Load forecasting model, which predicts future load distribution based on historical patterns; 8) Multi-level task queues, supporting priority and device affinity scheduling; 9) Cross-stage queue linkage control: When the queue depth in the post-processing stage is >10, after receiving the back pressure signal in the inference stage, the Batchsize will be reduced from 8 to 1, and the queue depth will be checked in each subsequent scheduling cycle. If the queue depth is ≤2 and lasts for 2 cycles, the Batchsize will be restored to 8. If the queue depth is still >2, the Batchsize will be kept at 1. 10) Device affinity scheduling: The video stream sequence number stream_id is modulo the number of available NPUs n, i.e., target_npu=stream_id mod (n), to obtain the target NPU number, realizing fixed binding between stream and device. When the NPU fails or the load is abnormal, the corresponding video stream is automatically remapped to the available NPU with the lowest current load, balancing binding stability and load balancing.

[0027] RTSP stream collaborative optimization employs hardware-accelerated decoding and NPU DMA transfer optimization technology, achieving efficient data stream processing and NPU collaborative optimization through a three-layer collaborative architecture.

[0028] In this embodiment of the disclosure, the RTSP stream cooperative optimization specifically includes the following process: 1) Decoding layer: FFmpeg hardware decoding, utilizing the hardware acceleration interface of the NPU device to process RTSP streams; 2) Transport layer: The CPU generates contiguous memory in a single copy, and the contiguous memory is directly transferred to the NPU contiguous memory via DMA; 3) Processing layer: SIMD vectorization preprocessing and batch NMS postprocessing, supporting parallel processing.

[0029] Example 2 This embodiment discloses a real-time target detection system based on RTSP stream and NPU collaborative optimization. This system is used to implement the real-time target detection method based on RTSP stream and NPU collaborative optimization described in Embodiment 1.

[0030] The real-time target detection system based on RTSP stream and NPU co-optimization described in this embodiment, such as... Figure 1 As shown, it includes a dynamic memory optimization management module, a latency jitter control algorithm and priority queue scheduling module, an asynchronous-synchronous collaborative processing module with a hybrid pipeline architecture, a device-aware intelligent load balancing module, and an RTSP stream collaborative optimization module.

[0031] The dynamic memory optimization management module, based on a hybrid pipeline architecture, uses a single CPU copy to generate contiguous memory, which is then directly transferred to NPU contiguous memory via DMA. Memory reuse is achieved through the NPU memory pool API. This module is the core foundation of the hybrid pipeline architecture, providing efficient memory support for asynchronous-synchronous collaborative mechanisms. The latency jitter control algorithm and priority queue scheduling module, based on the LSTM latency prediction model and the priority queue latency jitter control algorithm, achieve latency stability optimization through time window aggregation and dynamic batch processing scheduling technology with partial batch processing. This module integrates dynamic batch processing scheduling and dynamic frame aggregation strategy mechanism, which achieves batch processing performance gain through dynamic frame aggregation mechanism and ensures low latency by combining intelligent frame discarding strategy.

[0032] The asynchronous-synchronous collaborative processing module of the hybrid pipeline architecture adopts the asynchronous-synchronous collaborative processing mechanism of the hybrid pipeline architecture to realize hybrid pipeline processing with asynchronous preprocessing, synchronous inference, and asynchronous postprocessing. The device-aware intelligent load balancing module, based on the NPU device status awareness intelligent load balancing algorithm, monitors the multi-dimensional indicators of each NPU device in real time, and dynamically adjusts the task allocation strategy by combining device performance profiles and historical load patterns to achieve accurate load balancing based on device awareness. The RTSP stream collaborative optimization module employs hardware-accelerated decoding and NPU DMA transmission optimization technology, achieving efficient data stream processing and NPU collaborative optimization through a three-layer collaborative architecture.

[0033] In this embodiment, the implementation process can be summarized into the following stages: Phase 1: System Initialization and Environment Configuration 1) Initialize the NPU device pool, load the optimized model with Batchsize=1 / Batchsize=8, and achieve unified device scheduling; 2) Create a hybrid memory pool, use dynamic allocation, generate a physically contiguous buffer through the memory alignment API to ensure memory contiguousness, and establish a task queue to connect each stage; 3) Start the LSTM latency prediction model. During the system initialization phase, collect historical latency samples (end-to-end latency, queue depth, NPU utilization) from the last 10 minutes. During operation, incrementally update the training set every 5 minutes, create a thread pool to manage preprocessing / postprocessing, and start the batch scheduler and load balancer. Phase Two: Real-time Data Stream Processing 4) RTSP stream reception and preprocessing: Establish RTSP connection, configure video stream parameters, use FFmpeg hardware decoding, and put the video image into the task queue after preprocessing; 5) Dynamic batch processing scheduling: 5.1) Monitor queue depth to achieve dynamic frame drop control; 5.2) The dynamic frame aggregation strategy is executed with the following priority: 5.2.1) Post-processing queue depth > 10: Immediately force a switch to Batchsize=1; 5.2.2) If the post-processing queue depth is ≤10 and the LSTM prediction P99 latency is >100ms: switch Batchsize=1; 5.2.3) If the post-processing queue depth is ≤2 and there are 3 consecutive scheduling cycles and the LSTM prediction P99 latency is <80 ms: restore Batchsize=8; 5.2.4) If the real-time NPU utilization is >90%, force a switch to Batchsize=1 to reduce latency jitter; 5.3) Assign tasks to the NPU; 6) The inference thread obtains batch processing tasks, transfers data to the NPU device memory via DMA, executes model inference, obtains detection results, and sends the results back to the host memory; 7) Bounding box decoding, coordinate system transformation, vectorized NMS algorithm, confidence threshold filtering for deduplication, and return of filtering results; Phase 3: Performance Monitoring and Dynamic Optimization 8) Real-time monitoring: Collect latency, queue status, and resource utilization at each stage, and record performance data; 9) Dynamic parameter adjustment mechanism: 9.1) Dynamically adjust batch size based on inference delay statistics (inference_delays); 9.2) Optimize the timeout strategy based on the end-to-end delay (total_delays); 9.3) Control frame dropping strategy based on queue depth and optimize device load distribution synchronously.

[0034] Phase 4: Stream Connection Management 10) RTSP Disconnection Recovery Mechanism: The FFmpeg hardware decoder restarts and adopts an exponential backoff reconnection strategy of 1 / 2 / 4 / 8 seconds. The maximum number of retries is ≤5. After timeout, device switching is triggered, the decoding health is monitored, and when the physical display is unavailable, the Xvfb virtual display is automatically enabled to ensure that the FFmpeg hardware decoder context is initialized normally. Phase 5: Dynamic Resource Regulation 11) Frame dropping strategy: When the post-processing queue depth is >10 frames, the frame dropping mechanism is triggered, and older frames are dropped first to maintain real-time performance. The input frame rate is dynamically adjusted according to the processing capacity. Xvfb virtual display is supported and adapted to headless server environment (when the physical display is unavailable, Xvfb virtual display is automatically enabled to ensure that the FFmpeg hardware decoder is initialized normally).

[0035] The real-time target detection method and system based on RTSP stream and NPU co-optimization described in this invention have the following breakthrough advantages compared with the prior art: Significant performance improvements: FPS increases by 60-90% in Batchsize=1 scenarios; FPS increases by 400-600% in Batchsize=8 scenarios; overall average FPS increases by over 200%; pipeline parallelism achieves a 3x performance gain.

[0036] Breakthrough technologies: Data transmission latency reduced by 80%; Multi-frame aggregation replaces single-frame copying, achieving true batch processing gains; Hybrid pipeline: NPU utilization increased to over 85%, overall FPS increased by 200%; Intelligent load balancing: NPU utilization increased to over 85%, fault recovery time shortened; Vectorization optimization: Improved preprocessing and post-processing speeds.

[0037] Ultra-low latency and high stability: End-to-end latency is reduced by 30-50%; latency stability of P95 and P99 is significantly improved; system stability is improved, and there is no performance degradation during long-term operation.

[0038] RTSP stream co-optimization: 85% reduction in frame loss rate; 40% improvement in memory usage efficiency; Flow control optimization: adaptive adjustment based on NPU processing capacity; end-to-end monitoring, real-time latency monitoring, and performance tuning.

[0039] Engineering and scalability advantages: Supports expansion to any number of NPU devices from 0 to n; supports batch processing optimization for various YOLO series models; suitable for various scenarios such as edge computing and cloud inference; integrates NPU-specific APIs and SIMD instruction set optimizations.

[0040] Enhanced fault tolerance and adaptability: RTSP automatic recovery success rate >99%; automatic isolation rate for device anomalies 100%; exponential backoff reconnection strategy avoids frequent reconnections; dynamic resource control, core functions maintain 95% performance during resource contention; virtual display support adapts to headless server environments.

Claims

1. A real-time target detection method based on RTSP stream and NPU collaborative optimization, characterized in that... include: Dynamic memory optimization management using a hybrid pipeline architecture is employed, which generates contiguous memory using a single copy by the CPU, and the contiguous memory is directly transferred to the NPU contiguous memory via DMA, and memory reuse is achieved through the NPU memory pool API; A latency stability optimization is achieved by using an LSTM-based latency prediction model combined with a priority queue latency jitter control algorithm and dynamic batch processing scheduling techniques involving time window aggregation and partial batch processing. An asynchronous-synchronous collaborative processing mechanism with a hybrid pipeline architecture is adopted to achieve hybrid pipeline processing with asynchronous preprocessing, synchronous inference, and asynchronous post-processing; The intelligent load balancing algorithm based on NPU device status awareness monitors multi-dimensional indicators of each NPU device in real time, and dynamically adjusts task allocation strategies by combining device performance profiles and historical load patterns to achieve accurate load balancing based on device awareness. RTSP stream collaborative optimization employs hardware-accelerated decoding and NPU DMA transfer optimization technology, achieving efficient data stream processing and NPU collaborative optimization through a three-layer collaborative architecture.

2. The real-time target detection method based on RTSP stream and NPU collaborative optimization according to claim 1, characterized in that, The dynamic memory optimization management specifically includes the following steps: 1) Hybrid pipeline memory pool pre-allocation, creating a dedicated memory pool that supports asynchronous-synchronous collaboration; 2) The CPU performs a single continuous copy, which is then directly transferred to the NPU's continuous memory via DMA. Data can be operated directly in the NPU's memory space, supporting pipeline-level inter-transfer. 3) Dynamic memory reuse mechanism, supporting frame-level and batch-level memory reuse to avoid duplicate allocation; 4) Intelligent zero-fill strategy, which optimizes the number of fill frames based on the LSTM latency prediction model; 5) Asynchronous memory management: Filling operations are executed in parallel with NPU computation, hiding memory operation latency; 6) The memory pool dynamically expands and shrinks, with priority dynamically calculated using a weighting formula: When Priority > 0.6, the system expands; when Priority < -0.4, the system shrinks. The initial value of queue weight α is 1.2; the initial value of NPU weight β is 1.0; QueueLoad represents the queue load rate; NPUUtilization represents the NPU utilization rate. 7) Cross-stage memory optimization: collaborative memory management across the preprocessing, inference, and postprocessing stages.

3. The real-time target detection method based on RTSP stream and NPU collaborative optimization according to claim 1, characterized in that, The LSTM-based latency prediction model combined with the priority queue latency jitter control algorithm specifically includes the following steps: 1) Construct a latency prediction model based on LSTM, and train it using historical latency data to predict future latency trends; 2) Priority queue design: Establish multi-level priority queues based on frame importance and timestamps; 3) Delay jitter control algorithm, which dynamically adjusts the batch processing strategy by predicting the delay; 4) Dynamic batch processing scheduling optimizes batch size and scheduling timing based on latency prediction results; 5) Dynamic frame aggregation strategy mechanism, intelligently selects filling strategy to reduce computational overhead; 6) Time window aggregation: Collect multiple frames within a fixed time window to construct a real batch; 7) Partial batch processing, supporting efficient processing of incomplete batches; 8) Intelligent frame discarding strategy: When the post-processing queue depth is >10 frames, old frames are discarded first to maintain real-time performance.

4. The real-time target detection method based on RTSP stream and NPU collaborative optimization according to claim 1, characterized in that, The asynchronous-synchronous collaborative processing mechanism of the hybrid pipeline architecture specifically includes the following steps: 1) Hybrid pipeline architecture design, establishing a hybrid pipeline that combines asynchronous and synchronous collaboration; 2) Cross-stage queue deep linkage control to achieve back pressure control and flow regulation; 3) Asynchronous preprocessing thread pool, supporting multi-threaded parallel image processing; 4) Synchronous inference scheduling engine to ensure determinism and low latency during the inference phase; 5) Asynchronous post-processing thread pool, supporting parallel vectorized NMS and coordinate transformation; 6) A pipeline coordination mechanism enables efficient data flow between stages; 7) Dynamic load balancing: The thread pool size is dynamically adjusted based on the load at each stage. 8) Abnormal isolation and recovery: a single-stage failure does not affect the overall pipeline operation.

5. The real-time target detection method based on RTSP stream and NPU collaborative optimization according to claim 1, characterized in that, The intelligent load balancing algorithm based on NPU device status awareness specifically includes the following steps: 1) Equipment perception and monitoring system: Establish multi-dimensional NPU device status monitoring; 2) Equipment performance profile construction: establishing equipment feature models based on historical data; 3) Intelligent load balancing algorithm, which combines real-time status and performance profile for dynamic allocation; 4) Equipment health assessment: comprehensively evaluate equipment availability using multiple indicators; 5) Adaptive weight adjustment: The weight allocation is dynamically adjusted based on the real-time NPU utilization of the device. 6) Fault prediction and switching, preventative equipment switching based on trend analysis; 7) Load forecasting model, which predicts future load distribution based on historical patterns; 8) Multi-level task queues, supporting priority and device affinity scheduling; 9) Cross-stage queue linkage control: When the queue depth in the post-processing stage is >10, after receiving the back pressure signal in the inference stage, the Batchsize will be reduced from 8 to 1, and the queue depth will be checked in each subsequent scheduling cycle. If the queue depth is ≤2 and lasts for 2 cycles, the Batchsize will be restored to 8. If the queue depth is still >2, the Batchsize will be kept at 1. 10) Device affinity scheduling: The video stream sequence number stream_id is modulo the number of available NPUs n, i.e., target_npu=stream_id mod (n), to obtain the target NPU number, realizing fixed binding between stream and device. When the NPU fails or the load is abnormal, the corresponding video stream is automatically remapped to the available NPU with the lowest current load, balancing binding stability and load balancing.

6. The real-time target detection method based on RTSP stream and NPU collaborative optimization according to claim 1, characterized in that, The RTSP stream collaborative optimization specifically The process includes the following: 1) Decoding layer: FFmpeg hardware decoding, utilizing the hardware acceleration interface of the NPU device to process RTSP streams; 2) Transport layer: The CPU generates contiguous memory in a single copy, and the contiguous memory is directly transferred to the NPU contiguous memory via DMA; 3) Processing layer: SIMD vectorization preprocessing and batch NMS postprocessing, supporting parallel processing.

7. A real-time target detection system based on RTSP stream and NPU collaborative optimization, characterized in that, It includes a dynamic memory optimization management module, a latency jitter control algorithm and priority queue scheduling module, an asynchronous-synchronous collaborative processing module with a hybrid pipeline architecture, a device-aware intelligent load balancing module, and an RTSP stream collaborative optimization module; The dynamic memory optimization management module is based on a hybrid pipeline architecture. It uses a single copy by the CPU to generate contiguous memory, and the contiguous memory is directly transferred to the NPU contiguous memory via DMA. Memory reuse is achieved through the NPU memory pool API. The latency jitter control algorithm and priority queue scheduling module, based on the LSTM latency prediction model and the priority queue latency jitter control algorithm, achieve latency stability optimization through time window aggregation and dynamic batch processing scheduling technology with partial batch processing. The asynchronous-synchronous collaborative processing module of the hybrid pipeline architecture adopts the asynchronous-synchronous collaborative processing mechanism of the hybrid pipeline architecture to realize hybrid pipeline processing with asynchronous preprocessing, synchronous inference, and asynchronous postprocessing. The device-aware intelligent load balancing module, based on the NPU device status awareness intelligent load balancing algorithm, monitors the multi-dimensional indicators of each NPU device in real time, and dynamically adjusts the task allocation strategy by combining device performance profiles and historical load patterns to achieve accurate load balancing based on device awareness. The RTSP stream collaborative optimization module employs hardware-accelerated decoding and NPU DMA transmission optimization technology, achieving efficient data stream processing and NPU collaborative optimization through a three-layer collaborative architecture.

8. The real-time target detection system based on RTSP stream and NPU co-optimization according to claim 7, characterized in that: The latency jitter control algorithm adopts an LSTM-based latency prediction model, which trains a neural network using historical latency data, predicts future latency trends in real time, and dynamically adjusts the processing strategy to achieve latency jitter control and priority queue scheduling.

9. The real-time target detection system based on RTSP stream and NPU collaborative optimization according to claim 8, characterized in that: The LSTM-based delay prediction model employs a time series neural network to achieve millisecond-level delay prediction accuracy and real-time delay jitter control.

10. The real-time target detection system based on RTSP stream and NPU co-optimization according to claim 7, characterized in that: The NPU device comprises several units.

Images