Sublimation hardware-based video multi-target intelligent detection method and system

Abstract

The invention discloses a video multi-target intelligent detection method and system based on mercuric chloride hardware. A hardware decoding module decodes an input video stream in real time, generates video frames and stores the video frames in a shared memory queue. And inputting the video frame into the YOLOv5 model converted by the mercuric chloride OMG tool, pre-loading a plurality of model instances into a memory by using an ACL interface of the mercuric chloride NPU, calling different model instances through a polling scheduling mechanism, and outputting structured data comprising a multi-target detection frame, a category label and confidence. And carrying out non-maximum suppression processing on the reasoning result, and judging whether the target is in an alarm monitoring area or not by adopting a central point detection method. According to the invention, real-time multi-target intelligent detection of the input video stream can be realized. And the decoded video frames are stored in a shared memory queue, so that efficient data transmission and processing are realized. The multi-model parallel reasoning improves the detection precision, and is suitable for the application scene of real-time video multi-target intelligent detection.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and more particularly to a method and system for intelligent multi-target video detection based on Ascend hardware. Background Art

[0002] With the rapid development of artificial intelligence (AI) technology, video surveillance systems have been widely used in public security, traffic management, business intelligence, and other fields. Traditional video surveillance systems rely heavily on manual monitoring, which is inefficient and prone to missed detections and misjudgments. To improve monitoring efficiency and accuracy, intelligent video analysis technology has emerged, with multi-target detection as a core component. However, existing multi-target detection methods often suffer from high computational resource consumption and lack of real-time performance when processing high-resolution video streams. Especially for implementations based on the Ascend hardware platform, optimizing algorithms to fully utilize the computing power of the Ascend processor while ensuring detection accuracy and speed is a pressing challenge that needs to be addressed.

[0003] As a chip designed specifically for artificial intelligence computing, the Ascend processor has powerful parallel processing capabilities and low power consumption, making it very suitable for compute-intensive tasks such as video processing. At present, although there are some video processing solutions based on Ascend hardware on the market, most of these solutions focus on video encoding, decoding, and simple image processing, and lack support for complex video multi-target intelligent detection functions. The patent with publication number CN119274117B discloses a sonar video target detection method and device for domestically produced Ascend AI processors. This method uses a two-stage training method to train the sonar video target detection network, and uses the Ascend inference framework to deploy the video target detection network, which promotes the application and development of domestically produced AI processors in the field of forward-looking sonar detection technology.

[0004] In the field of video surveillance, the accuracy and speed of multi-target detection technology directly impact the practicality and reliability of surveillance systems. While existing multi-target detection technologies, such as deep learning-based YOLO, SSD, and Faster R-CNN, have achieved promising results in certain areas, they still face limitations in practical applications. For example, these algorithms often require high-performance GPUs when processing large amounts of video data. However, Ascend hardware, as an emerging AI computing platform, is not suitable for directly applying these deep learning models. Therefore, developing an intelligent multi-target video detection method optimized for Ascend hardware will not only promote the application of the Ascend platform in video surveillance but also provide new ideas and solutions for the development of intelligent video analysis technology. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a video multi-target intelligent detection method and system based on Ascend hardware to realize intelligent detection of multiple targets in video streams to meet the needs of corresponding application scenarios.

[0006] The object of the present invention is achieved through the following technical solutions: A real-time video multi-target intelligent detection method based on Ascend hardware includes the following steps: Video stream acquisition and hardware decoding steps: Call the hardware decoding module to decode the input video stream in real time, generate video frames, and store the decoded video frames in the shared memory queue; Multi-target detection model inference steps: Input video frames into the YOLOv5 model (.om format) converted by the Ascend OMG tool, and use the Ascend NPU's ACL interface to preload multiple model instances into memory. Call different model instances through a polling scheduling mechanism (for example, alternating between personnel and vehicle detection tasks), and output structured data containing multi-target detection boxes, category labels, and confidence levels. Detection result post-processing steps: Perform non-maximum suppression on the inference results, and use the center point detection method to determine whether the target is in the alarm monitoring area based on the regional coordinate data in the predefined JSON configuration file.

[0007] As a preferred embodiment, the hardware decoding step specifically includes: FFmpeg software decoding combined with the Ascend DVPP pre-processing module parses the video stream into raw frame data in YUV420 format. Decoded data is directly stored in the NPU memory via the DVPP interface (avoiding manual memory copying), with low transmission latency; The DVPP hardware scaling engine dynamically adjusts the image resolution to fit the model input size.

[0008] As a preferred approach, the optimization method of the lightweight YOLOv5 model includes: replacing the Focus layer of the original model with a convolutional layer and removing the redundant C3 structure; integrating a dynamic non-maximum suppression algorithm in the model output layer and setting a category-related IOU threshold.

[0009] As a preferred method, the JSON configuration file defines the following contents: a set of polygon vertex coordinates of the monitoring area and the corresponding alarm type identifier; a confidence threshold and a minimum detection size for each detection target; and a combination of time conditions for triggering the alarm, including a retention time threshold and a cooling time parameter.

[0010] As a preferred method, the implementation of the center point detection method includes: calculating the geometric center coordinates (x, y) of the target detection box; using the ray method to determine whether the center point falls within a predefined polygonal area; performing time-series filtering on the detection results of N consecutive frames, and triggering an alarm when the number of frames in which the target continues to exist in the area exceeds a threshold.

[0011] As a preferred method, it also includes the model hot update step: Monitor the integrity of OM model files in the specified directory through hash verification; When a valid new model is detected, the backup model container is enabled to load the new model, and the original model continues to run; After the new model is loaded successfully, switch the inference thread to the new container; Keep a copy of the old model and automatically roll back if the new model fails verification.

[0012] A real-time video multi-target intelligent detection system based on Ascend hardware includes: a hardware decoding module, integrated with the Ascend 310B4 chip on the Orange Pie AI Pro development board, which decodes the video stream using FFmpeg software decoding and DVPP preprocessing, outputting BGR24 format video frames to a shared memory pool; a multi-model inference module, which deploys a lightweight YOLOv5 model optimized by OMG tools, uses preloaded model parallel containers to achieve dynamic context switching for multi-target detection tasks, and outputs detection box data with category labels; and a region analysis module, which parses predefined JSON region configuration files and generates region status events based on a double check of the intersection-over-union (IOU) between the target center point and the bounding box. The alarm decision module generates an alarm instruction based on the target residence time threshold and the dynamic backoff cooling algorithm, and triggers the asynchronous storage thread to save the alarm screen.

[0013] As a preferred embodiment, the hardware decoding module includes: a code stream parsing unit, which obtains the resolution and encoding format parameters of the video stream in real time through FFprobe; The hardware acceleration unit calls the DVPP pre-processing module of the Ascend NPU to complete decoding and format conversion; The memory management unit uses the Ascend memory mapping interface to write decoded data directly into the NPU dedicated memory area.

[0014] As a preferred embodiment, the multi-model reasoning module includes: Model warehouse, which stores quantized and compressed OM format model files; The context manager maintains the parallel container preloading status and NPU memory mapping relationship of multiple model instances; The inference execution engine performs multi-target detection tasks through the ACL aclmdlExecuteAsync interface combined with pipeline parallel technology.

[0015] As a preferred embodiment, the regional analysis module includes: Configuration file loader, parsing normalized polygon coordinates and alarm rules in JSON format; Geometric calculation unit, realizing dual verification of center point and bounding box; The state tracker maintains the state machine and residence time of the targets in each monitoring area.

[0016] The present invention has at least the following beneficial effects: The present invention is a real-time video multi-target intelligent detection method and system based on Ascend hardware, which can realize real-time multi-target intelligent detection of input video streams. By calling the hardware decoding module to decode the video stream in real time, the burden on the CPU is effectively reduced and the efficiency of video processing is improved. At the same time, the decoded video frames are stored in a shared memory queue to achieve efficient data transmission and processing. The ACL interface of the Ascend NPU is used to preload multiple YOLOv5 model instances into the memory, and different model instances are called for reasoning through a polling scheduling mechanism. The multi-model parallel reasoning method not only improves the detection accuracy, but also significantly shortens the detection time and realizes a rapid response to video frames. Intelligent alarm monitoring performs non-maximum suppression processing on the inference results, effectively reducing redundant detection frames. The present invention is suitable for application scenarios that require real-time video multi-target intelligent detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] To reveal the technical details of the embodiments of the present invention, the following is a brief introduction to the drawings involved in the embodiments. It should be emphasized that these drawings only illustrate several embodiments of the present invention and should not be considered as defining the scope of the invention. Those skilled in the art can deduce other relevant drawings based on these drawings without engaging in creative work.

[0018] Figure 1 This is a flowchart of a real-time video multi-target intelligent detection method based on Ascend hardware; Figure 2 This is a schematic diagram of the dynamic alarm triggering process; Figure 3 This is a diagram showing the fire detection operation effect of the embodiment. DETAILED DESCRIPTION

[0019] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings, but the protection scope of the present invention is not limited to the following.

[0020] In the following, embodiments of the present disclosure are described in detail with the aid of accompanying drawings. However, please be aware that the present disclosure is not limited to the specific forms shown herein. Rather, it should be understood to encompass various variations, equivalents, and / or alternatives to the embodiments of the present disclosure. In describing the drawings, the same reference numerals will be used to indicate similar components.

[0021] In the various embodiments of the present disclosure, expressions such as "first," "second," "the first," or "the second" are intended to modify different components, rather than to indicate order and / or importance, and do not limit the corresponding components. For example, a first user device and a second user device each represent different user devices, although they both fall within the scope of user devices. Similarly, a first component can be named a second component, and a second component can be named a first component, which does not change their essential attributes within the scope of the present disclosure.

[0022] In this disclosure, terms are used to illustrate specific embodiments and do not constitute limitations of this disclosure. In this context, the use of the singular also encompasses the plural, unless the text clearly indicates otherwise. In the process of explanation, it should be understood that terms such as "including" or "having" are intended to indicate the presence of a feature, quantity, step, operation, structural component, part, or combination thereof, and do not preclude the possibility or addition of one or more other features, quantities, steps, operations, structural components, parts, or combinations thereof.

[0023] It should be understood that while the following description provides extensive specific details intended to facilitate a comprehensive understanding of the example embodiments, those skilled in the art will appreciate that the example embodiments can be implemented without these specific details. For example, systems may be presented in block diagram form to avoid excessive detail that would obscure the clarity of the examples. In other cases, unnecessary details regarding well-known processes, structures, and techniques may be omitted to maintain clarity of the examples.

[0024] A real-time video multi-target intelligent detection method based on Ascend hardware includes the following steps (see Figure 1 ): Video stream acquisition and hardware decoding steps: S1 uses FFmpeg to call the Ascend NPU's hardware decoding module to decode the input video stream in real time, generating video frames in BGR24 format (using the FFmpeg software decoder to perform CPU soft decoding on the input video stream to generate raw frame data in YUV420 format; calling the Ascend DVPP module's hardware acceleration preprocessing unit to perform format conversion (YUV420 to BGR24)), and storing the decoded video frames in a shared memory queue; Multi-target detection model inference steps: S2 inputs video frames into a YOLOv5 model (.om format) converted using the Ascend OMG tool. Multiple model instances are preloaded into memory using the Ascend NPU's ACL interface. S3 calls different model instances through a round-robin scheduling mechanism (e.g., alternating between person and vehicle detection tasks), outputting structured data including multi-target detection bounding boxes, category labels, and confidence scores. Model instance switching is implemented in-memory, eliminating the need for repeated loading and unloading. Detection result post-processing steps: S4 performs non-maximum suppression on the inference results. Based on the regional coordinate data in a predefined JSON configuration file, S5 uses center point detection to determine whether the target is within the alarm monitoring area.

[0025] This invention leverages Ascend hardware to achieve efficient video object detection. The process consists of three stages: Hardware-accelerated decoding: The Ascend chip's built-in hardware decoding module directly processes video stream data (e.g., H.264 / H.265 formats), automatically performing decoding and color format conversion (e.g., YUV to BGR). The processed images are directly stored in a shared memory queue, eliminating the CPU processing overhead of traditional solutions and improving decoding speed. Intelligent model inference: This system utilizes an optimized, lightweight YOLOv5 model (e.g., replacing the Focus layer with a 3×3 convolution layer) to rapidly and parallelize video frames via an Ascend-specific interface. The model supports dynamic switching of detection modes (e.g., simultaneous identification of people, vehicles, and hazardous materials), reuses memory resources, enables multi-task collaborative detection, and reduces inference time. Precise area alerting: This system combines target center point location with bounding box overlap analysis to determine in real time whether a target has entered a preset monitoring area. The system incorporates a temporal filtering mechanism and supports complex polygonal area configurations, reducing false alarm rates compared to traditional solutions.

[0026] In one embodiment, the present invention further comprises: a dynamic alarm triggering step (refer to Figure 2 ): Generates an alarm event based on the relationship between the target's residence time in the monitoring area and the preset alarm threshold, controls the alarm frequency in combination with the cooling time mechanism, and saves the alarm screen and triggers external interface notifications through asynchronous threads.

[0027] The system continuously tracks the number of frames in which the target is present in the monitoring area. A sliding window mechanism is used. When the number of frames that meet the conditions in the window exceeds the threshold, it is determined to be effectively retained. Cooling time control: a dynamic cooling timer is started after the alarm is triggered. The initial cooling time is 30-60 seconds. If the alarm is triggered continuously in the same area, the cooling time is extended according to the exponential backoff algorithm to prevent high-frequency false alarms. Asynchronous processing: the alarm screen is saved using a double-buffered queue. The main thread writes to the temporary memory, and the independent thread pool is responsible for compression and storage. External interface notifications use a priority message queue to ensure that critical alarms (such as fire) are pushed first to avoid loss due to network congestion. Reference Figure 3When there is flame in the area, a fire alarm is triggered. When there is smoke in the area, a smoke alarm is triggered. The corresponding area will be drawn with a red frame, the detected flame will be marked with a red frame, and the smoke will be marked with a yellow frame. At the same time, the program will save the alarm screen locally and send the alarm information to the external interface.

[0028] This embodiment maintains a state machine (normal / pre-alarm / alarming / cooling) for each monitored target to avoid state conflicts. An atomic counter is used to count the number of frames a target has retained, preventing counting errors caused by multi-threaded contention. This embodiment's sliding window check prevents transient false triggers (such as pedestrians passing by briefly). The dynamic cooling mechanism is more suitable for complex scenarios. Double buffering and priority queues ensure that alarms are not lost even under high concurrency. The state machine design enables the system to trace the complete behavior trajectory of each target.

[0029] Multi-threaded collaboration steps: A pipeline architecture is constructed by independently running decoding threads, inference threads, and display threads. A ring buffer of Ascend Host-Device shared memory is used between threads to achieve isolated data transmission. The inference thread uses the Ascend CANN toolchain to preload and manage model memory, and uses an asynchronous execution interface to achieve concurrent inference.

[0030] The decoding thread leverages the Ascend chip's hardware decoding capabilities to quickly convert the video stream into processable image data, storing it directly in a memory area shared with the NPU, eliminating time-consuming data copying. The inference thread preloads the detection model into the NPU's memory and maintains it there for extended periods. It simultaneously processes multiple frames of data via an asynchronous interface, leveraging hardware parallel computing. The display thread reads images containing detection results from the shared memory area, renders them using the GPU, and overlays the detection results onto the video stream for output. This solution utilizes Ascend's dedicated shared memory area to achieve zero-copy data transfer between the CPU, NPU, and GPU. Keeping the model resident in memory eliminates the overhead of repeated model loading. The inference thread can execute three to four tasks simultaneously, improving throughput.

[0031] In a preferred embodiment, the hardware decoding step specifically includes: FFmpeg software decoding combined with the Ascend DVPP pre-processing module parses the video stream into raw frame data in YUV420 format. Decoded data is directly stored in the NPU memory via the DVPP interface (avoiding manual memory copying), with low transmission latency; The DVPP hardware scaling engine dynamically adjusts the image resolution to fit the model input size.

[0032] In this embodiment, video decoding and preprocessing uses FFmpeg software to decode the video stream (e.g., H.264 format) and the Ascend DVPP module to perform color format conversion (e.g., RGB to YUV420), generating standardized frame data. Furthermore, zero-copy memory transfer uses the Ascend-specific interface (DVPP) to directly store the processed video frames into the NPU memory area, skipping the conventional CPU memory transfer step. Leveraging DVPP's built-in hardware scaling engine, the input video resolution (e.g., 1920x1080) is detected in real time and automatically scaled to the model's required size, achieving intelligent resolution adaptation.

[0033] In a preferred embodiment, the optimization method for the lightweight YOLOv5 model includes: replacing the Focus layer of the original model with a convolutional layer and removing the redundant C3 structure; using the Ascend OMG tool to quantize the model, converting the FP32 weights to INT8 format after optimization using a calibration dataset; integrating the dynamic non-maximum suppression algorithm (dynamic NMS algorithm) in the post-processing stage and setting category-related IOU thresholds.

[0034] This example completes the Focus layer transformation. The original model extracted features through slicing (slicing and reassembling images), but this operation is inefficient on the Ascend NPU. This was replaced with a standard 3×3 convolutional layer, allowing the hardware to directly access the efficient convolutional computation unit, improving inference speed. The original model's C3 module contained repeated branch computations. By analyzing feature map redundancy, 30% of these repeated computation branches were removed, reducing the model size. Model weights were compressed from high-precision decimals (FP32) to integers (INT8), reducing data size. In actual use, 500 real-world scene images were used as a calibration set. The numerical distribution range of each layer was automatically analyzed to prevent the loss of key features (such as the edges of small objects) due to compression. Dynamic NMS algorithm: Traditional methods use the same overlap filtering threshold for all object types (such as people and vehicles), resulting in missed detection of densely packed small objects (such as crowds). This invention maintains a high threshold (interpretation of union = 0.6) for large objects such as vehicles and a lower threshold (interpretation of union = 0.4) for dense objects such as pedestrians. The model size is reduced, making it suitable for deployment on embedded devices, and the inference speed is improved. Through the coordinated optimization of structural transformation and quantization compression, a balance between accuracy and efficiency is achieved.

[0035] In a preferred embodiment, the JSON configuration file defines the following contents: the normalized polygon vertex coordinates of the monitoring area (based on the aspect ratio of the screen) and the corresponding alarm type identifier; the confidence threshold and minimum detection size (width × height pixel value) of each detection target; and a combination of time conditions for alarm triggering, including a retention time threshold (unit: seconds) and a cooling time parameter based on an exponential backoff algorithm (unit: seconds).

[0036] The configuration file defines a set of intelligent detection rules, including: monitoring area calibration, which uses normalized coordinates (e.g., [0.2, 0.3] for the upper left corner and [0.8, 0.7] for the lower right corner) to mark polygonal areas and automatically adapt to any screen resolution. Alarm type identification (e.g., "intrusion alarm" / "detention alarm") determines subsequent processing logic. Target filtering rules, with confidence thresholds (e.g., 0.8 for vehicles, 0.6 for pedestrians) filter out false detections; only those above the threshold are considered valid. A minimum detection size (e.g., 50×50 pixels) eliminates interference from small, distant objects. Alarm decision logic, with a retention time threshold (e.g., 10 seconds): An alarm is triggered only if the target remains within the area for a certain period of time, preventing instantaneous false alarms. A cooldown time (e.g., 30 seconds): After an alarm is triggered, the system must wait this long before responding again, preventing frequent false alarms. For example, when a vehicle with a width of more than 50 pixels and a confidence level ≥0.8 stays in a marked "no parking zone" for more than 10 seconds in the camera image, the system immediately triggers an alarm and locks the image, and then no longer detects the area within 30 seconds.

[0037] In a preferred embodiment, the inference results are subjected to non-maximum suppression processing, and based on the regional coordinate data in the predefined JSON configuration file, a dual detection method of center point and bounding box area overlap is used to determine whether the target is in the alarm monitoring area. The dual detection method of center point and bounding box area overlap includes: calculating the geometric center coordinates (x, y) of the target detection box, and calculating the intersection-over-union (IOU) ratio between the detection box and the monitoring area; Use the ray method to determine whether the center point falls within the predefined polygon area and whether the intersection-over-union ratio exceeds the set threshold (such as IOU ≥ 0.3); Perform temporal filtering on the detection results of N consecutive frames, and trigger an alarm when the number of frames in which the target center point and IOU meet the conditions at the same time exceeds the threshold.

[0038] The present invention ensures the accuracy of alarms through triple verification. ① Core data calculation includes center point positioning and area overlap analysis. Center point positioning: Find the center coordinates of the target frame (similar to marking the center point of the target on a map) Area overlap analysis: Calculate the overlap ratio between the target frame and the monitoring area (for example, 30% of the target is within the restricted area). ② Dual condition judgment includes center point verification and overlap threshold. Center point verification: Use the ray method to determine whether the center point is within the demarcated area. Overlap threshold: Requires that at least 30% of the target area is within the monitoring range (to avoid alarms when the target edge is grazing). ③ Continuous state verification includes timing filtering and anti-jitter design. Timing filtering: The alarm is triggered only when at least 8 frames out of 10 consecutive frames of the target meet the above conditions at the same time (to prevent false alarms caused by brief stray entry). Anti-jitter design: Allows detection fluctuations in the middle 2 frames (such as when the target is temporarily obscured). For example, if the center point of a car is in a no-parking zone and 40% of the car body is within the area, the system will trigger an illegal parking alarm if it continues to detect for 8 seconds.

[0039] In a preferred embodiment, the method for saving the alarm screen by the asynchronous thread includes: creating an independent result processing thread pool, receiving the alarm frame data through a double-buffered queue; after timestamp superimposing and detecting frame marking the alarm screen, saving it as a JPEG format file or a PNG format file; pushing the alarm metadata to the designated server via HTTP POST, the metadata including the target coordinates, category and confidence level.

[0040] Asynchronous alarm processing utilizes double buffering for efficient transmission. When the main thread detects an alarm, it quickly writes the image to buffer A, while an independent thread simultaneously reads data from buffer B for processing, avoiding read / write conflicts. When buffer A is full, it automatically switches to buffer B, ensuring uninterrupted data flow. Key information is consolidated, with precise timestamps (accurate to the millisecond) and detection box annotations (e.g., a red box + category label) superimposed on the image to ensure traceability. The JPEG format is used to balance image quality and storage. Structured data is reported, with metadata including target location (center point coordinates), type (e.g., "vehicle" / "pedestrian"), and confidence level (e.g., 0.92) for backend statistical analysis. HTTP POST utilizes a short connection and retry mechanism, with up to three retries during network fluctuations to ensure data loss. When the system detects an intruding vehicle, the image annotations are saved within 0.5 seconds, and data such as "coordinates (650,320) / vehicle / 0.94" is simultaneously pushed to the monitoring center.

[0041] In a preferred embodiment, the ring buffer design in the multi-threaded collaboration step includes: setting up three levels of Ascend hardware ring buffers corresponding to decoding output, inference input, and display input respectively; using an event semaphore mechanism to ensure buffer read and write atomicity; when any buffer reaches the capacity threshold, automatically discarding the oldest frame and marking the data integrity status, and recording the frame loss count.

[0042] This embodiment uses three dedicated transmission channels to achieve efficient data flow. The three-level hardware conveyor belts include a decoding conveyor belt, an inference conveyor belt, and a display conveyor belt. The decoding conveyor belt stores the raw, newly decompressed images. The inference conveyor belt stores the images to be recognized. The display conveyor belt stores the images with detection results. Each conveyor belt uses dedicated memory on the Ascend chip. When the decoding thread wants to add a new image, the hardware automatically turns green to allow writing. When the inference thread retrieves an image, an orange warning light is triggered, pausing the writing of new data. Transmission resumes immediately after the read is complete. We have also implemented a self-cleaning mechanism. When a conveyor belt exceeds 80% of its capacity, it automatically clears the oldest two frames (marked as "incomplete data") to free up space. The background statistics display the "frame loss rate" indicator (<0.1% is normal). During rush hour, even with a sudden influx of vehicle images, the system maintains stable operation. New images are continuously input, and old images are orderly retired.

[0043] In a preferred embodiment, a model hot update step is also included: the integrity of the OM model file in a designated directory is monitored through hash verification. When a valid new model is detected, the backup model container is activated to load the new model, while the original model continues to run. After the new model is successfully loaded, the inference thread is switched to the new container, discarding two frames of data during the switch. A copy of the old model is retained, and if the new model fails verification, it is automatically rolled back. This example utilizes a security verification mechanism. The system continuously scans model files and verifies file integrity through fingerprint comparison (hash value). The current model in the main cabin continues processing video frames (e.g., identifying a vehicle) while the backup cabin silently loads the new model. The two cabins are completely isolated, and the loading process does not affect the main cabin. When the backup cabin is loaded, the system completes the switch within a two-frame interval: the two frames at the moment of switching are discarded (to prevent misjudgments caused by the semi-finished model), and the new model immediately takes over subsequent frame processing. If the new model fails to perform properly (e.g., 10 consecutive frames fail detection), the system automatically switches back to the old model and notifies the administrator via SMS, ensuring system crash prevention.

[0044] In a preferred embodiment, the cooling time mechanism is specifically implemented as follows: maintaining an independent alarm state machine for each monitoring area (including normal / cooling / pending recovery states); starting a timer when an alarm is triggered, and only recording but not triggering repeated alarms in the same area during the cooling time; dynamically adjusting the cooling time parameters, and extending the cooling period (such as 30s, 60s, 120s) according to the exponential backoff algorithm when the number of consecutive alarms exceeds the threshold.

[0045] This mechanism manages alarm triggers like the "intelligent do not disturb mode". State partition management, each monitoring area independently maintains three states. Normal: real-time detection, and an alarm will be issued immediately if the conditions are met. Cooling down: countdown after the alarm is triggered (such as 30 seconds), during which only recording and no notification will be given. Waiting for recovery: 5 seconds before the end of the countdown, it will enter the transition period, and return to normal if there is no new alarm. Dynamic cooling adjustment, first trigger: cooling down for 30 seconds. Continuous triggering: If the same area is triggered 3 times within 1 hour, the cooling time will be extended by doubling rules (30s, 60s, 120s). Reset condition: no new alarm during the cooling period, and the time will return to the initial value. When there are frequent illegal parking in a no-parking zone (such as 5 triggers within 10 minutes), the system will automatically extend the cooling time from 30 seconds to 240 seconds, greatly reducing the notification frequency. At the same time, the background will continue to record data for administrators to trace.

[0046] A real-time video multi-target intelligent detection system based on Ascend hardware includes: a hardware decoding module, integrated with the Ascend 310B4 chip on the Orange Pie AI Pro development board, which decodes the video stream using FFmpeg software decoding and DVPP preprocessing, outputting BGR24 format video frames to a shared memory pool; a multi-model inference module, which deploys a lightweight YOLOv5 model optimized by OMG tools, uses preloaded model parallel containers to achieve dynamic context switching for multi-target detection tasks, and outputs detection box data with category labels; and a region analysis module, which parses predefined JSON region configuration files and generates region status events based on a double check of the intersection-over-union (IOU) between the target center point and the bounding box. The alarm decision module generates an alarm instruction based on the target residence time threshold and the dynamic backoff cooling algorithm, and triggers the asynchronous storage thread to save the alarm screen.

[0047] The hardware decoding module in this invention implements a "depacketization pipeline" for video streams. Input processing: The camera video stream (e.g., H.264 format) is transmitted to the Orange Pi development board. The Ascend 310B4 chip's DVPP module, in conjunction with the FFmpeg software decoder, quickly decompresses the compressed video into recognizable BGR format images. DVPP hardware accelerates preprocessing (e.g., scaling and color conversion). The decoded images are directly stored in the Ascend shared memory pool, eliminating data copying between the CPU and NPU. The multi-model inference module preloads models: After compression (e.g., INT8 quantization) using OMG tools, the YOLOv5 model is preloaded into parallel containers in the NPU memory, with each container handling one detection task. When complex scenes are detected, the system automatically allocates more container resources to key targets and outputs labeled detection boxes (e.g., "Vehicle: 0.95" confidence label). The region analysis module performs center point verification: The target center coordinates are calculated and a raycast method is used to determine whether it is within a predefined polygon (e.g., a no-parking zone). Area overlap verification: Calculate the intersection over union (IOU) of the detection bounding box and the monitored area (≥0.3) to ensure that the target portion is within the area. If the vehicle's center point is within the no-parking zone, but only 10% of the vehicle body enters, no alarm is triggered. If the vehicle's center point is offset, but 40% of the vehicle body is within the no-parking zone, an alarm is triggered. The alarm decision module implements a dynamic backoff cooldown mechanism. After the initial alarm is triggered, a 30-second cooldown period is implemented, during which only logging is performed without notification. For continuous alarms, if the same area is triggered three times within an hour, the cooldown period is doubled (30, 60, or 120 seconds). Alarm execution: An asynchronous thread saves the alarm screen (with timestamp and detection bounding box annotations) and pushes the alarm information (including target coordinates, type, and confidence level) to the server.

[0048] In one embodiment, the pipeline scheduling module constructs a processing pipeline with independently running decoding, inference, and display threads. Zero-copy data transfer is achieved between these threads through a triple ring buffer. The inference thread leverages the Ascend CANN toolchain to dynamically allocate model memory. Three dedicated ring buffers (buffers) include the decoding belt (Buffer A), where the decoding thread breaks down the video into raw images; the inference belt (Buffer B), where the model analyzes the images; and the display belt (Buffer C), which displays the resulting images (e.g., a labeled package). The belts are directly connected to the hardware memory, and the inference thread automatically allocates memory space based on image complexity. Simple images are allocated 50MB of memory, and complex scenes are dynamically increased to 200MB to avoid wasted resources.

[0049] The positions of the three conveyor belts correspond to each other. In this embodiment, the three ring buffers correspond to the following three processing stages: Decoding Conveyor Belt (Buffer A): Located between the decoding thread and the inference thread, it stores decoded raw video frames (e.g., BGR24 format data). Inference Conveyor Belt (Buffer B): Located within the inference thread, it stores pre-processed tensor data (e.g., normalized image matrices) awaiting model processing. Display Conveyor Belt (Buffer C): Located between the inference thread and the display thread, it stores rendered frames with detection results (e.g., bounding boxes and labels). When the decoding thread completes decoding a frame, it immediately pushes the data into Buffer A. When the inference thread retrieves data from Buffer A, it triggers the Ascend CANN aclrtMemcpy interface, directly mapping the data to NPU memory (without CPU copying). Specific implementation of dynamic memory allocation. The inference thread implements the core steps of dynamic memory allocation using the Ascend CANN toolchain. Initial pre-allocation: Use aclrtMalloc to pre-allocate a base memory pool (e.g., 500MB) on the NPU for regular inference. Dynamic adjustment at runtime: When a high-resolution frame (such as a 4K image) is detected, aclrtMalloc is called to temporarily expand the memory pool to 800MB. In simple scenarios (such as 720P), redundant memory is automatically released, reducing the memory pool to 300MB. After inference is complete, aclrtFree is called to release the current frame's memory, but the base pool is retained for subsequent reuse.

[0050] In a preferred embodiment, the hardware decoding module includes: The code stream analysis unit obtains the resolution and encoding format parameters of the video stream in real time through FFprobe; The hardware acceleration unit calls the DVPP pre-processing module (VPC / JPEGD) of the Ascend NPU to complete decoding, format conversion, and dynamic resolution adjustment; The memory management unit uses the Ascend memory mapping interface to write decoded data into the NPU dedicated memory area.

[0051] As soon as the video stream enters the system, the hardware decoding module's bitstream parsing unit uses the FFprobe tool to scan its basic information, including resolution (e.g., 1920×1080) and encoding format (e.g., H.265 compression). A processing solution (e.g., selecting the appropriate decoder) is automatically generated. The hardware acceleration unit activates the Ascend NPU's DVPP module: the JPEGD unit decompresses the H.264 / H.265 compression package, and the VPC unit converts the image to the required BGR24 format. Hardware acceleration enables faster decoding than CPU-based decoding. The memory management unit allocates the converted image to NPU memory using aclrtMalloc and copies it to NPU memory using aclrtMemcpy, skipping the traditional CPU transfer process and reducing memory transfer time.

[0052] In a preferred embodiment, the multi-model reasoning module includes: Model warehouse, which stores quantized and compressed OM format model files; The context manager maintains the parallel container preloading status and NPU memory mapping relationship of multiple model instances; The inference execution engine performs multi-target detection tasks through the ACL aclmdlExecuteAsync interface combined with pipeline parallel technology.

[0053] The multi-model inference module establishes a model repository to store models for different functions (such as vehicle recognition, face detection, and fire detection). Each model is presented as a prepackaged toolkit (OM format) compressed (INT8 quantization) to reduce its size. Each toolkit has a unique "identity card" (SHA-256 checksum), which is automatically scanned for integrity before loading to prevent corruption or tampering. A preloading mechanism is implemented for the context manager: frequently used models (such as vehicle detection) are preloaded into separate containers in the NPU memory, occupying a fixed memory area (e.g., 500MB per container). Dynamic mapping: When switching models (e.g., from vehicle to pedestrian), the preloaded container is directly called, minimizing switching time. The inference execution engine implements asynchronous triggering: using the aclmdlExecuteAsync interface, multiple models are simultaneously launched to process different frames (e.g., model 1 processes frame 1, model 2 processes frame 2). While model A processes the current frame, model B is already loading the next frame's data. The eight video streams share the model container, allocating computing resources on demand. Output: Each frame is annotated with a detection bounding box and a category label (e.g., "Truck: 96%").

[0054] In a preferred embodiment, the regional analysis module includes: Configuration file loader, which parses the normalized polygon coordinates (based on the screen aspect ratio) and alarm rules (including the region ID and alarm type mapping table) in JSON format; Geometric calculation unit, which implements dual verification of center point and bounding box (point-polygon inclusion detection and area intersection ratio ≥ threshold); The state tracker maintains the state machine (enter / continue / leave) and residence time (in seconds) of the targets in each monitoring area.

[0055] The region analysis module ensures alert accuracy through triple validation. A configuration file loader reads normalized coordinates from a JSON file (e.g., region vertex coordinates [0.2, 0.3], [0.8, 0.7]) and automatically adapts to camera images of varying resolutions. Alarm types are bound (e.g., region ID = 1 corresponds to a "No Parking Zone," triggering a parking violation alert). Geometric calculation unit center point initial screening: Calculates the target's center coordinates and uses raycasting to determine whether it is within the geofence. Region overlap review: Calculates the intersection-of-union ratio (IoU) between the target bounding box and the monitored area (e.g., a vehicle is considered illegally parked only if ≥30% of the vehicle is in the No Parking Zone), avoiding false alarms caused by wheels simply crossing the line. The state tracker includes state machine management. Entry: The target is flagged when it first meets dual conditions. Ongoing: The target accumulates the dwell time every 1 second (e.g., from 5 to 6 seconds). Exit: The target is reset if it disappears or if the conditions are not met for more than 3 seconds. Alarm triggering: When the dwell time exceeds a threshold (e.g., more than 30 seconds in a No Parking Zone), an alarm is immediately triggered and evidence is captured. If a vehicle's center point is outside the prohibited parking zone, but 40% of the vehicle body is within the area, the initial center point screening fails and no alarm is triggered. However, if the intersection-merge ratio is ≥ 30%, the system continues tracking. If the vehicle body remains within the area for a certain period of time, an alarm is still triggered.

[0056] In a preferred embodiment, the pipeline scheduling module includes: The thread controller dynamically adjusts the Ascend compute unit binding relationship and ACL task priority of each processing thread; the buffer monitor counts the Ascend hardware memory usage and DVPP transmission frame loss rate of each ring buffer in real time; and the exception handling unit calls aclrtResetDevice to reset the compute unit and load the breakpoint context snapshot when a thread exception is detected.

[0057] The pipeline scheduling module ensures efficient and stable video processing. Thread controller compute unit binding: Locks the decoding thread to NPU core 1 and the inference thread to core 2 to avoid resource contention. Task priority: Urgent tasks (such as alarm screen processing) are set to the highest priority, giving them priority in hardware resource allocation. Buffer monitor memory usage: Monitors NPU memory usage (e.g., issues an alert when the inference buffer is 80% full) and automatically cleans up redundant data. DVPP frame loss rate: Counts hardware transmission frame drops (alarms occur if >5 frames are lost per second) and triggers downgrade strategies (e.g., reducing resolution). Exception handling unit exception detection: When an inference thread freezes (e.g., no response for 3 seconds), aclrtResetDevice is immediately called to restart the NPU compute unit. Breakpoint snapshot: Automatically saves the context of the last three frames (e.g., detection box data) before an exception occurs, allowing processing to resume from the breakpoint after recovery.

[0058] This paper builds a complete video processing pipeline based on Ascend hardware: the hardware decoding module efficiently converts video streams into BGR24 format through DVPP preprocessing and zero-copy transmission, storing them in a shared memory pool. The multi-model inference module utilizes a preloaded lightweight YOLOv5 model and implements multi-target parallel detection through the ACL interface. The regional analysis module combines center point verification and intersection-over-union analysis to accurately determine the target's residence time within a predefined monitoring area. The alarm decision module triggers alarm instructions and asynchronously saves key evidence based on a dynamic backoff algorithm and timing filtering mechanism. Each module implements isolated data transmission through a three-level ring buffer, and a model hot update mechanism ensures continuous and stable system operation. Ultimately, a closed-loop processing process of decoding, inference, analysis, and alarming is formed, enabling real-time and accurate detection of multiple video streams under the hardware acceleration of the Ascend NPU.

[0059] Although preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they become aware of the basic inventive concepts. Therefore, the appended claims are intended to be interpreted as covering the preferred embodiments and all changes and modifications that fall within the scope of the invention. The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. It should be noted that any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A real-time video multi-target intelligent detection method based on Ascend hardware, characterized by: The following steps are involved: Video stream acquisition and hardware decoding steps: Call the hardware decoding module to decode the input video stream in real time, generate video frames, and store the decoded video frames in the shared memory queue; Multi-target detection model inference steps: Input video frames into the YOLOv5 model converted by the Ascend OMG tool, and use the Ascend NPU's ACL interface to preload multiple model instances into memory. Call different model instances through a polling scheduling mechanism and output structured data including multi-target detection boxes, category labels, and confidence levels; Post-processing steps of the detection results: perform non-maximum suppression processing on the inference results, and use the center point detection method to determine whether the target is in the alarm monitoring area based on the regional coordinate data in the predefined JSON configuration file.

2. The method for real-time video multi-target intelligent detection based on Ascend hardware according to claim 1, characterized in that: The hardware decoding step specifically includes: FFmpeg software decoding combined with the Ascend DVPP pre-processing module parses the video stream into raw frame data in YUV420 format. Decoded data is directly stored in NPU memory through the DVPP interface; The DVPP hardware scaling engine dynamically adjusts the image resolution to fit the model input size.

3. The method for real-time video multi-target intelligent detection based on Ascend hardware according to claim 1, characterized in that: The optimization method of the lightweight YOLOv5 model includes: replacing the Focus layer of the original model with a convolutional layer and removing the redundant C3 structure; integrating a dynamic non-maximum suppression algorithm in the model output layer and setting a category-related IOU threshold.

4. The method for real-time video multi-target intelligent detection based on Ascend hardware according to claim 1, characterized in that: The JSON configuration file defines the following: a set of polygon vertex coordinates of the monitoring area and the corresponding alarm type identifier; a confidence threshold and minimum detection size for each detection target; and a combination of time conditions for triggering the alarm, including a retention time threshold and a cooling time parameter.

5. The method for real-time video multi-target intelligent detection based on Ascend hardware according to claim 1, characterized in that: The implementation of the center point detection method includes: calculating the geometric center coordinates (x, y) of the target detection box; using the ray method to determine whether the center point falls within a predefined polygonal area; performing time-series filtering on the detection results of N consecutive frames, and triggering an alarm when the number of frames in which the target continues to exist in the area exceeds a threshold.

6. A real-time video multi-target intelligent detection method based on Ascend hardware according to any one of claims 1-5, characterized in that: It also includes the model hot update step: Monitor the integrity of OM model files in the specified directory through hash verification; When a valid new model is detected, the backup model container is enabled to load the new model, and the original model continues to run; After the new model is loaded successfully, switch the inference thread to the new container; Keep a copy of the old model and automatically roll back if the new model fails verification.

7. A real-time video multi-target intelligent detection system based on Ascend hardware, characterized by: include: The hardware decoding module, integrated into the Ascend 310B4 chip of the Orange Pi AI Pro development board, decodes the video stream using FFmpeg software decoding and DVPP preprocessing, outputting BGR24 format video frames to a shared memory pool. The multi-model inference module deploys a lightweight YOLOv5 model optimized by OMG tools, uses pre-loaded model parallel containers to implement dynamic context switching for multi-target detection tasks, and outputs detection box data with category labels. The region analysis module parses the predefined JSON region configuration file and generates region status events based on the intersection and union of the target center point and the bounding box. The alarm decision module generates an alarm instruction based on the target residence time threshold and the dynamic backoff cooling algorithm, and triggers the asynchronous storage thread to save the alarm screen.

8. The real-time video multi-target intelligent detection system based on Ascend hardware according to claim 7, characterized in that: The hardware decoding module includes: a code stream parsing unit, which obtains the resolution and encoding format parameters of the video stream in real time through FFprobe; The hardware acceleration unit calls the DVPP pre-processing module of the Ascend NPU to complete decoding and format conversion; The memory management unit uses the Ascend memory mapping interface to write decoded data directly into the NPU dedicated memory area.

9. The real-time video multi-target intelligent detection system based on Ascend hardware according to claim 7, characterized in that: The multi-model reasoning module includes: Model warehouse, which stores quantized and compressed OM format model files; The context manager maintains the parallel container preloading status and NPU memory mapping relationship of multiple model instances; The inference execution engine performs multi-target detection tasks through the ACL aclmdlExecuteAsync interface combined with pipeline parallel technology.

10. A real-time video multi-target intelligent detection system based on Ascend hardware according to any one of claims 7-9, characterized in that: The regional analysis module includes: Configuration file loader, parsing normalized polygon coordinates and alarm rules in JSON format; Geometric calculation unit, realizing dual verification of center point and bounding box; The state tracker maintains the state machine and residence time of the targets in each monitoring area.

Images