Domain controller based multi-path input target detection and secondary classification method

Abstract

The application discloses a multi-path input target detection and secondary classification method based on a domain controller, relates to the technical field of target detection and classification, and directly writes image data of a multi-path camera into a hardware buffer of a domain controller through a V4L2 drive interface; a detection thread reads available frames that have completed collection from the hardware buffer, aligns according to timestamps of the multiple available frames and a group batch synchronization time window, assembles the available frames into an inference batch when the number of available frames reaches a target batch or the waiting time reaches a group batch waiting timeout, and the application directly writes multi-path camera images into the hardware buffer of the domain controller through the V4L2 drive interface, avoids the process of first storing the images in CPU memory and then transferring the images to a GPU, and significantly reduces memory bandwidth occupation and data collection delay.

Description

Technical Field

[0001] This invention belongs to the field of target detection and classification technology, specifically a multi-input target detection and secondary classification method based on a domain controller. Background Technology

[0002] In real-world scenarios such as construction machinery, park security, and road inspection, the on-site environment is often accompanied by interference factors such as dust, ground ditches, backlighting, shadows, background occlusion, and uneven ground. Targets vary significantly in scale, pose, and visibility. In such environments, general single-stage target detection algorithms are prone to misidentifying dust, ditch edges, shadows, and background structures as targets, or misidentifying real targets as background, resulting in numerous false positives and false negatives.

[0003] Existing edge vision solutions typically acquire images from multiple cameras into CPU memory first, and then perform color space conversion, cropping, copying, and inference. This results in the coupling of acquisition, detection, classification, and data transfer, leading to additional memory copying and CPU overhead. When multiple cameras input concurrently, the CPU can easily become a bottleneck, causing frame rate drops and video stream choppyness.

[0004] Currently, most engineering solutions still rely on "object detection," where the model directly outputs the bounding box and category results, and then performs simple filtering based on a confidence threshold. While this approach is simple to implement, it lacks a further verification mechanism for false detections in complex environments, especially in scenarios with significant dust, ground gullies, and background interference, resulting in insufficient stability of single detection results. If multiple cameras share input and all results are directly output using the same confidence threshold, it can easily lead to alarm jitter, category confusion, and discontinuous results.

[0005] In addition, existing solutions do not make full use of dedicated acceleration units (such as deep learning accelerators DLA) on domain controllers, resulting in uneven allocation of GPU and CPU resources and low overall efficiency; detection and secondary classification are usually performed serially and coupled with data acquisition, which can easily block the acquisition link when the classification time increases, reducing video smoothness.

[0006] Therefore, how to provide a multi-input target recognition method that can reduce data copying, reduce CPU load, suppress false detections in complex backgrounds, and make full use of domain controller hardware resources has become a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0007] This invention aims to solve at least one of the technical problems existing in the prior art; Therefore, this invention proposes a multi-input target detection and secondary classification method based on a domain controller, comprising the following steps: Image data from multiple cameras can be directly written to the domain controller's hardware buffer via the V4L2 driver interface. The detection thread reads the available frames that have been collected from the hardware buffer, aligns them according to the timestamps of multiple available frames and the batch synchronization time window, and assembles the available frames into an inference batch when the number of available frames reaches the target batch or the waiting time reaches the batch waiting timeout. The assembled batch of images is input into a shared primary detection model, a target detection is performed on the GPU, and the detection results are post-processed to output a set of candidate target boxes for each viewpoint.

[0008] Furthermore, it also includes: for each candidate bounding box, extracting its first detection confidence score, removing candidate bounding boxes with confidence scores lower than the first candidate threshold, and performing non-maximum suppression to remove duplicates from the remaining candidate bounding boxes.

[0009] Furthermore, it also includes: for each retained candidate target box, cropping a region of interest image patch from its corresponding original image on the GPU side, scaling the region of interest image patch to the input size of the secondary classification model and normalizing it, and then feeding it into the secondary classification model to perform inference, outputting the probability that the region of interest belongs to the real target or the fine-grained sub-category probability.

[0010] Furthermore, the secondary classification model is preferentially deployed on the deep learning accelerator of the domain controller for execution, and falls back to GPU execution when the deep learning accelerator is unavailable.

[0011] Furthermore, it also includes: for each candidate target box, reading its primary detection confidence and secondary classification probability; when the secondary classification module is available, calculating the fusion score; if the secondary classification probability is greater than or equal to the classification confirmation threshold and the fusion score is greater than or equal to the fusion confirmation threshold, then the target is retained as a valid target; when the secondary classification module is unavailable and the primary detection confidence is greater than or equal to the first direct pass threshold, the target is directly retained as a valid target.

[0012] Furthermore, the fusion score is calculated by weighted summation, where the weight of the first detection confidence is α, the weight of the second classification probability is β, and α+β=1.

[0013] Furthermore, it also includes: for each retained valid target, recording its timestamp of appearance and the number of consecutive occurrence frames; when the same target is determined to exist for a consecutive number of stable alarm frames, outputting a stable alarm; when the target disappearance time reaches the alarm cancellation timeout, canceling the alarm.

[0014] Furthermore, the assembly triggering conditions for the dynamic inference batch include: the number of available frames reaches the target batch, the number of available frames reaches the minimum batch and the waiting time reaches the batch waiting timeout, or the number of available frames reaches the maximum batch.

[0015] Furthermore, the shared primary detection model is the TensorRT engine of the YOLO series object detection network, and the secondary classification model is the TensorRT engine of the MobileNetV2 lightweight classification network.

[0016] A multi-input target detection and secondary classification device based on a domain controller uses the method described above to perform multi-input target detection and secondary classification.

[0017] Compared with the prior art, the beneficial effects of the present invention are: This invention writes multi-camera images directly into the domain controller's hardware buffer via the V4L2 driver interface, avoiding the process of first writing the images into CPU memory and then transferring them to the GPU, thus significantly reducing memory bandwidth usage and data acquisition latency.

[0018] This invention enables the acquisition thread, detection thread, and classification thread to collaborate asynchronously through a hardware buffer pool, queue, and event mechanism, ensuring that they do not block each other. When secondary classification takes time, it does not block the acquisition link, effectively reducing the risk of stuttering and frame drops, and maintaining video smoothness.

[0019] This invention enables multiple cameras to share the same detection model and adopts a dynamic batch packetization mechanism (automatically adjusting the batch size based on the number of available frames and the time window). This not only improves throughput when multiple channels are basically synchronized, but also maintains low latency when some channels drop frames, thus avoiding single-channel stuttering that blocks the global detection link. Attached Figure Description

[0020] Figure 1 This is the overall block diagram of a multi-input target detection and secondary classification decoupling system.

[0021] Figure 2 A flowchart for V4L2 hardware buffer acquisition and sharing of a single detection process.

[0022] Figure 3 This is a schematic diagram of local cropping of candidate target boxes and DLA secondary classification.

[0023] Figure 4 A schematic diagram of the alarm hold and automatic reset state machine.

[0024] Figure 5 This diagram illustrates the collaboration between threads, queues, and hardware buffers. Detailed Implementation

[0025] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] To make the objectives, technical solutions, and advantages of this application clearer, the following description will be provided in conjunction with the accompanying drawings. Figures 1 to 5 This application provides a detailed description of its technical solution. It offers a multi-input target detection and secondary classification method based on a domain controller, which is particularly suitable for NVIDIA domain controller platforms. It should be noted that the following description is merely a preferred embodiment of this application. Those skilled in the art can make various improvements and modifications without departing from the principles of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

[0027] This application aims to provide a multi-input target detection and secondary classification method based on a domain controller; As an example of this application; This embodiment describes a multi-input target detection and secondary classification method based on a domain controller, including the following steps: Step A1: Multi-channel video acquisition and hardware buffering; Image data from multiple cameras is directly written to the domain controller's hardware buffer via the V4L2 driver interface. Each camera corresponds to one or more pre-allocated hardware buffer slots, and each buffer slot is bound to a ready_event event to indicate whether a frame of image has been acquired.

[0028] Step A2: Share dynamic batch inference of the single detection model; The detection thread reads the available frames that have been acquired and completed from the hardware buffer, and aligns them according to the timestamps of multiple available frames and the batch synchronization time window W_sync; When the number of available frames reaches the target batch B_opt, or the waiting time reaches the batch waiting timeout T_batch_wait, the available frames are assembled into an inference batch. The assembled batch of images is input into a shared primary detection model, and a single object detection is performed on the GPU, outputting a set of candidate object boxes for each viewpoint.

[0029] Step A3: Initial screening of candidate bounding boxes; For each candidate bounding box, extract its first detection confidence score P_det; remove candidate bounding boxes whose P_det is lower than the first candidate threshold T_det_low; sort the remaining candidate bounding boxes in descending order of confidence score, and use the intra-class greedy NMS algorithm (intersection over union threshold is T_nms) to remove duplicate boxes.

[0030] Step A4: ROI cropping and secondary classification; For each candidate target box retained after the initial screening, a local ROI image block is directly cropped from its corresponding original image or business ROI image on the GPU side; The ROI image patch is scaled to the input size of the secondary classification model and normalized. The processed ROI tensor is then fed into the secondary classification model for inference. The secondary classification model is preferentially deployed on the domain controller's DLA, and falls back to the GPU for execution when the DLA is unavailable. The secondary classification model outputs the probability P_cls that the ROI belongs to the true target or the fine-grained subclass probability.

[0031] Step A5: Merge the detection results with the classification results; For each candidate bounding box, read its detection confidence P_det and secondary classification probability P_cls; When the secondary classification module is available, the fusion score S_fuse is calculated using the following formula: S_fuse=α×P_det+β×P_cls, where α+β=1; If P_cls≥T_cls and S_fuse≥T_fuse, then retain the target as a valid target; If the secondary classification module is unavailable and P_det≥T_det_high, then the target is retained as a valid target; otherwise, the target is discarded.

[0032] Step A6: Alarm state machine output; For each retained valid target, record its timestamp of appearance and the number of consecutive frames it appears; when the same target is determined to exist for N consecutive stable frames, output a stability alarm; when the target disappears for T_release, clear the alarm.

[0033] Example 2

[0034] This embodiment focuses on describing the detailed implementation of dynamic batch detection in step A2. (Refer to...) Figure 1 , Figure 2 .

[0035] Step B1: The acquisition thread writes the images from the four cameras into the hardware buffer pool via V4L2, and the ready_event of each buffer slot is set after acquisition is completed.

[0036] Step B2: The detection thread polls each buffer slot to collect available frames that have been triggered by ready_event; and records the receiving timestamp of each available frame.

[0037] Step B3: The detection thread determines whether any of the following triggering conditions are met: Condition B31: The number of available frames reaches the target batch B_opt=4; Condition B32: The number of available frames reaches the minimum region B_min=1, and the waiting time since the first frame entered the inference queue reaches T_batch_wait=10 milliseconds; Condition B33: The number of available frames reaches the maximum batch size B_max=8.

[0038] Once any condition is met, proceed to step B4; otherwise, continue waiting or polling.

[0039] Step B4: Assemble all available frames into an inference batch, with each batch containing 1 to 8 frames; perform mailbox scaling, normalization, and NCHW format conversion on each frame in the batch via the GPU to form an input tensor of shape [batch, 3, 640, 640].

[0040] Step B5: Feed the input tensor into the shared YOLO series tensor runtime engine to perform a detection inference once, and obtain the original tensor of the candidate target box corresponding to each viewpoint.

[0041] Step B6: Perform coordinate decoding, category parsing, confidence filtering, and NMS post-processing on the original tensor of each viewpoint to obtain an ordered sequence of candidate target boxes saved by channel.

[0042] Example 3

[0043] This embodiment focuses on the detailed implementation of ROI clipping and DLA-accelerated secondary classification in step A4. (Refer to...) Figure 3 .

[0044] Step C1: From the candidate bounding boxes output by the detection module, read the boundary coordinates (left, top, right, bottom) of a candidate bounding box and the memory address of its source image.

[0045] Step C2: Clamp the boundary coordinates to the width and height range of the source image to ensure that the cropped area does not exceed the image boundary; if the width or height after cropping is less than 1 pixel, discard the candidate box.

[0046] Step C3: On the GPU side, directly crop the ROI image block from the source image data in the hardware buffer according to the clamped coordinates; this operation is completed through the Compute Unified Device Architecture kernel function or the cropping layer of TensorRT, without going through the CPU; Step C4: Scale the cropped ROI image block to the input size of the secondary classification model using the bilinear interpolation algorithm on the GPU, which is 224x224 pixels in this embodiment.

[0047] Step C5: Normalize the scaled ROI image: divide the pixel value by 255 to make its value range become [0, 1]; and perform standardization according to the channel mean and variance of the classification model, this step is optional as required.

[0048] Step C6: Send the normalized ROI tensor to the MobileNetV2 tensor runtime engine deployed on the DLA to perform classification inference; if multiple candidate boxes exist at the same time, multiple ROI tensors are submitted to the DLA in parallel through the classification thread pool.

[0049] Step C7: The secondary classification model outputs two types of probabilities: the real target probability P_cls and the non-target probability; P_cls is taken as the classification result.

[0050] Step C8: If the DLA hardware is unavailable or the driver is abnormal, the classification inference is automatically rolled back to the GPU for execution to ensure functional continuity.

[0051] Embodiment Four

[0052] This embodiment focuses on describing the detailed implementation of the fusion algorithm and the state machine in steps A5 and A6, with reference Figure 4 .

[0053] Step D1: The fusion thread reads a to-be-fused item from the fusion queue, and the to-be-fused item contains the primary detection confidence P_det, the secondary classification probability P_cls, the target category, coordinates and timestamp of the same candidate target box.

[0054] Step D2: If the secondary classification module is available, perform the following fusion judgment: If P_cls < T_cls, the candidate box is directly marked as a false detection and discarded, and T_cls = 0.5 in this embodiment; If P_cls ≥ T_cls, calculate S_fuse = 0.4 × P_det + 0.6 × P_cls; If S_fuse ≥ T_fuse, the target is retained, otherwise it is discarded, and T_fuse = 0.6 in this embodiment.

[0055] Step D3: If the secondary classification module is unavailable, perform a rollback judgment: if P_det≥T_det_high, then keep the target; otherwise, discard it. In this embodiment, T_det_high=0.7.

[0056] Step D4: The alarm module maintains a state machine for each target, with states including: non-existent, existent, stable existence, short-term loss, stable loss, and alarm cleared.

[0057] When a reserved target appears for the first time, the state transitions to "existence", and the occurrence time t0 and the consecutive count cnt=1 are recorded.

[0058] When the same target is retained again in the next frame (matched by coordinate IOU), cnt is incremented; if cnt ≥ N_stable, the state transitions to "stable existence" and an alarm is output. In this embodiment, N_stable = 3.

[0059] If the target does not appear in a certain frame, the state transitions to "short-term loss" and the loss time t1 is recorded; if t1-t0≤T_release and the target reappears in subsequent frames, the state returns to "existence". In this embodiment, T_release=1 second; if the target still does not appear after T_release, the state transitions to "stable loss" and "alarm cleared".

[0060] Step D5: The alarm module outputs the final result according to a 1-second statistical window: if the percentage of valid target frames in the window exceeds 50%, it outputs "Alarm valid"; otherwise, it outputs "Alarm invalid".

[0061] Example 5

[0062] This embodiment illustrates how the method of this application can be applied to different types of detection targets, such as vehicles, cones, helmets, and non-motorized vehicles.

[0063] Step E1: Based on the target type, select the corresponding primary detection model and secondary classification model. For example: For vehicle detection, the primary detection model category set includes "car", "bus", "truck", etc., and the secondary classification model outputs "real vehicle / false detection" or vehicle subtype (such as "car / SUV / truck"). For helmet detection, the primary detection model detects either "head" or "person", while the secondary classification model outputs "wearing a helmet / not wearing a helmet". For cone detection, the primary detection model detects "cone", and the secondary classification model outputs "real cone / false detection".

[0064] Step E2: Use steps A1 to A6 exactly the same as in Example 1, only changing the model file and category mapping table; the parameters of hardware buffer, dynamic batch, GPU-side ROI pruning, DLA secondary classification, fusion algorithm and alarm state machine (such as T_det_low, T_cls, T_fuse, N_stable, T_release, etc.) can be configured independently according to the scenario.

[0065] Step E3: During fusion, the final type is remapped based on the secondary classification result. For example, in a helmet detection scenario: if the primary detection outputs the category "person" and the secondary classification outputs "not wearing a helmet", then the fused output will be "person not wearing a helmet" and trigger an alarm; if the secondary classification outputs "wearing a helmet", then the output will be "person wearing a helmet" and no alarm will be triggered.

[0066] Step E4: This method does not limit the specific network structure of the primary detection model and the secondary classification model; the primary detection model can use YOLO series, SSD or other single-stage detection networks, and the secondary classification model can use lightweight networks such as ResNet, MobileNet, ShuffleNet, as long as the TensorRT engine can be exported and run on the domain controller.

[0067] The above embodiments are only used to illustrate the technical methods of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical methods of the present invention without departing from the spirit and scope of the technical methods of the present invention.

Claims

1. A multi-input target detection and secondary classification method based on a domain controller, characterized in that, Includes the following steps: Image data from multiple cameras can be directly written to the domain controller's hardware buffer via the V4L2 driver interface. The detection thread reads the available frames that have been collected from the hardware buffer, aligns them according to the timestamps of multiple available frames and the batch synchronization time window, and assembles the available frames into an inference batch when the number of available frames reaches the target batch or the waiting time reaches the batch waiting timeout. The assembled batch of images is input into a shared primary detection model, a target detection is performed on the GPU, and the detection results are post-processed to output a set of candidate target boxes for each viewpoint.

2. The multi-input target detection and secondary classification method based on a domain controller according to claim 1, characterized in that, Also includes: For each candidate bounding box, extract its first detection confidence score, remove candidate bounding boxes with confidence scores lower than the first candidate threshold, and perform non-maximum suppression to remove duplicates from the remaining candidate bounding boxes.

3. The multi-input target detection and secondary classification method based on a domain controller according to claim 2, characterized in that, Also includes: For each retained candidate bounding box, a region of interest (ROI) image patch is cropped from its corresponding original image on the GPU side. The ROI image patch is scaled to the input size of the secondary classification model and normalized. Then, it is fed into the secondary classification model to perform inference and outputs the probability that the ROI belongs to the real target or the probability of a fine-grained sub-class.

4. The multi-input target detection and secondary classification method based on a domain controller according to claim 3, characterized in that, The secondary classification model is preferentially deployed on the deep learning accelerator of the domain controller for execution, and falls back to GPU execution when the deep learning accelerator is unavailable.

5. The multi-input target detection and secondary classification method based on a domain controller according to claim 3, characterized in that, It also includes: for each candidate target box, reading its primary detection confidence and secondary classification probability; when the secondary classification module is available, calculating the fusion score; if the secondary classification probability is greater than or equal to the classification confirmation threshold and the fusion score is greater than or equal to the fusion confirmation threshold, then the target is retained as a valid target; when the secondary classification module is unavailable and the primary detection confidence is greater than or equal to the first direct pass threshold, the target is directly retained as a valid target.

6. The multi-input target detection and secondary classification method based on a domain controller according to claim 5, characterized in that, The fusion score is calculated by weighted summation, where the weight of the first detection confidence is α, the weight of the second classification probability is β, and α+β=1.

7. The multi-input target detection and secondary classification method based on a domain controller according to claim 5, characterized in that, It also includes: for each retained valid target, recording its timestamp of appearance and the number of consecutive occurrence frames; when the same target is determined to exist for a consecutive number of stable alarm frames, outputting a stable alarm; when the target disappearance time reaches the alarm cancellation timeout, canceling the alarm.

8. The multi-input target detection and secondary classification method based on a domain controller according to claim 1, characterized in that, The assembly triggering conditions for the dynamic inference batch include: the number of available frames reaches the target batch, the number of available frames reaches the minimum batch and the waiting time reaches the batch waiting timeout, or the number of available frames reaches the maximum batch.

9. The multi-input target detection and secondary classification method based on a domain controller according to claim 1, characterized in that, The shared primary detection model is the TensorRT engine of the YOLO series object detection network, and the secondary classification model is the TensorRT engine of the MobileNetV2 lightweight classification network.

10. A multi-input target detection and secondary classification device based on a domain controller, characterized in that, The device uses the method described in any one of claims 1-9 to perform multi-input target detection and secondary classification.

Images