Vehicle illegal passenger carrying intelligent detection method and system based on CLIP and D-Fine cascade framework

Abstract

The application discloses a vehicle illegal passenger-carrying intelligent detection method and system based on a CLIP and D-Fine cascaded framework, which comprises the following steps: collecting real-time video frame images of a preset traffic monitoring network and inputting the monitoring images into a CLIP-ILP model after processing; using the CLIP-ILP model as a filter and combining a Top-K screening strategy to screen out candidate images; inputting the candidate images into a constructed context-enhanced D-FINE detection model and positioning and multi-class detection on a preset target, then outputting multi-class detection results and executing spatial co-occurrence rules, license plate position heuristic rules, size consistency rules and context consistency rules verification, then marking the results that do not pass the verification as suspicious or eliminating them, and outputting the results that pass the verification as illegal passenger-carrying detection results to a target terminal. The method can solve the balance problem between recall rate and precision in passenger-carrying detection.

Description

Technical Field

[0001] This application relates to the field of image processing technology, specifically to an intelligent detection method and system for illegal passenger carrying in vehicles based on the CLIP and D-Fine cascade framework. Background Technology

[0002] Passenger transport, including carrying people inside the cargo boxes of trucks and tricycles, poses a significant challenge to urban traffic safety management. Due to a lack of necessary safety precautions, such behavior can easily lead to serious consequences. Currently, large-scale, 24 / 7 automated screening is conducted using surveillance cameras deployed along major traffic arteries; however, this task faces multiple challenges in the field of computer vision: targets (such as passengers) typically only expose parts of their bodies (e.g., head, shoulders), their features are weak, and they are often severely obscured by vehicle structure or cargo; furthermore, passenger transport incidents are low-probability events in massive traffic flows, leading to severe class imbalance. These factors place extremely high demands on the accuracy and stability of detection algorithms.

[0003] In recent years, to address the heavy reliance of traditional detectors on bounding box annotations, Visual Language Models (VLMs), represented by CLIP, have demonstrated tremendous potential. By pre-training on billions of image-text pairs, they have achieved unprecedented generalization and zero-shot recognition capabilities. This allows them to perform open-vocabulary detection using natural language descriptions, significantly reducing their dependence on bounding box annotations. In tasks such as detecting unauthorized passenger transport, this means that supervision can be conducted through natural language descriptions (e.g., "There's someone in the truck bed"), achieving high recall rates without the need for cumbersome bounding box annotations. However, the image-level feature matching mechanism of VLMs inherently limits their localization accuracy; their image-level global feature matching method results in poor localization accuracy and a high false detection rate.

[0004] Another technological route developing in parallel with VLM is the advanced target detector represented by D-FINE, which sets a new benchmark for high-precision real-time detection. D-FINE solves several bottlenecks in traditional bounding box regression through its two core components: fine-grained distribution optimization (FDR) and global optimal localization self-distillation (GO-LSD), achieving a high accuracy. However, its complex architecture and demand for computing resources are still significant, making it unsuitable for directly processing massive amounts of manned images to be sorted without discrimination. Furthermore, its performance is highly dependent on the iterative optimization of multi-layer decoders. When faced with extremely small or severely occluded targets, if the shallow network cannot form an effective preliminary prediction, the model may still miss detections due to insufficient initial confidence.

[0005] Therefore, there is an urgent need for an intelligent detection method for illegally carrying passengers in vehicles that can solve the problem of balancing recall rate and accuracy in passenger-carrying detection. Summary of the Invention

[0006] Purpose of the invention: In order to overcome the above shortcomings, the purpose of this application is to provide a vehicle illegal passenger-carrying intelligent detection method and system based on CLIP and D-Fine cascade framework. By combining the semantic filtering capability of CLIP with the precise positioning capability of D-FINE after context enhancement and supplemented by logical verification, a joint detection strategy integrating semantic and geometric verification is formed, which solves the problem of balancing recall and accuracy in passenger-carrying detection.

[0007] To address the aforementioned technical problems, this application provides an intelligent detection method for illegal passenger carrying in vehicles based on a CLIP and D-Fine cascaded framework, comprising:

[0008] S1: Collect real-time video frame images from the preset traffic monitoring network, and after frame segmentation, noise reduction and normalization, input the generated monitoring images into the CLIP model.

[0009] S2: Using the CLIP-ILP model as a filter and combining it with the Top-K filtering strategy, calculate the cosine similarity between each monitoring image and the text description, and use the category corresponding to the highest similarity as the judgment condition to filter out candidate images;

[0010] S3: The candidate image is input into the context-enhanced D-FINE detection model with the large kernel attention module, and the preset target is located and multi-class detected by multi-scale feature fusion and distributed regression methods, and then multi-class detection results are output.

[0011] S4: Perform spatial co-occurrence rules, license plate location heuristic rules, size consistency rules, and context consistency rules to verify multi-class detection results and mark multi-class detection results that fail the verification as suspicious or remove them;

[0012] S5: Output the verified multi-category detection results as illegal manned transport detection results to the target terminal and store them synchronously in the database.

[0013] In a preferred embodiment of this application, step S2 includes:

[0014] S21: The input surveillance image is fed into the visual encoder of the CLIP-ILP model and the preset text description is fed into the text encoder of the CLIP-ILP model as a classification label, so as to extract the image features and text features of the surveillance image respectively.

[0015] S22: Calculate the cosine similarity between image features and each text feature in the shared latent space and use the Top-K screening strategy to determine the image feature with the highest similarity to the prompt related to illegal passenger transport as the candidate image.

[0016] As a preferred embodiment of this application, in step S3, the context-enhanced D-FINE detection model includes an HGNetv2 backbone network, a feature pyramid network, a large kernel attention module, a Transformer encoder, a Transformer decoder, and a multi-task detection head.

[0017] As a preferred embodiment of this application, the HGNetv2 backbone network performs multi-scale feature extraction on candidate images, including 8×8×256, 16×16×512, and 32×32×1024.

[0018] As a preferred embodiment of this application, the feature pyramid network performs feature fusion and unifies the channels of the extracted multi-scale features from top to bottom to output candidate features at scales of 8×8×256, 16×16×256 and 32×32×256.

[0019] In a preferred embodiment of this application, the large kernel attention module is integrated between the feature pyramid network and the Transformer encoder. The large kernel attention module comprises three consecutive convolutional layers: a 5×5 depthwise convolution, a 7×7 dilated depthwise convolution, and a 1×1 pointwise convolution. The calculation process of the large kernel attention module for the input candidate feature X is as follows:

[0020] ,

[0021] ,in, It is a 5×5 depthwise convolution. It is a 7×7 depthwise convolution with holes. For 1×1 pointwise convolution, This represents the Sigmoid activation function. This represents element-wise multiplication;

[0022] The context-enhanced candidate features are then fed into the Transformer encoder.

[0023] As a preferred embodiment of this application, the context-enhanced D-FINE detection model employs a multi-task learning strategy and various data augmentation strategies, and is trained using the AdamW optimizer. The various data augmentation strategies include random cropping, horizontal flipping, color jittering, and MixUp-based image blending.

[0024] Furthermore, the loss function of the context-enhanced D-FINE detection model comprehensively considers the classification loss and regression loss of each category:

[0025] ,in For the total loss function, For a set of categories, and These are the classification and regression loss weights for each category. and These are the classification and regression loss functions for each category.

[0026] As a preferred embodiment of this application, the method for training the context-enhanced D-FINE detection model includes:

[0027] S31: Select surveillance images focusing on freight vehicles from the preset traffic monitoring network and generate a dataset;

[0028] S32: Manually label the bounding boxes of the subset of images containing illegal passenger transport behavior and the negative sample images in the dataset using the principle of tightly surrounding the target. The key categories of the labeling include: person, the main detection target, i.e., passengers illegally riding in the cargo box; truck_bed / tricycle_bed, key context targets, used for LVM spatial co-occurrence rule judgment; license_plate, key context targets, used for LVM license plate location heuristic rules.

[0029] As a preferred embodiment of this application, in step S4, the spatial co-occurrence rule verifies whether the person category detection is indeed located within the spatial range of the freight carrier truck_bed or tricycle_bed. This is achieved through a dual standard of center point inclusion determination and IoU overlap determination. If the center point of the person target falls within the bounding box of the bed category, or if its IoU is greater than a threshold, the person is considered to be within the bed category bounding box. When an IOU is detected, it is determined that there may be violations. The confidence level will be increased if the verified person detection is successful.

[0030] ;

[0031] The license plate position heuristic rule is based on the relative stability of license plate positions in traffic monitoring scenarios. It uses license plate detection results to infer the possible location area of ​​the person target. When the lower edge of the person target is lower than the lower edge of the nearest license plate minus the safety margin τoffset, it is inferred that it is more likely to be in the driver's cab or outside the vehicle, which does not constitute a violation.

[0032] The size consistency rule is based on the physical size constraints of personnel targets in the monitoring scenario, and filters obviously unreasonable detection results through absolute size constraints and relative size constraints.

[0033] The context consistency rule verifies the semantic and logical relationships between different categories of detection results, ensuring that there is a spatial association between truck_bed and truck, tricycle_bed and tricycle, and the rationality of the personnel-carrier association.

[0034] This application also provides a vehicle illegal passenger-carrying intelligent detection system based on a CLIP and D-Fine cascaded framework using the above method, comprising:

[0035] The image processing module is used to acquire real-time video frame images from the preset traffic monitoring network and, after frame segmentation, noise reduction and normalization, input the generated monitoring images into the CLIP model.

[0036] The high recall filtering module is used to use the CLIP-ILP model as a filter and combine it with the Top-K filtering strategy to calculate the cosine similarity between each monitoring image and the text description and use the category corresponding to the highest similarity as the judgment condition to filter out candidate images.

[0037] The high-precision verification module is used to input the candidate image into the context-enhanced D-FINE detection model constructed by introducing the large kernel attention module, and to locate and detect the preset target in multiple categories through multi-scale feature fusion and distributed regression methods, and then output multi-category detection results;

[0038] The multi-class verification module is used to perform spatial co-occurrence rules, license plate location heuristic rules, size consistency rules and context consistency rules verification on multi-class detection results, and mark multi-class detection results that fail the verification as suspicious or remove them;

[0039] The results output module is used to output the verified multi-category detection results as illegal manned transport detection results to the target terminal and store them synchronously in the database.

[0040] The technical solution described in this application has the following advantages over the prior art:

[0041] 1. This application introduces a CLIP-based visual language semantic screening module at the front end of the detection process, using natural language descriptions to achieve zero-shot recognition, effectively compensating for the missed detection problem of traditional object detection in small sample and weak feature scenarios. Through a screening-verification cascade mechanism, high-sensitivity screening is achieved in large-scale traffic monitoring data; at the same time, a context-enhanced D-FINE (CE-D-FINE) model is introduced to further optimize accuracy on the basis of high recall, achieving a two-way improvement in recall and accuracy.

[0042] 2. This application embeds a large kernel attention module into the D-FINE detection framework, which significantly enhances the model's ability to perceive and model the spatial relationship between cargo boxes and passengers. It can effectively capture the contextual dependence of small-scale, partially occluded targets and improve the detection capability of weak feature targets.

[0043] 3. This application introduces a logic verification module, which performs semantic and spatial logic verification on the model detection results through four levels: spatial co-occurrence, license plate heuristic, size consistency, and context rules. This not only effectively filters out abnormal detection results, but also provides logical verification basis for each detection result, thereby enhancing the interpretability of the detection results.

[0044] 4. Through the synergistic effect of multiple mechanisms such as visual language filtering, context-enhanced modeling, and logical verification, this application maintains stable performance in complex traffic environments, demonstrating strong stability and environmental adaptability. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0046] Figure 1 This is a schematic diagram of the CLIP and D-Fine cascade framework provided in the embodiments of this application.

[0047] Figure 2 This is a schematic diagram of the original D-FINE network framework provided in the embodiments of this application.

[0048] Figure 3 This is an improved D-FINE network framework provided in the embodiments of this application.

[0049] Figure 4 This is a schematic diagram of the image annotation style provided in the embodiments of this application; wherein Figure 4 (a) shows the three-round image annotation style. Figure 4 (b) is the labeling style for truck images.

[0050] Figure 5 This is a flowchart illustrating the intelligent detection method for illegally carrying passengers in vehicles provided in this application embodiment.

[0051] Figure 6 This is a schematic diagram of the module connections of the intelligent detection system for illegally carrying passengers in vehicles provided in this application embodiment.

[0052] Figure 7This is a schematic diagram of the performance confusion matrix of the CE-D-FINE model provided in this application embodiment on the test set.

[0053] Figure 8 This is a schematic diagram showing the cross-category accuracy comparison of different models provided in the embodiments of this application.

[0054] Figure 9 This is a thermal comparison diagram of LKA attention and original attention provided in an embodiment of this application; wherein, Figure 9 (a) is the LKA attention heatmap. Figure 9 (b) is the original attention heatmap.

[0055] Figure 10 This is a schematic diagram of a typical case where a CLIP false detection was successfully corrected by CE-D-FINE, as provided in the embodiments of this application; wherein Figure 10 (a) is a corrective case for the first truck. Figure 10 (b) is a corrective case for the second truck.

[0056] Figure 11 This is a schematic diagram illustrating a case analysis under a complex scenario provided in the embodiments of this application; wherein, Figure 11 (a) is a dark scene. Figure 11 (b) In the case of occlusion, Figure 11 (c) is a scene with insufficient lighting. Figure 11 (d) is a scene in rainy or foggy weather.

[0057] Figure 12 This is a schematic diagram of the application architecture of the illegal manned vehicle detection system provided in the embodiments of this application. Detailed Implementation

[0058] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0059] Therefore, this application does not attempt to modify the internal core mechanism of D-FINE; the core improvement lies in the transformation of the deployment paradigm, namely, drawing on the cascading idea from the Viola-Jones face detector and Faster R-CNN, and referencing... Figure 1 As shown, a two-stage detection framework is constructed:

[0060] The first stage (high recall screening) uses the CLIP model for rapid screening, sending only a small number of suspected violations to the second stage. This solves the problem that D-FINE is computationally expensive and unsuitable for large-scale processing.

[0061] The second stage (high-precision verification): In order to address the potential risk of missed detections when D-FINE is processing weak or occluded targets, this application constructs a context-enhanced version of D-FINE (CE-D-FINE). By strategically integrating a large kernel attention (LKA) module between the backbone network of D-FINE and the Transformer encoder, it provides richer and more context-aware feature inputs for the subsequent refining process, focusing on capturing richer contextual information and providing a new approach to processing occluded and weak targets.

[0062] The first stage aims to select candidate positive samples from a massive amount of monitoring images with a high recall rate, and then the second stage performs high-precision detection and localization on the candidate images.

[0063] By combining CLIP's semantic filtering capabilities with the precise localization capabilities of context-enhanced D-FINE, and supplementing it with logical verification, a modern cascaded detection strategy integrating semantic and geometric verification is formed, providing a more comprehensive and effective solution to passenger transport detection problems than a single model.

[0064] Specifically, the goal of the first phase of this application is to maximize the recall rate. Its core task is to quickly and efficiently identify all candidate images that may contain illegal passenger transport behavior from a massive, undifferentiated stream of surveillance images, at the cost of an extremely low false negative rate, so as to provide high-quality input for the refined analysis in the second phase.

[0065] To achieve this goal, this application adopts the CLIP-ILP (Illegal Passenger Detection) model previously proposed by team members (see "Wu, X., & Pan, W. (2025). Improved CLIP-ILP Model for Detecting Illegal Passenger Transport in Freight Trucks. IEEEAccess, 13, 14214-14224"), as the core technology for this stage. This model was chosen as the high-recall filter based on its unique advantages: unlike traditional object detectors (such as YOLO and Faster R-CNN) that require extensive, labor-intensive bounding box annotations, CLIP-ILP is built upon CLIP's powerful image-text pre-training capabilities, processing large numbers of image-text pairs without explicit object annotation. This approach not only greatly simplifies the data preparation process but also endows the model with excellent generalization capabilities, enabling it to adapt to varying truck types, passenger behaviors, and environmental conditions in illegal passenger transport scenarios.

[0066] The working mechanism and implementation workflow strictly follow the prediction mechanism of the CLIP-ILP model. The model input is a single surveillance image to be detected. This application uses four sets of carefully designed text descriptions that have been verified as effective as classification labels: "have person in four-wheeled truck container", "no person in four-wheeled truck container", "have person in three-wheeled truck container", and "no person in three-wheeled truck container". The image is fed into the model's visual encoder, while the four text prompts are fed into the text encoder, and their feature vectors are extracted respectively. Subsequently, the model calculates the cosine similarity between the image features and each text feature in the shared latent space. This application adopts a Top-K (K=1) selection strategy, that is, if the similarity calculated by any prompt related to "illegal human carrying" is the highest among the four groups, the image is judged as a candidate positive sample and is passed to the second stage for refinement and verification. In order to ensure that the first-stage filter has sufficiently high sensitivity and robustness, the CLIP-ILP model integrates three key enhancement techniques, namely, improving the CLIP framework by introducing enhancement functions such as multi-scale feature fusion, cross-modal self-attention mechanism and more robust text encoding.

[0067] In summary, the first phase, by employing a multi-enhanced CLIP-ILP model and combining it with an efficient Top-K screening strategy, constructed a highly sensitive high-recall filter. Its design goal is clear: to achieve near-perfect recall at the cost of an acceptable false positive rate, ensuring that no potential violation is overlooked, thus laying the foundation for precise targeting in subsequent phases.

[0068] In recent years, various Transformer-based end-to-end detectors have emerged in the field of object detection, such as DETR, Deformable DETR, and DINO. These models have made significant progress in object detection tasks by introducing attention and query mechanisms. However, these models still have limitations when facing tasks such as small object detection and accurate bounding box regression, especially in traffic monitoring scenarios that require high-precision positioning. After a comprehensive evaluation of these detectors, this application selected D-FINE as the basic detector.

[0069] D-FINE's unique distributed regression mechanism and two-stage decoding strategy make it particularly suitable for fine-grained detection tasks, such as identifying unauthorized passenger transport in truck cargo boxes. Its distributed regression capability allows for more detailed modeling of bounding boxes, while the two-stage decoding strategy provides better target localization accuracy. The D-FINE architecture consists of three main components:

[0070] The system includes a multi-scale feature extraction network, a Transformer encoder / decoder, and a distributed regression detection head. The multi-scale feature extraction network is based on the HGNetv2 backbone network and can extract feature representations at different scales.

[0071] The Transformer encoder employs a multi-scale deformable attention mechanism to effectively integrate cross-scale contextual information. The Transformer decoder uses a two-stage strategy: first, the encoder generates candidate targets, and then the decoder performs fine-grained localization.

[0072] Distributed Regression (DFR) mechanism addresses the common pitfalls of traditional bounding box regression, which typically involves directly regressing four boundary values. This approach is prone to localization errors when dealing with small targets or ambiguous boundaries. DFR significantly improves localization accuracy by transforming the bounding box regression problem into a distributed prediction problem. Specifically, DFR models the distances from the four boundaries to a reference point as a discrete distribution. For each boundary, the model predicts a probability distribution containing B bins. Where B is typically set to 16 or 32; by calculating the expected value of the distribution, the continuous boundary distance can be obtained:

[0073] ,

[0074] This distributed regression method has significant advantages over traditional L1 regression. First, it can better handle cases with ambiguous boundaries because distributed prediction provides richer boundary information. Second, it improves the consistency between localization accuracy and confidence, reducing cases of high confidence but inaccurate localization. The D-FINE training process performs optimal matching between the prediction results and the ground truth labels, and the matching cost function comprehensively considers classification loss, regression loss, and IoU loss.

[0075] ,

[0076] in, , , and These are the weight coefficients for each loss term. During the inference phase, D-FINE directly outputs the detection results in an end-to-end manner, without the need for post-processing steps such as non-maximum suppression (NMS). This design not only simplifies the inference process but also improves detection efficiency, making it particularly suitable for real-time monitoring applications.

[0077] However, despite its excellent performance in object detection tasks, D-FINE still faces some challenges when dealing with complex traffic scenes. First, when object features are weak or severely occluded, object queries in its Transformer decoder may not obtain sufficiently strong feature responses. Second, when faced with extremely imbalanced data classes, the model may favor the dominant class, affecting detection performance for the minority classes. (Original D-FINE network framework reference) Figure 2 As shown.

[0078] refer to Figure 3 As shown, to further improve the detection performance of D FINE in complex traffic scenes, especially its ability to identify small and occluded targets, this application designs a Context-Enhanced D FINE (CE D FINE). This improved version introduces a Large Kernel Attention (LKA) module on the basis of the original D FINE to enhance the model's ability to perceive contextual information. The core idea of ​​the LKA module is to capture semi-local contextual information through convolutional operations with a large receptive field while maintaining computational efficiency. The LKA module consists of three consecutive convolutional layers: a 5×5 depthwise convolution, a 7×7 dilated depthwise convolution, and a 1×1 pointwise convolution. This design gradually expands the receptive field, enabling the model to capture a wider range of spatial dependencies, while maintaining parameter efficiency through depthwise and pointwise convolutions. Specifically, for input features... Figure X The calculation process of the LKA module is as follows:

[0079] , ,in, It is a 5×5 depthwise convolution. It is a 7×7 depthwise convolution with holes. For 1×1 pointwise convolution, This represents the Sigmoid activation function. This represents element-wise multiplication. The LKA module is integrated between the Feature Pyramid Network (FPN) of D-FINE and the Transformer encoder. After each scale feature map (P3, P4, P5) output by the FPN, an LKA module is cascaded for context enhancement. The enhanced features are then fed into the multi-scale deformable attention layer of the Transformer encoder. This ensures that features at different scales receive enhanced contextual information, especially for high-resolution feature maps that are crucial for small object detection. The improvements in CE-D-FINE are not only reflected in the feature extraction stage but also in the design of the detection head. To adapt to the characteristics of the illegal manned vehicle detection task, CE-D-FINE detects multiple key categories simultaneously. This multi-category detection strategy provides rich contextual information for the subsequent logic verification module.

[0080] During training, CE-D-FINE employs a multi-task learning strategy to simultaneously optimize detection performance across four categories; the loss function comprehensively considers both classification and regression losses for each category.

[0081] ,in For the total loss function, For a set of categories, and These are the classification and regression loss weights for each category. and These are the classification and regression loss functions for each category.

[0082] The inference process of CE-D-FINE in this application is similar to that of the original D-FINE, but the output includes detection results in four categories. These results not only provide direct evidence of unauthorized manned transport (person category), but also provide important contextual information (truck_bed, tricycle_bed, license_plate), laying the foundation for the subsequent logical verification module; thus, the D-FINE network framework is improved as follows: Figure 3 As shown.

[0083] The dataset used in this application was set or adjusted by the designers according to actual needs and costs. This application selected images from it that focus on freight vehicles, covering the two major categories of four-wheeled trucks and three-wheeled motorcycles required for the task.

[0084] To train the CE-D-FINE model as a validation network and support rule judgment in the Logical Validation Module (LVM), more refined bounding box annotations are needed. Therefore, manual bounding box annotations were performed on 14,000 images, including a subset of images containing illegal human-carrying behavior and some negative sample images. The key categories annotated include:

[0085] Person: The primary target of detection is passengers who are illegally riding inside the cargo container.

[0086] truck_bed / tricycle_bed: Key contextual targets used for spatial co-occurrence rule determination in LVM.

[0087] license_plate: Key context target used for license plate location heuristics in LVM.

[0088] To ensure consistent labeling quality, detailed labeling specifications were established. For the "person" category, if the head, shoulders, or any part of the body is visible, it is labeled as illegally carrying passengers, even if there is partial obstruction. For the "bed" category, the cargo area of ​​the truck or tricycle is fully labeled, even if there are no passengers inside. As for the "license_plate" category, the four corners of the license plate are precisely labeled to ensure the accuracy of subsequent license plate recognition and spatial logic judgment.

[0089] All bounding boxes were labeled using the principle of tightly surrounding the target to avoid redundant backgrounds. To alleviate data imbalance, various data augmentation strategies were employed during the training of CE-D-FINE, including random cropping, horizontal flipping, color dithering, and MixUp-based image blending. (Reference) Figure 4 As shown, the final annotation style of the images in the dataset is displayed, including bounding boxes of the detected target (people) and contextual targets used for logical judgment (cargo box, license plate). The composition of the dataset labels is shown in Table 1:

[0090] Table 1: Dataset Label Composition Table

[0091]

[0092] Therefore, the training objective of this application is to train a D-FINE model as a high-precision validator, enabling it to accurately locate multiple targets such as person, truck_bed, tricycle_bed, and license_plate. The hyperparameter settings for training this application's model are shown in Table 2.

[0093] Table 2: Training Parameter Settings Table

[0094]

[0095] The batch size was set to 32, the optimizer was AdamW, the base learning rate was set to 2.5e-4, and the weight decay was 1.25e-4. A training strategy incorporating advanced data augmentation was employed, with a total of 74 training epochs (72 epochs using advanced augmentation, and the last 2 epochs disabling it). The total loss function of CE-D-FINE is a composite loss function consisting of a weighted sum of multiple components, designed to simultaneously optimize classification, localization, and internal knowledge distillation. Its main components include: standard classification loss (such as Focal Loss) and bounding box regression loss (such as L1 Loss and GIoU Loss); D-FINE's unique fine-grained localization loss (FGL Loss) to supervise the refinement process of the bounding box probability distribution; and D-FINE's unique decoupled distillation focus loss (DDF Loss) to achieve global optimal localization self-distillation from deep to layer. The weights of each loss component were set according to best practices in the D-FINE paper. Through these two independent training phases, a high-recall filter and a high-precision validator were obtained.

[0096] To further enhance the reliability of the system's results and its post-hoc correction capability, this application proposes a Logic Validation Module (LVM) as an interpretable rule verification mechanism to perform semantic rationality judgment and spatial logic verification on the detection results of CE-D-FINE.

[0097] The goal of LVM is to perform secondary validation using a series of interpretable rules after CE-D-FINE outputs the initial detection results. This validation aims to eliminate logically illogical false positives and improve the accuracy and reliability of the final results. LVM employs a multi-level rule validation architecture based on four design principles: interpretability, progressive validation, fault tolerance, and statistical monitoring. The input to LVM is the set of bounding boxes D = {di} detected by CE-D-FINE, where each detection result di = (ci, bi, si) includes the class label ci, bounding box coordinates bi, and confidence score si. The output is the validated detection set Dvalid and the set of detections marked as suspicious Dsuspicious. The core validation mechanism includes four levels of rule constraints:

[0098] The spatial co-occurrence rule verifies whether the "person" category detection is indeed located within the spatial range of the freight carrier (truck_bed or tricycle_bed). It uses a dual standard of center point inclusion and IoU overlap. When the center point of the person target falls within the bed category bounding box, or its IoU is greater than the threshold τiou, it is determined that there may be a violation. Person detections that pass this verification will receive a confidence boost. .

[0099] The license plate location heuristic rule is based on the relative stability of license plate locations in traffic monitoring scenarios. It uses license plate detection results to infer the possible location area of ​​a person target. When the lower edge of the person target is lower than the lower edge of the nearest license plate minus the safety margin τoffset, it is inferred that it is more likely to be located in the driver's cab or outside the vehicle, and does not constitute a violation.

[0100] The size consistency rule is based on the physical size constraints of personnel targets in the monitoring scenario. It filters obviously unreasonable detection results through absolute size constraints (area greater than τmin_area) and relative size constraints (ratio of area to vehicle area less than τmax_ratio).

[0101] Context consistency rules verify the semantic and logical relationships between different categories of detection results, ensuring spatial associations between truck_bed and truck, tricycle_bed and tricycle, and the rationality of personnel-vehicle associations. LVM employs a pipelined verification process, applying the four types of rules sequentially. Each detection result, after complete verification, is assigned to one of three categories: verified, suspicious, or exempted. The final determination of illegal passenger transport uses a weighted voting mechanism, combining the binary judgment results of each rule. The system integrates a complete statistical monitoring mechanism, recording key indicators such as rule coverage, pass rate, confidence improvement effect, and false alarm reduction rate in real time. Based on statistical information, rule parameters are dynamically adjusted to achieve adaptive optimization. Through multi-level logical verification using LVM, the cascaded detection framework proposed in this application not only achieves high-precision illegal passenger transport detection but, more importantly, provides clear logical basis for each detection decision, significantly improving its credibility in practical applications.

[0102] Therefore, for reference Figure 5 As shown in some embodiments of this application, this application provides an intelligent detection method for illegal passenger carrying in vehicles based on a CLIP and D-Fine cascaded framework, the method comprising:

[0103] S1: Acquire real-time video frame images from the preset traffic monitoring network, and after frame segmentation, noise reduction, and normalization processing, input the generated monitoring images into the CLIP model; wherein the preset traffic monitoring network is set and adjusted by the designer according to actual needs;

[0104] S2: Using the CLIP-ILP model as a filter and combining it with the Top-K filtering strategy, calculate the cosine similarity between each monitoring image and the text description, and use the category corresponding to the highest similarity as the judgment condition to filter out candidate images;

[0105] S3: The candidate image is input into the context-enhanced D-FINE detection model with the large kernel attention module, and the preset target is located and multi-class detected by multi-scale feature fusion and distributed regression methods, and then multi-class detection results are output.

[0106] S4: Perform spatial co-occurrence rules, license plate location heuristic rules, size consistency rules, and context consistency rules to verify multi-class detection results and mark multi-class detection results that fail the verification as suspicious or remove them;

[0107] S5: Output the verified multi-category detection results as illegal manned transport detection results to the target terminal and store them synchronously in the database.

[0108] Therefore, for reference Figure 6 As shown in some embodiments of this application, this application provides a vehicle illegal passenger-carrying intelligent detection system based on the CLIP and D-Fine cascaded framework using the method described above, comprising:

[0109] The image processing module 201 is used to acquire real-time video frame images from a preset traffic monitoring network and, after frame segmentation, noise reduction and normalization, input the generated monitoring images into the CLIP model.

[0110] The high recall filtering module 202 is used to use the CLIP-ILP model as a filter and combine it with the Top-K filtering strategy to calculate the cosine similarity between each monitoring image and the text description and use the category corresponding to the highest similarity as a judgment condition to filter out candidate images.

[0111] The high-precision verification module 203 is used to input the candidate image into the context-enhanced D-FINE detection model constructed by introducing the large kernel attention module, and to locate and detect the preset target in multiple categories through multi-scale feature fusion and distributed regression methods, and then output the multi-category detection results.

[0112] The multi-class verification module 204 is used to perform spatial co-occurrence rules, license plate location heuristic rules, size consistency rules and context consistency rules verification on the multi-class detection results and mark the multi-class detection results that fail the verification as suspicious or remove them;

[0113] The result output module 205 is used to output the verified multi-category detection results as illegal manned transport detection results to the target terminal and simultaneously store them in the database.

[0114] To comprehensively and objectively evaluate the performance of the proposed two-stage cascaded detection framework, this application designed a series of detailed experiments. These experiments not only verified the framework's final performance on real traffic monitoring datasets but also, through a comprehensive comparison with current mainstream detection models and progressive ablation studies, deeply analyzed the specific contributions of each innovative module. Furthermore, a detailed quantitative and qualitative analysis of the framework's computational efficiency and stability under various complex edge scenarios was conducted.

[0115] The experiment was conducted in a robust computing environment, as detailed in Table 3:

[0116] Table 3

[0117]

[0118] The hardware configuration includes an NVIDIA A800 GPU with 80GB of video memory, a dual 737532-core processor, and 512GB of RAM. The programming environment includes PyCharm, Python 3.9, CUDA 11.8, and the deep learning framework PyTorch 2.1.1.

[0119] After sufficient training, we evaluated the performance of the context-enhanced D-FINE (CE-D-FINE) model on the reserved test set. The evaluation results are visualized using a confusion matrix, which details the model's specific classification performance across seven key categories.

[0120] refer to Figure 7 As shown, the CE-D-FINE model achieved an overall accuracy of 92.4% in a multi-class detection task, demonstrating that the D-FINE model enhanced by the LKA module possesses powerful feature extraction and classification capabilities, effectively identifying multiple key targets in unauthorized passenger-carrying scenarios. The values ​​on the diagonal of the matrix (from top left to bottom right) represent the number of correctly classified samples, showing that the model in this application performs excellently in most categories.

[0121] Key category detection performance is crucial to the task, and the model demonstrated high accuracy in several key categories: Person (passenger): As the primary detection target, the model successfully identified 1196 samples as Person, the highest number of correct identifications among all categories, proving its strong ability to capture human features; Vehicle and cargo box categories: The model also accurately distinguished between the vehicle body and its cargo box. For example, it correctly identified 841 Truck, 854 Truck_bed, 962 Tricycle, and 845 Tricycle_bed. This discriminative ability is crucial for the subsequent Logical Verification Module (LVM) to determine spatial relationships.

[0122] The analysis of key confusion points reveals the challenges the model faces in decision-making by analyzing the off-diagonal elements of the confusion matrix. These analyses also indirectly confirm the necessity of the Logical Verification Module (LVM).

[0123] Core Confusion: Person vs. Driver: The most significant confusion in the model occurs between Person and Driver. Data shows that 92 real Drivers were incorrectly identified as Persons, while 61 real Persons (passengers who violated regulations) were incorrectly identified as Drivers. This confusion is visually perfectly reasonable, since drivers are also human.

[0124] Component-to-whole confusion: Another common type of confusion occurs between the whole vehicle and its components. For example, 81 Trucks were identified as Truck_bed and 48 Tricycles were identified as Tricycle_bed. This type of confusion has little impact on the final task because both belong to the target area of ​​concern.

[0125] Overall, CE-D-FINE, as a pure vision detection model, has achieved very successful training results, validating the effectiveness of contextual augmentation of its architecture.

[0126] To evaluate the end-to-end performance of the CE-D-FINE model under real-world conditions, key tests were conducted. The results are shown in Table 4. The model's prediction accuracy for unloaded passengers exceeds 97%, and its accuracy for trucks carrying passengers and three-wheeled vehicles carrying passengers reaches 97.3% and 95.8%, respectively, as detailed in Table 4.

[0127] Table 4: Inference Accuracy of the D-FINE Model

[0128]

[0129] To evaluate the end-to-end performance of the complete cascaded framework (CLIP and D-Fine) under real-world conditions, key tests were conducted, in which real datasets were input into the framework; the test results are shown in Table 5 (unloaded data was not subjected to two-stage detection):

[0130] Table 5: Inference Accuracy of Cascaded Models

[0131]

[0132] Therefore, to accurately assess the position of the proposed framework in the current technological landscape, this application selects four highly representative mainstream object detection models for comprehensive comparison. These models cover different technical schools: efficient single-stage detectors represented by YOLOv8 and YOLOv10, classic two-stage detectors represented by Faster R-CNN, and Transformer-based end-to-end detectors represented by DETR. This application analyzes the inherent advantages and bottlenecks of these models when handling the specific task of "trucks illegally carrying passengers," thereby highlighting the advancement of the proposed two-stage cascaded method. The results are referenced in the following sections. Figure 8 As shown.

[0133] The CLIP-ILP model, as the first stage of the cascaded framework in this application, has the core advantage of leveraging CLIP's powerful image-text matching capabilities to quickly identify all scenarios where someone is suspected to be on the truck at the semantic level; Figure 8 As can be seen, it exhibits extremely high recall rates in the two positive example categories of trucks carrying passengers and tricycles carrying passengers (not directly shown in the chart, but reflected in its final accuracy), effectively avoiding missed detections; however, the cost is lower precision, especially in negative example scenarios such as trucks carrying no passengers, where complex cargo or backgrounds are easily misclassified as passengers, resulting in limited overall accuracy; this clearly demonstrates that while using CLIP-ILP alone can ensure no missed detections, it will put enormous pressure on subsequent manual review, thus powerfully highlighting the necessity of introducing a second-stage high-precision validator and the superiority of the cascaded framework of this application.

[0134] The YOLO series, especially YOLOv8 and its latest evolution YOLOv10, represents the pinnacle of current real-time object detection technology, achieving an excellent balance between speed and accuracy. In tests, both models demonstrated extremely high processing efficiency (FPS), quickly identifying large and medium-sized objects with distinct features in a scene. However, in the refined task of identifying unauthorized passenger transport, their shared single-stage design revealed shortcomings: First, for small, partially occluded passengers frequently found inside cargo compartments, the YOLO model easily loses crucial details during feature downsampling in its deep network, directly leading to false negatives (low recall). Second, while YOLOv10 optimized the training process by introducing strategies such as consistent dual allocation, it still fundamentally relies on visual features for judgment, lacking the ability to understand complex spatial and semantic relationships, such as whether an object (person) is inside another object (cargo compartment). Therefore, they struggle to effectively distinguish between passengers inside the cargo compartment, the driver in the cab, and pedestrians beside the vehicle, thus affecting detection accuracy.

[0135] As the cornerstone of the two-stage detection paradigm, Faster R-CNN, through its unique Region Proposal Network (RPN) and subsequent fine-grained classification and regression heads, generally outperforms single-stage models in detection accuracy. On the dataset, Faster R-CNN does indeed demonstrate relatively high accuracy because it performs independent and detailed analysis on each candidate region. However, its core limitation lies precisely in the RPN. The RPN relies on pre-defined anchor boxes and sliding window mechanisms to generate candidate regions. When the offending passenger's pose or size is unconventional, or when it is visually highly confused with surrounding goods, the RPN may fail to generate effective candidate boxes covering the target, thus limiting its recall rate from the outset. Furthermore, the processing flow of Faster R-CNN (proposal + classification) makes its inference speed significantly slower than the YOLO series, making it difficult to meet the real-time monitoring needs of multiple high-definition video streams.

[0136] DETR pioneered a new paradigm of end-to-end detection without anchor boxes and without NMS. Its core self-attention mechanism gives it powerful global context modeling capabilities. Theoretically, this capability helps the model understand the layout of the entire scene, thereby distinguishing targets inside and outside the vehicle. However, in practical applications, DETR's performance bottleneck lies in its object query mechanism's insensitivity to weak, low-contrast targets. For passengers hidden in shadows or with only a small part of their bodies visible, sparse object query vectors are difficult to effectively match and decode, resulting in poor recall. Although its subsequent variants have improved this, the fundamental challenges remain. At the same time, DETR's huge computational cost and slow convergence speed also make its deployment and training expensive.

[0137] The above comparison reveals that, regardless of whether it's the speed-oriented YOLO series, the accuracy-focused Faster R-CNN, or the globally-oriented DETR, these general-purpose detection models all have significant shortcomings when facing the specific task of "illegal passenger transport," which is highly context-dependent, has weak target features, and suffers from extreme class imbalance. Specifically, they struggle to simultaneously achieve high recall and high precision. This application, however, does not solve all problems with a single model but rather decouples the tasks: First, the CLIP-ILP model does not rely on fragile visual features but leverages CLIP's powerful image-text matching capabilities to quickly identify all scenes suggesting someone is on the truck at the semantic level, overcoming the false negative problem of all comparison methods with near-perfect recall. Next, CE-D-FINE no longer receives massive amounts of irrelevant data but rather a small number of high-risk candidate images after the first-stage screening. This allows it to concentrate valuable computational resources, utilizing LKA-enhanced context awareness and the powerful modeling capabilities of the Transformer to perform high-precision target identification and localization, effectively eliminating false positives introduced in the first stage.

[0138] Ultimately, this application overcomes the performance bottleneck of a single model through a collaborative strategy, achieving an accuracy exceeding that of all comparative models while ensuring that no violation is missed, thus providing an efficient and stable new design for solving such complex public safety detection problems.

[0139] Therefore, in order to systematically verify the effectiveness of each core component of the CFC (CLIP-FINE Cascade) framework proposed in this application, this application also designed a progressively layered ablation experiment, as shown in Table 6:

[0140] Table 6

[0141]

[0142] This application starts with a robust baseline model, progressively adds modules from the CFC (CLIP-FINE Cascade) framework, and observes the changes in key performance indicators, where FPS (Frames Per Second) represents the number of inference images processed per second by each model; all experiments are conducted on the same dataset and in the same experimental environment to ensure the fairness of the comparison.

[0143] The complete two-stage cascaded framework (40.6 FPS) shows a slight decrease in inference speed compared to the CLIP-ILP model using only the first stage (44.9 FPS); this is expected, as the CE-DFINE model in the second stage requires additional depth analysis of the candidate images selected in the first stage. By adding the second stage of validation, the overall accuracy of the framework significantly improved from 91.1% to 97.9%, with a significant reduction in the false positive rate. In practical traffic enforcement applications, high accuracy is far more important than extreme inference speed, as it directly relates to the validity of enforcement evidence and the cost of subsequent manual review. A system that can reduce the false positive rate brings far more value than the impact of a 10% decrease in speed. Furthermore, the inference speed of 40.6 FPS is more than sufficient to meet the requirements, demonstrating that this application achieves a balance between accuracy, efficiency, and practical application value.

[0144] refer to Figure 9 As shown, by comparing the two attention maps, the two major improvements brought by the LKA module can be clearly seen, such as... Figure 9 As shown in (b), the original backbone network features can initially locate the area of ​​illegally carrying passengers on tricycles. After processing by the LKA module of this application, as shown in (b), Figure 9 As shown in (a), the model’s focus on the core objective is significantly enhanced, and the activation area of ​​the heatmap is brighter and more focused.

[0145] The LKA module demonstrates exceptional context awareness, focusing not only on the primary target but also on other pedestrians, vehicles, and potential occlusions in the background. This proves that the LKA module does not simply amplify salient features but rather builds a more comprehensive scene understanding, providing richer input for the subsequent Transformer decoder to make high-precision and robust judgments. It is this dual capability of core focus and global context awareness that enables our model to achieve a significant advantage in final detection performance.

[0146] refer to Figure 10 As shown, in some scenarios, the CLIP-ILP model in the first stage may mistakenly identify objects with shapes similar to human outlines, such as stacked blue oil drums or irregularly shaped tarpaulin covers, as suspected illegal human-carrying objects. This exposes the inherent limitations of the CLIP model, which relies on global semantics and fuzzy outlines for matching. It is good at capturing macroscopic concepts that may involve people, but it is prone to misjudgment in the absence of fine geometric and texture details.

[0147] However, when these misjudged candidate images enter the second stage, the CE-D-FINE model of this application, with its powerful fine-grained recognition capability, can perform accurate secondary verification. CE-D-FINE not only analyzes the contours of objects, but also pays attention to their surface texture (such as the smooth plastic feel of oil drums and the wrinkles of tarpaulins), rigid geometric structures, and contextual relationships with the surrounding environment. Therefore, it can easily determine that these targets are not real people, and thus remove them in the final output, effectively preventing false alarms.

[0148] Furthermore, the framework's performance was tested under various harsh conditions, including nighttime, low-light, rainy, foggy, and long-distance monitoring scenarios. Experimental results show that this application maintains high detection performance under diverse and challenging conditions, demonstrating excellent robustness. (Reference) Figure 11 As shown, the model can successfully handle:

[0149] Figure 11 (a) is a dark scene and Figure 11 (c) represents a low-light scenario: In low-light environments such as at night or with only dim headlights, the model is still able to effectively capture the outlines of vehicles and passengers and make accurate identifications.

[0150] Figure 11 (b) is an occlusion scenario: Even when the target is severely occluded by objects such as vegetation and cargo, and only part of the body outline is exposed, the model can still successfully infer the presence of the passenger thanks to its powerful contextual understanding ability.

[0151] Figure 11 (d) represents rainy or foggy scenes: In scenes where image clarity is reduced due to severe weather such as rain and fog, the model demonstrates stable detection capabilities without significant performance degradation.

[0152] These success stories collectively demonstrate the application's strong adaptability to real-world, ever-changing environments.

[0153] Furthermore, the intelligent detection method for illegal passenger carrying based on the CLIP and D-Fine cascaded framework provided in this application is applied to a practical illegal passenger carrying detection system and a layered architecture design is adopted, with reference to... Figure 12 As shown, the entire system consists of four core layers: data acquisition layer, model testing layer, business application layer, and relational database. This architecture fully considers the complexity of the actual deployment environment and the actual needs of law enforcement operations, ensuring that the system has high availability, high scalability, and high maintainability.

[0154] Among them, the data acquisition layer is responsible for acquiring data in real time from the urban traffic monitoring network, and performing unified data access and preprocessing through the big data platform. It is mainly responsible for data access and preprocessing to provide high-quality input for the subsequent model detection layer.

[0155] The model detection layer serves as the core, integrating the two-stage cascaded detection framework proposed in this application to achieve high-precision and robust identification of illegal passenger-carrying behavior. The CLIP-ILP model performs rapid semantic matching on the input image, filtering out a large number of redundant frames unrelated to trucks and passenger-carrying behavior, thus improving overall processing efficiency and recall. For the candidate images filtered by CLIP, an improved D-FINE model is used for deep analysis. A large kernel attention module (LKA) is introduced between the backbone network and the Transformer encoder to enhance long-distance dependency perception capabilities and improve detection performance for small targets and partially occluded scenes.

[0156] The business application layer provides various business functions for law enforcement personnel. This layer is geared towards traffic law enforcement business and provides multi-dimensional application functions: real-time display of detected violations in typical interfaces, support for the generation, archiving and push of relevant evidence, and linkage with the law enforcement platform to achieve closed-loop case processing; and provides statistical analysis, trend prediction and high-risk area warning to assist traffic management departments in optimizing law enforcement strategies.

[0157] The relational database employs a distributed storage architecture to achieve efficient storage, indexing, and retrieval of massive amounts of surveillance images, detection results, and metadata. The storage system supports hierarchical management of hot and cold data, redundant backups, and seamless integration with big data analytics platforms, ensuring the system's stability and scalability under massive data processing demands.

[0158] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0159] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A method for intelligent detection of illegal passenger carrying in vehicles based on a CLIP and D-Fine cascaded framework, characterized in that, Includes the following steps: S1: Collect real-time video frame images from the preset traffic monitoring network, and after frame segmentation, noise reduction and normalization, input the generated monitoring images into the CLIP-ILP model. S2: Using the CLIP-ILP model as a filter and combining it with the Top-K filtering strategy, calculate the cosine similarity between each monitoring image and the text description, and use the category corresponding to the highest similarity as the judgment condition to filter out candidate images; S3: The candidate image is input into the context-enhanced D-FINE detection model with the large kernel attention module, and the preset target is located and multi-class detected by multi-scale feature fusion and distributed regression methods, and then multi-class detection results are output. The context-enhanced D-FINE detection model includes an HGNetv2 backbone network, a feature pyramid network, a large kernel attention module, a Transformer encoder, a Transformer decoder, and a multi-task detection head; The large kernel attention module is integrated between the feature pyramid network and the Transformer encoder. The large kernel attention module includes three consecutive convolutional layers: a 5×5 depthwise convolution, a 7×7 dilated depthwise convolution, and a 1×1 pointwise convolution. The calculation process of the large kernel attention module for the input candidate feature X is as follows: ， ,in, It is a 5×5 depthwise convolution. It is a 7×7 depthwise convolution with holes. For 1×1 pointwise convolution, This represents the Sigmoid activation function. This represents element-wise multiplication; The context-enhanced candidate features are then fed into the Transformer encoder. S4: Perform spatial co-occurrence rules, license plate location heuristic rules, size consistency rules, and context consistency rules to verify multi-class detection results and mark multi-class detection results that fail the verification as suspicious or remove them; S5: Output the verified multi-category detection results as illegal manned transport detection results to the target terminal and store them synchronously in the database.

2. The vehicle violation occupancy intelligent detection method based on the CLIP and D-Fine cascade framework according to claim 1, characterized in that, In step S2, the method includes: S21: The input surveillance image is fed into the visual encoder of the CLIP-ILP model and the preset text description is fed into the text encoder of the CLIP-ILP model as a classification label, so as to extract the image features and text features of the surveillance image respectively. S22: Calculate the cosine similarity between image features and each text feature in the shared latent space and use the Top-K screening strategy to determine the image feature with the highest similarity to the prompt related to illegal passenger transport as the candidate image.

3. The vehicle violation occupancy intelligent detection method based on the CLIP and D-Fine cascade framework according to claim 1, characterized in that, The HGNetv2 backbone network performs multi-scale feature extraction on candidate images, including 8×8×256, 16×16×512, and 32×32×1024.

4. The vehicle violation occupancy intelligent detection method based on the CLIP and D-Fine cascade framework according to claim 1, characterized in that, The feature pyramid network fuses the extracted multi-scale features from top to bottom and unifies the channels to output candidate features at scales of 8×8×256, 16×16×256 and 32×32×256.

5. The vehicle violation occupancy intelligent detection method based on the CLIP and D-Fine cascade framework according to claim 1, characterized in that, The context-enhanced D-FINE detection model employs a multi-task learning strategy and various data augmentation strategies, and is trained using the AdamW optimizer. The various data augmentation strategies include random cropping, horizontal flipping, color jittering, and MixUp-based image blending. Furthermore, the loss function of the context-enhanced D-FINE detection model comprehensively considers the classification loss and regression loss of each category: ,in For the total loss function, For a set of categories, and These are the classification and regression loss weights for each category. and These are the classification and regression loss functions for each category.

6. The intelligent detection method for illegal passenger carrying in vehicles based on a CLIP and D-Fine cascaded framework according to claim 1, characterized in that, When training the context-enhanced D-FINE detection model, the method includes: S31: Select surveillance images focusing on freight vehicles from the preset traffic monitoring network and generate a dataset; S32: Manually label the bounding boxes of the subset of images containing illegal passenger transport behavior and the negative sample images in the dataset using the principle of tightly surrounding the target. The key categories of the labeling include: person, the main detection target, i.e., passengers illegally riding in the cargo box; truck_bed / tricycle_bed, key context targets, used for LVM spatial co-occurrence rule judgment; license_plate, key context targets, used for LVM license plate location heuristic rules.

7. The vehicle violation occupancy intelligent detection method based on the CLIP and D-Fine cascade framework according to claim 6, characterized in that, In step S4, the spatial co-occurrence rule verifies whether the person category detection is indeed located within the spatial range of the cargo carrier truck_bed or tricycle_bed. This is achieved through a dual criterion of center point inclusion and IoU overlap. If the center point of the person target falls within the bounding box of the bed category, or if its IoU is greater than a threshold, the detection is successful. When it is determined that there may be violations, the confidence level will be increased by verifying the person's identity. ； The license plate position heuristic rule is based on the relative stability of license plate positions in traffic monitoring scenarios. It uses license plate detection results to infer the possible location area of ​​the person target. When the lower edge of the person target is lower than the lower edge of the nearest license plate minus the safety margin τoffset, it is inferred that it is more likely to be in the driver's cab or outside the vehicle, which does not constitute a violation. The size consistency rule is based on the physical size constraints of personnel targets in the monitoring scenario, and filters obviously unreasonable detection results through absolute size constraints and relative size constraints. The context consistency rule verifies the semantic and logical relationships between different categories of detection results, ensuring that there is a spatial association between truck_bed and truck, tricycle_bed and tricycle, and the rationality of the personnel-carrier association.

8. A CLIP and D-Fine cascade framework based vehicle violation manned intelligent detection system using the method of any one of claims 1-7, characterized in that, include: The image processing module is used to acquire real-time video frame images from the preset traffic monitoring network and, after frame segmentation, noise reduction and normalization, input the generated monitoring images into the CLIP-ILP model. The high recall filtering module is used to use the CLIP-ILP model as a filter and combine it with the Top-K filtering strategy to calculate the cosine similarity between each monitoring image and the text description and use the category corresponding to the highest similarity as the judgment condition to filter out candidate images. The high-precision verification module is used to input the candidate image into the context-enhanced D-FINE detection model constructed by introducing the large kernel attention module, and to locate and detect the preset target in multiple categories through multi-scale feature fusion and distributed regression methods, and then output multi-category detection results; The multi-class verification module is used to perform spatial co-occurrence rules, license plate location heuristic rules, size consistency rules and context consistency rules verification on multi-class detection results, and mark multi-class detection results that fail the verification as suspicious or remove them; The results output module is used to output the verified multi-category detection results as illegal manned transport detection results to the target terminal and store them synchronously in the database.

Images