Traffic scene vehicle and event recognition method based on edge small model and cloud large model cooperation

Abstract

The present application relates to a kind of traffic scene vehicle and event identification method based on edge small model and cloud big model cooperation, belong to intelligent transportation technical field.The method is in view of the problems such as low recognition accuracy in complex environment, rely on a large number of labeled data, cannot identify unknown class in prior art, propose multimodal traffic visual perception coding system, big model enhanced small sample collaborative training algorithm and dynamic trigger type double model inference framework.Small target feature expression is enhanced by semantic visual joint coding, the dependence of small model on labeled data is reduced by using big model pseudo label and knowledge distillation, and cloud big model is dynamically called for secondary discrimination according to confidence and scene complexity.The present application effectively improves the recognition robustness of the system under harsh conditions, and realizes the balance of open vocabulary perception and low-resource efficient deployment.

Description

Technical Field

[0001] This invention belongs to the field of intelligent transportation technology and relates to a method for recognizing vehicles and events in traffic scenarios based on the collaboration of edge small models and cloud large models. Background Technology

[0002] With the acceleration of urbanization and the continuous growth of motor vehicle ownership, problems such as traffic congestion, frequent accidents, and low management efficiency have become increasingly prominent. Traditional technical systems have undergone three generations of evolution: The first generation, video motion detection methods based on image processing, employed background subtraction and inter-frame subtraction algorithms. While low in deployment cost, they had a high false alarm rate, exceeding 50% under complex lighting conditions, and could not distinguish vehicle types, with a near 100% false negative rate for stationary vehicles. The second generation, shallow machine learning detection methods, achieved vehicle recognition through manual feature extraction combined with classifiers. Accuracy reached 82% under ideal weather conditions, but feature representation capabilities were limited, with a recall rate of less than 55% for occluded vehicles, and poor model generalization ability, with average accuracy dropping by more than 30% when migrated to new intersections. The third generation, deep learning object detection systems, used convolutional neural networks, represented by the YOLOv5s model. Deployed on edge AI devices, they could achieve real-time detection and support multi-class vehicle recognition. However, these models still have significant limitations: model updates require full retraining, resulting in long iteration cycles.

[0003] Early object detection models used multi-scale sliding windows combined with histogram of oriented gradients (HARQ) features and support vector machine (SVM) classifiers, but faced the problem of exploding computational complexity, with inference latency as high as 1.5 seconds. Deep learning-based improvements used feature pyramid network structures to fuse multi-scale features, which improved the recall rate of small targets, but still relied on a large number of labeled samples, and the localization error for vehicles smaller than 30 pixels exceeded 35%.

[0004] Meanwhile, existing technologies have significant shortcomings at the manual data level: small targets account for over 40% of traffic monitoring videos, but manual annotation suffers from large boundary biases, low consistency, and a subjective mislabeling rate as high as 22%, resulting in poor dataset compatibility. At the model algorithm level, small models, due to limited parameter capacity, heavily rely on high-quality annotations. Small targets contribute less than 5% to the total loss during training, causing the model to ignore them. Furthermore, manual annotation has poor ability to preserve the identity of targets in consecutive frames, with a high identity switching rate, severely impacting multi-target tracking accuracy. The high cost of annotation also causes model iteration to lag behind changes in the traffic environment.

[0005] Existing annotation strategies typically mark highly occluded targets as ignored, but small models cannot distinguish between truly absent targets and those marked as ignored, leading to an 18% increase in false detection rate. More seriously, the inconsistent category systems and scale division standards used by different annotation teams further exacerbate performance degradation across datasets. In summary, traditional methods struggle to achieve high-precision small target recognition, open-category perception, and rapid deployment with low data dependencies in complex dynamic environments. There is an urgent need to overcome these technical bottlenecks and build more efficient, accurate, and highly adaptive intelligent traffic perception systems. Summary of the Invention

[0006] The purpose of this invention is to provide a traffic scene vehicle and event recognition method based on the collaboration of edge small models and cloud large models, aiming to solve the core technical problems in the existing technology such as low recognition accuracy, reliance on large-scale manual annotation, weak open category recognition capability, and large system latency.

[0007] To achieve the above objectives, this invention employs Grounding DINO 2.0 as the large-scale cloud model and YOLOv7 as the small-scale edge model. Leveraging the powerful zero-shot and few-shot learning capabilities of the large model, high-precision detection can be achieved with only a few hundred image data points, driven by prompt words. Simultaneously, pseudo-labels and knowledge distillation strategies output by the large model guide the training of the small model, significantly reducing its dependence on large-scale manually labeled data and enabling it to achieve high accuracy at low cost. During deployment, the small model is used for real-time monitoring across the entire road segment, and the triggering of the large model for secondary discrimination is dynamically determined based on confidence thresholds and scene complexity. This achieves a highly efficient collaborative mechanism of "normally relying on the small model to ensure efficiency, and calling the large model at critical moments to ensure accuracy."

[0008] To achieve the above objectives, the present invention provides the following technical solution:

[0009] A method for vehicle and event recognition in traffic scenes based on the collaboration of edge small models and cloud large models includes the following steps:

[0010] S1: Construct a multimodal traffic visual perception coding system to perform semantic-visual joint coding, small target multi-scale feature enhancement, and spatiotemporal context feature enhancement on the input traffic monitoring video stream;

[0011] S2: A small-sample collaborative training algorithm based on large model enhancement, which uses a large cloud model to generate pseudo-labels and guides the training of marginal small models through knowledge distillation.

[0012] S3: Deploy a dynamically triggered dual-model inference framework, using a trained edge small model on the edge side to perform real-time monitoring of the entire road segment, and dynamically triggering the cloud large model for secondary discrimination based on confidence and scene complexity.

[0013] Furthermore, the semantic-visual joint encoding in S1 specifically involves defining the input image. With text prompt set Joint features are extracted through a dual-encoder structure of a large model, where the visual encoder uses the Swin-L Transformer to extract hierarchical feature maps. The text encoder uses the BERT-large model to generate word-level embeddings. ,and Cross-modal attention module performs operations Output a semantically guided visual response map , used to generate open vocabulary detection suggestion boxes; where , , It is a learnable projection matrix.

[0014] Furthermore, the multi-scale feature enhancement of small targets in S1 is implemented using a weighted bidirectional feature pyramid network (W-BiFPN). The backbone network of the edge small model outputs feature maps C3, C4, and C5, with spatial resolutions of [missing information]. , , The number of channels are 256, 512, and 1024 respectively; W-BiFPN uses learnable weights To achieve cross-scale fusion, the calculation formula is as follows: ,in , Indicates the first p The input source layer of the layer.

[0015] Furthermore, the spatiotemporal context feature enhancement in S1 specifically involves: enhancing the continuous... T =5 frames of images, extract the P4 feature map sequence of the edge mini-model. The input bidirectional gated recurrent unit (GRU) captures timing dependencies, and its forward computation is as follows: Backward calculation is Final output Inject into the detection head.

[0016] Furthermore, S2 specifically includes:

[0017] S21: Utilize a frozen cloud-based large model to perform zero-shot inference on unlabeled video streams and generate a weakly supervised pseudo-label set. ,in For bounding box, As a category, ∈[0,1] represents the confidence level. Papers such as "DistillingGroundingDINO for an Edge-Cloud Collaborative Advanced Driver AssistanceSystem," "F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Models," "Zero-Shot Open-Ended Video Inference," and "Zero-Shot VideoMoment Retrieval from Frozen Vision-Language Models" all employ a frozen large model to perform zero-shot inference on unlabeled video streams, thereby generating a weakly supervised pseudo-label set (including bounding boxes, categories, and confidence levels) as a whole. This method references the algorithmic approach of these papers, generating a label set by taking an initial image and text instructions as input.

[0018] S22: Perform dual screening on the pseudo-label set, with screening criteria including visual confidence level. >0.75 Text alignment score By combining spatial consistency verification and geofence filtering, high-quality pseudo-labels are obtained.

[0019] S23: Integrating high-quality pseudo-tags with manually labeled sets Merge and build a fine-tuning dataset ;

[0020] S24: Design a cross-modal knowledge distillation loss function Guiding the training of small edge models, where For edge small models, the native loss, Weights for dynamic knowledge distillation, relative weight coefficients =0.35, =0.25, =0.15, This represents the distillation loss for each output head of the YOLOv7s model. Since student YOLOv7s models are prone to losing detail, this setting... > > These lost details can be recovered during the learning process. , , These three parameters are set manually and are used to fine-tune the loss function to improve the model's prediction accuracy. jValues ​​range from 1 to 3, representing the rear, center, and front detection heads, respectively. These three detection heads are... , , Dynamic weights The parameters are adaptively adjusted based on factors such as training phase, student-teacher differences, and task difficulty to prevent overfitting in the early stages of training. The calculation method is as follows:

[0021] , , ,in t Indicates the number of training sessions. T Indicates the total number of training sessions. e Indicates an index. This represents the maximum value of the dynamic weight; This represents the maximum value of the dynamic weights. Training starts at t=0, therefore... Will from Initially, the will gradually diminishes as training progresses. This enables a knowledge distillation strategy that involves "strong guidance in the early stages and weak constraints in the later stages."

[0022] Furthermore, the native loss of the edge small model Loss from predicted boxes Prediction box distribution loss Binary cross-entropy loss for predicted categories Composition, specifically represented as ;in , , express CIOU The loss, IOU The intersection-union ratio (IUU) represents the ratio of intersection to union, with values ​​ranging from [0,1]. The calculation formula is: , CIOU (CompleteIntersection over Union) represents the loss function used for bounding box regression in object detection, which optimizes the alignment and distance of the boxes by introducing an aspect ratio penalty term; Let Euclidean distance be the center point of the predicted bounding box and the ground truth bounding box. The diagonal distance is the minimum bounding rectangle containing both the predicted and ground truth boxes. For the weight function, Used to measure aspect ratio consistency; , k Calculation is performed only for the two non-zero label positions. To predict probabilities for the model, Category weights; , For real labels, Predict probabilities for the model.

[0023] Furthermore, the dynamic triggering condition in S3 is a two-factor logical AND operation: ,in s The confidence score output by the edge small model is s, where s is the bounding box predicted for each small object. A confidence score is output for each small object, and the formula is: Among them, target confidence is the probability of determining whether a detected object is the target object; threshold =0.4, SCI is the Scene Complexity Index, the threshold. =0.65; the formula for calculating SCI is: ,in For occlusion rate, For the variance of illumination, For target density, weight , , .

[0024] Furthermore, the edge small model is a YOLOv7s model, deployed on an edge AI device, with an input resolution of 640×640; the cloud large model is a Grounding DINO2.0 model.

[0025] Furthermore, after the cloud-based large model performs secondary discrimination, an Intersection over Union (IoU) threshold of 0.5 and a semantic consistency fusion strategy are used to fuse the output of the cloud-based large model with the output of the edge small model, updating the final structured detection result. The cloud-based large model performs secondary discrimination using Grounding DINO 2.0 as the cloud-based large model and YOLOv7 as the lightweight detection model at the edge. The large model plays a dual role in the system: a knowledge provider during the training phase and a key decision arbiter during the inference phase.

[0026] During the training phase, the cue word-driven capabilities and powerful multimodal priors of Grounding DINO 2.0 are fully utilized to generate high-quality detection results and pseudo-labels with only a few hundred labeled samples. Through pseudo-label supervision and cross-modal knowledge distillation (such as feature alignment and relation distillation), the YOLOv7 small model is effectively guided to learn, significantly reducing its dependence on large-scale manually labeled data and achieving high-performance training under small sample conditions.

[0027] During the deployment phase, the system primarily utilizes small edge models for real-time detection across all time domains and regions. Simultaneously, a dynamic triggering mechanism is introduced, combining the prediction confidence score output by the small model with the Scene Complexity Index (SCI) to adaptively determine whether to invoke a large cloud model for secondary, refined discrimination of suspicious areas. This mechanism achieves a collaborative reasoning paradigm of "relying on small models for efficiency in normal situations and invoking large models for accuracy in critical moments," significantly improving detection reliability in complex scenarios while maintaining overall system low latency.

[0028] A traffic scene vehicle and event recognition system for implementing the method includes:

[0029] The perception module deployed at the edge is used to execute the edge mini-model to perform real-time video stream processing and preliminary detection;

[0030] A semantic enhancement module deployed in the cloud is used to execute the large cloud model and perform fine-grained judgment of open lexical terms;

[0031] The dynamic trigger management module is used to calculate the scene complexity index and manage the collaborative reasoning process between the edge and the cloud.

[0032] The perception module, semantic enhancement module, and dynamic trigger management module are connected via network communication.

[0033] The beneficial effects of this invention are as follows:

[0034] (1) By constructing a semantic-visual joint coding system and a multi-scale feature enhancement mechanism, the system can effectively overcome the impact of insufficient lighting, nighttime environment and adverse weather conditions such as rain, snow and fog on recognition accuracy. After introducing spatiotemporal context modeling, the system enhances the consistency of detection between consecutive frames, effectively suppresses missed detections and misjudgments caused by target occlusion or motion blur, and thus maintains stable perception performance in dynamically changing traffic scenarios.

[0035] (2) With the help of high-quality pseudo-labels generated by large models and knowledge distillation mechanism, marginal small models can converge quickly with very few manually labeled samples, breaking through the bottleneck of traditional small models relying on large-scale labeled data to achieve high accuracy. This mechanism significantly shortens the model training and iteration cycle and reduces the data cost of system deployment and maintenance.

[0036] (3) Relying on the zero-shot reasoning and semantic understanding capabilities of the cloud-based large model, the system can respond to natural language commands and identify new vehicle categories or traffic events that have not appeared in the training set, such as temporary emergency facilities and specific behavioral events. This open vocabulary detection capability enables the system to have good scalability and scene adaptability, and can cope with the ever-evolving traffic control needs without retraining.

[0037] (4) Through a dynamic trigger-based collaborative reasoning mechanism, the system relies on small edge models to ensure real-time processing efficiency under normal traffic flow conditions, and intelligently calls large cloud models for secondary discrimination under low confidence or complex scenarios, forming an efficient collaborative paradigm of "lightweight edge perception - deep cloud analysis". This architecture effectively improves the recognition accuracy of key events while controlling response latency.

[0038] (5) The edge side adopts a lightweight model and hardware acceleration technology to meet the requirements of low power consumption and high efficiency; the cloud model uses a low-frequency calling strategy to balance computing resources and recognition effect. The overall system has good scalability and stability and is suitable for large-scale deployment in urban road network monitoring scenarios.

[0039] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0040] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein:

[0041] Figure 1 This is a schematic diagram of the model collaborative architecture provided by the present invention for vehicle and traffic event recognition in traffic scenarios;

[0042] Figure 2 This is a schematic diagram illustrating the vehicle detection performance of the collaborative model proposed in this invention at different times of night.

[0043] Figure 3 This is a schematic diagram illustrating the detection performance of the collaborative model proposed in this invention under different visibility conditions;

[0044] Figure 4 This is a schematic diagram illustrating the detection effect of the collaborative model proposed in this invention on different traffic events. Detailed Implementation

[0045] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0046] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0047] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0048] First of all Figure 1 The collaborative architecture of the present invention is illustrated in the diagram. This architecture diagram clearly demonstrates the collaborative relationship between the large cloud model Grounding DINO 2.0 and the small edge model YOLOv7. The left side shows the Grounding DINO 2.0 model, which receives images and text prompts, such as "car, truck, snow, spilled material," as input. Internally, it contains two parallel backbone networks: the Swin-L Transformer Backbone for extracting deep image features and the Bert-large Backbone for extracting deep text features. The features extracted by both are then fed into the "semantic-visual joint encoding" module after passing through a feature enhancer. This module is crucial for achieving open vocabulary understanding; it aligns textual semantics with visual information to generate Keys & Values ​​for subsequent detection.

[0049] The image on the right is the YOLOv7 model. It receives image input and passes it through a series of convolutional modules (Conv), ELAN modules, MP modules, and reparameterized convolutional modules (RepConv), forming its main detection network architecture. YOLOv7 performs detection at three different feature scales to accommodate targets of different sizes.

[0050] The synergy between the two is reflected in two aspects. First, during the training phase, the visually weighted features of Grounding DINO 2.0 are injected into the YOLOv7 training process through a "knowledge distillation" process, serving as additional supervisory signals to improve YOLOv7's performance. Second, during the inference phase, YOLOv7 performs real-time detection at the edge. After the detection results and alarm outputs are processed by a judgment module, which determines whether to initiate "secondary discrimination" based on preset conditions (triggers). Once triggered, the keyframes are uploaded to Grounding DINO 2.0 in the cloud for more refined analysis.

[0051] Example 1: Efficient and accurate vehicle detection based on real-time video stream

[0052] First, a trained YOLOv7s model is deployed on an edge computing device, such as the NVIDIA Jetson AGX Xavier. This model receives real-time video streams from roadside surveillance cameras with an input resolution of 640x640. To achieve real-time processing, the model is accelerated using TensorRT, achieving inference speeds of up to 62 frames per second and ensuring a single-frame processing latency of less than 16 milliseconds.

[0053] Secondly, the model performs multimodal traffic visual perception encoding on each frame of image. The YOLOv7s backbone network first extracts feature maps at three levels: C3, C4, and C5. To address the difficulty in detecting small, distant targets in traffic scenes, the system employs a weighted bidirectional feature pyramid network (W-BiFPN) to fuse these feature maps across scales. Through learnable weights, W-BiFPN adaptively integrates information from different levels, significantly enhancing the P3 layer's feature response capability for small targets. Simultaneously, to ensure the continuity and stability of detection, the system caches the most recent five consecutive frames, extracts their P4 feature map sequence, and inputs it into a lightweight bidirectional gated recurrent unit (GRU) network. This network can capture the temporal dependencies of vehicles, effectively suppressing ID switching issues caused by brief occlusions or rapid movement, thus significantly improving the MOTA (Motion Over-The-Air) accuracy of multi-target tracking.

[0054] Next, the model performs forward inference, outputting structured JSON data containing vehicle bounding boxes, categories, confidence scores, coordinates, and timestamps. Simultaneously, the dynamic trigger management module calculates the Scene Complexity Index (SCI) for the current frame in real time. This index is calculated using a weighted formula. Calculations were performed to comprehensively evaluate the occlusion rate, illumination changes, and target density in the scene.

[0055] The Scene Complexity Index (SCI) is a metric used to quantify the complexity of an image or visual scene. It has wide applications in computer vision, remote sensing, autonomous driving, image quality assessment, and human-computer interaction.

[0056] 1. Occlusion rate is directly annotated using the model: original target bounding box; visible portion bounding box. Then, the occlusion rate is calculated using the following formula:

[0057] O rate =1 - (Area of ​​the original target bounding box) / (Area of ​​the visible bounding box)

[0058] in O rate This indicates the occlusion rate.

[0059] 2. Changes in light intensity are calculated using the following formula:

[0060] (1) Average brightness:

[0061]

[0062] It is the calculated mean. W Indicates the width of the image. H Indicates image height; This means that the grayscale value is first calculated based on the pixel values ​​of the red, green, and blue channels, and then the average grayscale value of all images is calculated. The constant for calculating the grayscale value was determined by the International Commission on Illumination (CIE) through large-scale human vision experiments. It is the red channel value of the pixel. It is the green channel value of the pixel. These are the blue channel values ​​of the pixels, and the values ​​of these pixels range from 0 to 255.

[0063] (2) Illumination variance:

[0064]

[0065] in This represents the variance of illumination.

[0066] (3) Target density:

[0067]

[0068] in Indicates the target density.

[0069] Finally, dynamic triggering and secondary discrimination are performed. The system evaluates each detection result output by YOLOv7s. If the detection confidence of a certain target is... s Below the preset threshold τ If the SCI (Solution Complexity Index) of the current scene is greater than the threshold θ = 0.65, then the two-factor logical AND condition is satisfied, triggering the secondary discrimination mechanism. The dynamic trigger management module uploads the keyframe image and related text prompts, such as "car" and "truck," to the cloud server. Upon receiving the request, the cloud-deployed Grounding DINO2.0 large model utilizes its powerful semantic understanding and visual localization capabilities for refined recognition. The recognition results of the large model are transmitted back to the edge system via the network. The edge system employs an Intersection over Union (IoU) threshold of 0.5 and a semantic consistency strategy to fuse the cloud results with the local results, update and output the final, more confident detection result.

[0070] Figure 2 The performance of this collaborative model in vehicle detection at different times of night is demonstrated. Four images from 8:15 PM to 10:10 PM show that even in poor lighting conditions, including headlight glare and shadows, the system can still stably detect vehicles on the road and accurately mark their positions and confidence levels with blue boxes. This verifies that the method maintains strong robustness in low-light environments.

[0071] Example 2: Recognition of Complex Traffic Events under Low Light and Inclement Weather Conditions

[0072] First, the YOLOv7s model on the edge still performs real-time analysis of the surveillance video stream 24 / 7. When the spatiotemporal context feature enhancement module analyzes consecutive frame sequences, it may detect anomalies. For example, if a target remains stationary within a lane for more than a preset time threshold, this could be an illegal parking incident or a vehicle malfunction. Or, if a target's trajectory is opposite to the driving direction defined by the lane lines, this could be a wrong-way driving incident. In these cases, the YOLOv7s model might only provide a detection box categorized as "car," but the detection confidence for events like "illegal parking" or "wrong-way driving" would be very low.

[0073] Secondly, secondary discrimination is triggered in the cloud. Although the edge model has low confidence in the event, such events are usually of high risk. The system will combine this with the calculated Scene Complexity Index (SCI). For example, the SCI value will be high on smoggy days or at night, thus forcing the large cloud model to perform secondary discrimination. The dynamic trigger management module will upload keyframe images containing potential events, along with one or more descriptive text prompts, such as "a broken-down truck with hazard lights on at night," "a pedestrian riding an electric bike without a helmet," or "litter on the road," to the cloud server.

[0074] Secondly, deep semantic analysis is performed using a large cloud-based model. The Grounding DINO2.0 large model in the cloud leverages its superior zero-shot inference capabilities to perform deep semantic understanding of received image and text prompts. It can not only identify objects in images but also understand complex semantic concepts, such as "a broken-down truck with hazard lights on" or "not wearing a helmet," scenarios that require combining state and behavior for judgment. Thanks to its open-vocabulary detection feature, even novel event descriptions never seen in the training data, such as "temporary emergency facilities," can elicit reasonable judgments from the large model.

[0075] Finally, event confirmation and alerting. After the cloud-based big data model completes its precise judgment, its output structured event description information, such as event type (illegal parking), location, and target attributes, will be sent to the edge system. The edge system integrates the authoritative judgment results from the big data model, confirms the event, immediately generates standardized alert information, and sends it to the traffic management center via standard protocols such as ONVIF for rapid response.

[0076] Figure 3 The detection performance of this collaborative model under different visibility conditions is demonstrated. In dense fog where visibility drops from 300 meters to 86 meters, image quality is severely degraded, but the system is still able to successfully detect distant vehicles using green boxes. Figure 4 The figure specifically demonstrates the model's detection performance on various traffic events. The figure clearly marks complex scenarios such as "shelf falling events," "paint spill events," "vehicle collision events," and "pedestrian guardrail climbing events," proving the powerful event recognition capabilities of this method.

[0077] Example 3: Rapid Adaptation and Iteration of Models Based on Few-Shot Learning

[0078] Suppose that traffic management departments need to add the ability to identify "electric scooters," a new type of traffic participant. Traditional methods require collecting tens of thousands of images containing electric scooters and manually labeling them, which is time-consuming and labor-intensive. This invention adopts the following efficient process:

[0079] The first step is automatic pseudo-label generation. Operations personnel do not need to perform any labeling. Instead, they utilize the Grounding DINO2.0 large model deployed in the cloud to perform batch-processing zero-shot inference on massive amounts of unlabeled historical traffic monitoring video streams. During inference, the large model is provided with a textual cue word for a new category, namely "electric scooter." The large model automatically searches for targets in the video frames that match this textual description and generates a weakly supervised pseudo-label set containing bounding boxes, categories, and confidence scores.

[0080] The second step is high-quality pseudo-label screening. Since large models can also make mistakes during zero-shot inference, the generated pseudo-labels need to be rigorously screened. The system will initiate a dual screening mechanism: first, only visual confidence scores are retained. The system first selects samples with scores greater than 0.75. Then, it calculates the alignment score between the text prompt and the image region, retaining samples with scores greater than 0.7. Building on this, the system also incorporates spatial consistency checks, such as requiring the Intersection over Union (IoU) of detection boxes for the same target in adjacent frames to be greater than 0.3, to eliminate drifting and flickering detection boxes. Simultaneously, it utilizes geofencing filtering, such as excluding illogical samples like "electric scooters" appearing on highway main roads. Through this series of filtering steps, high-quality pseudo-label data is obtained.

[0081] The third step is to construct the fine-tuning dataset. The selected high-quality pseudo-label data is combined with a very small set of manually labeled images, such as containing only 300 to 500 images, to form the training dataset for model fine-tuning.

[0082] The fourth step is knowledge distillation-guided training. A cross-modal knowledge distillation strategy is used to fine-tune the YOLOv7s model on the edge side. During training, the total loss function consists of two parts: .in, It is the native loss of YOLOv7, which consists of the predicted box loss. The loss consists of two parts: the prediction box distribution loss (DFL) and the prediction class binary cross-entropy loss, ensuring that the model can learn the basic detection information provided by the pseudo-labels. Another part is the knowledge distillation loss. It uses a feature alignment loss function to force the feature representation of YOLOv7 to align with the feature representation of Grounding DINO2.0. In this way, more robust and generalized knowledge about "electric scooters" in the large model is "distilled" and transferred to the small model.

[0083] Through the workflow of this embodiment, the YOLOv7s model only needs about 400 high-quality pseudo-label images to improve the average accuracy mAP@0.5 for the "car" category. Compared with traditional training using only this small amount of data, this greatly improves performance, reduces data costs and model update cycles, and demonstrates the significant advantages of this invention in terms of low data dependence and rapid adaptation.

[0084] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for vehicle and event recognition in traffic scenes based on the collaboration of edge small models and cloud large models, characterized in that: Includes the following steps: S1: Construct a multimodal traffic visual perception coding system to perform semantic-visual joint coding, small target multi-scale feature enhancement, and spatiotemporal context feature enhancement on the input traffic monitoring video stream; S2: A small-sample collaborative training algorithm based on large model enhancement, which utilizes a large cloud model to generate pseudo-labels and guides the training of marginal small models through knowledge distillation; specifically including: S21: Utilize a frozen cloud-based large model to perform zero-shot inference on unlabeled video streams and generate a weakly supervised pseudo-label set. ,in For bounding box, As a category, The confidence level is ∈[0,1]. S22: Perform dual screening on the pseudo-label set, with screening criteria including visual confidence level. >0.75 Text alignment score By combining spatial consistency verification and geofence filtering, high-quality pseudo-labels are obtained. S23: Integrating high-quality pseudo-tags with manually labeled sets Merge and build a fine-tuning dataset ; S24: Design a cross-modal knowledge distillation loss function Guiding the training of small edge models, where For edge small models, the native loss, Weights for dynamic knowledge distillation, relative weight coefficients =0.35, =0.25, =0.15, This refers to the distillation loss corresponding to each output head of the YOLOv7s model. j Values ​​range from 1 to 3, representing the rear, center, and front detection heads, respectively. These three detection heads are... , , ; S3: Deploy a dynamically triggered dual-model inference framework, using a trained edge small model on the edge side to perform real-time monitoring of the entire road segment, and dynamically triggering the cloud large model for secondary discrimination based on confidence and scene complexity.

2. The traffic scene vehicle and event recognition method based on the collaboration of edge small models and cloud large models according to claim 1, characterized in that: The semantic-visual joint encoding in S1 specifically involves defining the input image. With text prompt set Joint features are extracted through a dual-encoder structure of a large model, where the visual encoder uses the Swin-L Transformer to extract hierarchical feature maps. The text encoder uses the BERT-large model to generate word-level embeddings. ,and ; Cross-modal attention module performs operations Output a semantically guided visual response map , used to generate open vocabulary detection suggestion boxes; where , , It is a learnable projection matrix.

3. The traffic scene vehicle and event recognition method based on the collaboration of edge small models and cloud large models according to claim 1, characterized in that: The multi-scale feature enhancement of small targets in S1 is achieved using a weighted bidirectional feature pyramid network (W-BiFPN). The backbone network of the edge small model outputs feature maps C3, C4, and C5, with spatial resolutions of [missing information]. , , The number of channels are 256, 512, and 1024 respectively; W-BiFPN uses learnable weights To achieve cross-scale fusion, the calculation formula is as follows: ,in , Indicates the first p The input source layer of the layer.

4. The traffic scene vehicle and event recognition method based on the collaboration of edge small models and cloud large models according to claim 1, characterized in that: The spatiotemporal context feature enhancement in S1 specifically involves: enhancing the continuous... T =5 frames of images, extract the P4 feature map sequence of the edge mini-model. The input bidirectional gated recurrent unit (GRU) captures timing dependencies, and its forward computation is as follows: Backward calculation is Final output Inject into the detection head.

5. The traffic scene vehicle and event recognition method based on the collaboration of edge small models and cloud large models according to claim 1, characterized in that: The native loss of the edge small model Loss from predicted boxes Prediction box distribution loss Binary cross-entropy loss for predicted categories Composition, specifically represented as ;in , express CIOU The loss, IOU Indicates intersection, union, and ratio. CIOU This represents the loss function used for bounding box regression in object detection. Let Euclidean distance be the center point of the predicted bounding box and the ground truth bounding box. The diagonal distance is the minimum bounding rectangle containing both the predicted and ground truth boxes. For the weight function, Used to measure aspect ratio consistency; , k Calculated only for the two non-zero label positions. To predict probabilities for the model, Category weights; , For real labels, Predict probabilities for the model.

6. The traffic scene vehicle and event recognition method based on the collaboration of edge small models and cloud large models according to claim 1, characterized in that: The dynamic triggering condition in S3 is a two-factor logical AND operation: ,in s The detection confidence score and threshold output by the edge small model. =0.4, SCI is the Scene Complexity Index, the threshold. =0.65; the formula for calculating SCI is: ,in For occlusion rate, For the variance of illumination, For target density, weight , , .

7. The traffic scene vehicle and event recognition method based on the collaboration of edge small models and cloud large models according to claim 1, characterized in that: The edge small model is a YOLOv7s model, deployed on an edge AI device, with an input resolution of 640×640; the cloud large model is a Grounding DINO2.0 model.

8. The traffic scene vehicle and event recognition method based on the collaboration of edge small models and cloud large models according to claim 1, characterized in that: After the large cloud model is triggered to perform secondary discrimination, the output results of the large cloud model and the small edge model are fused using an Intersection over Union (IoU) threshold of 0.5 and a semantic consistency fusion strategy to update the final structured detection result.

9. A traffic scene vehicle and event recognition system for implementing the method of any one of claims 1 to 8, characterized in that: include: The perception module deployed at the edge is used to execute the edge mini-model to perform real-time video stream processing and preliminary detection; A semantic enhancement module deployed in the cloud is used to execute the large cloud model and perform fine-grained judgment of open lexical terms; The dynamic trigger management module is used to calculate the scene complexity index and manage the collaborative reasoning process between the edge and the cloud. The perception module, semantic enhancement module, and dynamic trigger management module are connected via network communication.

Images