Traffic scene vehicle and event identification method based on cooperation of edge small model and cloud large model

Abstract

The invention relates to a traffic scene vehicle and event identification method based on cooperation of an edge small model and a cloud large model, and belongs to the technical field of intelligent traffic. Aiming at the problems of low recognition precision, dependence on a large amount of labeled data, incapability of recognizing unknown categories and the like in a complex environment in the prior art, the method provides a multi-modal traffic visual perception coding system, a large model enhanced small sample cooperative training algorithm and a dynamic trigger type double-model reasoning framework. Small target feature expression is enhanced through semantic and visual joint coding, dependence of a small model on annotation data is reduced by using a large model pseudo tag and knowledge distillation, and a cloud large model is dynamically called according to confidence and scene complexity for secondary discrimination. According to the method, the recognition robustness of the system under severe conditions is effectively improved, and the balance between open vocabulary perception and low-resource efficient deployment is realized.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of intelligent transportation, and relates to a traffic scene vehicle and event identification method based on edge small model and cloud large model cooperation. BACKGROUND

[0002] With the acceleration of urbanization and the continuous growth of motor vehicle ownership, traffic congestion, frequent accidents and low management efficiency have become increasingly prominent. The traditional technical system has undergone three generations of evolution: the first generation of video motion detection method based on image processing adopts background difference method and inter-frame difference algorithm, although the deployment cost is low, the false positive rate is high, which is more than 50% under complex lighting conditions, and it cannot distinguish vehicle types, and the missing detection rate of static vehicles is close to 100%. The second generation of shallow machine learning detection method realizes vehicle identification through manual feature extraction combined with a classifier, and the accuracy can reach 82% under ideal weather conditions, but the feature expression ability is limited, the recall rate of occluded vehicles is less than 55%, and the model generalization ability is poor, and the average precision decreases by more than 30% when migrating to a new intersection. The third generation of deep learning target detection system adopts a convolutional neural network, represented by the YOLOv5s model, which can realize real-time detection under the deployment of edge artificial intelligence devices and support multi-class vehicle identification. However, this type of model still has significant limitations: model updates require full retraining, with a long iteration cycle.

[0003] Early target detection small models use multi-scale sliding windows combined with histogram of oriented gradients features and support vector machine classifiers, but face the problem of computational complexity explosion, with a reasoning delay of up to 1.5 seconds. Deep learning improvement schemes use feature pyramid network structures to integrate multi-scale features, which improve small target recall rates, but still rely on a large number of labeled samples, and the positioning error for vehicles below 30 pixels is more than 35%.

[0004] At the same time, existing technologies have significant defects in artificial data: small targets account for more than 40% in traffic monitoring videos, but artificial labeling boundary deviation is large, consistency is low, and subjective mislabeling rate is as high as 22%, resulting in poor data set compatibility. At the model algorithm level, small models are highly dependent on high-quality labeling due to limited parameter capacity, and small targets contribute less than 5% to the total loss during training, causing the model to ignore small targets. In addition, artificial labeling has poor identity maintenance ability for consecutive frames, with a high identity switching rate, which seriously affects the accuracy of multi-target tracking. The high labeling cost leads to model iteration lagging behind changes in the traffic environment.

[0005] Existing labeling strategies usually mark high-occluded targets as ignored, but small models cannot distinguish between real no-targets and labeled ignored, resulting in an increase of 18% in false detection rate. More seriously, different labeling teams use different classification systems and scale division standards, further exacerbating the performance degradation of models across datasets. In summary, traditional methods are difficult to achieve high-precision small target recognition, open-class perception, and low-data-dependent rapid deployment in complex dynamic environments. It is urgent to break through the technical bottleneck and build a more efficient, accurate, and self-adaptive intelligent traffic perception system. SUMMARY

[0006] The purpose of the present application is to provide a traffic scene vehicle and event recognition method based on the cooperation of edge small models and cloud large models, aiming to solve the core technical problems of low recognition accuracy, dependence on large-scale manual labeling, weak open-class recognition ability, and large system delay in the prior art.

[0007] To achieve the above-mentioned purpose, the present application uses Grounding DINO 2.0 as a cloud large model and YOLOv7 as an edge small model. By using the powerful zero-shot and few-shot learning ability of the large model, high-precision detection can be achieved with only hundreds of image data through prompt word driving. At the same time, the pseudo-labels and knowledge distillation strategies output by the large model are used to guide the training of the small model, thereby significantly reducing the dependence of the small model on large-scale manual labeling data, enabling the small model to achieve a high level of accuracy at a low cost. In the deployment stage, the small model is used for real-time monitoring of the entire road section, and the confidence threshold and scene complexity are used to dynamically determine whether to trigger the large model for secondary discrimination, realizing an efficient cooperation mechanism of "relying on the small model to ensure efficiency in normal times, and calling the large model to ensure accuracy in critical times".

[0008] To achieve the above-mentioned purpose, the present application provides the following technical solutions: A traffic scene vehicle and event recognition method based on the cooperation of edge small models and cloud large models, comprising the following steps: S1: Constructing a multi-modal traffic visual perception coding system, performing semantic-visual joint coding, small target multi-scale feature enhancement, and spatio-temporal context feature enhancement on the input traffic monitoring video stream; S2: Based on the small sample cooperative training algorithm enhanced by the large model, generating pseudo-labels using the cloud large model and guiding the training of the edge small model through knowledge distillation; S3: Deploying a dynamic trigger type dual-model inference framework, using the trained edge small model for real-time monitoring of the entire road section on the edge side, and dynamically triggering the cloud large model for secondary discrimination according to the confidence and scene complexity.

[0009] Further, the semantic-visual joint coding in S1 is specifically defined as: defining the input image With text prompt set Joint features are extracted through a dual-encoder structure of a large model, where the visual encoder uses the Swin-L Transformer to extract hierarchical feature maps. The text encoder uses the BERT-large model to generate word-level embeddings. ,and Cross-modal attention module performs operations Output a semantically guided visual response map , used to generate open vocabulary detection suggestion boxes; where , , It is a learnable projection matrix.

[0010] Furthermore, the multi-scale feature enhancement of small targets in S1 is implemented using a weighted bidirectional feature pyramid network (W-BiFPN). The backbone network of the edge small model outputs feature maps C3, C4, and C5, with spatial resolutions of [missing information]. , , The number of channels are 256, 512, and 1024 respectively; W-BiFPN uses learnable weights To achieve cross-scale fusion, the calculation formula is as follows: ,in , Indicates the first p The input source layer of the layer.

[0011] Furthermore, the spatiotemporal context feature enhancement in S1 specifically involves: enhancing the continuous... T =5 frames of images, extract the P4 feature map sequence of the edge mini-model. The input bidirectional gated recurrent unit (GRU) captures timing dependencies, and its forward computation is as follows: Backward calculation is Final output Inject into the detection head.

[0012] Furthermore, S2 specifically includes: S21: Utilize a frozen cloud-based large model to perform zero-shot inference on unlabeled video streams and generate a weakly supervised pseudo-label set. ,in For bounding box, As a category, ∈[0,1] represents the confidence level. Papers such as "DistillingGroundingDINO for an Edge-Cloud Collaborative Advanced Driver AssistanceSystem," "F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Models," "Zero-Shot Open-Ended Video Inference," and "Zero-Shot VideoMoment Retrieval from Frozen Vision-Language Models" all employ a frozen large model to perform zero-shot inference on unlabeled video streams, thereby generating a weakly supervised pseudo-label set (including bounding boxes, categories, and confidence levels) as a whole. This method references the algorithmic approach of these papers, generating a label set by taking an initial image and text instructions as input.

[0013] S22: Perform dual screening on the pseudo-label set, with screening criteria including visual confidence level. >0.75 Text alignment score By combining spatial consistency verification and geofence filtering, high-quality pseudo-labels are obtained. S23: Integrating high-quality pseudo-tags with manually labeled sets Merge and build a fine-tuning dataset ; S24: Design a cross-modal knowledge distillation loss function Guiding the training of small edge models, where For edge small models, the native loss, Weights for dynamic knowledge distillation, relative weight coefficients =0.35, =0.25, =0.15, This represents the distillation loss for each output head of the YOLOv7s model. Since student YOLOv7s models are prone to losing detail, this setting... > > These lost details can be recovered during the learning process. , , These three parameters are set manually and are used to fine-tune the loss function to improve the model's prediction accuracy. j Values ​​range from 1 to 3, representing the rear, center, and front detection heads, respectively. These three detection heads are... , , Dynamic weight The parameter is adaptively adjusted according to the training stage, student-teacher difference, task difficulty, etc. to prevent overfitting in the early training stage, and its calculation method is as follows: , , , wherein t represents the number of training times, T represents the total number of training, e represents the exponential, represents the maximum value of the dynamic weight; represents the maximum value of the dynamic weight. Training starts from t=0, so will start from , gradually decaying with the training, achieve the knowledge distillation strategy of "early strong guidance and late weak constraint".

[0014] Further, the edge small model original loss is composed of a prediction box loss , a prediction box distribution loss and a prediction class binary cross entropy loss , and is specifically represented as ; wherein , , represents the loss of CIOU , IOU represents the intersection over union, the value range is [0, 1], and the calculation formula is: , CIOU (Complete Intersection over Union) represents the loss function used for bounding box regression in target detection, and the alignment and distance of the box are optimized by introducing the aspect ratio penalty term; is the Euclidean distance between the center points of the predicted box and the real box, is the diagonal distance of the minimum enclosing rectangle containing the predicted box and the real box, is a weight function, for measuring the aspect ratio consistency; , k only two non-zero label positions are calculated, is the model prediction probability, is the class weight; , is the real label, is the model prediction probability.

[0015] Further, the dynamic triggering condition in the S3 is a double-factor logical and operation: , wherein sThe detection confidence output by the edge small model, s is a confidence score output for each small target prediction bounding box, and the formula is: ; wherein the target confidence is a probability of judging whether the detected object is a target object; the threshold = 0.4, SCI is a scene complexity index, and the threshold = 0.65; the calculation formula of the SCI is , wherein is an occlusion rate, is a light variance, is a target density, and the weight , , .

[0016] Further, the edge small model is a YOLOv7s model, deployed on an edge AI device, and the input resolution is 640x640; the cloud large model is a Grounding DINO2.0 model.

[0017] Further, after triggering the cloud large model for secondary discrimination, the output results of the cloud large model and the output results of the edge small model are fused using an intersection over union (IoU) threshold of 0.5 and a semantic consistency fusion strategy to update the final structured detection result. The cloud large model performs secondary discrimination, and Grounding DINO 2.0 is used as the cloud large model, and YOLOv7 is used as the edge lightweight detection model. The large model plays a dual role in the system: a knowledge provider in the training stage and a key decision arbiter in the reasoning stage.

[0018] In the training stage, the prompt word driving capability and powerful multi-modal prior of Grounding DINO 2.0 are fully utilized to generate high-quality detection results and pseudo-labels under the condition of only a few hundred labeled samples. Through pseudo-label supervision and cross-modal knowledge distillation (such as feature alignment and relationship distillation), the YOLOv7 small model is effectively guided to learn, significantly reducing its dependence on large-scale artificial labeled data, and achieving high-performance training under the condition of small samples.

[0019] In the deployment stage, the system mainly uses the edge small model for full-time and full-area real-time detection. At the same time, a dynamic triggering mechanism is introduced, which combines the prediction confidence output by the small model and the comprehensive complexity index (Scene Complexity Index, SCI) of the scene to adaptively determine whether to call the cloud large model for secondary fine discrimination of suspicious areas. This mechanism realizes the collaborative reasoning paradigm of "relying on the small model to ensure efficiency in normal times, and calling the large model to ensure accuracy in critical moments", which guarantees the overall low latency of the system while significantly improving the detection reliability in complex scenes.

[0020] A traffic scene vehicle and event recognition system for implementing the method comprises: A perception module deployed on the edge side is used to execute the edge small model to perform real-time video stream processing and preliminary detection; A semantic enhancement module deployed in the cloud is used to execute the cloud large model to perform open vocabulary fine judgment; A dynamic trigger management module is used to calculate a scene complexity index and manage the collaborative inference process between the edge side and the cloud; The perception module, the semantic enhancement module and the dynamic trigger management module are connected through network communication.

[0021] The present application has the following advantages: (1) By constructing a semantic-visual joint coding system and a multi-scale feature enhancement mechanism, the system can effectively overcome the influence of insufficient light, night environment and bad weather such as rain, snow and fog on recognition accuracy. After introducing the spatio-temporal context modeling, the system enhances the consistency of detection between consecutive frames, effectively suppresses the missed detection and misjudgment caused by target occlusion or motion blur, and thus maintains stable perception performance in dynamic traffic scenes.

[0022] (2) With the help of high-quality pseudo-labels generated by the large model and the knowledge distillation mechanism, the edge small model can quickly converge under the condition of very few manually labeled samples, breaking the bottleneck that traditional small models rely on large-scale labeled data to achieve high accuracy. This mechanism significantly shortens the model training and iteration period, and reduces the data cost of system deployment and maintenance.

[0023] (3) Relying on the zero-shot inference and semantic understanding ability of the cloud large model, the system can respond to natural language instructions and recognize new vehicle categories or traffic events that have not appeared in the training set, such as temporary emergency facilities and specific behavior events. This open vocabulary detection capability makes the system have good scalability and scene adaptability, and can respond to the evolving traffic control needs without retraining.

[0024] (4) Through the dynamic trigger collaborative inference mechanism, the system relies on the edge small model to ensure real-time processing efficiency under normal traffic flow, intelligently calls the cloud large model for secondary discrimination under low confidence or complex scenes, forming an efficient collaborative paradigm of "edge lightweight perception-cloud depth judgment". This architecture effectively improves the recognition accuracy of key events while controlling the response delay.

[0025] (5) The edge side uses lightweight models and hardware acceleration technology to meet the needs of low power consumption and high efficiency; the cloud model uses a low-frequency calling strategy to balance computing resources and recognition effect, and the overall system has good scalability and stability, and is suitable for large-scale deployment in urban road network monitoring scenarios.

[0026] Additional advantages, objects, and features of the application will be apparent to those skilled in the art upon examination of the following specification. It is intended that the application not be limited by the disclosed implementation, but that it include all such variations and modifications to the full extent allowed by law. BRIEF DESCRIPTION OF DRAWINGS

[0027] In order to make the objects, technical solutions and advantages of the present application clearer, the preferred embodiments of the present application will be described in detail below with reference to the drawings, in which: Figure 1 A model coordination architecture applied to traffic scene vehicle and traffic event recognition provided by the present application is shown in the figure; Figure 2 A vehicle detection effect diagram of the coordination model provided by the present application at different times at night is shown in the figure; Figure 3 A detection effect diagram of the coordination model provided by the present application under different visibility conditions is shown in the figure; Figure 4 A detection effect diagram of the coordination model provided by the present application on different traffic events is shown in the figure. DETAILED DESCRIPTION

[0028] The advantages and effects of the present application can be easily understood by those skilled in the art from the content disclosed in the specification. The present application can also be implemented or applied in other different specific embodiments, and the details in the specification can be modified or changed in various ways based on different views and applications without departing from the spirit of the present application. It should be noted that the diagrams provided in the following embodiments only illustrate the basic concept of the present application in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0029] The drawings are only used for illustrative explanation, and the representation is only a schematic diagram, not a physical diagram, and should not be understood as a limitation of the present application. In order to better illustrate the embodiments of the present application, some components in the drawings may be omitted, enlarged or reduced, and do not represent the actual size of the product. It is understandable for those skilled in the art that some known structures and their descriptions in the drawings may be omitted.

[0030] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0031] First of all Figure 1 The collaborative architecture of the present invention is illustrated in the diagram. This architecture diagram clearly demonstrates the collaborative relationship between the large cloud model Grounding DINO 2.0 and the small edge model YOLOv7. The left side shows the Grounding DINO 2.0 model, which receives images and text prompts, such as "car, truck, snow, spilled material," as input. Internally, it contains two parallel backbone networks: the Swin-L Transformer Backbone for extracting deep image features and the Bert-large Backbone for extracting deep text features. The features extracted by both are then fed into the "semantic-visual joint encoding" module after passing through a feature enhancer. This module is crucial for achieving open vocabulary understanding; it aligns textual semantics with visual information to generate Keys & Values ​​for subsequent detection.

[0032] The image on the right is the YOLOv7 model. It receives image input and passes it through a series of convolutional modules (Conv), ELAN modules, MP modules, and reparameterized convolutional modules (RepConv), forming its main detection network architecture. YOLOv7 performs detection at three different feature scales to accommodate targets of different sizes.

[0033] The synergy between the two is reflected in two aspects. First, during the training phase, the visually weighted features of Grounding DINO 2.0 are injected into the YOLOv7 training process through a "knowledge distillation" process, serving as additional supervisory signals to improve YOLOv7's performance. Second, during the inference phase, YOLOv7 performs real-time detection at the edge. After the detection results and alarm outputs are processed by a judgment module, which determines whether to initiate "secondary discrimination" based on preset conditions (triggers). Once triggered, the keyframes are uploaded to Grounding DINO 2.0 in the cloud for more refined analysis.

[0034] Embodiment One: Efficient and Accurate Vehicle Detection Based on Real-time Video Stream First, a trained YOLOv7s model is deployed on an edge computing device, such as an NVIDIA Jetson AGX Xavier. This model receives real-time video streams from roadside surveillance cameras with an input resolution of 640x640. To achieve real-time processing, the model is accelerated using TensorRT, with an inference speed of up to 62 frames per second, ensuring that the single-frame processing delay is less than 16 milliseconds.

[0035] Second, the model performs multi-modal traffic visual perception encoding on each image frame. The backbone network of YOLOv7s first extracts feature maps at three levels: C3, C4, and C5. To address the difficulty of detecting small target vehicles at long distances in traffic scenarios, the system uses a weighted bidirectional feature pyramid network (W-BiFPN) to perform cross-scale fusion on these feature maps. Through learnable weights, W-BiFPN can adaptively integrate information from different levels, significantly enhancing the feature response capability of P3 layer to small targets. At the same time, to ensure the continuity and stability of detection, the system caches the last 5 consecutive image frames, extracts their P4 feature map sequences, and inputs them into a lightweight bidirectional gated recurrent unit (GRU) network. This network can capture the temporal dependence of vehicles, effectively suppressing the ID switching problem caused by temporary occlusion or rapid movement, greatly improving the multi-object tracking accuracy (MOTA) index.

[0036] Third, the model performs forward inference, outputting structured JSON data containing vehicle bounding boxes, classes, confidence, coordinates, and timestamps. While inferring, the dynamic trigger management module calculates the scene complexity index (SCI) in real time. This index is calculated using the weighted formula , which comprehensively evaluates the occlusion rate, lighting changes, and target density in the scene.

[0037] The Scene Complexity Index (SCI) is an indicator used to quantify the complexity of an image or visual scene. It has wide applications in computer vision, remote sensing, autonomous driving, image quality assessment, human-computer interaction, and other fields. 1. The occlusion rate is directly labeled by the model: original target bounding box; visible part bounding box. Then calculate the occlusion rate using the following formula: O rate =1-(original target bounding box area) / (visible part bounding box area) Where O rate represents the occlusion rate.

[0038] 2. The lighting change is calculated using the following formula: (1) Average brightness:

[0039] is the calculated mean value, W represents the width of the image, H represents the image height; represents that the gray value is first calculated according to the pixel value of the red, green and blue channels, and then the average value of the gray value of all images is calculated. The constant for calculating the gray value is determined by the International Commission on Illumination (CIE) through large-scale human visual experiments. is the red channel value of the pixel, is the green channel value of the pixel, is the blue channel value of the pixel, and the value range of these pixels is 0-255.

[0040] (2) Illumination variance:

[0041] wherein represents the illumination variance.

[0042] (3) Target density:

[0043] wherein represents the target density.

[0044] Finally, dynamic triggering and secondary discrimination are performed. The system judges each detection result output by YOLOv7s. If the detection confidence of a certain target s is lower than the preset threshold τ =0.4, and the complexity index SCI of the current scene is higher than the threshold θ=0.65, the double-factor logical AND condition is met, triggering the secondary discrimination mechanism. The dynamic triggering management module uploads the key frame image and related text prompts, such as "car" and "truck", to the cloud server. The Grounding DINO2.0 large model deployed in the cloud receives the request and uses its powerful semantic understanding and visual positioning capabilities for fine identification. The identification result of the large model is returned to the edge side through the network. The edge system uses the intersection over union (IoU) threshold of 0.5 and the semantic consistency strategy to fuse the cloud result and the local result, update and output the final detection result with higher confidence.

[0045] Figure 2The vehicle detection effect of the collaborative model at different times at night is shown. From the four effect pictures from 8:15 to 10:10 at night, it can be seen that even in the case of poor light conditions at night, there are car light glare and shadows, the system can still stably detect the vehicles on the road, and accurately mark their positions and confidence with blue marked boxes. This verifies that the method still has strong robustness in low light environment.

[0046] Example Two: Complex Traffic Incident Recognition in Low Light and Adverse Weather First, the YOLOv7s model on the edge side still performs real-time analysis on the monitoring video stream 7x24 hours. When the spatio-temporal context feature enhancement module analyzes the continuous frame sequence, it may find abnormal situations. For example, a target remains stationary in the driving lane for more than a preset time threshold, which may be a case of illegal parking or vehicle breakdown. Or, the motion trajectory of a target is opposite to the driving direction defined by the lane line, which may be a case of reverse driving. In this case, the YOLOv7s model may only give a detection box with a class of "car", but the detection confidence for events such as "illegal parking" or "reverse driving" is very low.

[0047] Second, trigger cloud secondary discrimination. Although the edge model has low confidence in the event, due to the high risk level of such events, the system will combine the calculated scene complexity index SCI, such as in foggy weather or at night, the SCI value will be high, thereby triggering the cloud large model for secondary discrimination. The dynamic trigger management module will upload the key frame image containing the potential event, along with one or more descriptive text prompt words, such as "faulty truck with double flash on at night", "pedestrian riding electric bike without helmet" or "road debris", to the cloud server.

[0048] Third, cloud large model for deep semantic analysis. The Grounding DINO2.0 large model in the cloud uses its excellent zero-shot reasoning ability to perform deep semantic understanding on the received images and text prompts. It not only can identify objects in the image, but also can understand complex semantic concepts, such as "faulty truck with double flash on" or "without helmet", which require the combination of state and behavior to make judgments. Thanks to its open vocabulary detection feature, even if the event description has never appeared in the training data, such as "temporary emergency facility", the large model can still give a reasonable judgment.

[0049] Finally, event confirmation and alarm. After the cloud large model completes the accurate judgment, the structured event description information output by it, such as event type (illegal parking), occurrence location, target attribute, etc., will be sent to the edge side. After the edge system fuses the authoritative judgment result of the large model, it generates standardized alarm information immediately after confirming the event, and sends it to the traffic management center through standard protocols such as ONVIF, realizing rapid response.

[0050] Figure 3 The detection effect of the collaborative model under different visibility conditions is shown. In dense fog weather, the visibility decreases from 300 meters to 86 meters, and the image quality is severely degraded, but the system can still successfully detect the vehicle in the distance through the green box. Figure 4 The detection effect of the model on various traffic events is specifically shown. The figure clearly marks various complex situations such as "shelf falling event", "paint splashing event", "vehicle collision event" and "pedestrian guardrail crossing event", proving the strong ability of the method in event recognition.

[0051] Example three: rapid adaptation and iteration of the model based on small sample learning Suppose the traffic management department needs to add the identification ability of "electric scooters", a new emerging traffic participant. The traditional method needs to collect tens of thousands of pictures containing electric scooters and perform manual labeling, which is time-consuming and labor-intensive. The invention adopts the following efficient process: First step, automatic generation of pseudo-labels. The operation and maintenance personnel do not need to perform any labeling, but use the Grounding DINO2.0 large model already deployed in the cloud to perform batch processing of zero-shot inference on a large number of unlabeled historical traffic monitoring video streams. When inferring, the large model is provided with a text prompt word of the new class, that is, "electric scooter". The large model will automatically find the target matching the text description in the video frame and generate a weakly supervised pseudo-label set containing the bounding box, class and confidence.

[0052] Second step, high-quality pseudo-label screening. Since the large model may also make mistakes when performing zero-shot inference, the generated pseudo-labels need to be strictly screened. The system will start a double screening mechanism: first, only samples with visual confidence >0.75 are retained; second, by calculating the alignment score of the text prompt and the image region, samples with a score greater than 0.7 are retained. On this basis, the system will also combine spatial consistency verification, such as requiring the detection box of the same target to have an intersection over union IoU greater than 0.3 between adjacent frames to eliminate drifting and flickering detection boxes. At the same time, geographic fence filtering is used, such as excluding "electric scooters" appearing on the main road of the highway, which is not reasonable. Through this series of screening, high-quality pseudo-label data is obtained.

[0053] Third, build the fine-tuning dataset. Merge the filtered high-quality pseudo-label data with a very small manually annotated dataset, for example, containing only 300 to 500 images, to form the training dataset for model fine-tuning.

[0054] Fourth, guided training by knowledge distillation. A cross-modal knowledge distillation strategy is adopted to fine-tune the YOLOv7s model on the edge side. During training, the total loss function is composed of two parts: . Wherein, is the original loss of YOLOv7, which is composed of bounding box loss , bounding box distribution loss DFL and predicted class binary cross-entropy loss, to ensure that the model can learn the basic detection information provided by the pseudo-label. The other part is the knowledge distillation loss , which forces the feature expression of YOLOv7 to align with that of Grounding DINO2.0 through a feature alignment loss function. In this way, the more robust and generalized knowledge about "scooters" in the large model is "distilled" and transferred to the small model.

[0055] Through the workflow of the embodiment, the YOLOv7s model only needs to use about 400 high-quality pseudo-label images to improve the average precision mAP@0.5 index of the "scooter" class. Compared with traditional training using only a small amount of data, the performance is greatly improved, the data cost and model update cycle are reduced, and the significant advantages of the present application in low data dependence and rapid adaptation are embodied.

[0056] Finally, it should be pointed out that the above embodiments are only used to illustrate the technical solutions of the present application and not to limit it. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced by equivalents without departing from the purpose and scope of the technical solutions, which should be covered in the scope of the claims of the present application.

Claims

1. A method for vehicle and event recognition in traffic scenes based on the collaboration of edge small models and cloud large models, characterized in that: Includes the following steps: S1: Construct a multimodal traffic visual perception coding system to perform semantic-visual joint coding, small target multi-scale feature enhancement, and spatiotemporal context feature enhancement on the input traffic monitoring video stream; S2: A small-sample collaborative training algorithm based on large model enhancement, which uses a large cloud model to generate pseudo-labels and guides the training of marginal small models through knowledge distillation. S3: Deploy a dynamically triggered dual-model inference framework, using a trained edge small model on the edge side to perform real-time monitoring of the entire road segment, and dynamically triggering the cloud large model for secondary discrimination based on confidence and scene complexity.

2. The traffic scene vehicle and event recognition method based on the collaboration of edge small models and cloud large models according to claim 1, characterized in that: The semantic-visual joint encoding in S1 specifically involves defining the input image. With text prompt set Joint features are extracted through a dual-encoder structure of a large model, where the visual encoder uses the Swin-L Transformer to extract hierarchical feature maps. The text encoder uses the BERT-large model to generate word-level embeddings. ,and ; Cross-modal attention module performs operations Output a semantically guided visual response map , used to generate open vocabulary detection suggestion boxes; where , , It is a learnable projection matrix.

3. The traffic scene vehicle and event recognition method based on the collaboration of edge small models and cloud large models according to claim 1, characterized in that: The multi-scale feature enhancement of small targets in S1 is achieved using a weighted bidirectional feature pyramid network (W-BiFPN). The backbone network of the edge small model outputs feature maps C3, C4, and C5, with spatial resolutions of [missing information]. , , The number of channels are 256, 512, and 1024 respectively; W-BiFPN uses learnable weights To achieve cross-scale fusion, the calculation formula is as follows: ,in , Indicates the first p The input source layer of the layer.

4. The traffic scene vehicle and event recognition method based on the collaboration of edge small models and cloud large models according to claim 1, characterized in that: The spatiotemporal context feature enhancement in S1 specifically involves: enhancing the continuous... T =5 frames of images, extract the P4 feature map sequence of the edge mini-model. The input bidirectional gated recurrent unit (GRU) captures timing dependencies, and its forward computation is as follows: Backward calculation is Final output Inject into the detection head.

5. The traffic scene vehicle and event recognition method based on the collaboration of edge small models and cloud large models according to claim 1, characterized in that: S2 specifically includes: S21: Utilize a frozen cloud-based large model to perform zero-shot inference on unlabeled video streams and generate a weakly supervised pseudo-label set. ,in For bounding box, As a category, The confidence level is ∈[0,1]. S22: Perform dual screening on the pseudo-label set, with screening criteria including visual confidence level. >0.75 Text alignment score By combining spatial consistency verification and geofence filtering, high-quality pseudo-labels are obtained. S23: Integrating high-quality pseudo-tags with manually labeled sets Merge and build a fine-tuning dataset ; S24: Design a cross-modal knowledge distillation loss function Guiding the training of small edge models, where For edge small models, the native loss, Weights for dynamic knowledge distillation, relative weight coefficients =0.35, =0.25, =0.15, This refers to the distillation loss corresponding to each output head of the YOLOv7s model. j Values ​​range from 1 to 3, representing the rear, center, and front detection heads, respectively. These three detection heads are... , , .

6. The traffic scene vehicle and event recognition method based on the collaboration of edge small models and cloud large models according to claim 5, characterized in that: The native loss of the edge small model Loss from predicted boxes Prediction box distribution loss Binary cross-entropy loss for predicted categories Composition, specifically represented as ;in , express CIOU The loss, IOU Indicates intersection, union, and ratio. CIOU This represents the loss function used for bounding box regression in object detection. Let Euclidean distance be the center point of the predicted bounding box and the ground truth bounding box. The diagonal distance is the minimum bounding rectangle containing both the predicted and ground truth boxes. For the weight function, Used to measure aspect ratio consistency; , k Calculation is performed only for the two non-zero label positions. To predict probabilities for the model, Category weights; , For real labels, Predict probabilities for the model.

7. The traffic scene vehicle and event recognition method based on the collaboration of edge small models and cloud large models according to claim 1, characterized in that: The dynamic triggering condition in S3 is a two-factor logical AND operation: ,in s The detection confidence score and threshold output by the edge small model. =0.4, SCI is the Scene Complexity Index, the threshold. =0.65; the formula for calculating SCI is: ,in For occlusion rate, For the variance of illumination, For target density, weight , , .

8. The traffic scene vehicle and event recognition method based on the collaboration of edge small models and cloud large models according to claim 1, characterized in that: The edge small model is a YOLOv7s model, deployed on an edge AI device, with an input resolution of 640×640; the cloud large model is a Grounding DINO2.0 model.

9. The traffic scene vehicle and event recognition method based on the collaboration of edge small models and cloud large models according to claim 1, characterized in that: After the large cloud model is triggered to perform secondary discrimination, the output results of the large cloud model and the small edge model are fused using an intersection-over-union (IoU) threshold of 0.5 and a semantic consistency fusion strategy to update the final structured detection result.

10. A traffic scene vehicle and event recognition system for implementing the method of any one of claims 1 to 9, characterized in that: include: The perception module deployed at the edge is used to execute the edge mini-model to perform real-time video stream processing and preliminary detection; A semantic enhancement module deployed in the cloud is used to execute the large cloud model and perform fine-grained judgment of open lexical terms; The dynamic trigger management module is used to calculate the scene complexity index and manage the collaborative reasoning process between the edge and the cloud. The perception module, semantic enhancement module, and dynamic trigger management module are connected via network communication.

Images