Method and system for identifying road event by using video large model

Abstract

The invention relates to a method and system for identifying a highway event by using a video large model, and the method comprises the steps: employing a three-stage processing architecture, firstly carrying out the real-time target detection and preliminary event judgment of a highway monitoring video stream through employing a YOLO algorithm, and generating an event candidate set; inputting the candidate events and the video clips thereof into a specially trained visual large model for deep semantic analysis and secondary reasoning; and finally, a reasoning result is rechecked through a rule engine, and false alarms are filtered by applying illusion suppression and a space-time association rule. According to the method, the real-time performance of traditional target detection and the deep reasoning capability of a visual large model are fused, so that the problems of high false alarm rate and high missing report rate of a traditional method are effectively solved, the accuracy and reliability of event identification in a complex traffic scene are remarkably improved, and meanwhile, the real-time processing capability of a system on multiple paths of high-definition video streams is ensured.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a method and system for recognizing highway events using large video models. Background Technology

[0002] With the rapid development of intelligent transportation systems, video analytics-based automatic highway incident identification technology has become a key means to improve the efficiency of road safety management. This technology analyzes real-time video streams collected by surveillance cameras throughout highways to automatically and promptly detect safety hazards such as pedestrian intrusion, abnormal parking, and road debris, thereby achieving proactive perception and rapid response to traffic conditions, reducing the workload of manual monitoring, and improving emergency response speed.

[0003] Currently, existing technologies mainly rely on building detection frameworks based on object detection models. First, object detection models are used to identify key targets such as vehicles and pedestrians in video frames; then, object tracking algorithms are combined to associate the same target in consecutive frames, and a rule engine based on human experience is used to make preliminary judgments about events.

[0004] However, the aforementioned technologies lack the ability to deeply understand the semantics of the scene and can only make shallow logical judgments based on fixed thresholds. They cannot effectively distinguish between real events and similar interference, resulting in high false alarm and false negative rates, which seriously restricts the practical application effect and reliability in real and complex highway monitoring scenarios.

[0005] Based on this, this application provides a method and system for identifying highway events using large video models. Summary of the Invention

[0006] To address the problem of insufficient deep semantic understanding of scenes, the inability to perform only shallow, fixed-threshold-based logical judgments, and the inability to effectively distinguish between real events and similar interference, resulting in high false alarm and false negative rates, which severely restricts the practical application effect and reliability in real and complex highway monitoring scenarios, this application provides a method and system for identifying highway events using a large video model.

[0007] Firstly, this application provides a method for identifying highway events using a large video model, employing the following technical solution: including: Collect highway video datasets containing various traffic event types, and annotate them according to the preset annotation guidelines and inference chain standards. Divide the annotated datasets into training and validation sets, and use them to supervise and fine-tune the pre-trained visual large model to obtain the trained visual large model. The system can access and preprocess highway monitoring video streams in real time, use the YOLO model deployed on the first set of graphics cards to perform real-time target detection on video frames, and preliminarily determine traffic events based on the target detection results and preset rules, and generate and output an event candidate set containing event type, location and time information. The event candidate set and its corresponding video segments are obtained. The structured results of the events and the extracted video keyframes are input into the trained visual big model deployed on the second set of graphics cards. The visual big model performs inference analysis to obtain the structured event judgment results. The system retrieves a pre-defined set of rules for suppressing large model illusions and determining spatiotemporal correlations. The rule engine is then used to filter and verify the event determination results output by the visual large model to obtain the final event determination results.

[0008] Preferably, the collection includes highway video datasets containing various traffic event types, and the datasets are labeled according to preset annotation guidelines and inference chain standards. The labeled datasets are then divided into training and validation sets, and the datasets are used to supervise and fine-tune a pre-trained visual model to obtain a trained visual model, including: Collect a highway video dataset containing various traffic scenarios and event types, including pedestrian intrusion, abnormal parking, and road littering. Develop a labeling guide that includes basic knowledge of highways, precise definitions of abnormal events, and event classification standards; Based on the annotation guidelines and according to the preset reasoning chain annotation standards, the collected video dataset is re-annotated. The reasoning chain annotation standards require the annotation process to follow the logical flow of observation-analysis-reasoning-conclusion, and the annotation results are output in a structured data format. The labeled dataset is divided into training and validation sets according to a preset ratio; The training and validation sets are used to supervise the fine-tuning of the pre-trained visual large model. The structured result text of the event is concatenated with the corresponding video keyframe image as a multimodal input, and the model parameters are optimized using parameter-efficient fine-tuning techniques to obtain the trained visual large model.

[0009] Preferably, the real-time access and preprocessing of highway monitoring video streams utilizes a YOLO model deployed on the first set of graphics cards to perform real-time target detection on video frames. Based on the target detection results and preset rules, traffic events are preliminarily determined, and a candidate set of events containing event type, location, and time information is generated and output, including: Multiple highway surveillance video streams are accessed in real time via streaming media protocols, and GPU hardware decoding technology is used to decode and extract frames from the video streams. The preprocessed video frames are input into the YOLO model deployed on the first set of graphics cards for real-time target detection, identifying targets including vehicles, pedestrians and debris; Based on the target detection results of the YOLO model, and combined with preset rule conditions, preliminary event determination is made. The rule conditions include: detecting a pedestrian continuously existing in the driving lane or emergency lane area, or a vehicle stopping in the driving lane for more than a preset time threshold, or detecting a large obstacle appearing in the lane. The preliminary event information is encapsulated into a structured event candidate set, which includes event type, location coordinates, timestamp, and video clips before and after the event.

[0010] Preferably, the step of obtaining the event candidate set and its corresponding video segments, and inputting the structured results of the events along with the extracted video keyframes into the trained visual big model deployed on the second set of graphics cards, allows the visual big model to perform inference analysis and obtain structured event determination results, including: Based on the video clip URLs contained in the event candidate set, obtain the corresponding video data, and extract keyframe images before and after the event from the video data; The structured results in the event candidate set are combined with the extracted video keyframe images, and a multimodal input is constructed according to a preset format. The multimodal input includes system instructions, highway background knowledge, structured information from a single detection, and keyframe images. The constructed multimodal input is sent to the trained visual large model deployed on the second set of graphics cards, and inference parameters are set to control the output stability. The visual large model performs inference analysis and generates a standardized, structured event determination result containing event type, event occurrence time and event location information.

[0011] Preferably, the step of retrieving a preset rule set for suppressing large model illusions and performing spatiotemporal correlation determination, and using a rule engine to filter and verify the event determination results output by the visual large model to obtain the final event determination result includes: The predefined rule set is retrieved, which includes a hallucination suppression rule for correcting inconsistencies in the output logic of the visual large model, a spatiotemporal correlation rule for merging multiple events with spatiotemporal continuity, and a confidence fusion rule for fusing the confidence scores of the YOLO model and the visual large model. The event determination results output by the visual big model are used as fact inputs to the rule engine service. The rule engine loads the rule set to perform pattern matching and logical reasoning, filters, corrects and merges the input event determination results, and outputs the final event determination results.

[0012] Preferably, the method further includes: The large visual model employs the TVM-R1 multimodal attention mechanism. It calculates the dot product of the query matrix and the key matrix, scales it using the square root of the key vector dimension, normalizes it using the softmax function, and then multiplies it with the value matrix to fuse multimodal feature information from text and video keyframes. The model training uses the cross-entropy loss function, which calculates the logarithmic difference between the model's predicted probability and the true label, and averages the results over all samples and categories to optimize the model parameters. The target detection confidence of the YOLO model is determined by the probability that the bounding box contains the target and the intersection-union ratio of the predicted box and the ground truth box. The loss function of the YOLO model comprehensively evaluates the bounding box coordinate prediction error, the confidence prediction error of whether the target is contained, and the category prediction error. For the calculation of the width and height errors of the bounding box, the square root is used to alleviate the scale sensitivity problem between large and small targets.

[0013] Preferably, the multiple graphics cards in the dedicated server are configured as follows: The first set of graphics cards consists of at least two graphics cards and is specifically designed for parallel execution of video decoding, frame extraction, and model inference tasks using the YOLO algorithm. The second set of graphics cards consists of at least one graphics card with large video memory, and is dedicated to running the inference task of the large visual model. The first set of graphics cards and the second set of graphics cards are isolated from each other in terms of hardware resources.

[0014] Secondly, this application discloses a device for recognizing highway events using a large video model, which employs the following technical solution, including: The model training module is used to collect highway video datasets containing various types of traffic events, and to annotate them according to the preset annotation guidelines and inference chain standards. The annotated dataset is divided into training set and validation set, and is used to supervise and fine-tune the pre-trained visual large model to obtain the trained visual large model. The target detection module is used to access and preprocess highway monitoring video streams in real time. It uses the YOLO model deployed on the first set of graphics cards to perform real-time target detection on video frames. Based on the target detection results and preset rules, it initially determines traffic events and generates and outputs a candidate set of events containing event type, location and time information. The event reasoning module is used to obtain the event candidate set and its corresponding video segments, and input the structured results of the events and the extracted video key frames into the trained visual big model deployed on the second set of graphics cards. The visual big model performs reasoning analysis to obtain the structured event judgment results. The result verification module is used to retrieve a preset set of rules for suppressing large model illusions and making spatiotemporal correlation determinations. It uses the rule engine to filter and verify the event determination results output by the visual large model to obtain the final event determination results.

[0015] Thirdly, this application also provides a control device, the device comprising: It includes a memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and executed, as described above, a method for identifying highway events using a large video model.

[0016] Fourthly, this application also provides a computer-readable storage medium storing a computer program that can be loaded by a processor and executed as described above in a method for identifying highway events using a large video model.

[0017] In summary, the system in this application utilizes the YOLO algorithm deployed on the first set of graphics cards to perform real-time object detection on the input video stream, initially generating a candidate set of events including pedestrian intrusion and abnormal parking based on preset rules. Next, the candidate events and their video clips are input into a large-scale visual model deployed on the second set of graphics cards for secondary inference. This large-scale visual model is specially trained and capable of deep semantic understanding based on multimodal input, outputting structured judgment results according to the observation-analysis-reasoning-conclusion thought chain. Finally, the output of the large-scale visual model is reviewed by a rule engine, applying rule sets such as hallucination suppression, spatiotemporal correlation, and confidence fusion to filter out erroneous judgments and output a final reliable event conclusion. Thus, by integrating the real-time performance of traditional object detection with the deep reasoning capabilities of the large-scale visual model, combined with post-processing optimization by the rule engine, the accuracy and reliability of event recognition in complex traffic scenarios are significantly improved, while ensuring the system's real-time processing capability for multiple high-definition video streams, providing effective technical support for intelligent traffic management. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating a method for identifying highway events using large video models.

[0019] Figure 2 This is a structural block diagram of a device that uses large video models to identify highway events. Detailed Implementation

[0020] The following combination Figure 1 - Figure 2 This application will be described in further detail.

[0021] With the development of intelligent transportation systems, the automatic identification and processing of highway events from camera videos has become increasingly important. Traditional event detection methods often rely on image feature-based object detection models, such as ResNet, VGG-19, and YOLO. These models, based on object detection, are supplemented by object tracking and rule engines to identify events such as pedestrian intrusion, parking, and road littering. However, these methods typically require manual feature design and have poor adaptability to different scenarios, making them difficult to handle complex traffic situations. Furthermore, traditional methods often suffer from numerous false positives and false negatives when dealing with dynamically changing traffic environments, making them ill-suited to complex and ever-changing traffic scenarios. Therefore, a new method and device are urgently needed to improve the automatic identification capability of highway events. In recent years, large-scale visual models have made significant progress in image and video analysis. Applying large-scale visual models to highway event detection can effectively improve the accuracy and robustness of event recognition, providing a new approach to solving the aforementioned problems.

[0022] Reference Figure 1 The embodiments of this application include at least steps S10 to S40.

[0023] S10 collects highway video datasets containing various types of traffic incidents, and annotates them according to the preset annotation guidelines and inference chain standards. The annotated dataset is divided into training set and validation set, and is used to supervise and fine-tune the pre-trained visual large model to obtain the trained visual large model.

[0024] S20 receives and preprocesses highway monitoring video streams in real time, uses the YOLO model deployed on the first set of graphics cards to perform real-time target detection on video frames, and preliminarily determines traffic events based on the target detection results and preset rules, generating and outputting a candidate set of events containing event type, location and time information.

[0025] S30: Obtain the event candidate set and its corresponding video segments. Input the structured results of the events and the extracted video keyframes into the trained visual big model deployed on the second set of graphics cards. The visual big model performs inference analysis to obtain the structured event judgment results.

[0026] S40: Retrieve the preset rule set for suppressing large model illusions and making spatiotemporal correlation determinations. Use the rule engine to filter and verify the event determination results output by the visual large model to obtain the final event determination results.

[0027] Specifically, a dedicated large-scale visual model is trained by collecting and finely annotating a highway video dataset, and then performing supervised fine-tuning based on it, enabling the model to master traffic event reasoning capabilities. The YOLO model is used to perform preliminary object detection and rule judgment on real-time video streams, generating an event candidate set containing event type, location, and time. The candidate set and its video keyframes are input into the large-scale visual model for deep semantic analysis and secondary reasoning to obtain more reliable structured judgment results. The reasoning results are finally reviewed by a rule engine, and hallucination suppression and spatiotemporal correlation rules are applied to filter false alarms, outputting highly reliable event conclusions.

[0028] In practice, the system first performs event detection using the YOLO algorithm after accessing the highway monitoring video stream, generating a preliminary event candidate set. Then, the detection results and corresponding video clips are input into a trained visual model for secondary analysis and verification. Finally, the rule engine verifies the model's output, suppresses illusions, and outputs the final event judgment result.

[0029] In some embodiments, step S10 specifically includes the following steps: Step 1.1 Data Collection The data source is historical AI-detected event recordings accumulated by the highway monitoring system. At least 50,000 video clips, each 10-40 seconds long, are collected, covering various weather conditions including daytime, nighttime, rain, and fog, and different scenarios such as urban expressways, mountain highways, tunnels, and toll stations. Event types must cover 10 major traffic event categories: pedestrian intrusion, abnormal parking, road littering, vehicles driving in the wrong direction, traffic accidents, congestion, smoke and fire, construction, slow driving, and abnormal lane changes. Simultaneously, corresponding structured AI event detection results are collected, including target detection bounding boxes, event type confidence scores, timestamps, and camera locations.

[0030] Step 1.2 Developing Annotation Guidelines Assemble a labeling team composed of traffic management experts and artificial intelligence algorithm engineers. Develop detailed labeling guidelines, including: Basic knowledge of highways: lane functions, meanings of traffic signs and markings, driving regulations under different weather and lighting conditions, etc.

[0031] Precise definition of abnormal events: For example, "pedestrian intrusion" needs to be clearly distinguished from maintenance and construction personnel, drivers and passengers of vehicles with breakdowns, and unrelated members of the public; "abnormal parking" needs to be comprehensively defined by combining factors such as parking location (driving lane, emergency lane), parking duration, and vehicle status (hazard lights on, warning signs placed).

[0032] Event classification criteria: Based on the urgency and potential harm of an event, events are classified into two levels: "highly concerning" and "not concerning".

[0033] Step 1.3 Event Scenario Definition and Classification The 10 categories of traffic incidents are further refined and classified, such as: Pedestrian intrusion: Pay close attention to ordinary-looking pedestrians within the highway area and along the guardrails. No need to pay attention to uniformed personnel such as road construction workers, road maintenance workers, and highway police.

[0034] Unusual parking: This requires close monitoring. Vehicles parked within the highway area that are not involved in road construction or patrol operations by highway police. Other parking incidents do not require attention and are not considered priority monitoring items.

[0035] Road debris: Pay close attention to large obstacles in the driving lane or emergency lane; no need to pay attention to obstacles on the shoulder that are small and do not affect driving safety.

[0036] Step 1.4 Criteria for Annotating the Reasoning Chain Annotators are required to follow the thought process of "observation-analysis-reasoning-conclusion" when annotating events. Example (pedestrian intrusion event): Observation: In the video footage, a moving target appears in the driving lane at K125+300 in frame 15.

[0037] Analysis: The target's shape and features resemble a human body, and its trajectory involves crossing the lane. Surrounding vehicles braked to avoid it.

[0038] Reasoning: According to the Road Traffic Safety Law, pedestrians are prohibited from entering highways. This behavior poses a significant threat to their own safety and the safety of other vehicles.

[0039] Conclusion: This incident is classified as a "pedestrian intrusion" and is classified as "high-priority concern." Even for incidents that are not reported to the police, a complete chain of reasoning must be recorded; for example, if the analysis determines the target to be a light and shadow illusion or an animal.

[0040] Step 1.5 Data Labeling Implementation Annotation was performed using a self-developed annotation platform. The annotation output was in structured JSON format. The annotation content included an image and a simulated dialogue. A human user, acting as a road safety officer, asked the AI ​​to identify abnormal traffic information in the image and output it in XML format, demonstrating their thought process. The AI's response consisted of two parts: first, an explanation of its step-by-step analysis, encoding, and confidence level judgment logic within its internal reasoning; and second, a detailed list of analysis conclusions for 12 traffic indicators (such as pedestrians, wrong-way driving, and abnormal parking) in structured XML format in the final result, ultimately concluding that there were no obvious safety anomalies requiring reporting at the toll station. A "one-review, one-proofreading" system was implemented throughout the annotation process to ensure annotation quality.

[0041] Step 1.6 Dataset Partitioning The labeled dataset was randomly divided into training, validation, and test sets in an 8:1:1 ratio. During the partitioning, it was ensured that the distribution ratios of different weather conditions, scenarios, and event types were approximately consistent across the three sets.

[0042] Step 1.7 Model Fine-tuning Choose an open-source visual language model (such as InternVL-Chat-V1.5 or Qwen-VL-Max) as the base model.

[0043] Hardware allocation: Model training and inference run on dedicated graphics cards with large amounts of video memory; Input: The structured results (text) of the AI ​​event are concatenated with the corresponding video keyframes (images) as multimodal input; Training: Supervised fine-tuning (SFT) is employed, using efficient parameter fine-tuning techniques such as Low-Rank Adaptation (LoRA) to train the model on the collected dataset. This allows the model to learn to follow the thought processes outlined in the annotation guidelines and output standardized, structured results. The loss function is cross-entropy loss. (TVM-R1 multimodal attention mechanism formula:) ; in For querying the matrix, The key matrix, For value matrices, is the dimension of the key vector.

[0044] Cross-entropy loss function: ; in For the sample size, For the number of categories, For real labels, To predict probabilities.

[0045] Evaluation: The model performance was evaluated on the validation set. The main metrics included event classification accuracy, recall, F1 score, and semantic similarity between the thought chain and human annotations.

[0046] In some embodiments, step S20 specifically includes the following steps: Step 2.1 Video Stream Access and Preprocessing Video source access: Access multiple video streams from the highway monitoring system via RTSP / RTMP protocol, with each video stream having a resolution of no less than 1080P and a frame rate of 25fps; Video Decoding: Utilizes GPU hardware decoding technology to decode the input video stream in real time, reducing CPU load; Frame Extraction and Preprocessing: Extracts video frames as needed and performs preprocessing operations such as size normalization and color space conversion on the frame images; Resource Allocation: Video decoding and preprocessing tasks are allocated to a dedicated YOLO detection graphics card for execution.

[0047] Step 2.2 YOLO Model Inference Model Deployment: The trained YOLOv11 model is deployed on two dedicated detection GPUs, supporting multi-instance parallel inference. YOLO object detection confidence formula: ; in This represents the probability that the bounding box contains the target. This is the intersection-union ratio (IU) between the predicted bounding box and the ground truth bounding box.

[0048] YOLO loss function: ; in For the number of grid cells, For the number of bounding boxes, Indicates the first The first grid Each bounding box contains the target. This indicates that the target is not included. For the coordinates of the predicted bounding box, For the coordinates of the true bounding box, For confidence level, This represents the category probability.

[0049] Object detection: Real-time object detection is performed on preprocessed video frames to identify the following main object categories: Vehicles: cars, trucks, buses, etc.

[0050] Pedestrians: Pedestrians in different poses.

[0051] Spilled materials: large obstacles, scattered objects, etc.

[0052] Performance optimization: Ensures that a single graphics card can handle 8-16 video streams simultaneously.

[0053] Step 2.3 Preliminary Event Assessment Based on the YOLO detection results, and in conjunction with the following rules and conditions, a preliminary event determination is made: Pedestrian intrusion event: A pedestrian is detected in the driving lane or emergency lane area for several frames.

[0054] Abnormal parking event: The vehicle stops in the driving lane for more than a preset time threshold (e.g., 10 seconds).

[0055] Road littering incident: Large, non-vehicle, non-pedestrian obstacles were detected in the lane.

[0056] Vehicle driving in the wrong direction: The vehicle's direction of movement is opposite to the lane's designated direction.

[0057] Traffic accident incident: Multiple vehicles gathered abnormally, accompanied by stopping, people getting out of the vehicles, etc.

[0058] Step 2.4 Structured Event Output The initially detected event information is encapsulated into structured data, which includes the event time, event type, 20-second video before and after the event, images of the event, and a bounding box indicating the event location.

[0059] In some embodiments, step S30 specifically includes the following steps: Step 3.1 Obtain a detection event Event reception: Receive event candidate sets generated by the YOLO detection module in real time through message middleware (such as Kafka, Redis).

[0060] Video clip retrieval: Retrieve the corresponding video clip based on the URL in the event information.

[0061] Keyframe extraction: Extracting keyframes from video clips, typically selecting N frames before and after the event (e.g., 2 frames per second).

[0062] Data assembly: The structured results of a single detection are combined with keyframe images to prepare for input into a large visual model.

[0063] Step 3.2 Model Inference Input Construction: Construct input prompts according to the format required by the fine-tuned visual model, including: System instructions: Define model roles and task requirements; Background knowledge: Basic knowledge of highways and event definitions; A single test result: Structured information detected by YOLO; Keyframe images: Extracted video keyframes; Inference execution: Sends the constructed input to the visual large model inference service deployed on a dedicated graphics card with large video memory; Parameter settings: Set appropriate inference parameters (temperature=0.1, top_p=0.9) to ensure output stability.

[0064] Step 3.3 Generate structured output Based on learned knowledge and reasoning ability, the large visual model outputs standardized, structured results: The actual model inference process is as follows: <think>Okay, I will think about it step by step. I've decided that for each thought process, I will first list the reasoning process and justifications, then provide the conclusion code, and the confidence interval for the judgment. After thinking about it internally, I have produced the final XML result. <scene> The image clearly shows two-way lanes with a median strip, but no ramps are visible, indicating a two-way traffic scene without ramps.< / scene> <fire_smoke> There is no flame or smoke in the footage. #NoEvent< / fire_smoke> <person> There is a person dressed in ordinary clothing on the left side of the emergency lane, not within the closed road area, and therefore not within the monitoring range of this system. #No incidents< / person> <animal> No animals were seen in the footage. #NoEvent< / animal> <scattered_debris> There were no spilled or dropped objects in the footage. #NoEvent< / scattered_debris><emergency_lane_parking> The image shows emergency lanes in both directions. No vehicles were seen parked in either direction's emergency lane. #NoIncident< / emergency_lane_parking><unusual_parking> No abnormal parking was observed in either direction of traffic in the footage. #No incident< / unusual_parking> <wrong-way> In the image, all vehicles whose front or rear is visible are facing in the same direction: all vehicles in the right-hand lane (going out) are facing that direction, and all vehicles in the left-hand lane (coming in) are facing that direction. Therefore, there are no vehicles traveling in the wrong direction. #No incident< / wrong-way> <traffic-accident> There was no traffic accident in the footage. #NoEvent< / traffic-accident> <construction> There's a traffic cone on the left side of the emergency lane guardrail, likely left after construction was completed; there was no construction activity. #NoEvent< / construction> <traffic-flow> Traffic flow is normal in both directions. #No incidents< / traffic-flow> <notice> Based on the above analysis, no anomalies were found, and there is no need to notify the security administrator. #No need to pay attention< / notice> .

[0065] In some embodiments, step S40 specifically includes the following steps: Step 4.1 Define the rule set The rule engine uses mature rule management systems such as Drools. Rule sets are defined based on business knowledge and historical false alarm analysis. Examples of key rules are as follows: Hallucination suppression rules: If the output of the large model does not meet the format requirements, modify the top_p parameter and re-infer; if the large model output description contains a description of a pedestrian in ordinary clothing, but the final result is no event, resulting in a logical inconsistency, the description shall prevail, and the result shall be revised to "pedestrian intrusion"; in other places where the conclusion is inconsistent with the single text description, the conclusion shall be revised to be consistent with the description.

[0066] Spatiotemporal association rules: If the same camera reports 3 or more "abnormal parking" events within 2 minutes and the cameras are the same, they will be automatically merged into one "congestion" event; if two adjacent cameras report "vehicle driving in the wrong direction" events one after another within 5 minutes and the driving direction is continuous, they will be confirmed as a real wrong-way trajectory and the events will be merged into one event.

[0067] Confidence fusion rule: When the YOLO detection confidence score is higher than 0.9 and the large model verification confidence score is higher than 0.8, it is directly confirmed as a real event. When the two detection results are inconsistent, the verification result of the large model is given priority. Step 4.2 Rule Engine Execution The rules engine is deployed as a standalone service running on the server CPU. The output of the large visual model is fed into the rules engine in real time. The rules engine loads the aforementioned rule set and performs pattern matching and inference on the input facts (i.e., the model output). After the engine finishes execution, it outputs the final verification result, stores it in the event database, and simultaneously pushes it to the traffic management command platform for alerts or display.

[0068] In some embodiments, a dedicated server and a processing system running thereon are included, wherein the dedicated server is equipped with multiple graphics cards, and: The first set of graphics cards is dedicated to running the YOLO algorithm and performing real-time event detection on the input video stream; The second set of graphics cards is dedicated to running large visual models and to reviewing and filtering the events detected by the first set of graphics cards.

[0069] To fully utilize the performance of dedicated server hardware, the following optimization strategies are adopted: Graphics card resource isolation: YOLO detection tasks and large model inference tasks run on separate graphics cards to avoid resource contention.

[0070] Hierarchical memory management: Frequently accessed video data is cached on SSDs, while historical data is stored on HDDs.

[0071] Pipeline processing: YOLO detection, large model verification, and rule engine processing form a pipeline to improve overall throughput.

[0072] Dynamic load balancing: dynamically adjusts the concurrency of each processing module based on real-time load.

[0073] Failover mechanism: When a graphics card fails, the system automatically migrates the task to another available graphics card.

[0074] The implementation principle of a method for identifying highway events using a large video model according to an embodiment of this application is as follows: The YOLO algorithm deployed on a first set of graphics cards performs real-time object detection on the input video stream, initially generating a candidate set of events including pedestrian intrusion and abnormal parking based on preset rules. Next, the candidate events and their video clips are input into a large visual model deployed on a second set of graphics cards for secondary inference. This large visual model is specially trained and capable of deep semantic understanding based on multimodal input, outputting structured judgment results according to the observation-analysis-reasoning-conclusion thought chain. Finally, a rule engine verifies the output of the large visual model, applying rule sets such as hallucination suppression, spatiotemporal correlation, and confidence fusion to filter out erroneous judgments and output a final reliable event conclusion. Thus, by integrating the real-time performance of traditional object detection with the deep reasoning capabilities of a large visual model, combined with post-processing optimization by a rule engine, the accuracy and reliability of event recognition in complex traffic scenarios are significantly improved, while ensuring the system's real-time processing capability for multiple high-definition video streams, providing effective technical support for intelligent traffic management.

[0075] Figure 1 This is a flowchart illustrating a method for identifying highway events using a large video model, as shown in one embodiment. It should be understood that, although... Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows; unless explicitly stated otherwise, there is no strict order requirement for the execution of these steps, and they can be executed in other orders; and Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0076] Based on the same technical concept, referring to Figure 2 This application also provides an apparatus for identifying highway events using a large video model, employing the following technical solution: the apparatus includes: The model training module is used to collect highway video datasets containing various types of traffic events, and to annotate them according to the preset annotation guidelines and inference chain standards. The annotated dataset is divided into training set and validation set, and is used to supervise and fine-tune the pre-trained visual large model to obtain the trained visual large model. The target detection module is used to access and preprocess highway monitoring video streams in real time. It uses the YOLO model deployed on the first set of graphics cards to perform real-time target detection on video frames. Based on the target detection results and preset rules, it initially determines traffic events and generates and outputs a candidate set of events containing event type, location and time information. The event reasoning module is used to obtain the event candidate set and its corresponding video segments. The structured results of the events and the extracted video key frames are input into the trained visual big model deployed on the second set of graphics cards. The visual big model performs reasoning analysis to obtain the structured event judgment results. The result verification module is used to retrieve a preset set of rules for suppressing large model illusions and making spatiotemporal correlation determinations. It uses the rule engine to filter and verify the event determination results output by the visual large model to obtain the final event determination results.

[0077] In some embodiments, the model training module is specifically used to collect a highway video dataset containing various traffic scenarios and event types, including pedestrian intrusion, abnormal parking, and road littering. Develop a labeling guide that includes basic knowledge of highways, precise definitions of abnormal events, and event classification standards; Based on the annotation guidelines and in accordance with the preset reasoning chain annotation standards, the collected video datasets were re-annotated. The reasoning chain annotation standards require the annotation process to follow the logical flow of observation-analysis-reasoning-conclusion, and the annotation results should be output in a structured data format. The labeled dataset is divided into training and validation sets according to a preset ratio; The training and validation sets are used to supervise the fine-tuning of the pre-trained visual large model. The structured text of the event and the corresponding video keyframe images are concatenated as multimodal inputs, and the model parameters are optimized using parameter-efficient fine-tuning techniques to obtain the trained visual large model.

[0078] In some embodiments, the target detection module is specifically used to access multiple highway monitoring video streams in real time via streaming media protocols, and to decode and extract frames from the video streams using GPU hardware decoding technology. The preprocessed video frames are input into the YOLO model deployed on the first set of graphics cards for real-time target detection, identifying targets including vehicles, pedestrians and debris; Based on the target detection results of the YOLO model, and combined with the preset rule conditions, the event is initially judged. The rule conditions include: a pedestrian is detected to be continuously present in the driving lane or emergency lane area, or a vehicle stops in the driving lane for more than a preset time threshold, or a large obstacle is detected to appear in the lane. The preliminary event information is encapsulated into a structured event candidate set, which includes event type, location coordinates, timestamp, and video clips before and after the event.

[0079] In some embodiments, the event reasoning module is specifically used to obtain corresponding video data based on the video clip URLs contained in the event candidate set, and extract key frame images before and after the event from the video data. The structured results in the event candidate set are combined with the extracted video keyframe images, and a multimodal input is constructed according to a preset format. The multimodal input includes system instructions, highway background knowledge, structured information from a single detection, and keyframe images. The constructed multimodal input is sent to the trained visual large model deployed on the second set of graphics cards, and inference parameters are set to control the output stability. The visual large model performs inference analysis and generates standardized, structured event judgment results containing event type, event occurrence time and event location information.

[0080] In some embodiments, the result verification module is specifically used to retrieve a predefined set of rules, which includes a hallucination suppression rule for correcting inconsistencies in the output logic of the visual large model, a spatiotemporal association rule for merging multiple events with spatiotemporal continuity, and a confidence fusion rule for fusing the confidence scores of the YOLO model and the visual large model. The event judgment results output by the visual big model are used as facts to input into the rule engine service. The rule engine loads the rule set to perform pattern matching and logical reasoning, filters, corrects and merges the input event judgment results, and outputs the final event judgment results.

[0081] In some embodiments, the object detection module is further used to employ the TVM-R1 multimodal attention mechanism in the large visual model. This mechanism calculates the dot product of the query matrix and the key matrix, scales it using the square root of the key vector dimension, normalizes it using the softmax function, and then multiplies it with the value matrix to fuse multimodal feature information from text and video keyframes. The loss function used for model training is the cross-entropy loss, which calculates the logarithmic difference between the model's predicted probability and the true label, and averages the results over all samples and categories to optimize the model parameters. The target detection confidence of the YOLO model is determined by the probability that the bounding box contains the target and the intersection-union ratio of the predicted box and the ground truth box. The loss function of the YOLO model comprehensively evaluates the bounding box coordinate prediction error, the confidence prediction error of whether the target is contained, and the category prediction error. For the calculation of the width and height errors of the bounding box, the square root is used to alleviate the scale sensitivity problem between large and small targets.

[0082] In some embodiments, the multiple graphics cards in the dedicated server are configured as follows: the first group of graphics cards consists of at least two graphics cards, which are dedicated to performing video decoding, frame extraction and model inference tasks of the YOLO algorithm in parallel; the second group of graphics cards consists of at least one graphics card with large video memory, which is dedicated to running inference tasks of large visual models, and the first group of graphics cards and the second group of graphics cards are isolated from each other in terms of hardware resources.

[0083] This application also discloses a control device.

[0084] Specifically, the control device includes a memory and a processor, the memory storing a computer program that can be loaded by the processor and executed as described above for a method of identifying highway events using a large video model.

[0085] This application also discloses a computer-readable storage medium.

[0086] Specifically, the computer-readable storage medium stores a computer program that can be loaded by a processor and executed as described above in a method for recognizing highway events using a large video model. The computer-readable storage medium includes, for example, various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0087] The above are all preferred embodiments of this application, and are not intended to limit the scope of protection of this application. Therefore, all equivalent changes made in accordance with the structure, shape and principle of this application should be covered within the scope of protection of this application.< / think>

Claims

1. A method for identifying highway events using a video large model, characterized in that, The method comprises the following steps: Collect a highway video dataset containing multiple traffic event types, and label the dataset according to a preset labeling guide and reasoning chain standard, divide the labeled dataset into a training set and a verification set, and use the dataset to supervise the fine-tuning of a visual large model pre-training model to obtain a trained visual large model; Real-time access and preprocessing of the expressway monitoring video stream, real-time target detection of the video frames by using the YOLO model deployed on the first group of graphics processing units (GPUs), preliminary determination of the traffic event based on the target detection result combined with a preset rule, generation and output of an event candidate set containing the event type, location and time information; Obtaining the event candidate set and its corresponding video segment, inputting the structured event result and the extracted video key frame into the trained visual large model deployed on the second group of GPUs for reasoning analysis to obtain a structured event determination result; Calling a preset rule set for suppressing large model hallucinations and spatiotemporal correlation determination, filtering and reviewing the event determination result output by the visual large model by using a rule engine to obtain a final event determination result.

2. The method for identifying highway events using a video large model according to claim 1, wherein, The method comprises the following steps: Collect a highway video dataset containing multiple traffic event types, and label the dataset according to a preset labeling guide and reasoning chain standard, divide the labeled dataset into a training set and a verification set, and use the dataset to supervise the fine-tuning of a visual large model pre-training model to obtain a trained visual large model, comprising: Collect a highway video dataset containing multiple traffic scenes and event types, wherein the event types include pedestrian intrusion, abnormal parking and road littering; Develop a labeling guide containing expressway basic knowledge, precise definition of abnormal events and event grading standards; According to the labeling guide, the video dataset collected is relabeled according to a preset reasoning chain labeling standard, which requires that the labeling process follow the logical process of observation-analysis-reasoning-conclusion, and the labeling result is output in a structured data format; Divide the labeled dataset into a training set and a verification set according to a preset ratio; 3. The method for identifying highway events using a video large model according to claim 2, wherein, The training set and the verification set are used to supervise the fine-tuning of a visual large model pre-training model, wherein the structured event result text and the corresponding video key frame image are spliced as multi-modal input, and a parameter-efficient fine-tuning technology is used to optimize the model parameters to obtain the trained visual large model. The method comprises the following steps: Real-time access and preprocessing of the expressway monitoring video stream, real-time target detection of the video frames by using the YOLO model deployed on the first group of GPUs, preliminary determination of the traffic event based on the target detection result combined with a preset rule, generation and output of an event candidate set containing the event type, location and time information, comprising: Real-time access to multiple expressway monitoring video streams through a streaming media protocol, and decoding and frame extraction of the video stream by using GPU hardware decoding technology; Input the preprocessed video frames into the YOLO model deployed on the first group of GPUs for real-time target detection to identify targets including vehicles, pedestrians and littering objects; Based on the target detection result of the YOLO model, an event is preliminarily determined according to preset rule conditions, the rule conditions including: a pedestrian continuously existing in a lane or an emergency lane is detected, or a vehicle stops in a lane for more than a preset time threshold, or a large obstacle is detected in the lane; The event information determined preliminarily is encapsulated as a structured event candidate set, the event candidate set including an event type, a location coordinate, a timestamp, and a video clip before and after the event.

4. The method for identifying highway events using a video large model according to claim 3, wherein, The event candidate set and the corresponding video clip are obtained, the structured result of the event and the extracted video key frame are input into the trained visual large model deployed on the second group of graphics cards, inference analysis is performed by the visual large model, a structured event determination result is obtained, including: Based on the video clip URL included in the event candidate set, corresponding video data is obtained, and key frame images before and after the event are extracted from the video data; The structured result in the event candidate set and the extracted video key frame images are combined, a multi-modal input is constructed in a preset format, the multi-modal input including a system instruction, a highway background knowledge, structured information of one detection, and key frame images; The constructed multi-modal input is sent to the trained visual large model deployed on the second group of graphics cards, inference parameters are set to control output stability, inference analysis is performed by the visual large model, and a standardized event determination result is generated, including an event type, an event occurrence time, and an event location information.

5. The method for identifying highway events using a video large model according to claim 4, wherein, The preset rule set for suppressing large model hallucinations and temporal-spatial correlation determination is called, the event determination result output by the visual large model is filtered and reviewed by a rule engine, and a final event determination result is obtained, including: The predefined rule set is called, the rule set including hallucination suppression rules for correcting inconsistent logic output by the visual large model, temporal-spatial correlation rules for merging multiple events with temporal-spatial continuity, and confidence fusion rules for fusing the confidence of the YOLO model and the visual large model; The event determination result output by the visual large model is input into a rule engine service as a fact, the rule engine loads the rule set for pattern matching and logical inference, filters, corrects, and merges the input event determination result, and outputs the final event determination result.

6. The method for identifying highway events using a video large model according to claim 5, wherein, The method further includes: The visual large model adopts a TVM-R1 multi-modal attention mechanism, calculates the dot product of a query matrix and a key matrix, scales by the square root of the dimension of the key vector, normalizes by a softmax function, and then multiplies a value matrix to fuse the multi-modal feature information of text and video key frames; a loss function used in model training is a cross-entropy loss, which calculates the logarithmic difference between the model prediction probability and the true label, and takes the average value of all samples and classes to optimize the model parameters. The target detection confidence of the YOLO model is determined by the probability of the target being contained in the bounding box and the intersection over union of the predicted box and the real box; the loss function of the YOLO model comprehensively evaluates the prediction error of the bounding box coordinates, the confidence prediction error of whether the target is contained, and the class prediction error, wherein the square root is used for the width and height error calculation of the bounding box to alleviate the scale sensitivity problem between large and small targets.

7. The method for identifying highway events using a video large model according to claim 1, wherein, The multiple graphics cards in the special server are configured as: The first group of graphics cards is composed of at least two graphics cards, which are specially used for parallel execution of video decoding, frame extraction and model inference tasks of the YOLO algorithm; The second group of graphics cards is composed of at least one large memory graphics card, which is specially used for running the inference task of the visual large model, and the first group of graphics cards and the second group of graphics cards are isolated from each other in hardware resources.

8. An apparatus for identifying a highway event using a video large model, the apparatus comprising: The device comprises: A model training module is configured to collect a highway video dataset containing multiple traffic event types, label the dataset according to a preset labeling guide and inference thinking chain standard, divide the labeled dataset into a training set and a validation set, and supervise the pre-training of a visual large model to obtain a trained visual large model; A target detection module is configured to access and preprocess a real-time highway monitoring video stream, use the YOLO model deployed on the first group of graphics cards to perform real-time target detection on the video frames, preliminarily determine a traffic event based on the target detection result combined with a preset rule, and generate and output an event candidate set containing event type, location and time information; An event inference module is configured to obtain the event candidate set and its corresponding video segment, input the structured event result and the extracted video key frame into the trained visual large model deployed on the second group of graphics cards, and perform inference analysis by the visual large model to obtain a structured event determination result; A result review module is configured to call a preset rule set for suppressing large model hallucinations and temporal and spatial correlation determination, filter and review the event determination result output by the visual large model using a rule engine, and obtain a final event determination result.

9. A control device, characterized by The device comprises: A memory and a processor, wherein the memory stores a computer program capable of being loaded and executed by the processor to perform the method of any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, A computer program capable of being loaded and executed by the processor to perform the method of any one of claims 1 to 7.

Images