Two-stage road traffic abnormal event identification method and system based on visual large model

Abstract

The invention relates to a two-stage road traffic abnormal event identification method and system based on a visual large model, and the method comprises the steps: collecting the monitoring image and video data of an expressway and an urban expressway, and building a static image semantic data set and a dynamic video traffic semantic data set; utilizing the static image semantic data set to train a visual large model to obtain a first visual large model; constructing a same-preference data pair, and performing direct preference optimization training of the first visual large model by using the same-preference data pair to obtain a second visual large model; intercepting an abnormal video key frame based on the dynamic video traffic semantic data set, and performing parameter fine tuning on the second visual large model based on the abnormal video key frame to obtain a road traffic abnormal event recognition model; and collecting a monitoring image or video sequence in real time, and performing abnormal event identification by using the road traffic abnormal event identification model. Compared with the prior art, the traffic abnormal event identification method provided by the invention can effectively combine dynamic and static characteristics of data and is efficient.

Description

Technical Field

[0001] This invention relates to the field of road traffic anomaly event recognition, and in particular to a two-stage road traffic anomaly event recognition method and system based on a large visual model. Background Technology

[0002] In modern urban transportation systems, timely identification and early warning of traffic anomalies are crucial for maintaining road safety and improving traffic efficiency. Current traffic monitoring primarily relies on manual patrols and fixed-rule algorithms, which are ill-suited to the complex and dynamic environments of urban expressways and highways. With the digital transformation of transportation infrastructure, urban road video surveillance has achieved widespread coverage, generating massive amounts of traffic images and video data that enable automated analysis. Deep learning-based technologies such as image recognition, vehicle detection, and multi-target tracking have been applied in intelligent transportation, significantly reducing the cost of manual patrols and supporting traffic management and emergency decision-making.

[0003] In recent years, the development of large-scale visual models has provided new solutions for semantic understanding and abnormal event detection in traffic scenarios. Compared with traditional computer vision models, large-scale visual models have stronger feature extraction and semantic modeling capabilities, and can identify complex events by combining scene context. However, existing methods struggle to accurately distinguish and locate abnormal event objects in complex traffic environments; moreover, most technologies rely on "single static recognition" or "single dynamic reasoning." For example, Chinese patent application CN119229388A provides a traffic accident recognition method based on unsupervised pre-training and a multimodal large-scale model. It uses video data to perform unsupervised training on a mask autoencoder to capture dynamic and static changes in the video, and reconstructs the video based on the learned features. The video type is labeled by calculating the difference between the reconstructed video and the original video. The labeled data is then used to train a multimodal large-scale model, and finally, the trained multimodal large-scale model is used to identify traffic accidents. However, this type of method has the following problems: 1) It consumes huge amounts of computing resources and has low recognition efficiency: The fine-tuning of large models driven by a single type of video data has high requirements for computing resources, which limits its practical application; 2) It is not sensitive to static features: The above-mentioned technical solution is essentially a dynamic anomaly detector, which is good at detecting abnormal motion but has difficulty in recognizing abnormal stillness. It excessively binds anomalies to unpredictable motion and ignores the static anomalies of many accidents at the spatial semantic level. It is not sensitive to static features, which affects the model's recognition accuracy. Similarly, if only a single static data is used for training and recognition, it will lead to the model being insensitive to dynamic features, which will also limit the model's recognition accuracy.

[0004] Therefore, providing a highly efficient method for identifying traffic anomalies that effectively combines the dynamic and static features of data is a technical problem that needs to be solved. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the existing technology by providing a two-stage road traffic anomaly event recognition method and system based on a large visual model. The large visual model is fine-tuned and trained from the perspectives of static images and dynamic videos to improve the model's discrimination and generalization capabilities, element recognition and semantic reasoning capabilities in traffic anomaly event detection. Combined with a structural and semantic evaluation index system, the invention enables automatic detection and result evaluation of traffic anomalies.

[0006] The objective of this invention can be achieved through the following technical solutions: According to a first aspect of the present invention, a two-stage road traffic anomaly event identification method based on a large visual model is provided, the method comprising: Collect monitoring images and video data of highways and urban expressways, and establish a static image semantic dataset and a dynamic video traffic semantic dataset covering multiple types of typical abnormal events; A large visual model is trained using the aforementioned static image semantic dataset to obtain the first large visual model. Based on the aforementioned traffic anomaly event semantic dataset, construct similar preference data pairs, and use the similar preference data pairs to perform direct preference optimization training of the first visual large model to obtain the second visual large model. The video data in the dynamic video traffic semantic dataset is extracted at preset intervals to extract abnormal video key frames. Based on the abnormal video key frames and the dynamic video traffic semantic dataset, the parameters of the second vision model are fine-tuned to obtain the road traffic abnormal event recognition model. Real-time acquisition of monitoring images or video sequences, and use of the road traffic anomaly event recognition model to identify anomalies.

[0007] As a preferred technical solution, both the static image semantic dataset and the dynamic video traffic semantic dataset include event types, involved objects, event locations, and event impact information based on manual annotation of monitoring images and videos; the event types include multi-vehicle accidents, abnormal parking, two-wheeled vehicles running in, and spilled materials.

[0008] As a preferred technical solution, the annotation method is as follows: using a large visual model to perform preliminary annotation on the collected monitoring images and video data based on judgment rules, and manually correcting the preliminary annotation results to complete the marking; For the aforementioned multi-vehicle accidents, the corresponding judgment rules are as follows: two or more vehicles show collision marks; the vehicles are in abnormal positions; people can be seen getting out of their vehicles to negotiate or traffic police arrive at the scene to handle the situation; other vehicles take emergency evasive action or stop due to the accident. The corresponding judgment rules for the aforementioned abnormal parking are as follows: the vehicle is completely stationary in a non-parking area; there are no traffic lights or signs allowing parking; the vehicle has not turned on its hazard lights or has them on but still does not meet the temporary parking rules; the driver is not in the vehicle or the vehicle has an obvious malfunction; vehicles behind are forced to detour or slow down. The corresponding criteria for determining the spilled material are: there is a foreign object of sufficient size in the lane; there is visible vehicle braking or detouring behavior; the foreign object may have come from the cargo box of a truck or accident residue. The corresponding rules for determining whether a two-wheeled vehicle has entered the restricted area are as follows: the motorcycle / electric vehicle / bicycle is in a prohibited area; there are dangerous behaviors such as driving against traffic or crossing the lane; there is no dedicated two-wheeled lane or obvious prohibition signs; the vehicle has no license plate or the driver is not wearing a helmet.

[0009] As a preferred technical solution, the method for constructing the same preference data pairs is as follows: Based on the static image semantic dataset and the dynamic video traffic semantic dataset, a first-choice answer and a second-choice answer are generated for each abnormal traffic event. The first-choice answer and the second-choice answer are a pair of data with the same preference. Furthermore, compared to the first choice, the secondary answer introduces redundant and ambiguous information in its event category, location, and element description, including lengthy and repetitive irrelevant traffic anomaly information.

[0010] As a preferred technical solution, the method further includes a two-stage evaluation of the road traffic abnormal event identification model; The first stage of evaluation involves: based on the road traffic anomaly event identification model, identifying static events from static images and calculating structural and semantic indicators. The structural indicators include BERTS score and accuracy. The semantic indicators include redundancy in the static event description, completeness in the description of static scene elements, and accuracy in identifying static event types. The second stage of evaluation involves calculating the authenticity and completeness of the dynamic event descriptions obtained from the road traffic anomaly event identification model based on the dynamic video. The authenticity is calculated based on the degree of consistency between the dynamic event descriptions and the dynamic video, and the completeness is calculated based on the coverage of the dynamic scene elements by the dynamic event descriptions.

[0011] According to a second aspect of the present invention, a two-stage road traffic anomaly event recognition system based on a large visual model is provided, the system comprising: The data acquisition and processing module collects monitoring images and video data from highways and urban expressways, and establishes a static image semantic dataset and a dynamic video traffic semantic dataset covering multiple types of typical abnormal events. The model training module trains a large visual model using the static image semantic dataset to obtain a first large visual model. It then constructs similar preference data pairs based on the traffic anomaly event semantic dataset and uses these pairs for direct preference optimization training of the first large visual model to obtain a second large visual model. Finally, it extracts keyframes of abnormal videos from the video data in the dynamic video traffic semantic dataset at preset intervals and fine-tunes the parameters of the second large visual model based on these keyframes and the dynamic video traffic semantic dataset to obtain a road traffic anomaly event recognition model. An abnormal event identification module acquires monitoring images or video sequences collected in real time by the data acquisition and processing module, and uses the road traffic abnormal event identification model to identify abnormal events.

[0012] Compared with the prior art, the present invention has the following beneficial effects: 1) This invention proposes a two-stage large model training method. In the first stage, a large visual model is trained using a static image dataset. In this stage, the model only needs to learn the static spatial semantic features of identifying abnormal events from a single image, avoiding complex temporal calculations and greatly reducing the computational burden of initial training. Based on the training in the first stage, the model parameters are fine-tuned using the LoRA method based on dynamic video data. Since the model already has a strong static feature perception capability, it only needs to combine lightweight temporal analysis to make judgments on dynamic events, improving the overall processing speed and real-time performance without consuming a large amount of computational resources.

[0013] 2) When identifying traffic anomalies, this invention uses static identification as the foundation and dynamic identification as a supplement. During the dynamic video-based identification training phase, key frames of the anomaly video are extracted to capture the most representative static anomaly moments in the dynamic sequence. These key frames are then used to fine-tune the statically optimized model, enabling the model to ultimately possess the analytical capabilities of combining static and dynamic analysis. Compared with traditional video-based anomaly identification methods, this invention improves the sensitivity to static data in dynamic sequences, thereby achieving more accurate anomaly event identification.

[0014] 3) When training the model, this invention also introduces data pairs with similar preferences for preference learning, so that the model learns the difference between the first choice and the second choice, directly guiding and strengthening the recognition model's ability to understand and judge static anomaly semantics, and making its sensitivity to static features optimal. Attached Figure Description

[0015] Figure 1 This forms the method framework of the present invention; Figure 2These are example images of major traffic events related to the static images of this invention; Figure 3 This is a schematic diagram of the dynamic video processing flow of the present invention; Figure 4 This is a schematic diagram of the visual large model architecture in the static image fine-tuning stage of the present invention. Figure 5 This is a schematic diagram of the visual large model architecture for the dynamic video fine-tuning stage of the present invention. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0017] Example 1 To address the challenge of accurately distinguishing and locating abnormal events in complex environments, this invention leverages the semantic understanding capabilities of large-scale models to design judgment rules that assist in scene recognition. A traffic semantic dataset covering core judgment features such as event type, involved objects, location, and impact is constructed to train the model and improve recognition accuracy. To address the high computational resource requirements of large-scale model fine-tuning driven by local data, this invention employs a strategy combining event type-based optimization and efficient parameter fine-tuning. For typical abnormal events such as spilled materials and heavy fog, static image understanding is considered; for accident-type events such as multi-vehicle accidents and abnormal parking, static recognition and dynamic video inference are integrated. Simultaneously, low-rank adaptation (LoRA) and its variants (such as rsLoRA / PiSSA) fine-tuning strategies are used, updating only some parameters of the model's attention layer, significantly reducing memory usage and training time, thus adapting to local deployment requirements.

[0018] Specifically, the process of the method provided by this invention is as follows: Figure 1 As shown, it includes the following steps: S1. Collect monitoring images and video data of highways and urban expressways, and establish a static image semantic dataset and a dynamic video traffic semantic dataset covering multiple types of typical abnormal events.

[0019] First, traffic images and video data are collected by surveillance cameras deployed on urban expressways and highways, such as... Figure 2 As shown. To ensure the representativeness of the data, several typical abnormal events were selected from actual road operations, such as multi-vehicle accidents, abnormal parking, road construction, two-wheeled vehicles entering the road, and debris spillage.

[0020] The static images and dynamic video data of the aforementioned abnormal events are preprocessed uniformly. The process for extracting keyframes from the dynamic video data is as follows: Figure 3 As shown, the process includes: manually extracting representative video segments from abnormal videos, setting the starting point of extraction as when the event has warning signs and the ending point as when the event ends, with each segment lasting approximately 10 seconds (this duration is based on statistical analysis of two types of key abnormal events in the traffic semantic dataset constructed in this invention, which can cover more than 95% of the complete event process, balancing information integrity and redundancy); then extracting keyframes from the 10-second segment at an interval of 1 frame / second, ultimately forming a dataset with 10 keyframes corresponding to each video sample, which is used for dynamic video training.

[0021] Size normalization is performed on static image data, including scaling the images to a uniform resolution of 384*384 to ensure the consistency of data size in the input model and avoid feature extraction deviations caused by resolution differences.

[0022] After preprocessing, the acquired surveillance images and video data are initially labeled using the Visual Large Model API based on decision rules and basic information. The initial labeling results are then manually corrected to complete the labeling process, as follows: For multi-vehicle accidents, the corresponding judgment rules are: two or more vehicles show collision marks; the vehicles are in abnormal positions; people can be seen getting out of their vehicles to negotiate or traffic police arrive at the scene to handle the situation; other vehicles take emergency evasive action or stop due to the accident.

[0023] The corresponding rules for judging abnormal parking are as follows: the vehicle is completely stationary in a non-parking area; there are no traffic lights or signs that allow parking; the vehicle has not turned on its hazard lights or has them on but still does not meet the temporary parking rules; the driver is not in the vehicle or the vehicle has an obvious malfunction; vehicles behind are forced to detour or slow down.

[0024] The corresponding rules for determining spilled material are: there is a foreign object of sufficient size in the lane; there is visible vehicle braking or detouring behavior; the foreign object may have come from the cargo box of a truck or accident residue.

[0025] The corresponding rules for judging two-wheeled vehicles entering the restricted area are as follows: motorcycles / electric vehicles / bicycles appear in prohibited areas; there are dangerous behaviors such as driving against traffic or crossing lanes; there is no dedicated two-wheeled lane or obvious prohibition signs; the vehicle has no license plate or the driver is not wearing a helmet.

[0026] Finally, a traffic event semantic dataset was constructed, which includes a static image semantic dataset and a dynamic video traffic semantic dataset. Both the static image semantic dataset and the dynamic video traffic semantic dataset include information on event type, involved objects, event location, and event impact based on manually annotated surveillance images and videos.

[0027] S2. Train a large visual model using a static image semantic dataset to obtain the first large visual model.

[0028] This step is the first stage of training in this invention. In this stage, the large visual model is optimized and trained using a static image dataset. The entire process adopts a low-rank adaptation strategy (LoRA) (applied to the q_proj / v_proj module of the model, with a rank value r=4~16): First, supervised fine-tuning is used to enable the model to learn the mapping relationship between "visual features of abnormal events - category labels - event description and elements". Then, direct preference optimization training (DPO) is performed. With manually designed preference pairs that have differences in recognition accuracy and semantic generalization conciseness and completeness as constraints, the text quality of the model output is optimized to achieve the recognition and semantic generalization of abnormal events in static images.

[0029] In detail, the parameter configuration for the pre-trained large visual model (such as Qwen-2.5-VL) during the efficient fine-tuning of parameters is as follows: i) LoRA adapter parameters.

[0030] The rank (r) of the low-rank matrix ranges from 4 to 16. The selection criteria are as follows: when r < 4, the model's expressive power is insufficient and it cannot capture the fine-grained features of traffic anomalies (such as the shape of spilled objects and the posture of accident vehicles); when r > 16, the memory usage increases by more than 30% (based on tests using NVIDIA A100 graphics cards), and the training efficiency decreases, while the improvement in recognition accuracy is less than 2%. Therefore, 4 to 16 is the optimal range for performance and cost.

[0031] Scaling factor ( ): Set to 2r (e.g., when r=8) =16), used to balance the update magnitude of the low-rank matrix and avoid overfitting caused by excessive fluctuations in model parameters during fine-tuning.

[0032] Target-acting module: Only for the model's attention layer (Query projection) and The (value projection) module is inserted into the LoRA adapter because these two modules directly affect the feature extraction of image semantics, and targeted fine-tuning can reduce the number of trainable parameters by 70% (compared to full fine-tuning).

[0033] Dropout probability: ranging from 0.1 to 0.2, used to suppress overfitting, especially suitable for traffic scenarios where similar abnormal events have different appearances, such as the characteristics of accident postures of different vehicle models.

[0034] ii) SFT supervised fine-tuning parameters.

[0035] Optimizer: The AdamW optimizer is used, with the weight decay coefficient set to 1e-4 to avoid the model overfitting to the noise of the training data.

[0036] Learning rate: The initial learning rate is 1e-4 to 5e-4; a linear learning rate decay strategy is adopted, and the learning rate is reduced to 1 / 10 of the initial value in the later stage of training (epoch>10) to ensure stable convergence of the model.

[0037] Batch size: Set to 8~16 based on a single NVIDIA A100 (40GB VRAM); if using multiple cards in parallel (e.g., 4 cards), it can be expanded to 32~64, while maintaining the gradient accumulation step count at 4, balancing training speed and VRAM usage.

[0038] Training epochs: Set to 8-15 epochs, and trigger early stopping when the F1 score on the validation set does not improve for 3 consecutive epochs to avoid ineffective training.

[0039] S3. Construct similar preference data pairs based on the semantic dataset of traffic anomalies, and use these similar preference data pairs to perform direct preference optimization training on the first visual large model to obtain the second visual large model.

[0040] Specifically, the process of constructing the same preference data pair is as follows: Based on the static image semantic dataset and the dynamic video traffic semantic dataset, a first choice answer and a second choice answer are generated for each abnormal traffic event. The first choice answer and the second choice answer constitute a same preference data pair. Compared with the first choice answer, the second choice answer introduces redundant and ambiguous information in the event category, location and feature description, including lengthy and repetitive irrelevant traffic anomaly information to ensure the distinguishability of the preference signal, such as irrelevant information or factual errors such as descriptions of green belts, weather, etc., such as misjudgment of accident type, incorrect description of key elements, etc.

[0041] In this embodiment, the preference optimization DPO parameter is set as follows: Preference data pair ratio: Each static image sample corresponds to 2-3 pairs of data with the same preference.

[0042] Temperature coefficient ( ): Set to 0.1~0.3; used to adjust the influence of preference weights in the DPO loss function. Too large a value will cause the model to be overly biased towards the first choice and lose its generalization ability, while too small a value will prevent it from effectively learning the preference signal.

[0043] Training batches: consistent with SFT (batch size=12), with half the number of training rounds as SFT (6~8 rounds) to avoid DPO training over-covering the basic semantic capabilities of SFT.

[0044] By training the model using Direct Preference Optimization (DPO) based on manually constructed preference data, the semantic accuracy and generalizability of the model's output text can be improved, enabling the identification and semantic understanding of abnormal events in static images.

[0045] Through the first stage of training using static images in steps S2-S3, the second large-scale visual model can achieve the following: Figure 4 The effect shown.

[0046] S4. Extract abnormal video keyframes from the video data in the dynamic video traffic semantic dataset at preset intervals. Based on the abnormal video keyframes and the dynamic video traffic semantic dataset, fine-tune the parameters of the second vision model to obtain the road traffic abnormal event recognition model.

[0047] This step is the second stage of the invention, which introduces a multi-frame video modeling method. The video data in the dynamic video semantic dataset is extracted at an interval of 1 frame / second to extract abnormal video key frames (forming a sequence of 8 to 16 frames). The LoRA algorithm is used to efficiently fine-tune the parameters of the model trained in the first stage in combination with the semantic annotations in the dataset. The cross-frame action coherence and trajectory relationship are modeled through the inter-frame attention layer to realize the recognition and reasoning of complex dynamic traffic events.

[0048] During this training phase, the core parameters of LoRA from the first phase (r=4~16, α=2r, dropout=0.15) are used, and the following parameters are added for video modeling scenarios: i) Video keyframe processing parameters.

[0049] Keyframe capture interval: 1~5 frames / second, determined based on the dynamic characteristics of traffic anomalies (e.g., a vehicle collision process lasting about 2~4 seconds). This interval can reduce redundant frame data by 50% while preserving the complete event trajectory.

[0050] Frame sequence length: 8 to 16 key frames are extracted from each video sample to form a fixed-length frame sequence input model, ensuring the consistency of cross-frame semantic fusion, such as the trajectory continuity of a multi-vehicle collision.

[0051] Frame normalization: All keyframes are uniformly scaled to 224×224 or 384×384 resolution to match the input size of the second vision large model, avoiding the loss of accuracy caused by feature scaling.

[0052] ii) Low-rank adaptation and LoRA variant (rsLoRA / PiSSA) parameters.

[0053] rsLoRA (Rank-Stable LoRA): The sparsity is set to 0.3~0.5, that is, only 40% of the elements in the low-rank matrix of LoRA are activated for updating, reducing the training computation by 25% while maintaining the recognition accuracy (decreased by ≤1%).

[0054] PiSSA (LoRA based on SVD initialization): During initialization, the first r singular values ​​of the pre-trained model weights are used (e.g., the first 8 when r=8), and the initial learning rate is reduced to 1e-4 (lower than the 2e-4 of conventional LoRA), avoiding convergence difficulties caused by excessive fluctuations in initialization parameters.

[0055] Cross-frame attention parameters: The learning rate of the newly added inter-frame attention layer is 1.5e-4, which is 1 / 4 of that of the basic LoRA layer (2e-4), prioritizing the optimization of the modeling ability of inter-frame temporal relationships (such as the trajectory association of vehicle lane changes).

[0056] Through this stage of training, the large visual model can not only identify objects in videos, but also understand their temporal relationships and behavioral logic, thereby improving its ability to identify abnormal events in dynamic scenes. The results are as follows: Figure 5 As shown.

[0057] In addition, a two-stage evaluation will be conducted on the road traffic anomaly event identification model after the model training is completed; The first phase of evaluation involves calculating structural and semantic metrics based on static event descriptions obtained from static images using a road traffic anomaly event recognition model. Structural metrics include BERTS score and accuracy; semantic metrics include redundancy in the static event description, completeness of the description of static scene elements, and accuracy in identifying static event types.

[0058] The second phase of evaluation involves calculating the authenticity and completeness of dynamic event descriptions obtained from dynamic videos based on the road traffic anomaly event recognition model. Authenticity is calculated based on the degree of consistency between the dynamic event description and the dynamic video, while completeness is calculated based on the coverage of the dynamic scene elements by the dynamic event description.

[0059] In detail, the structural indicators mentioned above include: BERTScore: The cosine similarity between the model output text and the labeled text is calculated using a pre-trained BERT model, and the mean value is taken as the final score (similarity range 0-1, the higher the score, the stronger the consistency).

[0060] Precision = .

[0061] Recall = .

[0062] F1 score = 2 × .

[0063] Regarding the semantic indicators mentioned above: Authenticity: This measures whether the information described in the generated text matches the image content, focusing on the accuracy of key information such as event type, location, vehicle behavior, and personnel behavior. The scoring criteria are shown in Table 1, and a bonus system is introduced to assist in verification. To avoid scoring bias, the initial score is set at 5 points. Different scores are assigned to different types of abnormal accidents based on their severity: 3 points are deducted for incorrect accident type, 1 point for incorrect description of the vehicle / location involved, 1 point for incorrect description of the specific accident location (e.g., lane number / mileage), and at least 3 points are awarded for a correct event type description.

[0064] Table 1 Authenticity Score Table Completeness: This measures whether the generated text covers all key scene elements in the image, such as accident location, human behavior, vehicle behavior, and event type. The scoring criteria are shown in Table 2. Table 2 Integrity Scoring Table Redundancy: This reflects whether the generated text contains information unrelated to the traffic accident, such as background descriptions, timestamps, and location names. Redundant content is counted in units of clauses (short sentences). One point is added for each piece of irrelevant content, up to a maximum of 5 points. The higher the score, the more serious the redundancy. For details, see Table 3.

[0065] Table 3 Redundancy Scoring Table Event Accuracy: Reflects whether the model can correctly identify the type of abnormal event. 1 point is awarded for a correct identification and 0 points for an incorrect identification.

[0066] Ultimately, the optimal model was selected as the road traffic anomaly event identification model based on the two-stage evaluation results.

[0067] S5. Real-time acquisition of monitoring images or video sequences, and use of road traffic abnormal event recognition model to identify abnormal events.

[0068] Example 2 To verify the feasibility and superiority of the method provided by the present invention, this embodiment takes the traffic of a certain city as an example and conducts the following experiment using the method provided in the above embodiment.

[0069] S1. Collect monitoring images and video data of highways and urban expressways, and establish a static image semantic dataset and a dynamic video traffic semantic dataset covering multiple types of typical abnormal events.

[0070] A large amount of traffic monitoring data was collected using surveillance cameras on urban expressways and highways in a certain city, including approximately 20,000 images and 1,500 video clips, covering typical scenarios such as traffic congestion, multi-vehicle accidents, abnormal parking, littering, and severe weather. In this embodiment, to verify the effectiveness of the two-stage optimization method, 876 static image samples and 471 video samples were obtained by filtering and cleaning from the above-mentioned original dataset.

[0071] Targeted preprocessing was performed on the selected samples: static images were uniformly scaled to 384×384 resolution; keyframes of video samples were extracted at intervals of 1 frame / second. Subsequently, the images and videos were initially labeled using a general-purpose visual large model API, including event type, involved objects, event location, and event impact. Then, manual correction was performed to finally establish a semantic dataset of traffic anomaly events, which was divided into training set, validation set, and test set.

[0072] S2. Train a large visual model using a static image semantic dataset to obtain the first large visual model.

[0073] S3. Construct similar preference data pairs based on the semantic dataset of traffic anomalies, and use these similar preference data pairs to perform direct preference optimization training on the first visual large model to obtain the second visual large model.

[0074] In the first stage of training (steps S2-S3), a pre-trained Qwen-2.5-VL model was used as the baseline model. The LoRA parameter efficient fine-tuning and supervised fine-tuning (SFT) method were used to train a subset of static images. Simultaneously, based on the same preference data pair construction rule, a total of 1752 preference data pairs were generated, including 876 static samples * 2 pairs / sample. Among these, 70% contained redundant secondary answers, and 30% contained factual errors. After training, the model redundancy decreased from 0.044 in the SFT stage to 0.007, validating the effectiveness of the construction rule.

[0075] In this embodiment, the specific parameters for static image fine-tuning are set as follows: base model: Qwen-2.5-VL (7B parameter version); LoRA parameter: r=8. =16, dropout=0.1, only applies to / Module; SFT parameters: AdamW optimizer (weight decay=1e-4), learning rate 3e-5 (cosine annealing), batch size=2 (single card operation, gradient accumulation steps=2), training epochs 10.

[0076] DPO parameters: Preference data ratio 1:2 (1 sample corresponds to 2 preference pairs), temperature coefficient =0.2, 3 training rounds Consistent with SFT; Hardware environment: NVIDIA vGPU-32GB (32GB VRAM), PyTorch 2.5.1 framework.

[0077] The model performance after training based on the above parameters is shown in the table below: Table 4 Performance Evaluation Table for the First Stage Model Training As can be seen, after the first stage of optimization training, the model has made significant improvements in static image recognition and semantic understanding.

[0078] S4. Extract abnormal video keyframes from the video data in the dynamic video traffic semantic dataset at preset intervals. Based on the abnormal video keyframes and the dynamic video traffic semantic dataset, fine-tune the parameters of the second vision model to obtain the road traffic abnormal event recognition model.

[0079] In the second stage, typical abnormal event segments were extracted from 471 video samples. Keyframe sequence modeling and cross-frame semantic fusion methods were used to further optimize the large visual model, enhancing its ability to understand cross-frame trajectory relationships and temporal logic. Specifically, the parameter settings for dynamic video fine-tuning are as follows: Video preprocessing: keyframe extraction interval 1 frame / second, frame sequence length 10 frames, resolution 384×384; Different LoRA variant parameters: ①LoRA: r=4, α=8, dropout=0.1, learning rate 5e-5; ②rsLoRA: r=4, α=8, sparsity 0.4, learning rate 5e-5; ③PiSSA: r=4, α=8, based on SVD initialization (taking the first 8 singular values), learning rate 5e-5.

[0080] Common parameters: AdamW optimizer (weight decay=1e-4), batch size=2 (single card operation), 1.5 training epochs, learning rate of 1.5e-4 for cross-frame attention layers.

[0081] Hardware environment: NVIDIA H20 (96GB video memory).

[0082] After training based on the above parameters, the performance of the model in dynamic video scenes is shown in Table 5: Table 5 Performance Evaluation of Model Training in the Second Stage It is evident that the method provided by this invention can effectively improve the realism, completeness, and accuracy of the baseline model representation under different LoRA variant parameters.

[0083] S5. Collect monitoring images or video sequences, and use the road traffic abnormal event recognition model to identify abnormal events.

[0084] In this embodiment, the operability of the manual evaluation rules is verified through the following typical samples. The specific process is as follows: A1. Static image sample verification.

[0085] Image content: On the K100 lane 3 of the highway, two cars collided. Both vehicles had their hazard lights on, and no one got out of the cars. Following vehicles were affected and slowed down to avoid them.

[0086] Model output: "The surveillance footage at 5:55 on December 30, 2023 shows that two cars collided in lane 3 of K100 on the highway. There are guardrails on both sides of the road and the road markings are clear." Semanticity metrics evaluation: Authenticity: The accident type (rear-end collision), location (K100 lane 3), and number of vehicles (2) are all accurate and do not contain any errors → Originally 5 points on a 5-point scale, converted to 1 point on a 1-point scale, it becomes 1.0.

[0087] Completeness: Covered the three core elements of "rear-end collision, lane 3 of K100, and two cars", but omitted "no one got out of the car", "following vehicles were affected", and "both vehicles had their hazard lights on" → originally 3 points out of 5, converted to 0.6 out of 1 point.

[0088] Redundancy: The text is split into 4 clauses (separated by periods), of which "timestamp" and "guardrail description" are irrelevant clauses → originally 5 points, 2 points, converted to 1 point, 0.4.

[0089] Accuracy of event identification: Accurate identification of event type → 1 point.

[0090] A2. Dynamic video sample verification.

[0091] Video content: Cardboard boxes were scattered on the left lane of an urban expressway; no vehicle collision occurred.

[0092] Model output: "The image is from a city expressway surveillance camera. There is a cardboard box scattered in the left lane. Vehicles are driving normally in the distance. The time is 10:00 AM." Semanticity metrics evaluation: Authenticity: The type of spilled material (cardboard box) and location (left lane) are accurate and do not contain any incorrect information → Originally 5 points on a 5-point scale, converted to 1 point on a 1-point scale, it becomes 1.0.

[0093] Completeness: Covering the two core elements "scattered objects (cardboard boxes) and left lane" (no other core elements) → Originally 5 points on a 5-point scale, converted to 1.0 on a 1-point scale.

[0094] Redundancy: Irrelevant clauses are "monitoring source, vehicle driving status, time" (3 items) → Originally 3 points out of 5, converted to 0.6 out of 1.

[0095] Accuracy of event identification: Accurate identification of event type → 1 point.

[0096] Example 3 This invention also provides a two-stage road traffic anomaly event recognition system based on a large visual model, the system comprising: The data acquisition and processing module collects monitoring images and video data from highways and urban expressways, and establishes a static image semantic dataset and a dynamic video traffic semantic dataset covering multiple types of typical abnormal events. The model training module trains a large visual model using a static image semantic dataset to obtain a first large visual model. Based on a traffic anomaly event semantic dataset, it constructs similar preference data pairs and uses these pairs to perform direct preference optimization training on the first large visual model to obtain a second large visual model. The module then extracts key frames of abnormal videos from the video data in the dynamic video traffic semantic dataset at preset intervals. Based on these key frames and the dynamic video traffic semantic dataset, it fine-tunes the parameters of the second large visual model to obtain a road traffic anomaly event recognition model. The abnormal event identification module acquires monitoring images or video sequences collected in real time by the data acquisition and processing module, and uses the road traffic abnormal event identification model to identify abnormal events.

[0097] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the described module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0098] Furthermore, the present invention provides an electronic device including a central processing unit (CPU) that can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) or loaded from a storage unit into random access memory (RAM). The RAM may also store various programs and data required for device operation. The CPU, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.

[0099] Multiple components in the device are connected to the I / O interface, including: input units such as keyboards and mice; output units such as various types of displays and speakers; storage units such as disks and optical discs; and communication units such as network interface cards (NICs), modems, and wireless transceivers. The communication unit allows the device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0100] The processing unit executes the various methods and processes described above, such as methods S1 to S5. For example, in some embodiments, methods S1 to S5 may be implemented as computer software programs tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on the device via ROM and / or a communication unit. When the computer program is loaded into RAM and executed by the CPU, one or more steps of methods S1 to S5 described above may be performed. Alternatively, in other embodiments, the CPU may be configured to execute methods S1 to S5 by any other suitable means (e.g., by means of firmware).

[0101] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0102] The program code used to implement the methods of the present invention can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code can be executed entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0103] In the context of this invention, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0104] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A two-stage road traffic anomaly event identification method based on a large visual model, characterized in that, The method includes: Collect monitoring images and video data of highways and urban expressways, and establish a static image semantic dataset and a dynamic video traffic semantic dataset covering multiple types of typical abnormal events; A large visual model is trained using the aforementioned static image semantic dataset to obtain the first large visual model. Based on the aforementioned traffic anomaly event semantic dataset, construct similar preference data pairs, and use the similar preference data pairs to perform direct preference optimization training of the first visual large model to obtain the second visual large model. The video data in the dynamic video traffic semantic dataset is extracted at preset intervals to extract abnormal video key frames. Based on the abnormal video key frames and the dynamic video traffic semantic dataset, the parameters of the second vision model are fine-tuned to obtain the road traffic abnormal event recognition model. Real-time acquisition of monitoring images or video sequences, and use of the road traffic anomaly event recognition model to identify anomalies.

2. The two-stage road traffic anomaly event identification method based on a large visual model according to claim 1, characterized in that, Both the static image semantic dataset and the dynamic video traffic semantic dataset include event types, involved objects, event locations, and event impact information based on manually annotated surveillance images and videos; the event types include multi-vehicle accidents, abnormal parking, two-wheeled vehicles running in, and spilled materials.

3. The two-stage road traffic anomaly event identification method based on a large visual model according to claim 2, characterized in that, The annotation method is as follows: using a large visual model to perform preliminary annotation on the collected monitoring images and video data based on judgment rules, and then manually correcting the preliminary annotation results to complete the labeling; For the aforementioned multi-vehicle accidents, the corresponding judgment rules are: two or more vehicles show collision marks; the vehicle positions are abnormal. It is evident that people got out of their vehicles to negotiate or that traffic police arrived at the scene to handle the situation. Other vehicles took emergency evasive maneuvers or came to a stop due to the accident; The corresponding judgment rules for the aforementioned abnormal parking are as follows: the vehicle is completely stationary in a non-parking area; there are no traffic lights or signs that allow parking; the vehicle has not turned on its hazard lights or has them on but still does not meet the temporary parking rules; the driver is not in the vehicle or the vehicle has an obvious malfunction; vehicles behind are forced to detour or slow down. The corresponding criteria for determining the spilled material are: there is a foreign object of sufficient size in the lane; there is visible vehicle braking or detouring behavior; the foreign object may have come from the cargo box of a truck or accident residue. The corresponding judgment rule for the aforementioned two-wheeled vehicle intrusion is: a motorcycle / electric vehicle / bicycle appears in a prohibited area; Dangerous behaviors such as driving against traffic and crossing lanes are present; There is no dedicated lane for two-wheeled vehicles or obvious prohibition signs; The vehicle had no license plate or the driver was not wearing a helmet.

4. The two-stage road traffic anomaly event identification method based on a large visual model according to claim 1, characterized in that, The method for constructing the aforementioned similar preference data pairs is as follows: Based on the static image semantic dataset and the dynamic video traffic semantic dataset, a first-choice answer and a second-choice answer are generated for each abnormal traffic event. The first-choice answer and the second-choice answer are a pair of data with the same preference. Furthermore, compared to the first choice, the secondary answer introduces redundant and ambiguous information in its event category, location, and element description, including lengthy and repetitive irrelevant traffic anomaly information.

5. The two-stage road traffic anomaly event identification method based on a large visual model according to claim 1, characterized in that, The method also includes a two-stage evaluation of the road traffic anomaly event identification model; The first stage of evaluation involves: based on the road traffic anomaly event identification model, identifying static events from static images and calculating structural and semantic indicators. The structural indicators include BERTS score and accuracy. The semantic indicators include redundancy in the static event description, completeness in the description of static scene elements, and accuracy in identifying static event types. The second stage of evaluation involves calculating the authenticity and completeness of the dynamic event descriptions obtained from the road traffic anomaly event identification model based on the dynamic video. The authenticity is calculated based on the degree of consistency between the dynamic event descriptions and the dynamic video, and the completeness is calculated based on the coverage of the dynamic scene elements by the dynamic event descriptions.

6. A two-stage road traffic anomaly event recognition system based on a large visual model, characterized in that, The system includes: The data acquisition and processing module collects monitoring images and video data from highways and urban expressways, and establishes a static image semantic dataset and a dynamic video traffic semantic dataset covering multiple types of typical abnormal events. The model training module trains a large visual model using the static image semantic dataset to obtain a first large visual model. It then constructs similar preference data pairs based on the traffic anomaly event semantic dataset and uses these pairs for direct preference optimization training of the first large visual model to obtain a second large visual model. Finally, it extracts keyframes of abnormal videos from the video data in the dynamic video traffic semantic dataset at preset intervals and fine-tunes the parameters of the second large visual model based on these keyframes and the dynamic video traffic semantic dataset to obtain a road traffic anomaly event recognition model. An abnormal event identification module acquires monitoring images or video sequences collected in real time by the data acquisition and processing module, and uses the road traffic abnormal event identification model to identify abnormal events.

7. A two-stage road traffic anomaly event recognition system based on a large visual model according to claim 6, characterized in that, Both the static image semantic dataset and the dynamic video traffic semantic dataset include event types, involved objects, event locations, and event impact information based on manually annotated surveillance images and videos; the event types include multi-vehicle accidents, abnormal parking, two-wheeled vehicles running in, and spilled materials.

8. A two-stage road traffic anomaly event recognition system based on a large visual model according to claim 7, characterized in that, The annotation method is as follows: using a large visual model to perform preliminary annotation on the collected monitoring images and video data based on judgment rules, and then manually correcting the preliminary annotation results to complete the labeling; For the aforementioned multi-vehicle accidents, the corresponding judgment rules are: two or more vehicles show collision marks; the vehicle positions are abnormal. It is evident that people got out of their vehicles to negotiate or that traffic police arrived at the scene to handle the situation. Other vehicles took emergency evasive maneuvers or came to a stop due to the accident; The corresponding judgment rules for the aforementioned abnormal parking are as follows: the vehicle is completely stationary in a non-parking area; there are no traffic lights or signs that allow parking; the vehicle has not turned on its hazard lights or has them on but still does not meet the temporary parking rules; the driver is not in the vehicle or the vehicle has an obvious malfunction; vehicles behind are forced to detour or slow down. The corresponding criteria for determining the spilled material are: there is a foreign object of sufficient size in the lane; there is visible vehicle braking or detouring behavior; the foreign object may have come from the cargo box of a truck or accident residue. The corresponding judgment rule for the aforementioned two-wheeled vehicle intrusion is: a motorcycle / electric vehicle / bicycle appears in a prohibited area; Dangerous behaviors such as driving against traffic and crossing lanes are present; There is no dedicated lane for two-wheeled vehicles or obvious prohibition signs; The vehicle had no license plate or the driver was not wearing a helmet.

9. A two-stage road traffic anomaly event recognition system based on a large visual model according to claim 6, characterized in that, The method for constructing the aforementioned similar preference data pairs is as follows: Based on the static image semantic dataset and the dynamic video traffic semantic dataset, a first-choice answer and a second-choice answer are generated for each abnormal traffic event. The first-choice answer and the second-choice answer are a pair of data with the same preference. Furthermore, compared to the first choice, the secondary answer introduces redundant and ambiguous information in its event category, location, and element description, including lengthy and repetitive irrelevant traffic anomaly information.

10. A two-stage road traffic anomaly event recognition system based on a large visual model according to claim 6, characterized in that, The method also includes a two-stage evaluation of the road traffic anomaly event identification model; The first stage of evaluation involves: based on the road traffic anomaly event identification model, identifying static events from static images and calculating structural and semantic indicators. The structural indicators include BERTS score and accuracy. The semantic indicators include redundancy in the static event description, completeness in the description of static scene elements, and accuracy in identifying static event types. The second stage of evaluation involves calculating the authenticity and completeness of the dynamic event descriptions obtained from the road traffic anomaly event identification model based on the dynamic video. The authenticity is calculated based on the degree of consistency between the dynamic event descriptions and the dynamic video, and the completeness is calculated based on the coverage of the dynamic scene elements by the dynamic event descriptions.

Images