Natural driving accident scene key element extraction method based on visual large model

Abstract

The invention relates to the technical field of computer vision and automatic driving, in particular to a natural driving accident scene key element extraction method based on a visual large model. Comprising the steps of data collection and preprocessing, accident data cue word design, ontology-based accident scene description and data annotation, pre-training model fine tuning and optimization, key element identification and post-processing and the like, the efficiency of natural driving accident analysis is remarkably improved, and the time and cost of manual examination are reduced. Through an ontology-based accident scene description method, dynamic feature extraction based on an optical flow method and more accurate cue word design, the recognition accuracy of the model on key elements is improved. Through a data enhancement technology and reasonable loss function design, the robustness and generalization ability of the model in different scenes are improved. The method is not only suitable for analysis of natural driving accidents, but also can be expanded to other scenes, and a high-efficiency scene understanding method is provided for management of intelligent traffic, smart cities and safe cities.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and autonomous driving technology, and in particular to a method for automatically identifying and extracting key elements from natural driving accident videos based on a large visual model. Background Art

[0002] Smart cars are the strategic commanding heights of automotive technology. With the increasing development of 5G, the Internet, neural networks, and large model technologies, smart cars have gradually turned from theory into reality. However, in the process of smart car development, safety has always been the core reason restricting its promotion and application. Smart cars need to deal with various complex road environments and emergencies during driving. How to ensure their reliability and safety in various scenarios is the key to research in the field of smart cars. For this reason, it is particularly important to conduct comprehensive testing and evaluation of smart cars.

[0003] Accident data is one of the main sources of key test scenario data for smart cars. By analyzing real accident data, potential problems of smart cars in specific scenarios can be discovered, and then the algorithm and system design can be optimized. Traditional accident analysis methods mainly rely on manual review, which is not only time-consuming and labor-intensive, inefficient, and lacks standardization, but is also easily affected by personal subjective factors, making it difficult to accurately extract core elements. Therefore, it is of great practical significance to develop a technology that can automatically and quickly extract key information from accident videos. Visual big models are emerging deep learning algorithms with long video understanding and analysis capabilities. Although existing open source visual big models (such as Qwen-VL, CLIP, etc.) perform well in image recognition and natural language processing tasks, they still have the following limitations in the task of extracting key elements from natural driving accident scenes:

[0004] (1) Insufficient domain specificity and diversity. Pre-trained models are usually pre-trained on general image datasets. The datasets lack consideration of the diversity and complexity of accident scenes and lack targeted training for specific fields (such as traffic accidents). As a result, they are unable to effectively identify and understand the complex elements in accident scenes, resulting in reduced performance when processing accident videos.

[0005] (2) Weak dynamic information processing capabilities. The key elements in accident scenes often have dynamic changes, and the existing models are insufficient in processing time series information, making it difficult to capture the dynamic changes in accident scenes. Summary of the invention

[0006] In order to solve the above technical problems, the present invention provides a method for extracting key elements of natural driving accident scenes based on a large visual model, comprising the following steps:

[0007] Step 1: Data collection and preprocessing

[0008] 11) Multi-source accident data collection: Collect natural driving accident video data from different sources to ensure the diversity and representativeness of data sources, covering different types of accident scenes and environmental conditions; typical conditions of the accident scenes include but are not limited to rear-end collisions, pedestrians crossing the road, and vehicles changing lanes, and environmental conditions include but are not limited to sunny days, rainy days, nights, and snowy days;

[0009] 12) Video format conversion: Convert the collected video files into a unified format to ensure that the encoding format of the video files is consistent for subsequent processing;

[0010] 13) Video resolution adjustment: adjust the video resolution to a size suitable for model input, keep the aspect ratio of the video unchanged, and avoid image distortion caused by resolution adjustment;

[0011] 14) Video denoising: Use video processing filters to remove noise from the video and improve video quality; adjust the video frame rate to an appropriate level to ensure smooth video and moderate data volume.

[0012] Step 2: Design of accident data prompt words

[0013] 21) Prompt word library design: Build a prompt word library containing several accident scenarios. Each prompt word is designed to guide the model to focus on specific key elements, covering the accident scenarios described in step 1; the design of prompt words includes element analysis, accident subject analysis, accident cause analysis, and avoidance measures analysis;

[0014] 22) Design of dynamic prompt word generation rules: Dynamically generate prompt words according to video content;

[0015] 23) Preliminary experiments on prompt words: Verify the effectiveness of prompt words through experiments to ensure that prompt words can significantly improve the recognition accuracy and robustness of the model; collect user feedback, continuously optimize the prompt word library, and improve the effectiveness of prompt words.

[0016] Step 3: Ontology-based accident scene description and data annotation

[0017] 31) Ontology model construction: define a hierarchical ontology model, including various entities and their attributes in the accident scene, the entities include vehicles, pedestrians, obstacles, road signs, and several types of the environment, and the attributes include several types of speed, position, direction, color, size, and type; at the same time, the ontology model includes the relationship between entities, including one or more of the relative position of vehicles and pedestrians, the distance between vehicles and obstacles, and the relationship between pedestrians and road signs;

[0018] 32) Optical flow method to enhance dynamic features of video data: Based on feature annotation, the dense optical flow Farneback method is used to gradually match and align the video area, calculate the pixel-level motion vectors between adjacent frames, and form a dense optical flow field, which can describe the instantaneous motion of all points in the scene; Due to the different sizes of different traffic participants, there are performance differences in the optical flow matching process, so a multi-resolution strategy is applied to increase the scale of the smallest motion unit of the object in the video to the largest motion unit scale, and extract optical flow information at different scales, so that the global motion trend can be obtained while maintaining local details; Based on the VAE algorithm, feature encoding of optical flow data is performed to obtain compact and compressed dynamic data features;

[0019] 33) Definition and actual development of annotation tool functions: Develop a user-friendly annotation tool to support annotators to annotate accident videos according to the ontology model. The annotation tool should have the following functions:

[0020] Entity recognition and labeling: allows labelers to mark vehicles, pedestrians or obstacles in the video;

[0021] Attribute annotation: allows annotators to fill in the attribute information of the entity;

[0022] Real-time optical flow visualization: To help annotators understand the dynamic characteristics of videos more intuitively, the real-time optical flow visualization function is integrated into the annotation tool to help annotators accurately mark entities and their attributes, so that they can evaluate whether the annotations are reasonable in the dynamic context.

[0023] Integrate prompt words into the annotation tool to guide the annotators in annotation;

[0024] 34) Recruitment of labelers and data labeling: Organize a professional labeling team to carry out detailed labeling of the collected accident videos, ensure the accuracy and consistency of the labeled data, and improve data quality through review and proofreading.

[0025] Step 4: Fine-tune and optimize the pre-trained model

[0026] 41) Pre-training model selection: Select a suitable pre-training model to ensure that the pre-training model has strong feature extraction and generalization capabilities;

[0027] 42) Fine-tuning strategy design: Fine-tune the pre-trained model on the accident scenario dataset, adjust hyperparameters including learning rate and batch size, and find the best model configuration; use early stopping to avoid overfitting and ensure the model has the best performance on the validation set;

[0028] 43) Loss function design and fine-tuning execution: In order to make the model have multi-task understanding ability, a multi-task loss function is designed to balance the weights of multiple tasks, and the fine-tuning process is executed based on the above process;

[0029] Step 5: Key element identification and post-processing

[0030] 51) Model reasoning: The preprocessed video frames are fed into the fine-tuned model, and the model outputs the key elements in each frame;

[0031] 52) Time series fusion: Considering the dynamic time series characteristics of the test scene, at the result output level, time series analysis technology is used to combine the model output information between multiple frames to improve recognition accuracy;

[0032] 53) Result integration: Based on the key elements of the model output, integrate them into an overall description of the accident scene and output the accident report, including the time, location, objects involved and their behavior information.

[0033] 54) Model optimization: continuously optimize the model through feedback mechanism to improve the performance and robustness of the model; regularly collect new accident data, retrain and fine-tune the model to ensure that the model always maintains the best state.

[0034] Furthermore, in step 21), the design of the prompt word includes the following steps:

[0035] ① First, perform factor analysis to identify the key elements in the accident scene, including vehicles, pedestrians, obstacles, road environment, weather and lighting; provide detailed attribute descriptions for each key element, which can express the most core parameters and semantic features of the element, including position, speed, direction, color and / or action:

[0036] The key elements of the vehicle description include: location, speed, direction, color and / or model;

[0037] The key elements of the pedestrian description include: position, action, direction and / or clothing color; the action includes walking, running and / or standing;

[0038] The key elements of the obstacle description include: type, location and / or size; the type includes fixed obstacles and / or mobile obstacles;

[0039] The key elements of the road environment include: traffic signs, lane lines and / or road conditions;

[0040] The key elements of weather and light description include: weather conditions and light conditions, weather conditions include daytime, nighttime and / or dusk, and light conditions include sunny days, rainy days and / or snowy days;

[0041] ② Secondly, conduct an accident subject analysis to identify the main participants in the accident, including vehicles, pedestrians and / or animals; provide a detailed description of each accident subject, including the subject's behavior, location and / or status:

[0042] The accident subject description of the rear-end collision accident includes: the front vehicle and the rear vehicle;

[0043] The subject description of the pedestrian crossing accident includes: pedestrians, vehicles;

[0044] The subject description of the accident involving the vehicle changing lanes includes: the lane-changing vehicle and the lane-changed vehicle;

[0045] ③ Next, conduct an accident cause analysis to analyze the causes that led to the accident, including driver error, the sudden appearance of pedestrians, and / or poor road conditions; provide a detailed description of each accident cause, including specific behavioral and environmental factors:

[0046] The description of the causes of the rear-end collision includes: the rear vehicle failed to brake in time;

[0047] The descriptions of the causes of pedestrian crossing accidents include: pedestrians suddenly stepping out from the roadside;

[0048] The description of the cause of the accident involving the vehicle changing lanes includes: the vehicle changing lanes failed to notice the vehicle behind;

[0049] ④ Then conduct an avoidance measure analysis and propose possible avoidance measures, including slowing down in advance, keeping a safe distance and / or using warning signs; provide a detailed description of each avoidance measure, including specific actions and suggestions:

[0050] The description of the measures to avoid rear-end collision accidents includes: the front vehicle slows down in advance and / or the rear vehicle keeps a safe distance;

[0051] Descriptions of avoidance measures for pedestrians crossing the road include: pedestrians using crosswalks and / or vehicles slowing down;

[0052] The description of the avoidance measures for vehicles changing lanes includes: the vehicle changing lanes turns on the turn signal in advance and changes lanes only after confirming that it is safe;

[0053] ⑤Finally, construct a prompt word library. Based on the above analysis, design a prompt word template to ensure that the prompt words can guide the model to focus on specific key elements.

[0054] Furthermore, in step 22), a dynamic prompt generation function is designed to dynamically generate prompt words according to the video content; based on the content of the video frame, a rule model is used to implement the method, including:

[0055] Write a rule model to generate corresponding prompt words according to the video content; the rule model is built based on text similarity. When the similarity is less than the set threshold, it means that an element related to the scene appears in the image, and the prompt word related to the element in the prompt word library will be automatically given for the user to choose.

[0056] Furthermore, the ontology model definition step in step 31) includes:

[0057] ①Define entity:

[0058] Vehicles: including cars, trucks and / or motorcycles;

[0059] Pedestrians: include people walking and / or cycling;

[0060] Obstacles: including fixed obstacles and / or mobile obstacles. Fixed obstacles include road signs and trees, and mobile obstacles include animals.

[0061] Road signs: including traffic lights and / or traffic signs;

[0062] Environment: including weather and / or lighting conditions; weather includes sunny, rainy and / or snowy days, and lighting conditions include daytime, nighttime and / or dusk;

[0063] ②Define attributes:

[0064] Vehicle attributes include: location, speed, direction, color and / or model;

[0065] Pedestrian attributes include: location, motion, direction, and / or clothing color; motion includes walking, running, and / or standing;

[0066] Obstacle attributes include: type, location and / or size;

[0067] Road sign attributes include: type, location and / or status;

[0068] Environmental attributes include: weather conditions and / or lighting conditions;

[0069] ③Define the relationship:

[0070] The relative position of the vehicle and the pedestrian: including the front of the vehicle, the rear of the vehicle, the left side of the vehicle and / or the right side of the vehicle;

[0071] The distance between the vehicle and the obstacle: including the distance n meters;

[0072] Pedestrians in relation to road markings: including pedestrians approaching the crosswalk and / or pedestrians moving away from the crosswalk.

[0073] Further, the process of enhancing the dynamic features of video data using the optical flow method in step 32) is as follows:

[0074] ① Construct the Farneback method architecture, realize optical flow estimation based on the frequency domain, approximate the inter-frame motion through polynomial expansion, and use Gaussian window weighted averaging to improve the estimation accuracy; this method can calculate the pixel-level motion vector between adjacent frames to form a dense optical flow field, so it can describe the instantaneous motion of all points in the scene, providing dynamic information for subsequent analysis;

[0075] ② Considering the different sizes of different traffic participants, there are performance differences in the optical flow matching process. In order to meet this challenge, a multi-resolution strategy is introduced. First, starting from the smallest motion unit scale of the object in the video, it increases to the largest motion unit scale in sequence, extracting optical flow information at different scales, while maintaining local details. Get the global motion trend, so as to ensure that targets of different sizes can be accurately represented;

[0076] ③ In order to obtain compact and compressed dynamic data features, a feature encoding scheme based on the VAE algorithm is adopted. First, a neural network structure consisting of an encoder and a decoder is designed. The encoder is responsible for mapping the original optical flow data to a low-dimensional latent space; the decoder reconstructs the original input from this latent space; during the training process, the latent variables output by the encoder are required to obey the standard normal distribution, which not only helps to prevent overfitting, but also makes the generated features smoother and more natural, thereby improving the performance of the model; in addition to the traditional reconstruction error, the KL divergence is introduced as a regular term and added to the loss function. The KL divergence measures the difference between the distribution generated by the encoder and the prior distribution. By minimizing this difference, it can ensure that the distribution of the latent variables is close to the expected prior distribution, and at the same time enable the model to generate reasonable samples in the latent space; once the VAE model is trained, its encoder part can be used to extract features from new optical flow data; the compact and compressed dynamic features obtained are directly used as the input of the large video model, or further combined with other static features to provide more comprehensive data support for accident scenes.

[0077] Preferably, the multi-resolution strategy described in ② of step 32) includes the following three parts:

[0078] High-resolution layer: Focuses on capturing the fine movements of small objects and the subtle changes around these objects;

[0079] Medium-resolution layer: balances local and global information, captures medium-sized targets, and reveals complex motion patterns;

[0080] Low-resolution layer: used to capture overall motion trends over a large area, including large objects or large-scale movement in a scene.

[0081] Furthermore, the development and design process of the annotation tool in step 33) is as follows:

[0082] ①Front-end development: Use HTML, CSS, and JavaScript to develop user interfaces, and use modern front-end frameworks such as React or Vue.js to improve development efficiency and user experience;

[0083] ②Backend development: Use the backend framework Node.js, Flask or Django to handle data storage and transmission, and use the database MySQL, PostgreSQL or MongoDB to store annotation data.

[0084] Furthermore, the fine-tuning strategy step in step 42) includes:

[0085] ① Divide the dataset into training set, validation set and test set, with the ratio set to 80%, 10%, and 10%; use data enhancement technology to increase the diversity of training data;

[0086] ②The initial learning rate is set to a small value and gradually adjusted to find the optimal value;

[0087] ③Choose an appropriate batch size based on computing resources;

[0088] ④Use Adam optimizer with default parameters;

[0089] ⑤Choose a suitable loss function;

[0090] ⑥ Use early stopping to avoid overfitting, that is, stop training when the performance on the validation set no longer improves.

[0091] Furthermore, step 43) loss function design and fine-tuning execution steps include:

[0092] First, define multiple tasks, including vehicle detection, pedestrian detection, obstacle detection, and / or environment classification. According to the designed task type, construct a multi-task loss function to balance the weights of each task.

[0093] The loss functions include:

[0094] Vehicle detection loss vehicle :The goal of the vehicle detection task is to identify vehicles in video frames; the cross entropy loss is used to measure the difference between the probability distribution predicted by the model and the true label. The calculation method is shown in the following formula. If the model's prediction is exactly the same as the true label, the loss is 0; otherwise, the loss will increase:

[0095]

[0096] Among them, y iis the true label, p i is the probability predicted by the model;

[0097] Pedestrian detection loss pedestrian :The goal of the pedestrian detection task is to identify pedestrians in video frames, using cross entropy loss to measure the difference between the probability distribution predicted by the model and the true label; the calculation method is consistent with that of vehicles, replacing the label with pedestrians:

[0098]

[0099] Obstacle detection loss obstacle :The goal of the obstacle detection task is to identify obstacles in video frames, using cross entropy loss to measure the difference between the probability distribution predicted by the model and the true label:

[0100]

[0101] Environmental classification detection loss environment :The goal of the environment classification task is to identify the environmental conditions in the video frames, using the cross entropy loss to measure the difference between the probability distribution predicted by the model and the true label:

[0102]

[0103] Total loss function Loss total : The total loss is obtained by weighted summation of the loss functions of multiple tasks, thereby optimizing multiple tasks at the same time and balancing the importance of each task;

[0104] Loss total =w1×Loss vehicle +w2×Loss pedestrian +w3×Loss obstacle +w4×Loss environment

[0105] Among them, w1, w2, w3 and w4 are the weights of each task;

[0106] To meet different scene understanding requirements, the weights of different tasks are different; the design weights are as follows: vehicle detection weight is 0.4, pedestrian detection weight is 0.3, obstacle detection weight is 0.2, and environment classification weight is 0.1; based on the above process, the open source model is fine-tuned.

[0107] Furthermore, step 52) the time series fusion step includes:

[0108] First, the description is parsed to convert the model output into structured data for subsequent processing. Second, the changes and correlations of key elements are extracted by comparing the descriptions of adjacent frames. The dynamic changes of key elements are tracked through time series analysis to improve the coherence of recognition. The extracted correlations and dynamic trajectories are integrated into an overall description of the accident scene.

[0109] 53) The results integration steps include:

[0110] According to the key elements of the model output, an overall description of the accident scene is integrated; an accident report is output, including the time, location, objects involved and their behavior information of the accident. The overall description of the accident scene is as follows:

[0111] Time: record the time when the accident occurred;

[0112] Location: record the location where the incident occurred;

[0113] Objects involved: Record the vehicles, pedestrians and / or obstacles involved in the accident;

[0114] Behavior: describes the behavior of each object, including the speed of vehicles and / or the movements of pedestrians;

[0115] 54) The model optimization steps include:

[0116] ① Collect user feedback on model output, identify potential problems and improvements, regularly evaluate the model using new data sets, and monitor model performance;

[0117] ② Regularly collect new accident data, expand the data set, and use the new data set to retrain the model and update the model parameters; fine-tune the model for specific scenarios to improve the performance of the model on specific tasks. Beneficial effects of the present invention:

[0118] The present invention provides a method for extracting key elements of natural driving accident scenes based on a large visual model, which can significantly improve the efficiency of natural driving accident analysis and reduce the time and cost of manual review. At the same time, the accuracy of the model's recognition of key elements is improved through an ontology-based accident scene description method and a more accurate prompt word design. Through data enhancement technology and a reasonable loss function design, the robustness and generalization ability of the model in different scenarios are improved. The method of the present invention is not only suitable for the analysis of natural driving accidents, but can also be extended to other scenarios, such as traffic monitoring, smart city management, etc. Provide an efficient scene understanding method for intelligent transportation, smart cities, and safe city management. BRIEF DESCRIPTION OF THE DRAWINGS

[0119] Figure 1 It is a schematic diagram of the overall architecture of the method of the present invention;

[0120] Figure 2 A schematic diagram of the process flow of accident data processing of the method of the present invention;

[0121] Figure 3 Schematic diagram of the video understanding process based on prompt words in the method of the present invention. DETAILED DESCRIPTION

[0122] See also Figure 1-3 The present invention provides a method for extracting key elements of natural driving accident scenes based on a large visual model, comprising the following steps:

[0123] Step 1: Data collection and preprocessing

[0124] 11) Multi-source accident data collection: Collect natural driving accident video data from multiple sources (such as vehicle-mounted cameras, surveillance cameras, drone footage, relevant videos on video sites (YouTube, Bilibili, etc.), and public accident datasets). Extensively investigate multiple data sources to ensure the diversity and representativeness of data sources, covering different types of accident scenarios and environmental conditions. Specifically, accident scenarios should include typical conditions such as rear-end collisions, pedestrians crossing the road, and vehicles changing lanes. Environmental conditions should include a variety of complex conditions such as sunny days, rainy days, nights, and snowy days.

[0125] 12) Video format conversion: Based on video tools or software such as FFmpeg, PR, moviepy, etc., the collected video files are converted into a unified format. In this embodiment, the MP4 format is used. Ensure that the encoding format of the video files is consistent for subsequent processing. Specifically, select "Format" as "H.264" and "Preset" as "Match Source-High Bitrate" in the software export settings. Save the file in MP4 format.

[0126] 13) Video resolution adjustment: Based on the above tools, adjust the video resolution to a size suitable for model input (such as 224x224). Keep the aspect ratio of the video unchanged to avoid image distortion caused by resolution adjustment.

[0127] 14) Video denoising: Use the video processing filter in FFmpeg to remove noise from the video and improve the video quality. Adjust the video frame rate to a suitable level (10 frames per second in this embodiment) to ensure smooth video and moderate data volume.

[0128] Step 2: Design of accident data prompt words

[0129] 21) Prompt word library design: Build a prompt word library containing various accident scenarios. Each prompt word should be designed to guide the model to focus on specific key elements, covering the accident scenarios in step 1, such as rear-end collisions, pedestrians crossing the road, vehicles changing lanes, etc. Prompt word design starts from the nature and purpose of accident video research, and mainly includes the following parts: factor analysis, accident subject analysis, accident cause analysis, and avoidance measures analysis. The specific introduction is as follows:

[0130] ① First, perform factor analysis to identify the key elements in the accident scene, such as vehicles, pedestrians, obstacles, road environment, weather and lighting, etc. Provide detailed attribute descriptions for each key element, which can express the most core parameters and semantic features of the element, including position, speed, direction, color, action, etc.

[0131] Here is an example:

[0132] The key elements of a vehicle are described as follows: location, speed, direction, color, model, etc.

[0133] The key elements of pedestrian description include: location, action (walking, running, standing, etc.), direction, clothing color, etc.

[0134] The key elements of obstacles include: type (fixed obstacles, mobile obstacles, etc.), location, size, etc.

[0135] The key elements of the road environment include: traffic signs, lane lines, road conditions, etc.

[0136] The key elements of weather and lighting description include: weather conditions (sunny, rainy, snowy, etc.), lighting conditions (day, night, dusk, etc.).

[0137] ② Secondly, conduct an accident subject analysis to identify the main participants in the accident, such as vehicles, pedestrians, animals, etc. Provide a detailed description of each accident subject, including the subject's behavior, location, status, etc.

[0138] Here is an example:

[0139] The description of the accident subjects in a rear-end collision includes: the front vehicle and the rear vehicle.

[0140] The subject description of pedestrian crossing accident includes: pedestrians and vehicles.

[0141] The description of the main body of a vehicle lane-changing accident includes: the lane-changing vehicle and the vehicle whose lane is changed.

[0142] ③Next, conduct an accident cause analysis to analyze the causes that led to the accident, such as driver error, sudden appearance of pedestrians, poor road conditions, etc. Provide a detailed description of each accident cause, including specific behavioral and environmental factors.

[0143] Here is an example:

[0144] Descriptions of the causes of rear-end collisions include: the rear vehicle failed to brake in time.

[0145] Descriptions of the causes of pedestrian crossing accidents include: The pedestrian suddenly walks out from the roadside.

[0146] Descriptions of the causes of accidents involving vehicle lane changes include: the vehicle changing lanes did not notice the vehicle behind it.

[0147] ④ Then conduct avoidance measures analysis and propose possible avoidance measures, such as slowing down in advance, keeping a safe distance, using warning signs, etc. Provide a detailed description of each avoidance measure, including specific actions and suggestions.

[0148] Here is an example:

[0149] The description of measures to avoid rear-end collisions includes: the front vehicle slows down in advance and the rear vehicle maintains a safe distance.

[0150] Descriptions of avoidance measures for pedestrians crossing the road include: pedestrians use crosswalks and vehicles slow down.

[0151] The description of avoidance measures for vehicles changing lanes includes: the vehicle changing lanes should turn on the turn signal in advance and change lanes only after confirming that it is safe.

[0152] ⑤Finally, construct a prompt word library. Based on the above analysis, design a prompt word template to ensure that the prompt words can guide the model to focus on specific key elements.

[0153] The overall prompt word template design example is as follows:

[0154] 1-What are the elements of this accident?

[0155] 2-Who are the main participants in the accident and what are their attributes?

[0156] 3-Analyze the causes of the accident?

[0157] 4- Based on the cause, propose possible avoidance measures.

[0158] 22) Design of dynamic prompt word generation rules: In order to facilitate practical engineering applications, a dynamic prompt generation function is designed to dynamically generate prompt words based on the video content. For example, when a pedestrian is detected, prompt words related to the pedestrian's behavior are generated. This is implemented using a rule model based on the content of the video frame.

[0159] Write a rule model to generate corresponding prompt words according to the video content. The rule model is built based on text similarity. When the similarity is less than the set threshold, it means that an element related to the scene appears in the image, and the prompt word related to the element in the prompt word library is automatically given for the user to choose.

[0160] 23) Preliminary experiments on prompt words: Verify the effectiveness of prompt words through experiments to ensure that prompt words can significantly improve the recognition accuracy and robustness of the model. Collect user feedback, continuously optimize the prompt word library, and improve the effectiveness of prompt words.

[0161] Based on the existing open-source video model and language model, we manually evaluate the effect of the prompt words and manually check the output of the model to ensure that the prompt words can guide the model to focus on the correct key elements, that is, to identify the main elements of the accident. We manually check whether the model correctly identifies key elements such as vehicles, pedestrians, and obstacles. We also check whether the model correctly understands the contextual information of the accident scene. We understand the actual effect of the prompt words and suggestions for improvement through user feedback, and continuously optimize the prompt word library based on user feedback to improve the effectiveness of the prompt words.

[0162] Step 3: Ontology-based accident scene description and data annotation

[0163] 31) Ontology model construction: Based on ontology, define a hierarchical ontology model, which includes various entities in the accident scene (such as vehicles, pedestrians, road signs, etc.) and their attributes (such as speed, position, direction, etc.). At the same time, the ontology model should include the relationship between entities, such as the relative position of vehicles and pedestrians, the distance between vehicles and obstacles, etc.

[0164] The details are as follows:

[0165] ①Define entity

[0166] Vehicles: This includes cars, trucks, motorcycles, etc.

[0167] Pedestrians: including people walking, cycling, etc.

[0168] Obstacles: include fixed obstacles (such as road signs, trees) and mobile obstacles (such as animals).

[0169] Road signs: including traffic lights, traffic signs, etc.

[0170] Environment: including weather (sunny, rainy, snowy, etc.) and lighting conditions (day, night, dusk, etc.).

[0171] ②Define attributes

[0172] Vehicle: location, speed, direction, color, model, etc.

[0173] Pedestrians: location, action (walking, running, standing, etc.), direction, clothing color, etc.

[0174] Obstacles: type, location, size, etc.

[0175] Road signs: type, location, status, etc.

[0176] Environment: weather conditions, lighting conditions, etc.

[0177] ③Define the relationship

[0178] The relative positions of vehicles and pedestrians: such as in front of the vehicle, behind the vehicle, on the left side of the vehicle, on the right side of the vehicle, etc.

[0179] The distance between the vehicle and the obstacle: such as 10 meters, 20 meters, etc.

[0180] The relationship between pedestrians and road signs: such as pedestrians approaching crosswalks, pedestrians away from crosswalks, etc.

[0181] 32) Optical flow method to enhance dynamic features of video data: Based on feature annotation, the dense optical flow Farneback method is used to gradually match and align the video area, and the pixel-level motion vectors between adjacent frames are calculated to form a dense optical flow field, which can describe the instantaneous motion of all points in the scene; a multi-resolution strategy is applied to extract optical flow information at different scales from the smallest motion unit scale of objects in the video to the largest motion unit scale, and the global motion trend is obtained while maintaining local details; the optical flow data is feature encoded based on the VAE algorithm to obtain compact and compressed dynamic data features.

[0182] The details are as follows:

[0183] ①Build the Farneback method architecture, realize optical flow estimation based on the frequency domain, approximate the inter-frame motion through polynomial expansion, and use Gaussian window weighted averaging to improve the estimation accuracy. This method can calculate the pixel-level motion vector between adjacent frames to form a dense optical flow field. Therefore, it can describe the instantaneous motion of all points in the scene, providing dynamic information for subsequent analysis.

[0184] ② Considering the different sizes of different traffic participants, the performance of the optical flow matching process varies. Therefore, a multi-resolution strategy is adopted. First, starting from the minimum motion unit scale of the object in the video, and then increasing to the maximum motion unit scale, the optical flow information is extracted at different scales; thus, the global motion trend can be obtained while maintaining local details, thereby ensuring that objects of different sizes can be accurately represented. It includes the following three parts:

[0185] High-resolution layer: focuses on capturing the fine movements of small objects (such as motorcycles and bicycles) and the subtle changes around these objects;

[0186] Medium-resolution layer: balances local and global information, is suitable for medium-sized targets (such as cars and pedestrians), and can reveal more complex motion patterns;

[0187] Low-resolution layer: used to capture the overall motion trend over a large area, suitable for large objects (such as trucks, buses) or large-scale movement in the scene;

[0188] Each layer generates a corresponding optical flow field, and then the information of these optical flow fields is fused together to form a comprehensive, multi-scale dynamic feature representation.

[0189] ③ In order to obtain compact and compressed dynamic data features, a feature encoding scheme based on the VAE algorithm is adopted. First, a neural network structure consisting of an encoder and a decoder is designed. The encoder is responsible for mapping the original optical flow data to a low-dimensional latent space; while the decoder attempts to reconstruct the original input from this latent space.

[0190] During the training process, the latent variables output by the encoder are required to follow a standard normal distribution, which not only helps prevent overfitting, but also makes the generated features smoother and more natural, thereby improving the performance of the model.

[0191] In addition to the traditional reconstruction error, the KL divergence is introduced as a regularization term into the loss function. The KL divergence measures the difference between the distribution generated by the encoder and the prior distribution. By minimizing this difference, it can ensure that the distribution of the latent variables is close to the expected prior distribution, while enabling the model to generate reasonable samples in the latent space. Once the VAE model is trained, its encoder part can be used to extract features from new optical flow data. The resulting compact and compressed dynamic features can be used directly as input to a large video model, or further combined with other static features to provide more comprehensive data support for accident scenes.

[0192] 33) Definition and actual development of annotation tool functions: Develop a user-friendly annotation tool to support annotators to annotate accident videos according to the ontology model. The annotation tool should have the following functions:

[0193] Entity recognition and labeling: allows labelers to mark vehicles, pedestrians, obstacles, etc. in the video.

[0194] Attribute labeling: allows the labeler to fill in the attribute information of the entity, such as speed, position, etc.

[0195] Real-time optical flow visualization: In order to allow annotators to more intuitively understand the dynamic characteristics of the video, the real-time optical flow visualization function is integrated into the annotation tool. It can help annotators accurately mark entities and their attributes, allowing them to evaluate whether the annotation is reasonable in the dynamic context.

[0196] Integrate prompt words into the annotation tool to guide the annotators in annotation.

[0197] The specific development and design process is as follows:

[0198] ①Front-end development: Use HTML, CSS, and JavaScript to develop user interfaces. Use modern front-end frameworks such as React or Vue.js to improve development efficiency and user experience.

[0199] ②Backend development: Use backend frameworks such as Node.js, Flask, or Django to handle data storage and transmission. Use databases such as MySQL, PostgreSQL, or MongoDB to store annotation data.

[0200] 34) Recruitment of labelers and data labeling: Organize a professional labeling team to label the collected accident videos in detail. Ensure the accuracy and consistency of the labeled data, and improve the data quality through multiple reviews and proofreading.

[0201] ① First, recruit annotation personnel who have relevant field knowledge, such as traffic engineering, computer vision, etc. Train the annotation personnel to ensure that they are familiar with the ontology model and the use of annotation tools.

[0202] ② Assign annotation tasks and assign video files to annotators to ensure that each person has an appropriate amount of tasks. Provide detailed annotation standards and examples to ensure the consistency of annotation data. Ensure the accuracy and consistency of annotation data through multiple reviews and proofreading. Establish a feedback mechanism to collect the opinions and suggestions of annotators and continuously optimize the annotation process.

[0203] ③ Randomly check the annotation situation and regularly randomly check the annotation data to ensure data quality. For key data, use multi-person annotation to improve data reliability. And use automated tools for preliminary verification, such as detecting the overlap of annotation boxes and the rationality of attribute information.

[0204] Step 4: Fine-tune and optimize the pre-trained model

[0205] 41) Pre-trained model selection: Choose a suitable pre-trained model (such as Qwen-VL), which should perform well on large-scale image recognition tasks. The pre-trained model should have strong feature extraction capabilities and be able to extract key features in complex accident scenes. At the same time, the pre-trained model should have good generalization capabilities and be able to perform well on new data.

[0206] Qwen-VL is a multimodal pre-trained model that is good at handling joint tasks of images and texts, and has performed well on large-scale image recognition tasks. In addition to Qwen-VL, other pre-trained models can also be considered, such as CLIP, ViT (Vision Transformer), ResNet, etc.

[0207] 42) Fine-tuning strategy design: Fine-tune the pre-trained model on the accident scenario dataset. Adjust hyperparameters such as learning rate and batch size to find the best model configuration. Use early stopping to avoid overfitting and ensure the model performs optimally on the validation set.

[0208] ① Divide the data set into training set, validation set and test set, with the ratio set to 80%, 10%, and 10%. Use data enhancement techniques (such as rotation, scaling, flipping, etc.) to increase the diversity of training data, thereby greatly expanding the original data and improving the generalization ability of the model.

[0209] ② The initial learning rate is set to a smaller value (1e-5 in the present invention), and is gradually adjusted to find the optimal value.

[0210] ③ Select an appropriate batch size based on computing resources (32 is used in this invention)

[0211] ④ The present invention uses the Adam optimizer, and the default parameters usually perform well.

[0212] ⑤ Select a suitable loss function. The present invention selects cross-entropy loss (Cross-Entropy Loss).

[0213] ⑥ Use early stopping to avoid overfitting, that is, stop training when the performance on the validation set no longer improves.

[0214] 43) Loss function design and fine-tuning execution: In order to make the model have multi-task understanding capabilities, a multi-task loss function is designed to balance the weights of multiple tasks.

[0215] First, define multiple tasks, such as vehicle detection, pedestrian detection, obstacle detection, environment classification, etc. According to the designed task type, construct a multi-task loss function to balance the weights of each task.

[0216] An example loss function is as follows:

[0217] Vehicle detection loss vehicle :The goal of the vehicle detection task is to identify vehicles in video frames. The cross entropy loss is used to measure the difference between the probability distribution predicted by the model and the true label. The calculation method is shown in the following formula. If the model's prediction is exactly the same as the true label, the loss is 0; otherwise, the loss will increase.

[0218]

[0219] Among them, y i is the true label, p i is the probability predicted by the model.

[0220] Pedestrian detection loss pedestrian : The goal of the pedestrian detection task is to identify pedestrians in video frames. The cross entropy loss is also used to measure the difference between the probability distribution predicted by the model and the true label. The calculation method is the same as that of the vehicle, except that the label is changed to pedestrians.

[0221]

[0222] Obstacle detection loss obstacle : The goal of the obstacle detection task is to identify obstacles in video frames. Similarly, the cross entropy loss can be used to measure the difference between the probability distribution predicted by the model and the true label.

[0223]

[0224] Environmental classification detection loss environment : The goal of the environment classification task is to identify environmental conditions (such as weather, lighting, etc.) in video frames. Similarly, the cross entropy loss can be used to measure the difference between the probability distribution predicted by the model and the true label.

[0225]

[0226] Total loss function Loss total : The total loss is obtained by weighted summation of the loss functions of multiple tasks, thereby optimizing multiple tasks at the same time and balancing the importance of each task:

[0227] Loss total =w1×Loss vehicle +w2×Loss pedestrian +w3×Loss obstacle +w4×Loss environment

[0228] Among them, w1, w2, w3 and w4 are the weights of each task.

[0229] In order to meet different scene understanding requirements, the weights of different tasks are also different. In this embodiment, the weights are designed as follows: vehicle detection weight is 0.4, pedestrian detection: weight is 0.3, obstacle detection: weight is 0.2, and environment classification: weight is 0.1. Based on the above process, the open source model is fine-tuned.

[0230] Step 5: Key element identification and post-processing

[0231] 54) Model reasoning: The preprocessed video frames are fed into the fine-tuned model, and the model outputs the key elements in each frame, such as vehicle position, pedestrian position, obstacles, etc.

[0232] The model outputs a description of each frame, for example:

[0233] Frame 1: The vehicle is in the center of the picture, and pedestrians are walking from left to right.

[0234] Frame 2: The vehicle continues to move forward and the pedestrians are still walking.

[0235] Frame 3: The vehicle slows down and the pedestrian approaches the crosswalk.

[0236] 55) Time series fusion: At the result level, time series analysis technology is used to combine the information between multiple frames to further improve the recognition accuracy.

[0237] First, the description is parsed to parse the description output by the model into structured data for subsequent processing. Secondly, by comparing the descriptions of adjacent frames, the changes and correlations of key elements are extracted. Through time series analysis, the dynamic changes of key elements are tracked to improve the coherence of recognition. The extracted correlations and dynamic trajectories are integrated into an overall description of the accident scene.

[0238] 56) Result integration: Based on the key elements of the model output, integrate them into an overall description of the accident scene. Output the accident report, including important information such as the time, location, objects involved and their behaviors.

[0239] The specific accident scene is described as follows:

[0240] Time: Record the time when the accident occurred.

[0241] Location: Record the location where the incident occurred.

[0242] Objects involved: record the vehicles, pedestrians, obstacles, etc. involved in the accident.

[0243] Behavior: describes the behavior of each object, such as the speed of a vehicle, the movements of a pedestrian, etc.

[0244] 54) Model optimization: Continuously optimize the model through feedback mechanism to improve the performance and robustness of the model. Regularly collect new accident data, retrain and fine-tune the model to ensure that the model always maintains the best state.

[0245] ① Collect user feedback on model output, identify potential problems and improvement points, regularly evaluate the model using new data sets, and monitor model performance.

[0246] ② Regularly collect new accident data, expand the data set, and use the new data set to retrain the model and update the model parameters. Fine-tune the model for specific scenarios to improve the performance of the model on specific tasks.

[0247] The preferred embodiments of the present invention are described in detail above in conjunction with the accompanying drawings. However, the protection scope of the present invention is not limited to the specific details in the above embodiments. Within the technical concept of the present invention, any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention. These simple variations all belong to the protection scope of the present invention.

[0248] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, the present invention will not further describe various possible combinations.

Claims

1. A method for extracting key elements of natural driving accident scenes based on a large visual model, characterized by: The following steps are involved: Step 1: Data collection and preprocessing 11) Multi-source accident data collection: Collect natural driving accident video data from different sources to ensure the diversity and representativeness of data sources, covering different types of accident scenes and environmental conditions; typical conditions of the accident scenes include but are not limited to rear-end collisions, pedestrians crossing the road, and vehicles changing lanes, and environmental conditions include but are not limited to sunny days, rainy days, nights, and snowy days; 12) Video format conversion: Convert the collected video files into a unified format to ensure that the encoding format of the video files is consistent for subsequent processing; 13) Video resolution adjustment: adjust the video resolution to a size suitable for model input, keep the aspect ratio of the video unchanged, and avoid image distortion caused by resolution adjustment; 14) Video denoising: Use video processing filters to remove noise from the video and improve video quality; adjust the video frame rate to an appropriate level to ensure smooth video and moderate data volume; Step 2: Design of accident data prompt words 21) Prompt word library design: Build a prompt word library containing several accident scenarios. Each prompt word is designed to guide the model to focus on specific key elements, covering the accident scenarios described in step 1; the design of prompt words includes element analysis, accident subject analysis, accident cause analysis, and avoidance measures analysis; 22) Design of dynamic prompt word generation rules: Dynamically generate prompt words according to video content; 23) Preliminary experiments on prompt words: Verify the effectiveness of prompt words through experiments to ensure that prompt words can significantly improve the recognition accuracy and robustness of the model; Collect user feedback, continuously optimize the prompt word library, and improve the effectiveness of prompt words; Step 3: Ontology-based accident scene description and data annotation 31) Ontology model construction: define a hierarchical ontology model, including various entities and their attributes in the accident scene, the entities include vehicles, pedestrians, obstacles, road signs, and several types of the environment, and the attributes include several types of speed, position, direction, color, size, and type; at the same time, the ontology model includes the relationship between entities, including one or more of the relative position of vehicles and pedestrians, the distance between vehicles and obstacles, and the relationship between pedestrians and road signs; 32) Optical flow method to enhance dynamic features of video data: Based on feature annotation, the dense optical flow Farneback method is used to gradually match and register the video area, calculate the pixel-level motion vectors between adjacent frames, form a dense optical flow field, and describe the instantaneous motion of all points in the scene; apply a multi-resolution strategy, increase the scale of the smallest motion unit of the object in the video to the largest motion unit scale, extract optical flow information at different scales, and obtain the global motion trend while maintaining local details; based on the VAE algorithm, feature encode the optical flow data to obtain compact and compressed dynamic data features; 33) Definition and actual development of annotation tool functions: Develop a user-friendly annotation tool to support annotators to annotate accident videos according to the ontology model. The annotation tool should have the following functions: Entity recognition and labeling: allows labelers to mark vehicles, pedestrians or obstacles in the video; Attribute annotation: allows annotators to fill in the attribute information of the entity; Real-time optical flow visualization: Integrate real-time optical flow visualization into the annotation tool to more intuitively understand the dynamic characteristics in the video and help annotators accurately mark entities and their attributes; Integrate prompt words into the annotation tool to guide the annotators in annotation; 34) Recruitment of annotation personnel and data annotation: Organize a professional annotation team to annotate the collected accident videos in detail, ensure the accuracy and consistency of the annotated data, and improve the data quality through review and proofreading; Step 4: Fine-tune and optimize the pre-trained model 41) Pre-training model selection: Select a suitable pre-training model to ensure that the pre-training model has strong feature extraction and generalization capabilities; 42) Fine-tuning strategy design: Fine-tune the pre-trained model on the accident scenario dataset, adjust hyperparameters including learning rate and batch size, and find the best model configuration; Use early stopping to avoid overfitting and ensure the model performs best on the validation set; 43) Loss function design and fine-tuning execution: In order to make the model have multi-task understanding ability, a multi-task loss function is designed to balance the weights of multiple tasks, and the fine-tuning process is executed based on the above process; Step 5: Key element identification and post-processing 51) Model reasoning: The preprocessed video frames are fed into the fine-tuned model, and the model outputs the key elements in each frame; 52) Time series fusion: Considering the dynamic time series characteristics of the test scene, at the result output level, time series analysis technology is used to combine the model output information between multiple frames to improve recognition accuracy; 53) Result integration: Based on the key elements of the model output, integrate them into an overall description of the accident scene and output the accident report, including the time, location, objects involved and their behavior information; 54) Model optimization: continuously optimize the model through feedback mechanism to improve the performance and robustness of the model; regularly collect new accident data, retrain and fine-tune the model to ensure that the model always maintains the best state.

2. The method for extracting key elements of natural driving accident scenes based on a visual large model according to claim 1, characterized in that: In step 21), the design of the prompt word includes the following steps: ① First, perform factor analysis to identify the key elements in the accident scene, including vehicles, pedestrians, obstacles, road environment, weather and lighting; provide detailed attribute descriptions for each key element, which can express the most core parameters and semantic features of the element, including position, speed, direction, color and / or action: The key elements of the vehicle description include: location, speed, direction, color and / or model; The key elements of the pedestrian description include: position, action, direction and / or clothing color; the action includes walking, running and / or standing; The key elements of the obstacle description include: type, location and / or size; the type includes fixed obstacles and / or mobile obstacles; The key elements of the road environment include: traffic signs, lane lines and / or road conditions; The key elements of weather and light description include: weather conditions and light conditions, weather conditions include daytime, nighttime and / or dusk, and light conditions include sunny days, rainy days and / or snowy days; ② Secondly, conduct an accident subject analysis to identify the main participants in the accident, including vehicles, pedestrians and / or animals; provide a detailed description of each accident subject, including the subject's behavior, location and / or status: The accident subject description of the rear-end collision accident includes: the front vehicle and the rear vehicle; The subject description of the pedestrian crossing accident includes: pedestrians, vehicles; The subject description of the accident of the vehicle changing lanes includes: the lane-changing vehicle and the lane-changed vehicle; ③ Next, conduct an accident cause analysis to analyze the causes that led to the accident, including driver error, the sudden appearance of pedestrians, and / or poor road conditions; provide a detailed description of each accident cause, including specific behavioral and environmental factors: The description of the causes of the rear-end collision includes: the rear vehicle failed to brake in time; The descriptions of the causes of pedestrian crossing accidents include: pedestrians suddenly stepping out from the roadside; The description of the cause of the accident involving the vehicle changing lanes includes: the vehicle changing lanes failed to notice the vehicle behind; ④ Then conduct an avoidance measure analysis and propose possible avoidance measures, including slowing down in advance, keeping a safe distance and / or using warning signs; provide a detailed description of each avoidance measure, including specific actions and suggestions: The description of the measures to avoid rear-end collision accidents includes: the front vehicle slows down in advance and / or the rear vehicle keeps a safe distance; Descriptions of avoidance measures for pedestrians crossing the road include: pedestrians using crosswalks and / or vehicles slowing down; The description of the avoidance measures for vehicles changing lanes includes: the vehicle changing lanes turns on the turn signal in advance and changes lanes only after confirming that it is safe; ⑤Finally, construct a prompt word library. Based on the above analysis, design a prompt word template to ensure that the prompt words can guide the model to focus on specific key elements.

3. The method for extracting key elements of natural driving accident scenes based on a visual large model according to claim 1 is characterized by: Step 22) Design a dynamic prompt generation function to dynamically generate prompt words based on the video content; Based on the content of the video frame, the rule model is used to implement the method, including: Write a rule model to generate corresponding prompt words based on the video content; The rule model is built based on text similarity. When the similarity is less than the set threshold, it means that elements related to the scene appear in the image. Then the prompt words related to the elements in the prompt word library will be automatically given for the user to choose.

4. The method for extracting key elements of natural driving accident scenes based on a visual large model according to claim 1 is characterized by: The ontology model definition step in step 31) includes: ①Define entity: Vehicles: including cars, trucks and / or motorcycles; Pedestrians: include people walking and / or cycling; Obstacles: including fixed obstacles and / or mobile obstacles. Fixed obstacles include road signs and trees, and mobile obstacles include animals. Road signs: including traffic lights and / or traffic signs; Environment: including weather and / or lighting conditions; weather includes sunny, rainy and / or snowy days, and lighting conditions include daytime, nighttime and / or dusk; ②Define attributes: Vehicle attributes include: location, speed, direction, color and / or model; Pedestrian attributes include: location, motion, direction, and / or clothing color; motion includes walking, running, and / or standing; Obstacle attributes include: type, location and / or size; Road sign attributes include: type, location and / or status; Environmental attributes include: weather conditions and / or lighting conditions; ③Define the relationship: The relative position of the vehicle and the pedestrian: including the front of the vehicle, the rear of the vehicle, the left side of the vehicle and / or the right side of the vehicle; The distance between the vehicle and the obstacle: including the distance n meters; Pedestrians in relation to road markings: including pedestrians approaching the crosswalk and / or pedestrians moving away from the crosswalk.

5. The method for extracting key elements of natural driving accident scenes based on a visual large model according to claim 1, characterized in that: The process of enhancing the dynamic features of video data using the optical flow method in step 32) is as follows: ① Construct the Farneback method architecture, realize optical flow estimation based on the frequency domain, approximate the inter-frame motion through polynomial expansion, and use Gaussian window weighted averaging to improve the estimation accuracy; this method can calculate the pixel-level motion vector between adjacent frames to form a dense optical flow field; ②Introducing a multi-resolution strategy, starting from the smallest motion unit scale of the object in the video, and then increasing to the largest motion unit scale, extracting optical flow information at different scales, obtaining the global motion trend while maintaining local details, and ensuring that objects of different sizes can be accurately represented; ③ Using a feature encoding scheme based on the VAE algorithm, first design a neural network structure consisting of an encoder and a decoder. The encoder is responsible for mapping the original optical flow data into a low-dimensional latent space; The decoder reconstructs the original input from this latent space. During the training process, the latent variables output by the encoder are required to obey the standard normal distribution. The KL divergence is introduced as a regular term and added to the loss function. The KL divergence measures the difference between the distribution generated by the encoder and the prior distribution. By minimizing this difference, the distribution of the latent variables is ensured to be close to the expected prior distribution, while enabling the model to generate reasonable samples in the latent space. After the VAE model is trained, its encoder part is used to extract features from new optical flow data. The resulting compact and compressed dynamic features are directly used as input to the large video model.

6. The method for extracting key elements of natural driving accident scenes based on a visual large model according to claim 5, characterized in that: The multi-resolution strategy described in step ② of step 32) includes the following three parts: High-resolution layer: Focuses on capturing the fine movements of small objects and the subtle changes around these objects; Medium-resolution layer: balances local and global information, captures medium-sized targets, and reveals complex motion patterns; Low-resolution layer: used to capture overall motion trends over a large area, including large objects or large-scale movement in a scene.

7. The method for extracting key elements of natural driving accident scenes based on a visual large model according to claim 1 is characterized by: The development and design process of the annotation tool described in step 33) is as follows: ①Front-end development: Use HTML, CSS, and JavaScript to develop user interfaces, and use modern front-end frameworks such as React or Vue.js to improve development efficiency and user experience; ②Backend development: Use the backend framework Node.js, Flask or Django to handle data storage and transmission, and use the database MySQL, PostgreSQL or MongoDB to store annotation data.

8. The method for extracting key elements of natural driving accident scenes based on a visual large model according to claim 1, characterized in that: The fine-tuning strategy step in step 42) includes: ① Divide the dataset into training set, validation set and test set, with the ratio set to 80%, 10%, and 10%; use data enhancement technology to increase the diversity of training data; ②The initial learning rate is set to a small value and gradually adjusted to find the optimal value; ③Choose an appropriate batch size based on computing resources; ④Use Adam optimizer with default parameters; ⑤Choose a suitable loss function; ⑥ Use early stopping to avoid overfitting, that is, stop training when the performance on the validation set no longer improves.

9. The method for extracting key elements of natural driving accident scenes based on a visual large model according to claim 1, characterized in that: Step 43) The loss function design and fine-tuning execution steps include: First, define multiple tasks, including vehicle detection, pedestrian detection, obstacle detection, and / or environment classification. According to the designed task type, construct a multi-task loss function to balance the weights of each task. The loss functions include: Vehicle detection loss vehicle :The goal of the vehicle detection task is to identify vehicles in video frames; the cross entropy loss is used to measure the difference between the probability distribution predicted by the model and the true label. The calculation method is shown in the following formula. If the model's prediction is exactly the same as the true label, the loss is 0; otherwise, the loss will increase: Among them, y i is the true label, p i is the probability predicted by the model; Pedestrian detection loss pedestrian :The goal of the pedestrian detection task is to identify pedestrians in video frames, using cross entropy loss to measure the difference between the probability distribution predicted by the model and the true label; the calculation method is consistent with that of vehicles, replacing the label with pedestrians: Obstacle detection loss obstacle :The goal of the obstacle detection task is to identify obstacles in video frames, using cross entropy loss to measure the difference between the probability distribution predicted by the model and the true label: Environmental classification detection loss environment :The goal of the environment classification task is to identify the environmental conditions in the video frames, using the cross entropy loss to measure the difference between the probability distribution predicted by the model and the true label: Total loss function Loos total : The total loss is obtained by weighted summation of the loss functions of multiple tasks, thereby optimizing multiple tasks at the same time and balancing the importance of each task; Loss total =w1×Loss vehicle +w2×Loss pedestrian +w3×Loss obstacle +w4×Loss environment Among them, w1, w2, w3 and w4 are the weights of each task; To meet different scene understanding requirements, the weights of different tasks are different; the design weights are as follows: vehicle detection weight is 0.4, pedestrian detection weight is 0.3, obstacle detection weight is 0.2, and environment classification weight is 0.1; based on the above process, the open source model is fine-tuned.

10. The method for extracting key elements of natural driving accident scenes based on a visual large model according to claim 1, characterized in that: Step 52) The timing fusion step includes: First, the description is parsed to convert the model output into structured data for subsequent processing. Second, the changes and correlations of key elements are extracted by comparing the descriptions of adjacent frames. The dynamic changes of key elements are tracked through time series analysis to improve the coherence of recognition. The extracted correlations and dynamic trajectories are integrated into an overall description of the accident scene. 53) The results integration steps include: According to the key elements of the model output, an overall description of the accident scene is integrated; an accident report is output, including the time, location, objects involved and their behavior information of the accident. The overall description of the accident scene is as follows: Time: record the time when the accident occurred; Location: record the location where the incident occurred; Objects involved: Record the vehicles, pedestrians and / or obstacles involved in the accident; Behavior: describes the behavior of each object, including the speed of vehicles and / or the movements of pedestrians; 54) The model optimization steps include: ① Collect user feedback on model output, identify potential problems and improvements, regularly evaluate the model using new data sets, and monitor model performance; ② Regularly collect new accident data, expand the data set, and use the new data set to retrain the model and update the model parameters; fine-tune the model for specific scenarios to improve the performance of the model on specific tasks.

Images