Traffic accident identification method based on unsupervised pre-training and multi-modal large model

Abstract

The invention discloses a traffic accident identification method based on unsupervised pre-training and a multi-modal large model, and the method comprises the following steps: S1, collecting and sorting video data in a traffic monitoring scene, selecting a video clip without a traffic accident as a data sample, extracting a video key frame, carrying out the unsupervised pre-training of a mask auto-encoder, and carrying out the unsupervised pre-training of the mask auto-encoder; reconstructing a video clip which is shielded randomly; s2, calculating a reconstruction error between the original video and the reconstructed video to evaluate the reconstruction quality, and screening out key frame video clips with poor reconstruction quality; according to the invention, through unsupervised training of the mask auto-encoder, even under the condition of limited annotation data, more traffic accident-free segments can be filtered, video segments in which traffic anomalies possibly exist can be rapidly screened out, and by constructing fine-tuning training of multi-modal traffic data on a multi-modal large model, the real-time performance of the model is improved. The method can improve the recognition accuracy and efficiency of a traffic accident in a complex scene, provides real-time alarm and detailed analysis, and takes countermeasures in time.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent traffic accident identification, and in particular to a traffic accident identification method based on unsupervised pre-training and a multi-modal large model. Background Art

[0002] With the rapid development of social economy and the accelerated advancement of urbanization, the number of urban residents and the number of motor vehicles have exploded, causing a sharp increase in road traffic volume. This phenomenon has directly increased the frequency of traffic accidents, making it a major hidden danger affecting public safety. In order to detect traffic accidents in a timely manner and take timely measures, people are currently trying to build a large-scale and comprehensive accident data set to build a model, which requires the model to obtain more sample data during the training stage;

[0003] However, the low frequency of traffic accidents and the diversity of traffic accidents currently result in limited data samples, making it difficult to generalize the general laws of traffic accidents from limited examples. The performance of the model will drop significantly, so a traffic accident recognition method based on unsupervised pre-training and multimodal large models is needed. Summary of the invention

[0004] The present invention provides a traffic accident identification method based on unsupervised pre-training and a multimodal large model, which can effectively solve the problem raised in the above background technology that the low frequency of traffic accidents and the diversity of traffic accidents currently lead to limited data samples, making it difficult to generalize the general laws of traffic accidents from limited examples, and the performance of the model will be significantly reduced.

[0005] To achieve the above object, the present invention provides the following technical solution: a traffic accident recognition method based on unsupervised pre-training and multimodal large model, comprising the following steps:

[0006] S1. Collect and organize video data in traffic monitoring scenes, select video clips without traffic accidents in different environments, locations and lighting conditions as data samples, extract video key frames, perform unsupervised pre-training of masked autoencoders, learn video feature representation, capture dynamic changes and static scene details in the video, and the model will learn the inherent structure and dynamic change rules of traffic data without abnormalities, and reconstruct randomly occluded video clips;

[0007] S2. Calculate the reconstruction error between the original video and the reconstructed video to evaluate the reconstruction quality. Videos under normal traffic conditions have high predictability, while abnormal conditions will lead to large reconstruction errors. Based on the complex traffic scenes, fast-moving objects, and abnormal traffic events in the video clips with large reconstruction errors, the video clips with large reconstruction errors are marked as potential traffic accidents.

[0008] S3. Collect traffic videos of normal traffic and abnormal accidents, perform key frame single frame annotation, use pre-trained visual language model and large language model to synthesize the analysis results of video clips and construct multi-round traffic accident perception guidance dialogue data, thereby constructing multi-modal data;

[0009] S4. Use the constructed multimodal data as input to train the preset multimodal large model, obtain a multimodal large model for the field of traffic accident identification, and enhance the model's ability to understand complex traffic scenes;

[0010] S5. Reconstruct the video clips and instruction prompts with errors higher than the preset threshold in the masked autoencoder results and input them into the trained multimodal traffic accident recognition large model to obtain corresponding detailed response results, including the recognition judgment, severity, and impact range of the specific accident.

[0011] According to the above technical solution, in S1, unsupervised pre-training of the masked autoencoder is performed, and the training goal is to minimize the reconstruction loss between the original video and the reconstructed video clip, including the imitation loss and the mask reconstruction loss, which is expressed as follows:

[0012]

[0013] Among them, L rec represents the reconstruction loss;

[0014] |M| represents the number of mask blocks;

[0015] represents the i-th input mask block;

[0016] represents the i-th reconstruction mask block;

[0017] |V| represents the number of visible blocks;

[0018] represents the representation of the i-th visible block;

[0019] represents the representation of the i-th high-level feature;

[0020] In S2, the reconstruction error is calculated to filter out key frame video clips with poor reconstruction quality. The expression is as follows:

[0021]

[0022] Where L P represents the reconstruction error;

[0023] |F| represents the number of mask positions;

[0024] x iRepresents the original value of the i-th pixel;

[0025] Represents the reconstructed value of the i-th pixel.

[0026] According to the above technical solution, in S1, the network structure of the masked autoencoder is:

[0027] The embedding layer divides the input image into multiple small blocks, randomly masks these small blocks, and embeds the unmasked parts into the vector space as the input of the encoder;

[0028] The encoding layer consists of stacked encoding blocks and self-attention mechanisms to extract and aggregate deep features of the image through multi-layer transformations;

[0029] The decoding layer, through stacked decoding blocks and self-attention mechanism, gradually restores the detailed features of the image from mask reconstruction and visible imitation reconstruction, ensuring the high quality and detail fidelity of the reconstruction task.

[0030] According to the above technical solution, in S3, the collected video is annotated and multiple rounds of traffic accident perception guidance dialogue data are constructed by semi-automatic annotation, which specifically includes:

[0031] Collect traffic video data from open source datasets or surveillance camera data, which contains various traffic conditions in multiple scenarios;

[0032] Use manual annotation and pre-trained multimodal large models to perform high-quality single-frame annotation of abnormal video frames;

[0033] Generate reliable traffic video annotation segments by expanding around annotation frames;

[0034] Generate video analysis results using the pre-trained end-to-end sequence-to-sequence visual language model Vid2Seq;

[0035] The video analysis results are summarized and analyzed through a pre-trained large language model, and the video is structured in a conversational format. Finally, the data constructed above is manually screened.

[0036] According to the above technical solution, in S4, the preset multimodal large model includes three components:

[0037] Visual encoder, which uses a pre-trained Vision Transformer-based encoder in a multimodal alignment framework to align images and video frames to the text feature space;

[0038] The video sampler randomly samples one frame of the input video sequence according to a fixed length, inputs the sampled features into the pre-trained video anomaly detection network, screens out potential abnormal frames based on the model output scores, and optimizes the model using instruction tuning methods;

[0039] The visual projection layer and the large language model (LLM) input the above-screened potential abnormal frame features into the projector. The projector consists of three layers of multi-layer perceptrons (MLPs), which are used to adjust the dimensions of the video abnormal frame features to align them with the input dimensions of the large language model. The adjusted features are input into the multimodal LLM to generate descriptions and explanations of abnormal events in the video frames.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] 1. The traffic accident recognition method based on unsupervised pre-training and multimodal large model adopted by the present invention utilizes large-scale unlabeled video data and unsupervised pre-training of masked autoencoders. Even when the labeled data is limited, it can quickly filter out more video clips without traffic accidents.

[0042] 2. The traffic accident recognition method based on unsupervised pre-training and multimodal large model adopted by the present invention minimizes the reconstruction loss between the original video and the reconstructed video through the masked autoencoder, quickly screens out video clips that may contain abnormal traffic events, saves computing resources, and improves processing speed and traffic accident recognition efficiency.

[0043] 3. The traffic accident identification method based on unsupervised pre-training and multimodal large model adopted by the present invention can automatically identify and understand complex traffic scenes by constructing multimodal traffic data to fine-tune the multimodal large model, and provide real-time alerts and detailed analysis for traffic management departments, which helps to take timely response measures and ensure road safety.

[0044] In summary, through unsupervised training of video data, it is possible to quickly filter out clips without traffic accidents. Even when the labeled data is limited, more video clips without traffic accidents can be quickly filtered out. The masked autoencoder minimizes the reconstruction loss between the original video and the reconstructed video, and quickly screens out video clips that may contain abnormal traffic events. The multimodal large model integrates multimodal information, can automatically identify and understand complex traffic scenes, improve the accuracy and efficiency of traffic accident recognition in complex scenes, and provide real-time alerts and detailed analysis to facilitate timely response measures and ensure road safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0046] In the attached picture:

[0047] Figure 1 It is a schematic diagram of the overall process of an example of the present invention;

[0048] Figure 2 A schematic diagram of the network structure of a masked autoencoder according to an example of the present invention;

[0049] Figure 3 A schematic diagram of screening and reconstructing video frames with poor quality according to an example of the present invention;

[0050] Figure 4 A schematic diagram of multimodal data construction for an example of the present invention;

[0051] Figure 5 Schematic diagram of multimodal large model network components of an example of the present invention. DETAILED DESCRIPTION

[0052] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0053] Example: Figure 1 As shown, the present invention provides a technical solution, a traffic accident recognition method based on unsupervised pre-training and multimodal large model, comprising the following steps:

[0054] S1. Collect and organize video data in traffic monitoring scenes, select video clips without traffic accidents in different environments, locations and lighting conditions as data samples, extract video key frames, perform unsupervised pre-training of masked autoencoders, learn video feature representation, capture dynamic changes and static scene details in the video, and the model will learn the inherent structure and dynamic change rules of traffic data without abnormalities, and reconstruct randomly occluded video clips;

[0055] S2. Calculate the reconstruction error between the original video and the reconstructed video to evaluate the reconstruction quality. Videos under normal traffic conditions have high predictability, while abnormal conditions will lead to large reconstruction errors. Based on the complex traffic scenes, fast-moving objects, and abnormal traffic events in the video clips with large reconstruction errors, the video clips with large reconstruction errors are marked as potential traffic accidents.

[0056] S3. Collect traffic videos of normal traffic and abnormal accidents, perform key frame single frame annotation, use pre-trained visual language model and large language model to synthesize the analysis results of video clips and construct multi-round traffic accident perception guidance dialogue data, thereby constructing multi-modal data;

[0057] S4. Use the constructed multimodal data as input to train the preset multimodal large model, obtain a multimodal large model for the field of traffic accident identification, and enhance the model's ability to understand complex traffic scenes;

[0058] S5. Reconstruct the video clips and instruction prompts with errors higher than the preset threshold in the masked autoencoder results and input them into the trained multimodal traffic accident recognition large model to obtain corresponding detailed response results, including the recognition judgment, severity, and impact range of the specific accident.

[0059] In S1, video key frames are extracted from video segments by combining time interval method, inter-frame difference change method, histogram difference detection method and open source multimedia processing framework. This method can fully capture the dynamic changes of video segments and more accurately reflect the main content of the video, providing a basis for subsequent video analysis and traffic accident identification.

[0060] like Figure 2 As shown, in S1, unsupervised pre-training of the masked autoencoder is performed, and the model structure includes:

[0061] The embedding layer divides the input image into multiple small blocks (patches), of which 75% of the image patches are randomly masked, and 25% of the visible patches are converted into 512-dimensional vectors as the input of the encoder through linear projection. The network can learn more robust and generalized features during the training process, improving the understanding and reconstruction capabilities of image content;

[0062] The encoding layer consists of stacked encoding blocks that use a multi-head self-attention mechanism and multiple layers of transformation to extract and aggregate the deep feature information of the image and convert the input visible patch into an intermediate representation EV.

[0063] The decoding layer consists of stacked decoding blocks and a self-attention mechanism. The goal is to minimize the loss between the reconstructed image Om generated by the decoding block and the input image, and to minimize the loss between the high-level features Mv of the target image extracted by the pre-trained model DINO or CLIP and the EV output by the encoding layer.

[0064] In S1, unsupervised pre-training of the masked autoencoder is performed. The training goal is to minimize the reconstruction loss between the original video and the reconstructed video clip, including the imitation loss and the mask reconstruction loss, which is expressed as follows:

[0065]

[0066] Among them, L rec represents the reconstruction loss;

[0067] |M| represents the number of mask blocks;

[0068] represents the i-th input mask block;

[0069] represents the i-th reconstruction mask block;

[0070] |V| represents the number of visible blocks;

[0071] represents the representation of the i-th visible block;

[0072] represents the representation of the i-th high-level feature.

[0073] The above decoding layer uses RGB pixels, DINO features, and CLIP features to capture the details and semantic information of the image at multiple levels, and restores the multi-level features of the image from two aspects: mask reconstruction and high-level feature imitation, ensuring the high quality and detail fidelity of the reconstruction task;

[0074] like Figure 3 As shown, in S3, the reconstruction error is calculated to filter out key frame video clips with poor reconstruction quality. The expression is as follows:

[0075]

[0076] Where L P represents the reconstruction error;

[0077] |F| represents the number of mask positions;

[0078] x i Represents the original value of the i-th pixel;

[0079] Represents the reconstructed value of the i-th pixel.

[0080] like Figure 4 As shown, in S3, the collected videos are annotated and multi-round traffic accident perception guidance dialogue data is constructed by semi-automatic annotation, including:

[0081] Collect traffic video data from open source datasets or surveillance camera data. The data contains various traffic conditions in multiple scenarios, including normal traffic and various traffic accidents.

[0082] Use manual annotation and pre-trained multimodal large models to perform high-quality single-frame annotation of abnormal video frames and obtain the annotated time points;

[0083] Generate reliable traffic video annotation segments by expanding around annotation frames;

[0084] Use the pre-trained end-to-end sequence-to-sequence visual language model Vid2Seq to generate video analysis results, including time description and traffic situation description;

[0085] The video analysis results are summarized and analyzed through a pre-trained large language model, and the video is structured in a conversational format. Finally, the constructed data is manually screened to ensure the accuracy and quality of the data.

[0086] like Figure 5 As shown in S4, the preset multimodal large model includes three components:

[0087] The visual encoder uses the specific encoder ViT-L / 14 model based on the CLIP framework in the pre-trained multimodal alignment framework to align images and video frames to the text feature space, where L represents the number of layers, 14 means that the image is divided into 14×14 image blocks (patch), and the visual encoder input is a video frame sequence S with a shape of N×H×W×C, where N represents the frame sequence length, H and W represent the height and width of the video sequence, and C represents the color channel. The output is a feature vector for each frame, including class label features (cls tokens) and image block features (patch tokens);

[0088] The video sampler randomly samples one frame of the input video sequence according to a fixed length, and inputs the sampled features into the pre-trained video anomaly detection network UR-DMU. According to the model output score, when the anomaly score exceeds the threshold of 0.8, it is considered that the video frame has potential anomalies, and the model is optimized using the instruction tuning method;

[0089] The visual projection layer and the large language model (LLM) input the above-screened potential abnormal frame features into the projector. The projector consists of three layers of multi-layer perceptrons (MLPs) to adjust the dimensions of the video abnormal frame features to align them with the input dimensions of the large language model. The adjusted features are input into the multimodal LLM to generate descriptions and explanations of abnormal events in the video frames.

[0090] By inputting the video clips and instruction prompts in the masked autoencoder results with reconstruction errors higher than the preset threshold into the trained multimodal traffic accident recognition large model, the corresponding detailed response results of the video clips are obtained, including the identification judgment, severity and impact scope details of the specific accident.

[0091] Finally, it should be noted that the above description is only a preferred example of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A traffic accident recognition method based on unsupervised pre-training and multimodal large model, characterized by: The steps include: S1. Collect and organize video data in traffic monitoring scenes, select video clips without traffic accidents in different environments, locations and lighting conditions as data samples, extract video key frames, perform unsupervised pre-training of masked autoencoders, learn video feature representation, capture dynamic changes and static scene details in the video, and the model will learn the inherent structure and dynamic change rules of traffic data without abnormalities, and reconstruct randomly occluded video clips; S2. Calculate the reconstruction error between the original video and the reconstructed video to evaluate the reconstruction quality. Videos under normal traffic conditions have high predictability, while abnormal conditions will lead to large reconstruction errors. Based on the complex traffic scenes, fast-moving objects, and abnormal traffic events in the video clips with large reconstruction errors, the video clips with large reconstruction errors are marked as potential traffic accidents. S3. Collect traffic videos of normal traffic and abnormal accidents, perform key frame single frame annotation, use pre-trained visual language model and large language model to synthesize the analysis results of video clips and construct multi-round traffic accident perception guidance dialogue data, thereby constructing multi-modal data; S4. Use the constructed multimodal data as input to train the preset multimodal large model, obtain a multimodal large model for the field of traffic accident identification, and enhance the model's ability to understand complex traffic scenes; S5. Reconstruct the video clips and instruction prompts with errors higher than the preset threshold in the masked autoencoder results and input them into the trained multimodal traffic accident recognition large model to obtain corresponding detailed response results, including the recognition judgment, severity, and impact range of the specific accident.

2. The traffic accident identification method based on unsupervised pre-training and multimodal large model according to claim 1 is characterized in that: In S1, unsupervised pre-training of the masked autoencoder is performed, and the training goal is to minimize the reconstruction loss between the original video and the reconstructed video clip, including the imitation loss and the mask reconstruction loss, which is expressed as follows: Among them, L rec represents the reconstruction loss; |M| represents the number of mask blocks; represents the i-th input mask block; represents the i-th reconstruction mask block; |V| represents the number of visible blocks; represents the representation of the i-th visible block; represents the representation of the i-th high-level feature; In S2, the reconstruction error is calculated to filter out key frame video clips with poor reconstruction quality. The expression is as follows: Where L P represents the reconstruction error; |F| represents the number of mask positions; x i Represents the original value of the i-th pixel; Represents the reconstructed value of the i-th pixel.

3. The traffic accident identification method based on unsupervised pre-training and multimodal large model according to claim 1 is characterized in that: In S1, the network structure of the masked autoencoder is: The embedding layer divides the input image into multiple small blocks, randomly masks these small blocks, and embeds the unmasked parts into the vector space as the input of the encoder; The encoding layer consists of stacked encoding blocks and self-attention mechanisms to extract and aggregate deep features of the image through multi-layer transformations; The decoding layer, through stacked decoding blocks and self-attention mechanism, gradually restores the detailed features of the image from mask reconstruction and visible imitation reconstruction, ensuring the high quality and detail fidelity of the reconstruction task.

4. The traffic accident identification method based on unsupervised pre-training and multimodal large model according to claim 1 is characterized in that: In S3, the collected videos are annotated and multi-round traffic accident perception guidance dialogue data is constructed by semi-automatic annotation, which specifically includes: Collect traffic video data from open source datasets or surveillance camera data, which contains various traffic conditions in multiple scenarios; Use manual annotation and pre-trained multimodal large models to perform high-quality single-frame annotation of abnormal video frames; Generate reliable traffic video annotation segments by expanding around annotation frames; Generate video analysis results using a pre-trained end-to-end sequence-to-sequence visual language model Vid2Seq; The video analysis results are summarized and analyzed through a pre-trained large language model, and the video is structured in a conversational format. Finally, the data constructed above is manually screened.

5. The traffic accident identification method based on unsupervised pre-training and multimodal large model according to claim 1 is characterized in that: In S4, the preset multimodal large model includes three components: Visual encoder, which uses a pre-trained Vision Transformer-based encoder in a multimodal alignment framework to align images and video frames to the text feature space; The video sampler randomly samples one frame of the input video sequence according to a fixed length, inputs the sampled features into the pre-trained video anomaly detection network, screens out potential abnormal frames based on the model output scores, and optimizes the model using instruction tuning methods; The visual projection layer and the large language model (LLM) input the above-screened potential abnormal frame features into the projector. The projector consists of three layers of multi-layer perceptrons (MLPs), which are used to adjust the dimensions of the video abnormal frame features to align them with the input dimensions of the large language model. The adjusted features are input into the multimodal LLM to generate descriptions and explanations of abnormal events in the video frames.

Images