Vehicle collision detection and description system and method based on monitoring video

By using the CLIP model and Transformer time series modeler to extract and analyze features from surveillance videos, and combining this with a language generation model to automatically generate natural language descriptions, the problem of existing systems being unable to finely distinguish accident types and automate descriptions has been solved. This enables accurate judgment of accident severity and fully automated analysis, improving the efficiency and readability of traffic accident assessment.

CN121236660APending Publication Date: 2025-12-30AUTOMOBILE RES INST OF TSINGHUA UNIV IN SUZHOU XIANGCHENG
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202511334866.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-09-18
Publication Date
2025-12-30

AI Technical Summary

Technical Problem

Existing video traffic analysis systems are unable to effectively determine the severity of accidents, lack fine-grained differentiation of accident types, and cannot automatically generate natural language descriptions that can be understood and referenced. As a result, emergency warnings and post-event analysis rely heavily on manual interpretation, which is inefficient and prone to errors.

Method used

A vehicle collision detection and description method based on surveillance video is adopted. The CLIP model and Transformer temporal modeler are used to extract features and capture temporal relationships in video frames. Combined with a language generation model, natural language descriptions are automatically generated to achieve accurate judgment and multi-level differentiation of accident severity, and to trigger early warnings immediately before or when an accident occurs.

Benefits of technology

It automates the entire process from video input to generating structured analysis reports, improving the efficiency and objectivity of traffic accident assessment. It provides multi-level severity classification and detailed natural language descriptions, supports pre-event warnings and real-time in-event responses, reduces human intervention, and improves the accuracy and readability of the analysis.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN121236660A_ABST
    Figure CN121236660A_ABST
Patent Text Reader

Abstract

The invention discloses a vehicle collision detection and description system and method based on a monitoring video, and belongs to the technical field of artificial intelligence and intelligent traffic. The system supports automatic analysis of monitoring videos, and realizes accident detection, severity grading, natural language description generation and real-time early warning. The system extracts video frame semantic features through a CLIP model, performs time sequence modeling and dynamic feature weighting by using Transform, and introduces an attention mechanism to focus key accident information; a classifier is adopted to output multi-level accident judgment results of'no accident ', 'slight', 'medium ', 'serious' and the like; and integrating target detection and a language generation model, and automatically generating a labeled video with a detection frame and a structured accident description text. The method effectively solves the problems that a traditional method depends on manual work, output is rough, and a semantic report cannot be automatically generated, has high automation, high accuracy and high interpretability, and is suitable for intelligent traffic monitoring and accident emergency processing scenes.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence, intelligent transportation, computer vision and natural language processing, and in particular to a vehicle collision detection and description system and method based on surveillance video. Background Technology

[0002] In the fields of artificial intelligence, intelligent transportation, computer vision, and natural language processing, video monitoring and analysis of traffic accidents is a crucial component of intelligent transportation systems. By processing and identifying driving records or surveillance videos, the scene of an accident can be quickly reconstructed and damage assessed, which is of great significance for road safety management and liability determination.

[0003] Currently, commonly used video traffic analysis technologies are mostly based on two-dimensional images or video frames, using methods such as object detection and motion analysis to identify and reconstruct vehicle behavior during accidents. However, these technologies have many drawbacks:

[0004] On the one hand, relying on manually set rules or a single model for judgment often fails to effectively determine the severity of an accident and makes it difficult to generate natural language descriptions that can be understood and referenced.

[0005] On the other hand, with the development of deep learning, although many systems have introduced image recognition and classification models, most of them can only make coarse-grained judgments on whether an accident has occurred, lacking fine distinctions on the type and severity of accidents, and most of them only output "label" results, which cannot fully describe the accident process.

[0006] Meanwhile, traditional methods cannot automatically extract semantic information from videos, and emergency warnings and post-event analysis still heavily rely on manual interpretation and secondary summarization, which is inefficient and prone to errors.

[0007] In practical applications, when a traffic accident occurs, a monitoring system that can promptly identify the type and severity of the accident and clearly indicate the accident status in natural language will greatly improve response and dispatch capabilities. However, most existing video analytics systems, due to limitations in their analysis granularity and output format, struggle to meet the requirements of automation, readability, and effectiveness in real-world scenarios. Summary of the Invention

[0008] The purpose of this invention is to provide a vehicle collision detection and description system and method based on surveillance video. This system enables accurate detection of vehicle collision accidents, effective judgment of severity, automatic generation of natural language descriptions, and automated processing throughout the entire process. This meets the requirements of intelligent transportation systems in practical applications for efficiency, accuracy, automation, and readability in emergency warning and post-accident analysis. Specifically, it addresses the following issues:

[0009] 1. Improve the ability to judge the severity of accidents: Change the current situation where traditional methods rely on manually set rules or single models to judge the severity of accidents, which cannot effectively judge the severity of accidents, and realize accurate reasoning about the severity of accidents.

[0010] 2. Achieve refined accident type differentiation: This addresses the problem that most existing deep learning-based traffic video analysis systems can only make coarse-grained judgments on "whether an accident has occurred," lacking refined differentiation of accident types.

[0011] 3. Automatically generate natural language descriptions: Overcome the shortcomings of traditional methods that cannot automatically extract semantic information from videos, which lead to high dependence on manual interpretation and secondary summarization for emergency warnings and post-event analysis, resulting in low efficiency and large errors. Automatically generate natural language descriptions that can be understood and cited, and fully describe the accident process.

[0012] 4. Meets the needs of real-world scenarios for automation and readability: Addressing the issue that the analysis granularity and output format of existing video analysis systems cannot meet the requirements of automation, readability, and effectiveness in real-world application scenarios, we provide highly reliable and interpretable technical support to achieve full automation from perception to understanding to output.

[0013] To achieve the above objectives, the present invention provides a vehicle collision detection and description method based on surveillance video, comprising:

[0014] Step S1: Decode the input raw video from the monitoring perspective, and uniformly extract frames according to the set frame rate parameters to output an image frame sequence.

[0015] Step S2: Adjust each frame of the image frame sequence in step S1 to a preset size and perform normalization processing. Then, encode each frame of the image into a first semantic feature vector through the image encoder of the CLIP model, and stack N first semantic feature vectors into a first temporal tensor.

[0016] Step S3: The first temporal tensor is dynamically weighted by the Transformer temporal modeler, and the weighted feature sequence is averaged by pooling to generate an aggregated feature vector that can capture the temporal relationship between frames and output the entire video segment. The aggregated feature vector is input into the classifier to obtain the prediction result of the accident severity level category. The prediction result is one of several predefined accident severity level categories.

[0017] Step S4: Extract the overall visual features of the first temporal tensor from Step S2 to obtain the second semantic feature vector at the video level; perform similarity matching between the second semantic feature vector and a preset text tag set to select one or more semantic tags that are closest to the video scene, thus obtaining a semantic tag set; construct prompt words based on the semantic tag set, input the prompt words into the language generation model, and generate natural language description text; wherein, the preset text tags include multi-category tags for describing weather, road conditions, and accident types;

[0018] Step S5: When the prediction result output in step S3 represents the level of collision, the early warning mechanism is triggered, an early warning signal is output, and the target detection model is called to overlay detection boxes and labels on the original video frames from the monitoring perspective, generating and saving an labeled video file.

[0019] Furthermore, in step S2, if the number of video frames is less than N frames, zero vectors are used to fill the gaps to form the first temporal tensor.

[0020] Furthermore, step S3, "inputting the aggregated feature vector into the classifier," also includes the following:

[0021] Step 31: Input the aggregated feature vector V into an attention network consisting of at least one fully connected layer and a non-linear activation function. The attention network learns to generate a weight vector W with the same dimension as V, and uses element-wise multiplication to weight the aggregated feature vector V to obtain a weighted aggregated feature vector V-weighted that focuses on key features, where V-weighted = V*W.

[0022] The method for "obtaining the predicted accident severity category" in step S3 specifically includes:

[0023] Step 32: The weighted aggregated feature vector V_weighted output from step 31 is compressed and projected into a low-dimensional space through the fully connected layer in the classifier. The number of dimensions of this low-dimensional space is equal to the number of categories in the classification task.

[0024] Step 33: The values ​​output in step 32 are converted into occurrence probability values ​​corresponding to each classification task by the Softmax function in the classifier; wherein, the classification task is one or more of a number of predefined accident severity level categories;

[0025] Step 34: Identify the category corresponding to the highest probability value output in step 33, and use it as the prediction result.

[0026] Furthermore, the method for "obtaining the prediction result of the accident severity level category" in step S3 specifically includes:

[0027] Step 35: During the training of the collision severity classification module, based on the difference between the prediction results output in step S34 and the pre-labeled true accident level labels in the dataset, the module is trained using a focus loss function and a parameter optimizer unit composed of a specific optimizer and a specific scheduler. At least one of the following strategies is employed to address the class imbalance problem in the training data:

[0028] Strategy 1: During training, use the focus loss function and adjust the hyperparameters to increase the weight of difficult or error-prone samples in the total loss;

[0029] Strategy 2: In the data preprocessing stage, random oversampling is used to replicate minority class samples in the training set to increase their frequency of occurrence in each training batch, so that the collision severity classification module can learn its features more fully.

[0030] Furthermore, the method for generating natural language description text in step S4 specifically includes:

[0031] Step S41: The semantic tag set from step S4, the vehicle type and collision location output from the target detection model, and the prediction results from step S3 are integrated and analyzed in parallel, and then combined into structured key-value pairs.

[0032] Step S42: Based on the preset text template, fill the corresponding placeholders in the preset text template with the structured key-value pairs from step S41 to dynamically construct a text prompt containing explicit instructions.

[0033] Step S43: After inputting the text prompts obtained in step S42 into a large language generation model to generate initial description text, perform a post-processing procedure that includes at least content cleaning, validity verification, and redundancy removal.

[0034] Step S44: After the post-processing in step S43 is completed, determine whether the generated description text meets the preset quality standards. If it does not meet the standards, activate a predefined fallback template and combine it with the core structured key-value pairs in step S41 to generate a structurally stable baseline description text as the final prediction result.

[0035] The present invention also provides a vehicle collision detection and description system based on surveillance video, comprising:

[0036] The video parsing module is used to decode the original video from the input monitoring perspective, and uniformly extract frames according to the set frame rate parameters to output an image frame sequence.

[0037] The image feature extraction module includes an image preprocessing unit, a CLIP model, and a multi-frame feature stitcher. The image preprocessing unit is used to adjust each frame of the image frame sequence in the video parsing module to a preset size and perform normalization processing. Then, the image encoder of the CLIP model encodes each frame of the image into a first semantic feature vector. The multi-frame feature stitcher is used to stack N first semantic feature vectors into a first temporal tensor.

[0038] The collision severity classification module includes a Transformer temporal modeler and a classifier. The Transformer temporal modeler includes a multi-layer encoder, a multi-head attention mechanism unit, and a temporal feature integration unit. The first temporal tensor is dynamically weighted by the multi-layer encoder and attention mechanism unit, and then the weighted feature sequence is averaged by the temporal feature integration unit to generate an aggregated feature vector that can capture the temporal relationship between frames and output the entire video segment. The classifier is used to classify the aggregated feature vector to obtain the prediction result of the accident severity level category. The prediction result is one of several predefined accident severity level categories.

[0039] The video semantic description generation module includes a semantic feature analysis unit and a text generation unit. The semantic feature analysis unit is used to extract the overall visual features of the first temporal tensor to obtain the second semantic feature vector at the video level. The second semantic feature vector is then matched with a preset set of text tags to select one or more semantic tags that are closest to the video scene, thus obtaining a set of semantic tags. The text generation unit is used to construct prompt words based on the semantic tags and call a language generation model to generate natural language description text.

[0040] The pre-collision recognition and early warning module is used to trigger an early warning mechanism when the prediction result output by the collision severity classification module represents the level of a collision. It outputs an early warning signal and calls the target detection model to overlay detection boxes and labels on the original video frames from the monitoring perspective, generating and saving labeled video files.

[0041] Furthermore, the image feature extraction module also includes a vector padding unit. If the number of video frames is less than N frames, the vector padding unit uses zero vectors to pad and form the first temporal tensor.

[0042] Furthermore, the collision severity classification module also includes an attention network, which includes at least one fully connected layer and a non-linear activation function. The aggregated feature vector V is input into the attention network, which learns to generate a weight vector W with the same dimension as V. The aggregated feature vector V is weighted by element-wise multiplication to obtain a weighted aggregated feature vector V-weighted that focuses on key features, where V-weighted = V*W.

[0043] The fully connected layer of the classifier is used to compress and project the weighted aggregated feature vector V_weighted output from the attention network into a low-dimensional space, the number of dimensions of which is equal to the number of categories in the classification task; the softmax function of the classifier transforms the numerical values ​​output by the fully connected layer into occurrence probability values ​​corresponding to each classification task; where the classification task is one or more of a number of predefined accident severity level categories; the last layer of the collision severity classification module identifies the category corresponding to the highest occurrence probability value as the prediction result.

[0044] Furthermore, the collision severity classification module also includes:

[0045] The model training unit, used during the training of the collision severity classification module, calculates the difference between the predicted result output by the collision severity classification module after processing the aggregated feature vector and the pre-labeled true accident level labels in the dataset. It uses either the focus loss function or the cross-entropy loss function, combined with the AdamW optimizer and the OneCycleLR scheduler, to train the model. At least one of the following strategies is employed to address the class imbalance problem in the training data:

[0046] Strategy 1: During training, use the focus loss function instead of the standard cross-entropy loss function, and adjust the hyperparameters to increase the weight of difficult or error-prone samples in the total loss.

[0047] Strategy 2: In the data preprocessing stage, random oversampling is used to replicate minority class samples in the training set to increase their frequency of occurrence in each training batch, so that the collision severity classification module can learn its features more fully.

[0048] Furthermore, the text generation unit specifically includes:

[0049] The multimodal information synthesis subunit is used to integrate and analyze the semantic tag set of the video semantic description generation module, the vehicle type and collision location output from the target detection model, and the prediction results from the collision severity classification module in parallel, and then combine them into structured key-value pairs.

[0050] The text prompt generation subunit is used to dynamically construct a text prompt containing explicit instructions by filling the structured key-value pairs in the multimodal information integration subunit into the corresponding placeholders in the preset text template based on the preset text template.

[0051] The Natural Language Description Subunit is used to input the text prompts obtained from the Text Prompt Generation Subunit into the large language generation model to generate the initial description text. Then, it performs a post-processing process that includes at least content cleaning, validity verification, and redundancy removal to determine whether the generated description text meets the preset quality standards. If it does not meet the standards, a predefined fallback template is activated. Combined with the core structured key-value pairs in the multimodal information synthesis subunit, a structurally stable baseline description text is generated as the final prediction result.

[0052] The present invention has the following advantages due to the adoption of the above technical solutions:

[0053] 1. This invention can automatically complete the entire process from video input to generating a structured analysis report, replacing the inefficient traditional model that relies on manual video viewing, subjective judgment, and report writing. The output not only includes a binary judgment of whether an accident occurred, but also provides multi-level severity classifications such as "no accident," "minor," "moderate," and "serious," as well as natural language descriptions containing details such as weather, road conditions, accident type, and vehicle information. Therefore, this invention solves the problems of low automation and reliance on manual interpretation in traditional methods: traditional technical solutions are inefficient and prone to introducing subjective errors. This invention achieves fully automated analysis, greatly reducing human intervention and improving the efficiency and objectivity of traffic accident assessment.

[0054] 2. This invention, by introducing a "pre-collision" recognition mechanism and sliding window temporal modeling, enables the system to analyze the dynamic frame sequence before an accident occurs, triggering an early warning immediately before or during a collision. This extends the functionality from post-accident backtracking to pre-accident warning or real-time response, providing the possibility for proactive safety intervention. It solves the problems of coarse-grained analysis and lack of detailed differentiation in existing methods: overcoming the limitation of most existing systems that can only make coarse-grained judgments of "whether an accident has occurred," it achieves multi-level fine-grained differentiation of accident severity and a deep understanding of scene elements.

[0055] 3. By fusing the Visual-Language Model (CLIP) with a large-scale language model, this invention enables the system to transform machine vision features into natural language descriptions that humans can intuitively understand. It also generates labeled videos with bounding boxes in conjunction with object detection results (YOLO). The output includes labels, text, and visual videos. Therefore, this invention changes the problem that traditional systems only output machine labels or coordinate boxes, which cannot provide intuitive and detailed semantic information for human decision-makers. By automatically generating accurate and complete natural language descriptions, the machine analysis results become readable, usable, and interpretable.

[0056] 4. To address practical challenges such as sample imbalance, lighting variations, and angular differences, this invention employs strategies such as data augmentation, focal loss, class resampling, and targeted attention mechanisms during training. These optimizations effectively improve the model's generalization ability under different conditions and its accuracy in identifying rare but important serious incidents. Attached Figure Description

[0057] Figure 1 This is a schematic diagram of the workflow of the vehicle collision detection and description method based on surveillance video according to an embodiment of the present invention.

[0058] Figure 2 This is a schematic diagram of the collision warning effect according to an embodiment of the present invention.

[0059] Figure 3 This is a schematic diagram illustrating the collision recognition effect according to an embodiment of the present invention.

[0060] Figure 4 This is a schematic diagram of a collision severity classification confusion matrix from a monitoring perspective according to an embodiment of the present invention. Detailed Implementation

[0061] In the accompanying drawings, the same or similar reference numerals are used to denote the same or similar elements or elements having the same or similar functions. The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0062] In the description of this invention, the terms "center," "longitudinal," "lateral," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting the scope of protection of this invention.

[0063] like Figure 1 As shown, the vehicle collision detection and description method based on surveillance video provided in this embodiment of the invention includes:

[0064] Step S1, Video Analysis: The input raw video from the surveillance perspective is decoded. Initial decoding processing is supported for mainstream video file formats such as MP4 and AVI. Frames are uniformly extracted according to the set frame rate parameters to output an image frame sequence, ensuring temporal integrity. The raw video from the surveillance perspective mentioned above comes from the Kaggle Road Accidents from CCTVFootages Dataset, available at: https: / / www.kaggle.com / datasets / suryaprabhakaran2005 / road-accidents-from-cctv-footages-dataset.

[0065] Step S2, Image Feature Extraction: Adjust each frame of the image frame sequence in Step S1 to a preset size and perform normalization processing. Then, use the image encoder of CLIP (Contrastive Language-Image Pretraining) model to encode each frame of the image into a first semantic feature vector, and stack N first semantic feature vectors into a first temporal tensor.

[0066] In one embodiment, the operation of adjusting each frame of the image frame sequence in step S1 to a preset size and performing normalization processing in step S2 is a process of image preprocessing, which specifically includes:

[0067] The dimensions of each frame in the image frame sequence of step S1 are uniformly adjusted to a preset size, such as 224×224 pixels. During image preprocessing, the open-source CLIP model is required. CLIP is an open-source model developed by OpenAI. Developers can easily obtain it through mainstream AI model platforms (such as Hugging Face). In the code provided in this project, the model identifier is clip-vit-base-patch32. In this embodiment, mean and standard deviation normalization is performed according to the requirements of the open-source CLIP model. Of course, if in training mode, random cropping, color perturbation, and flipping are also performed on the input image to improve the model's generalization ability and robustness. By randomly transforming the training image, it is equivalent to artificially creating more diverse training data. If not in the training phase, these random enhancement operations are not performed, and the system directly processes the original, unmodified video frames.

[0068] In one embodiment, the encoding of each frame of image into a first semantic feature vector by the image encoder of the CLIP model in step S2 in step S1 is illustrated by the following example:

[0069] The image encoder backbone network employs a ViT-B / 32 structure to encode each preprocessed image frame into a 512-dimensional semantic feature vector, which is then output in a standardized format. This standardized format involves: CLIP encoding N frames of images into N frames of 512-dimensional semantic feature vectors, and then stacking these N frames into an [N×512] two-dimensional tensor, i.e., the first temporal tensor. Subsequent processing models require input data with a fixed size. Standardizing the first temporal tensor to an [N×512] format ensures that the model can process arbitrary video segments and function correctly.

[0070] In the process of stacking N frames of 512-dimensional semantic feature vectors into a [N×512] two-dimensional tensor structure, if the number of video frames is less than N frames, zero vectors are used to fill in the gaps to form the first temporal tensor.

[0071] Step S3, Collision Severity Classification: The first temporal tensor from Step S2 is input into the Transformer temporal modeler. The encoder and self-attention mechanism of the Transformer temporal modeler dynamically weight the first temporal tensor, and the temporal feature integration unit of the Transformer temporal modeler performs average pooling on the weighted feature sequence to generate an aggregated feature vector that captures the temporal relationships between frames and outputs a representation of the entire video segment. The aggregated feature vector is input into the classifier to obtain the predicted accident severity level category. The predicted result is one of several predefined accident severity level categories.

[0072] In one embodiment, a number of predefined accident severity level categories include at least non-accident, light, medium, and severe. For example, the category with the highest Softmax output probability is taken as the final category, with light being the classification label for cosmetic damage only, medium being the classification label for component deformation, and severe being the classification label for structural damage.

[0073] Specifically, step S3, which involves dynamically weighting the first temporal tensor using the encoder and self-attention mechanism unit of the Transformer temporal modeler, includes:

[0074] The Transformer temporal modeler models the first temporal tensor to capture and understand the temporal relationships between image frames. We know that a traffic accident is a dynamic process, not a static image. The Transformer temporal modeler's self-attention mechanism analyzes the entire frame sequence, determining which frames are most important for the final accident assessment and understanding the relationships between them. After processing by the Transformer temporal modeler, a higher-level video semantic representation incorporating contextual information is obtained—a weighted feature sequence. Therefore, compared to the original feature vectors representing only a single frame, the weighted feature sequence output by the Transformer temporal modeler contains the dynamic information of the entire time series.

[0075] The weighted feature sequence output by the Transformer temporal modeler is then average-pooled by the temporal feature integration unit to generate an aggregated feature vector that captures the temporal relationships between frames and represents the entire video segment. In other words, the feature sequence output by the Transformer—an [N×512] tensor—is aggregated into a single, fixed-length feature vector, i.e., a [1×512] vector. This aggregated feature vector represents the final semantic summary of the entire video segment, containing a video segment-level semantic representation of temporal dynamic information. For example, the semantic representation of a 512-dimensional vector, condensing the core information of the video, can be understood as the mathematical expression "a car is traveling at high speed on a wet road and a serious rear-end collision occurs."

[0076] In one embodiment, the Transformer temporal modeler uses a 2-layer encoder, which is a lightweight and efficient choice for the 10-frame sequence modeling task in this embodiment. It captures the necessary temporal relationships without causing training difficulties or overfitting due to excessive model depth. The Transformer temporal modeler uses a 4-head attention mechanism unit, allowing the model to simultaneously focus on information in the frame sequence from four different "subspaces," enhancing its ability to capture diverse feature relationships. Ultimately, the number of encoder layers and attention mechanism units is determined by testing different combinations (e.g., 1 / 2 / 4 layers, 2 / 4 / 8 heads) on a validation set and selecting the best-performing configuration.

[0077] In one embodiment, step S3, "inputting the aggregated feature vector into the classifier," further includes the following:

[0078] Step 31, Attention Focusing: The aggregated feature vector V is input into an attention network consisting of at least one fully connected layer and a non-linear activation function. This attention network learns to generate a weight vector W with the same dimension as V, and then weights the aggregated feature vector V using element-wise multiplication to obtain a weighted aggregated feature vector V-weighted that focuses on key features, where V-weighted = V * W. V-weighted, instead of the original V, is then fed into the subsequent classification network. This process allows the model to adaptively "focus" on the features it deems most critical. For example, in complex scenarios involving multiple vehicles, the attention network may learn to increase the weight of features related to the dynamics of the colliding vehicles while suppressing the weight of background or irrelevant vehicle information, thereby improving the signal-to-noise ratio and accuracy of accident level determination.

[0079] The method for "obtaining the predicted accident severity category" in step S3 specifically includes:

[0080] Step 32: The weighted aggregated feature vector V_weighted output from step 31 is compressed and projected into a low-dimensional space through the fully connected layer in the classifier. The number of dimensions of this low-dimensional space is equal to the number of categories in the classification task.

[0081] Step 33: The values ​​output in step 32 are converted into occurrence probability values ​​corresponding to each classification task using the Softmax function in the classifier. The classification task is one or more predefined accident severity level categories.

[0082] Step 34: Identify the category corresponding to the highest probability value output in step 33, and use it as the prediction result.

[0083] For example, the fully connected layer first compresses and projects the 512-dimensional aggregated feature vector representing video semantics into a low-dimensional space. The number of dimensions in this low-dimensional space is exactly equal to the number of categories in the classification task. For example, 512→2 means that the aggregated feature vector is mapped to two values, representing the two classification tasks of "accident occurred" and "no accident occurred". The Softmax function transforms the values ​​output by the fully connected layer into probability values ​​for "accident occurred" and "no accident occurred". Or, for example, 512→4 means that the aggregated feature vector is mapped to four values, representing the four classification tasks of "no accident", "minor", "moderate", and "serious". The Softmax function transforms the values ​​output by the fully connected layer into probability values ​​for "no accident", "minor", "moderate", and "serious". By comparing the probability values ​​of all categories, the category with the highest probability is selected as the final prediction result.

[0084] In fact, other linear mapping forms are also possible. Here, 512 is determined by the structure of the CLIP model, while 2 or 4 is determined by the specific task requirements (number of classification categories). Although 512 → number of categories is the most straightforward form, deeper networks can also be designed, such as 512 → 256 → 4, i.e., adding one or more hidden layers. Doing so can enhance the model's non-linear expressive power and may achieve better results on more complex problems, but it will also increase the model's complexity and computational cost.

[0085] In one embodiment, if it is a training process, backpropagation training is completed through the model training and saving unit, which calls the loss function calculator, parameter optimizer, and model breakpoint saving logic. Based on this, the method for "obtaining the prediction result of the accident severity level category" in step S3 further includes:

[0086] Step 35: During the training of the root collision severity classification module, based on the difference between the prediction results output in step S34 and the pre-labeled real accident level labels in the dataset, the focus loss function or cross-entropy loss function is used, combined with the AdamW optimizer and OneCycleLR scheduler, to train the model, making the model predictions increasingly accurate. The final saved "model breakpoint" represents a mature model capable of completing a specific task.

[0087] In tasks such as vehicle collision damage assessment, a challenge is the class imbalance of the data samples. This means that in a large amount of video data, there are far more "no accident" or "minor accident" scenarios than "serious accident" scenarios. The standard cross-entropy loss function treats all samples equally during model training, which leads to a serious problem: the model is dominated by a large number of easily classified "simple samples." As a result, the model's optimization direction tends to predict everything as the majority class, thus neglecting the learning of the few but crucial "serious accident" samples, resulting in low accuracy in identifying key events. To overcome this limitation, this embodiment introduces a focus loss function during model training in step S35. By adjusting the hyperparameters, the weight of difficult or misclassifiable samples (usually the minority class of serious accidents) in the total loss is increased. The core idea is to dynamically adjust the weight of each sample in the total loss. By introducing an adjustment factor, the weight of simple samples that are easily and correctly classified by the model can be significantly reduced, while maintaining or even increasing the weight of difficult samples that are hard to classify. In this way, the model can focus more on learning and understanding those rare but key accident features during training, thereby effectively improving the recognition accuracy under extremely imbalanced data.

[0088] Similarly, to address the class imbalance issue in the training data during model training in step S35, in addition to the above embodiments, random oversampling can be used in the data preprocessing stage to duplicate minority class samples in the training set. This increases their frequency of appearance in each training batch, allowing the collision severity classification module to learn their features more fully. This operation is a one-time preparatory step set before the model officially begins its training cycle. Its purpose is to artificially increase the proportion of minority class features in the entire training set by duplicating them, thereby constructing a more balanced training dataset. This allows the subsequent model training to learn the features of the minority class more fully and avoids model bias.

[0089] Step S4, Generate Video Semantic Description: Extract the overall visual features of the image frame sequence from Step S2 to obtain a video-level second semantic feature vector; perform similarity matching between the second semantic feature vector and a preset text tag set, and select one or more semantic tags that best match the video scene to obtain a semantic tag set; construct a prompt word based on the semantic tag set, and input the prompt word into the language generation model to generate natural language description text. The preset text tags include multi-category tags for describing weather, road conditions, and accident types.

[0090] In one embodiment, the process of obtaining a video-level second semantic feature vector from the overall visual features of the image frame sequence in step S2 in step S4 is illustrated by the following example:

[0091] By extracting the 512-dimensional feature vectors of each of the N frames extracted from the image frame sequence in step S2, and then aggregating them through averaging and other methods, the features of multiple frames are summarized into a unified vector, namely the second semantic feature vector. This achieves a holistic description of the entire video content.

[0092] The preset set of text tags contains multiple categories of tags for describing weather, road conditions, and accident types. Weather includes daytime or nighttime, road conditions include dry or wet roads, and accident types include "rear-end collision" or "side collision," which can cover almost all common traffic accident scene elements.

[0093] "Match the second semantic feature vector with the preset set of text labels to select one or more semantic labels that are closest to the video scene." For example, the second semantic feature vector can be compared with the preset set of labels to select the set of semantic labels that are closest to the scene. For example, in the "road conditions" category, "wet road surface" scores 0.9 and "dry road surface" scores 0.05, so the system will select "wet road surface" as the final label for the scene.

[0094] In step S4, the second semantic feature vector is matched with a preset set of text tags to select one or more semantic tags that are closest to the video scene. For example, for a video of a rear-end collision that occurred during the day after rain, the present invention calculates a similarity score between the video features and all candidate tags under each category through image-text matching. Then, the tag with the highest similarity score is selected from the preset set of text tags as one or more semantic tags that are closest to the scene.

[0095] In one embodiment, the method for generating natural language description text in step S4 specifically includes:

[0096] Step S41, Multimodal Information Synthesis: The semantic label set from step S4, the vehicle type and collision location output from the target detection model, and the prediction results from step S3 are integrated and analyzed in parallel, and then aggregated into structured key-value pairs. That is, the multi-dimensional information collected in the previous step is organized into a flat or logically clear key-value pair structure.

[0097] The "parallel integration and analysis" can be implemented using existing conventional programming methods. Specifically, this process involves first creating a standard data container and then filling it with results from different analysis modules. This includes the accident severity prediction results obtained from step S3, scene semantic labels (such as weather, road conditions, and accident type) obtained through visual-language model comparison, and information such as vehicle type and collision location identified by object detection models (such as YOLO). The entire process integrates data from various independent analysis steps into a unified structure. The final output of this operation is a structured dataset that encapsulates all key information dimensions about a single accident event, providing a data foundation for subsequent generation of structured key-value pairs and the construction of natural language prompts.

[0098] A specific embodiment of step S41 below:

[0099] After incorporating information from the object detection model, the system can identify the specific vehicles in the video as "a blue sedan" and "a white truck," and may determine the collision locations as "the front of the sedan" and "the rear of the truck" through analysis. Therefore, the constructed prompts can become more specific and precise. For example: Please generate a traffic accident description based on the following information: Accident type: rear-end collision, time: daytime, vehicles involved: [blue sedan, white truck], collision locations: [front of sedan, rear of truck]. The generated natural language description will be more accurate than when there is no object detection information.

[0100] Step S42, Dynamic Construction of Structured Prompts: Based on a preset text template, the structured key-value pairs from step S41 are filled into the corresponding placeholders in the preset text template to dynamically construct a text prompt containing explicit instructions. For example, the core information of the structured key-value pairs in step S41 includes:

[0101] Structured scene tags: ["Daytime", "Wet road surface", "Rear-end collision"];

[0102] Target detection results: ["truck", "car"]; and

[0103] Accident severity level: "Severe";

[0104] The above preset text template includes content requirements and constraints, and is preset by the Prompt constructor. The preset text template not only includes placeholders for filling in information, but also provides explicit output specifications for the language model. For example, the template could be: "Generate an objective description of this traffic accident based on the following information: accident severity: {severity}, accident type: {type}, time and weather: {time_weather}, road condition: {road_condition}, vehicles involved: {vehicles}. Content requirements: 1. Must begin with 'The scene'." 1. Start directly with "shows...". 2. Keep the description entirely based on facts. 3. Avoid content: any speculation about the cause of the accident, any suggestions or warnings. Dynamically construct the prompt word "Prompt" by filling the corresponding positions in the text template with the information from step S41. For example, replace {severity} with "serious", {type} with "rear-end collision", {time_weather} with "daytime", {road_condition} with "wet road", and {vehicles} with "truck, car", generating specific text prompts. For example, the final prompt is: Generate an objective description of this traffic accident based on the following information: Accident severity: serious, Accident type: rear-end collision, Time and weather: daytime, Road condition: wet road, Vehicles involved: truck, car.

[0105] Step S43, Natural Language Generation and Multi-Level Post-Processing: The text prompts obtained in Step S42 are input into a large-scale language generation model (such as the GPT series models) to generate an initial natural language description. To ensure the professionalism and accuracy of the final output, the system does not directly use the initial text, but instead performs a multi-level post-processing process. This process first automatically removes the remaining input prompts used to guide the model from the text and cleans up special characters or numbers. Subsequently, the system strictly verifies whether the generated text begins with a preset starting phrase (such as "The scene shows...") and filters out invalid sentences that are too short or lack sufficient information. Finally, it also uses algorithms to analyze and remove unnecessary repeated words or phrases in the text to improve the fluency and information density of the overall description.

[0106] Step S44, Fallback Fault Tolerance Mechanism: In addition, to address the extreme case where a large language model might occasionally generate suboptimal content, this embodiment sets up a fallback fault tolerance mechanism. After post-processing in step S43, the system evaluates whether the final text meets preset quality standards, such as the minimum word count. If the text does not meet this standard, the system will not output the substandard text, but instead will use a predefined, more structurally stable fallback template. This template combines the core structured information obtained in step S41 with a set of backup phrases used to supplement scene details, thereby generating a grammatically correct and informationally accurate fallback description as the final prediction result. This mechanism ensures that the system can provide a stable and usable output under any circumstances, greatly enhancing the system's reliability. Here, the "core" structured key-value pairs refer to the key factual information that constitutes the most basic skeleton of the incident description. This information ensures that even if the large language model generates suboptimal content, the system can still produce a grammatically correct, factually accurate, and informationally complete fallback description. These include: vehicle type, accident type, severity of damage, visible damaged parts, road conditions, accident environment, and time or lighting conditions.

[0107] Step S5, Pre-collision identification and early warning: When the classification label output in step S3 represents the level of collision, an early warning mechanism is triggered, an early warning signal is output, and a notification is sent to the console or extended to an API push. Additionally, the object detection model is invoked to overlay detection boxes and labels onto the original video frame from the monitoring perspective, generating and saving an labeled video file. For example, when the prediction result is "crash" or "severe," the object detection model is automatically invoked to overlay a red detection box and a "SevereCrash" label onto the video frame, simultaneously activating the API alarm push interface and outputting an MP4 format video file with semantic descriptive text.

[0108] In one embodiment, step S5 includes the following warning mechanism: real-time overlay of warning text on video frames, pushing alarm information via API interface, and locally saving annotated videos with detection boxes, such as... Figure 2 and Figure 3 As shown.

[0109] This invention also provides a vehicle collision detection and description system based on surveillance video. This system includes a video parsing module, an image feature extraction module, a collision severity classification module, a video semantic description generation module, and a pre-collision identification and early warning module, wherein:

[0110] The video parsing module is used to decode the original video from the input monitoring perspective, and to uniformly extract frames according to the set frame rate parameters to output an image frame sequence.

[0111] The image feature extraction module includes an image preprocessing unit, a CLIP model, and a multi-frame feature stitcher. The image preprocessing unit is used to adjust each frame of the image frame sequence in the video parsing module to a preset size and perform normalization processing. Then, the image encoder of the CLIP model encodes each frame of the image into a first semantic feature vector. The multi-frame feature stitcher is used to stack N first semantic feature vectors into a first temporal tensor.

[0112] The collision severity classification module includes a Transformer temporal modeler and a classifier. The Transformer temporal modeler includes a multi-layer encoder, a multi-head attention mechanism unit, and a temporal feature integration unit. The first temporal tensor is dynamically weighted by the multi-layer encoder and attention mechanism unit, and then the weighted feature sequence is averaged by the temporal feature integration unit to generate an aggregated feature vector that can capture the temporal relationship between frames and output the entire video segment. The classifier is used to classify the aggregated feature vector to obtain the prediction result of the accident severity level category. The prediction result is one of several predefined accident severity level categories.

[0113] The video semantic description generation module includes a semantic feature analysis unit and a text generation unit. The semantic feature analysis unit is used to extract the overall visual features of the first temporal tensor to obtain the second semantic feature vector at the video level. The second semantic feature vector is then matched with a preset set of text tags to select one or more semantic tags that are closest to the video scene, thus obtaining a semantic tag set. The text generation unit is used to construct prompt words based on the semantic tags and call a language generation model to generate natural language description text.

[0114] The pre-collision recognition and early warning module is used to trigger an early warning mechanism when the prediction result output by the collision severity classification module represents the level of a collision. It outputs an early warning signal and calls the target detection model to overlay detection boxes and labels on the original video frames from the monitoring perspective, generating and saving labeled video files.

[0115] In one embodiment, the image feature extraction module further includes a vector padding unit. If the number of video frames is less than N frames, the vector padding unit uses zero vectors to pad and form a first temporal tensor.

[0116] In one embodiment, the collision severity classification module further includes an attention network, which includes at least one fully connected layer and a non-linear activation function. The aggregated feature vector V is input into the attention network, which learns to generate a weight vector W with the same dimension as V. The aggregated feature vector V is weighted by element-wise multiplication to obtain a weighted aggregated feature vector V-weighted that focuses on key features, where V-weighted = V*W.

[0117] The fully connected layer of the classifier is used to compress and project the weighted aggregated feature vector V_weighted output from the attention network into a low-dimensional space, the number of dimensions of which is equal to the number of categories in the classification task; the softmax function of the classifier transforms the numerical values ​​output by the fully connected layer into occurrence probability values ​​corresponding to each classification task; where the classification task is one or more of a number of predefined accident severity level categories; the last layer of the collision severity classification module identifies the category corresponding to the highest occurrence probability value as the prediction result.

[0118] In one embodiment, the collision severity classification module further includes a model training unit, wherein:

[0119] During the training of the collision severity classification module, the model training unit uses the difference between the predicted result output by the collision severity classification module after processing the aggregated feature vector and the pre-labeled true accident level labels in the dataset. It employs either the focus loss function or the cross-entropy loss function, combined with the AdamW optimizer and the OneCycleLR scheduler, to train the model. At least one of the following strategies is used to address the class imbalance problem in the training data:

[0120] Strategy 1: During training, use the focus loss function instead of the standard cross-entropy loss function, and adjust the hyperparameters to increase the weight of difficult or error-prone samples in the total loss.

[0121] Strategy 2: In the data preprocessing stage, random oversampling is used to replicate minority class samples in the training set to increase their frequency of occurrence in each training batch, so that the collision severity classification module can learn its features more fully.

[0122] In one embodiment, the text generation unit specifically includes a multimodal information synthesis subunit, a text prompt generation subunit, and a natural language description subunit, wherein:

[0123] The multimodal information synthesis subunit is used to integrate and analyze the original video from the monitoring perspective input by the video parsing module, the semantic tag set from the video semantic description generation module, the vehicle type and collision location output from the target detection model, and the prediction results from the collision severity classification module in parallel, and then aggregate them into structured key-value pairs.

[0124] The text prompt generation subunit is used to dynamically construct a text prompt containing explicit instructions by filling the corresponding placeholders in the multimodal information integration subunit with the structured key-value pairs in the preset text template.

[0125] The Natural Language Description Subunit is used to input the text prompts obtained from the Text Prompt Generation Subunit into the large language generation model to generate initial description text. It then performs a post-processing process that includes at least content cleaning, validity verification, and redundancy removal to determine whether the generated description text meets the preset quality standards. If it does not meet the standards, a predefined fallback template is used, which is combined with the core structured key-value pairs in the Multimodal Information Synthesis Subunit to generate a structurally stable baseline description text as the final prediction result.

[0126] like Figure 4 As shown, Figure 4 This diagram presents a confusion matrix used to evaluate the classification performance of a model, primarily describing the comparison between the model's predictions and the actual situation. The vertical axis (True) represents the true severity level of the incident, and the horizontal axis (Predicted) represents the model's predicted severity level. The categories include "Minor," "Moderate," "No Incident," and "Severe." Each box in the matrix represents a combination of prediction and reality; the number inside indicates the sample size, and the color intensity is proportional to the quantity—darker colors indicate a larger sample size. As can be seen from the diagram, the diagonal boxes are the darkest and have the largest numbers (e.g., 22, 22, 18, 29), indicating that the model performs well in correctly identifying most categories (i.e., predictions are consistent with reality). The off-diagonal boxes reveal confusion points, indicating that the model faces challenges in distinguishing moderate-level incidents.

[0127] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Those skilled in the art should understand that modifications can be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for vehicle collision detection and description based on monitoring video, characterized in that, The method comprises the following steps: Step S1, decoding the input monitoring view original video and uniformly frame-skipping according to the set frame rate parameter to output a sequence of image frames; Step S2, adjusting each frame image in the sequence of image frames of step S1 to a preset size and performing normalization processing, then encoding each frame image into a first semantic feature vector through an image encoder of a CLIP model, and stacking N first semantic feature vectors into a first time sequence tensor; Step S3, performing dynamic weighting processing on the first time sequence tensor through a Transformer time sequence modeler, and performing average pooling on the weighted feature sequence to generate an aggregated feature vector capable of capturing the time sequence relationship between frames and outputting a representative aggregated feature vector of the entire video segment; inputting the aggregated feature vector into a classifier to obtain a prediction result of the accident severity level category, and the prediction result is one of a plurality of pre-defined accident severity level categories; Step S4, extracting the overall visual features of the first time sequence tensor of step S2 to obtain a video-level second semantic feature vector; performing similarity matching between the second semantic feature vector and a pre-set text label set to filter out one or more semantic labels closest to the video scene to obtain a semantic label set; constructing a prompt word based on the semantic label set, inputting the prompt word into a language generation model to generate a natural language description text; wherein the pre-set text label includes multi-category labels for describing weather, road conditions, and accident types; Step S5, when the prediction result output by step S3 is a level representing a collision, triggering an early warning mechanism, outputting a warning signal, and calling a target detection model to superimpose a detection box and a label on the monitoring view original video frame to generate and save a labeled video file.

2. The method for vehicle collision detection and description based on monitoring video according to claim 1, characterized in that, In step S2, if the number of video frames is less than N frames, a zero vector is used to fill in to form the first time sequence tensor.

3. The method of claim 1 or 2, wherein the method is characterized by, Before "inputting the aggregated feature vector into a classifier" in step S3, it further comprises the following steps: Step 31, inputting the aggregated feature vector V into an attention network composed of at least one fully connected layer and a nonlinear activation function, the attention network learning to generate a weight vector W with the same dimension as V, and weighting the aggregated feature vector V by element-wise multiplication to obtain a weighted aggregated feature vector V-weighted that focuses on key features, wherein V-weighted = V*W; The method of "obtaining a prediction result of the accident severity level category" in step S3 specifically comprises the following steps: Step 32, compressing and projecting the weighted aggregated feature vector V_weighted output by step 31 into a low-dimensional space through the fully connected layer in the classifier, and the number of dimensions of the low-dimensional space is equal to the number of categories of the classification task; Step 33, converting the value output by step 32 into occurrence probability values corresponding to each classification task through the Softmax function in the classifier; wherein the classification task is one or more of a plurality of pre-defined accident severity level categories; Step 34, identifying the category corresponding to the highest occurrence probability value output by step 33 as the prediction result.

4. The method for vehicle collision detection and description based on monitoring video according to claim 3, wherein, The method of "obtaining a prediction result of the accident severity level category" in step S3 specifically further comprises: In the training process of the collision severity classification module, according to the difference between the prediction result output in step S34 and the true accident level label pre-labeled in the data set, the parameter optimizer unit composed of a specific optimizer and a specific scheduler is trained using a focal loss function, and at least one of the following strategies is used to handle the class imbalance problem in the training data: Strategy one: in the training process, the focal loss function is used, and the weight of difficult or mistaken classification samples in the total loss is increased by adjusting the hyperparameters; Strategy two: in the data preprocessing stage, a random oversampling technique is used to duplicate the minority class samples in the training set to increase their frequency in each training batch, so that the collision severity classification module can learn more about its features.

5. The method for vehicle collision detection and description based on monitoring video according to claim 4, wherein, The method for generating a natural language description text in step S4 specifically includes: Step S41, the semantic tag set in step S4, the vehicle type and the collision part from the output of the target detection model, and the prediction result from step S3 are integrated and analyzed in parallel, and then collected into a structured key-value pair; Step S42, based on the preset text template, the structured key-value pair in step S41 is filled into the corresponding placeholder of the preset text template to dynamically construct a text prompt containing clear instructions; Step S43, after inputting the text prompt obtained in step S42 into the large language generation model to generate an initial description text, a post-processing procedure including content cleaning, validity checking and redundancy removal is performed; Step S44, after the post-processing in step S43 is completed, it is judged whether the generated description text meets the preset quality standard; if not, a pre-defined fallback template is enabled, and a structured guaranteed description text is generated based on the core structured key-value pair in step S41, as the final prediction result.

6. A vehicle collision detection and description system based on monitoring video, characterized by, Comprise: a video analysis module for decoding the input monitoring angle original video and uniformly frame extracting according to the set frame rate parameter, outputting an image frame sequence; an image feature extraction module including an image preprocessing unit, a CLIP model and a multi-frame feature splicer, wherein the image preprocessing unit is used to adjust each frame image in the image frame sequence of the video analysis module to a preset size and perform normalization processing, then encode each frame image into a first semantic feature vector through the image encoder of the CLIP model, and the multi-frame feature splicer is used to stack N first semantic feature vectors into a first time sequence tensor; The collision severity classification module comprises a Transformer time sequence modeler and a classifier, wherein the Transformer time sequence modeler comprises a multi-layer encoder, a multi-head attention mechanism unit and a time sequence feature integration unit; the first time sequence tensor is dynamically weighted by the multi-layer encoder and the attention mechanism unit, and then the weighted feature sequence is subjected to average pooling by the time sequence feature integration unit to generate an aggregated feature vector capable of capturing the time sequence relationship between frames and outputting a representative aggregated feature vector of the entire video segment; the classifier is used for classifying the aggregated feature vector to obtain a prediction result of the accident severity level category, and the prediction result is one of a plurality of predefined accident severity level categories; The video semantic description generation module comprises a semantic feature analysis unit and a text generation unit, the semantic feature analysis unit is used for extracting the overall visual features of the first time sequence tensor to obtain a video-level second semantic feature vector; the second semantic feature vector is subjected to similarity matching with a preset text label set to filter out one or more semantic labels closest to the video scene to obtain a semantic label set; and the text generation unit is used for constructing prompt words according to the semantic labels and calling a language generation model to generate a natural language description text; The pre-collision identification and early warning module is used for triggering a warning mechanism, outputting a warning signal and calling a target detection model to superimpose a detection box and a label on the original video frame in the monitoring angle to generate and save a labeled video file when the prediction result output by the collision severity classification module represents a collision level.

7. The vehicle collision detection and description system based on monitoring video according to claim 5, characterized in that, The image feature extraction module further comprises a vector padding unit, which fills zero vectors to form the first time sequence tensor if the number of video frames is less than N frames.

8. A vehicle collision detection and description system based on monitoring video according to claim 5 or 6, characterized in that, The collision severity classification module further comprises an attention network, which comprises at least one fully connected layer and a nonlinear activation function; the aggregated feature vector V is input into the attention network, which learns to generate a weight vector W of the same dimension as V, and weights the aggregated feature vector V by element-wise multiplication to obtain a weighted aggregated feature vector V-weighted that focuses on key features, wherein V-weighted = V*W; The fully connected layer of the classifier is used for compressing and projecting the weighted aggregated feature vector V_weighted output by the attention network into a low-dimensional space, and the number of dimensions of the low-dimensional space is equal to the number of categories of the classification task; the Softmax function of the classifier converts the values output by the fully connected layer into occurrence probability values corresponding to each classification task; wherein the classification task is one or more of a plurality of predefined accident severity level categories; the last layer of the collision severity classification module identifies the category corresponding to the highest occurrence probability value as the prediction result.

9. The vehicle collision detection and description system based on monitoring video as claimed in claim 8, wherein, The collision severity classification module further comprises: The model training unit is used in the training process of the collision severity classification module, and the difference between the prediction result output by the collision severity classification module after processing the aggregated feature vector and the real accident level label pre-labeled in the data set is used to train the model using the focal loss function or the cross-entropy loss function, combined with the AdamW optimizer and the OneCycleLR scheduler. At least one of the following strategies is used to handle the class imbalance problem in the training data: Strategy one: during the training process, use the focal loss function instead of the standard cross-entropy loss function, and adjust the hyperparameters to increase the weight of difficult or error-prone classification samples in the total loss; Strategy two: in the data preprocessing stage, use random oversampling technology to duplicate the minority class samples in the training set to increase their frequency in each training batch, so that the collision severity classification module can learn more about its features.

10. The vehicle collision detection and description system based on monitoring video as claimed in claim 9, wherein, The text generation unit specifically includes: A multi-modal information integration sub-unit that integrates and analyzes the semantic label set of the video semantic description generation module, the vehicle type and collision site from the target detection model output, and the prediction result from the collision severity classification module in parallel, and then combines them into a structured key-value pair; A text prompt generation sub-unit that fills the structured key-value pair in the multi-modal information integration sub-unit into the corresponding placeholder of the preset text template based on the preset text template to dynamically construct a text prompt containing explicit instructions; A natural language description sub-unit that inputs the text prompt obtained by the text prompt generation sub-unit into a large language generation model to generate an initial description text, and then performs a post-processing procedure including content cleaning, effectiveness verification, and redundancy removal to determine whether the generated description text meets the preset quality standard. If not, a pre-defined fallback template is used to generate a structured guaranteed description text based on the core structured key-value pair in the multi-modal information integration sub-unit as the final prediction result.