Traffic event detection system based on multi-modal data

Abstract

The invention discloses a traffic incident detection system based on multi-modal data, and belongs to the technical field of traffic incident detection. The invention aims to solve the problems of poor traffic event detection capability and incapability of effectively associating context information caused by insufficient video data in a training process of an existing traffic event detection method. Comprising the following steps: acquiring high-dimensional visual features of a sample event image and a sample event video by adopting a visual encoder; obtaining a text feature vector of the cue word description text of the sample event by adopting a text encoder; a feature mapping layer is adopted to obtain high-dimensional visual features aligned with the text feature vectors; common multi-modal features are obtained through a multi-modal joint learning module; obtaining a semantic prediction result and an answer type text sequence of the sample event through a large language module; and obtaining a traffic event classification result through a traffic event classification module. The method is used for on-line detection of traffic events.

Description

Technical Field

[0001] The invention relates to a traffic event detection system based on multimodal data, and belongs to the technical field of traffic event detection. Background Art

[0002] Existing traffic incident detection technologies mainly rely on video data, and detect traffic incidents by analyzing scenes and moving targets in the video. This detection usually uses computer vision and machine learning algorithms, such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), and the recently emerging Transformer. Although these technologies have played an important role in improving the efficiency and safety of traffic monitoring, they still have the following shortcomings:

[0003] Dependence on a single data source: Currently, most traffic incident detection systems rely mainly on video data, while ignoring other data modalities that may contain rich information, such as images and text. This reliance on a single data source limits the detection system's ability to obtain information and the depth of scene understanding.

[0004] Insufficient data: For deep learning models, a large amount of labeled data is the key to training efficient models. However, in practical applications, it is very difficult to obtain a large amount of high-quality traffic event video data, especially those rare but important traffic events.

[0005] Lack of effective data fusion methods: Although some studies have attempted to combine videos with other modal data, these methods often lack effective data fusion mechanisms, resulting in the model being unable to fully utilize the complementarity of multimodal data, thus affecting the detection effect.

[0006] Insufficient model generalization ability: Existing traffic incident detection models often perform well in specific scenarios, but the model's generalization ability is insufficient in different environments or lighting conditions, which can easily lead to false positives and false negatives.

[0007] Limitations of fine-tuning strategies: When video data is insufficient to effectively fine-tune the pre-trained model, existing strategies often fail to effectively improve the performance of the model, especially when dealing with specific traffic events.

[0008] Inaccurate event recognition and classification: Existing traffic event recognition and classification methods are often based directly on video data, lacking in-depth understanding of event context information, resulting in inaccurate classification results.

[0009] In order to solve the above problems, it is necessary to provide a traffic incident detection method to improve the accuracy and efficiency of traffic incident detection. Summary of the invention

[0010] In view of the problems in existing traffic event detection methods that insufficient video data during training leads to poor traffic event detection capabilities and inability to effectively associate context information, the present invention provides a traffic event detection system based on multimodal data.

[0011] A traffic incident detection system based on multimodal data of the present invention comprises:

[0012] Visual encoder: During the training process, deep semantic feature extraction is performed on sample event images and sample event videos to obtain high-dimensional visual features of sample event images and sample event videos;

[0013] Text encoder: During the training process, the prompt word description text of the sample event is converted into a context-sensitive text feature vector;

[0014] Feature mapping layer: During the training process, the high-dimensional visual features of the sample event images and sample event videos are mapped to the language feature space to obtain the high-dimensional visual features of the language feature space, and aligned with the text feature vector to obtain the high-dimensional visual features after text alignment;

[0015] Multimodal joint learning module: During the training process, a joint training method is used to extract the common multimodal features greater than the similarity threshold in the high-dimensional visual features and text feature vectors after text alignment;

[0016] Large language module: during the training process, obtaining semantic prediction results for sample events based on the common multimodal features; and obtaining answer-type text sequences for sample events based on the semantic prediction results;

[0017] Traffic event classification module: during the training process, a traffic event classification result is obtained based on the semantic prediction result;

[0018] The trained traffic event detection system is used to obtain the answer text sequence of the current event and the classification result of the current traffic event based on the current event image or the current event video and the prompt word description text of the current event.

[0019] According to the traffic event detection system based on multimodal data of the present invention, the method for the visual encoder to obtain high-dimensional visual features of sample event images and sample event videos includes:

[0020] The sample event images and sample event videos are processed frame by frame and resized into image blocks of a set size. The image blocks are converted into encoding vectors using a linear projection layer, and then the encoding vectors are converted into high-dimensional visual features using a self-attention mechanism.

[0021] In the process of processing the encoding vector corresponding to the sample event video by the self-attention mechanism, the low-rank adaptation method of the temporal attention mechanism is adopted to apply temporal embedding to capture the temporal dependencies between video frames and configure the temporal information for the corresponding high-dimensional visual features.

[0022] According to the traffic event detection system based on multimodal data of the present invention, the method for the text encoder to obtain a text feature vector includes: the text encoder adopts a multi-layer Transformer architecture, and through a self-attention mechanism and a feedforward neural network, captures the long-distance dependencies and complex semantic structures in the prompt word description text, represents the prompt word description text with multiple set tags, and then indexes it through a vocabulary to generate a discrete ID sequence, and then converts the discrete ID sequence into a context-sensitive word embedding representation, and the word embedding representation is subjected to multiple rounds of self-attention calculations of the Transformer layer to generate a vector containing contextual semantic information corresponding to each set tag, thereby obtaining a context-sensitive text feature vector corresponding to the prompt word description text.

[0023] According to the traffic incident detection system based on multimodal data of the present invention, the method for obtaining high-dimensional visual features after text alignment in the feature mapping layer includes:

[0024] The feature mapping layer adopts a multi-layer perceptron MLP architecture, uses multiple fully connected layers to perform linear transformation on the high-dimensional visual features of sample event images and sample event videos, and combines the ReLU nonlinear activation function to obtain the high-dimensional visual features of the language feature space;

[0025] During the training process, the weight and bias parameters are optimized through the back-propagation method, and the high-dimensional visual features and text feature vectors in the language feature space are adaptively aligned to obtain the high-dimensional visual features after text alignment.

[0026] According to the traffic incident detection system based on multimodal data of the present invention, the multimodal joint learning module is optimized by a contrast loss function.

[0027] According to the traffic incident detection system based on multimodal data of the present invention, the large language module includes an encoder and a decoder;

[0028] The encoder encodes the common multimodal features through multi-head self-attention, and outputs the semantic prediction results of the sample events through the decoder after fully understanding the semantics of the common multimodal features; and then obtains the answer text sequence of the sample event based on the semantic prediction results.

[0029] According to the traffic incident detection system based on multimodal data of the present invention, the large language module adopts the Transformer architecture, and its encoder includes an embedding layer, a position encoding layer, a multi-head attention layer and a feedforward neural network layer;

[0030] The common multimodal features are used to generate text embeddings through the embedding layer, and then the position encoding is added through the position encoding layer to obtain a text sequence with sequential information; the multi-head attention layer is used to calculate the correlation between text sequences, and the semantic features of multiple texts are processed in parallel to capture the global dependencies of text sequences; the feedforward neural network layer performs a nonlinear transformation on the output of the multi-head attention layer to obtain the encoding result.

[0031] According to the traffic event detection system based on multimodal data of the present invention, the decoder of the large language module uses an attention layer to align the encoding results of the encoder.

[0032] According to the traffic event detection system based on multimodal data of the present invention, the traffic event classification module performs feature classification based on the semantic prediction result to obtain a traffic event classification result.

[0033] According to the traffic event detection system based on multimodal data of the present invention, the traffic event classification module adopts multiple fully connected layers to integrate and classify the semantic prediction results, and uses the Softmax function to calculate the confidence of each category. The confidence calculation is normalized to generate a probability distribution for each event category, and the classification result with high confidence is output as the current traffic event classification result.

[0034] Beneficial effects of the present invention: The present invention realizes the detection of traffic incidents based on computer vision technology and artificial intelligence, and improves the accuracy and efficiency of traffic incident detection by fusing multi-source data, optimizing fine-tuning strategies and deeply understanding event context information.

[0035] The system of the present invention can effectively fuse video, image and text data, and can process inputs from different modalities simultaneously. By learning the joint visual features of video and image, it can more efficiently extract information from different media and improve the accuracy and robustness of traffic event detection.

[0036] The system of the present invention effectively reduces the information fusion barrier between different modalities by integrating image and video data into a unified visual feature space. It not only improves the visual comprehension ability of the model in multimodal tasks, but also enhances the model's processing ability for images and videos through a unified visual feature integration method.

[0037] The system of the present invention can perform image compensation training under small sample video data. When the video training set has few samples, the image data set and the video data set are uniformly processed with features, and the image data is used to compensate for the lack of video data. This comprehensive training strategy significantly improves the performance of the model in traffic event perception and processing, ensuring that the model can still effectively identify and analyze traffic conditions even when video data is scarce.

[0038] The system of the present invention performs event perception in the text feature space. On the basis of understanding multimodal data, the event perception module composed of a large language module and a traffic event classification module performs classification perception of traffic events to obtain accurate analysis results of abnormal traffic events. In view of the classification accuracy and real-time feedback of the task, the event perception module accepts the aligned visual feature vectors. At the same time, the large language model is connected to the module. On the basis of fully understanding the image content, it can output the category of the traffic event and a detailed description of the traffic event through a reference prompt. The system of the present invention can respond quickly when an event occurs and convey the analysis results to the supervisor in real time; at the same time, the system can perform fine-grained classification of traffic events, so that the system can more accurately identify different types of traffic events, such as vehicle violations, traffic accidents, traffic congestion, etc., thereby providing more refined data support for traffic management. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 is a principle block diagram of the traffic incident detection system based on multimodal data according to the present invention;

[0040] Figure 2 It is a flow chart of the traffic incident detection system based on multimodal data described in the present invention. DETAILED DESCRIPTION

[0041] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0042] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0043] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, but they are not intended to limit the present invention.

[0044] Specific implementation method 1. Combination Figure 1 and Figure 2 As shown, the present invention provides a traffic incident detection system based on multimodal data, comprising:

[0045] Visual encoder: During the training process, deep semantic feature extraction is performed on sample event images and sample event videos to obtain high-dimensional visual features of sample event images and sample event videos;

[0046] Text encoder: During the training process, the prompt word description text of the sample event is converted into a context-sensitive text feature vector;

[0047] Feature mapping layer: During the training process, the high-dimensional visual features of the sample event images and sample event videos are mapped to the language feature space to obtain the high-dimensional visual features of the language feature space, and aligned with the text feature vector to obtain the high-dimensional visual features after text alignment;

[0048] Multimodal joint learning module: During the training process, a joint training method is used to extract the common multimodal features greater than the similarity threshold in the high-dimensional visual features and text feature vectors after text alignment;

[0049] Large language module: during the training process, obtaining semantic prediction results for sample events based on the common multimodal features; and obtaining answer-type text sequences for sample events based on the semantic prediction results;

[0050] Traffic event classification module: during the training process, a traffic event classification result is obtained based on the semantic prediction result;

[0051] The trained traffic event detection system is used to obtain the answer text sequence of the current event and the classification result of the current traffic event based on the current event image or the current event video and the prompt word description text of the current event.

[0052] The traffic incident detection system described in this embodiment can be trained by collecting a large amount of image data to achieve fine-tuning of the overall model when there is less video data. Fine-tuning the multimodal large model using image features can improve its performance in traffic incident perception.

[0053] When using the trained traffic event detection system to perform traffic event recognition and classification, multimodal features are extracted from the traffic event data to be recognized; the multimodal features are input into the trained detection model to obtain traffic event recognition results and achieve classification.

[0054] In this embodiment, the visual encoder and the text encoder form a multimodal feature encoder, which realizes the reasoning and training of multimodal data such as images, videos, and texts, and sets the visual encoder and text encoder respectively for different modal data.

[0055] Furthermore, the method for the visual encoder to obtain high-dimensional visual features of the sample event image and the sample event video includes:

[0056] The visual encoder extracts features from image and video data, which can effectively extract deep semantic features from images and videos. First, the sample event images and sample event videos are processed frame by frame and resized into image blocks of a set size. The linear projection layer is used to convert the image blocks into encoding vectors, and then the self-attention mechanism is used to convert the encoding vectors into high-dimensional visual features.

[0057] For video data, an additional temporal attention mechanism is introduced;

[0058] In the process of processing the encoding vector corresponding to the sample event video by the self-attention mechanism, the low-rank adaptation method of the temporal attention mechanism is adopted to apply temporal embedding to capture the temporal dependencies between video frames and configure the temporal information for the corresponding high-dimensional visual features.

[0059] In this embodiment, the temporal attention layer is initialized from the spatial attention layer through a low-rank adaptation technique, and before each attention layer starts calculation, temporal embedding is applied to capture the temporal dependencies between video frames, so that the visual encoder can better retain the dynamic temporal information and spatial structure information in the visual features.

[0060] Furthermore, the method for obtaining a text feature vector by a text encoder includes: the text encoder adopts a multi-layer Transformer architecture, and through a self-attention mechanism and a feedforward neural network, captures long-distance dependencies and complex semantic structures in the prompt word description text, the text encoder receives natural language text input, represents the prompt word description text with multiple set tags inside the encoder, and then indexes it through a vocabulary to generate a discrete ID sequence, and then converts the discrete ID sequence into a context-sensitive word embedding representation, and the word embedding representation is subjected to multiple rounds of self-attention calculations of the Transformer layer to generate a vector containing contextual semantic information corresponding to each set tag, thereby obtaining a context-sensitive text feature vector corresponding to the prompt word description text. The text feature vector allows fusion with visual features for subsequent multimodal task processing.

[0061] The feature mapping layer and the multimodal joint learning module constitute a multimodal feature mapping module. The feature mapping layer maps the aligned visual features to the text feature space to ensure that multimodal data can be expressed and processed in the language model.

[0062] In this implementation, the method for obtaining the high-dimensional visual features after text alignment in the feature mapping layer includes:

[0063] In order to achieve multimodal feature alignment between images and videos, a feature mapping layer is introduced to map the visual features of images and videos into a unified feature space, thereby converting the high-dimensional visual features generated by the visual encoder into unified features and completing the feature alignment between multimodal features.

[0064] The feature mapping layer uses a multi-layer perceptron (MLP) architecture to map high-dimensional visual feature vectors to language feature space, aligning features of different modalities in the same semantic space and eliminating inconsistencies between images and videos due to modality differences. The MLP architecture uses multiple fully connected layers to perform linear transformations on high-dimensional visual features of sample event images and sample event videos, and combines the ReLU nonlinear activation function to enhance feature extraction, obtaining high-dimensional visual features in the language feature space;

[0065] During the training process, the feature mapping layer optimizes the weights and bias parameters through the back-propagation method, adaptively aligns the high-dimensional visual features and text feature vectors in the language feature space, realizes the mapping between the feature spaces, and obtains the high-dimensional visual features after text alignment.

[0066] The multimodal joint learning module uses a joint training method to simultaneously train image, video and text data on a multimodal dataset. The module introduces a joint training loss function, and by continuously optimizing the joint training loss function during the training process, the model can learn common feature representations between multimodal data.

[0067] As an example, optimization is performed through a contrast loss function. The multimodal joint learning module is trained on a large number of positive sample pairs and negative sample pairs. Assuming that the positive sample pairs have high similarity and the negative sample pairs have low similarity, the higher the similarity, the lower the designed contrast loss function. By continuously optimizing the contrast loss, the module can represent multimodal data in a unified embedding space, improve the semantic similarity of data pairs in the space, and reduce the similarity of unrelated data pairs, thereby realizing the mapping of image-video multimodal features to the unified feature space.

[0068] The big language module and the traffic event classification module form the event perception module, which can process multimodal features and output the classification results of traffic events. The event perception module is connected to the big language module, which can process the text features after multimodal alignment and complete the classification of traffic event categories through linear projection layer and softmax confidence classification. Through this design, the context information output by the big language module can be deeply processed to extract more advanced features. This architecture enables the model to pay attention to important contextual content in the image, such as the location, time, event type and related background information of the traffic event, so as to achieve effective perception of the event type.

[0069] Furthermore, the large language module includes an encoder and a decoder;

[0070] The encoder encodes the common multimodal features through multi-head self-attention, and outputs the semantic prediction results of the sample events through the decoder after fully understanding the semantics of the common multimodal features; and then obtains the answer text sequence of the sample event based on the semantic prediction results.

[0071] The large language module adopts the Transformer architecture, and its encoder includes an embedding layer, a position encoding layer, a multi-head attention layer, and a feedforward neural network layer;

[0072] The common multimodal features are used to generate text embeddings through the embedding layer, and then the position encoding is added through the position encoding layer to retain the order information of the feature sequence and obtain a text sequence with order information; the multi-head attention layer is used to calculate the correlation between text sequences, and the semantic features of multiple texts are processed in parallel to capture the global dependencies of text sequences; the feedforward neural network layer is composed of two linear layers and a nonlinear activation function, which performs a nonlinear transformation on the output of the multi-head attention layer to obtain the encoding result.

[0073] The decoder of the large language module is similar to the encoder structure, but an additional cross-attention layer is designed to align the encoder output. The decoder of the large language module uses an attention layer to align the encoder's encoding results.

[0074] After encoding the context, the event perception module classifies the processed features to determine the specific category of the traffic event. The traffic event classification module classifies the features based on the semantic prediction results to obtain the traffic event classification results.

[0075] In this implementation, the traffic event classification module uses multiple fully connected layers to integrate and classify the semantic prediction results, and uses the Softmax function to calculate the confidence of each category to provide a more intuitive classification result. The confidence calculation is normalized to generate a probability distribution for each event category, and the classification result with high confidence is output as the current traffic event classification result.

[0076] Example:

[0077] Traffic surveillance video reasoning logic design:

[0078] The traffic event detection system receives the traffic video to be analyzed. The first step is to implement video preprocessing, deconstruct the video frame by frame, resample and adjust the video frame size according to the initial setting parameters, and then divide the video frame into patches, so as to generate a fixed number of patch correspondences for each frame impact; the second step is to implement visual encoding, the linear projection layer converts the patch into an embedding vector, at this time the embedding vector is the preliminary feature representation of the input video, add time encoding and perform self-attention calculation, and output a high-dimensional feature vector containing temporal information and deep spatial features; the third step is to implement multimodal mapping, through the video-text multimodal feature alignment module, the generated high-dimensional feature vector is projected to the text feature space to achieve multimodal alignment; the fourth step is to implement event perception, by accessing the large language module to process the aligned multimodal feature vector, the large language module encodes the feature vector, understands its semantics, and outputs its accurate category through linear projection and Softmax classification of the vector encoded by the large language module, and the large language module decoder module can output a detailed description of the event.

[0079] Traffic surveillance video training logic design:

[0080] The traffic event detection system receives a multimodal traffic event dataset in a training environment, which includes a large amount of video and image data and their corresponding text annotation data, where the text annotation data is a detailed description of the corresponding image, in order to improve the model's perception of various traffic events. The video and image data are first preprocessed to generate patches of the corresponding images, and the patches are sent to the linear projection layer, the time encoding layer, the self-attention mechanism layer, and the multimodal feature alignment layer in sequence to generate the initial feature vector; the text encoder receives the text annotation and generates the feature vector corresponding to the text. By introducing the joint learning loss function, the feature vector and the initial feature vector are calculated for loss, and the network parameters of the above layers are updated in reverse, and it is continuously iterated until the model training converges. The event perception module receives a classified text dataset in a training environment, which contains detailed descriptions of traffic events and classified text pairs. The large language module and the event classification module are fine-tuned by Lora, and multiple rounds of training are performed to improve the output accuracy.

[0081] Although the present invention is described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the present invention. It should therefore be understood that many modifications may be made to the exemplary embodiments and that other arrangements may be devised without departing from the spirit and scope of the present invention as defined by the appended claims. It should be understood that the various dependent claims and features described herein may be combined in a manner different from that described in the original claims. It should also be understood that the features described in conjunction with a single embodiment may be used in other described embodiments.

Claims

1. A traffic incident detection system based on multimodal data, characterized in that: include: Visual encoder: During the training process, deep semantic feature extraction is performed on sample event images and sample event videos to obtain high-dimensional visual features of sample event images and sample event videos; Text encoder: During the training process, the prompt word description text of the sample event is converted into a context-sensitive text feature vector; Feature mapping layer: During the training process, the high-dimensional visual features of the sample event images and sample event videos are mapped to the language feature space to obtain the high-dimensional visual features of the language feature space, and aligned with the text feature vector to obtain the high-dimensional visual features after text alignment; Multimodal joint learning module: During the training process, a joint training method is used to extract the common multimodal features greater than the similarity threshold in the high-dimensional visual features and text feature vectors after text alignment; Large language module: during the training process, obtaining semantic prediction results for sample events based on the common multimodal features; and obtaining answer-type text sequences for sample events based on the semantic prediction results; Traffic event classification module: during the training process, a traffic event classification result is obtained based on the semantic prediction result; The trained traffic event detection system is used to obtain the answer text sequence of the current event and the classification result of the current traffic event based on the current event image or the current event video and the prompt word description text of the current event.

2. The traffic incident detection system based on multimodal data according to claim 1, characterized in that: The method for the visual encoder to obtain high-dimensional visual features of sample event images and sample event videos includes: The sample event images and sample event videos are processed frame by frame and resized into image blocks of a set size. The image blocks are converted into encoding vectors using a linear projection layer, and then the encoding vectors are converted into high-dimensional visual features using a self-attention mechanism. In the process of processing the encoding vector corresponding to the sample event video by the self-attention mechanism, the low-rank adaptation method of the temporal attention mechanism is adopted to apply temporal embedding to capture the temporal dependencies between video frames and configure the temporal information for the corresponding high-dimensional visual features.

3. The traffic incident detection system based on multimodal data according to claim 2, characterized in that: The method for a text encoder to obtain a text feature vector includes: the text encoder adopts a multi-layer Transformer architecture, captures long-distance dependencies and complex semantic structures in a prompt word description text through a self-attention mechanism and a feedforward neural network, represents the prompt word description text with multiple set tags, and then indexes it through a vocabulary to generate a discrete ID sequence, and then converts the discrete ID sequence into a context-sensitive word embedding representation, and the word embedding representation is subjected to multiple rounds of self-attention calculations of the Transformer layer to generate a vector containing contextual semantic information corresponding to each set tag, thereby obtaining a context-sensitive text feature vector corresponding to the prompt word description text.

4. The traffic incident detection system based on multimodal data according to claim 3, characterized in that: Methods for obtaining high-dimensional visual features after text alignment in the feature mapping layer include: The feature mapping layer adopts a multi-layer perceptron MLP architecture, uses multiple fully connected layers to perform linear transformation on the high-dimensional visual features of sample event images and sample event videos, and combines the ReLU nonlinear activation function to obtain the high-dimensional visual features of the language feature space; During the training process, the weight and bias parameters are optimized through the back-propagation method, and the high-dimensional visual features and text feature vectors in the language feature space are adaptively aligned to obtain the high-dimensional visual features after text alignment.

5. The traffic incident detection system based on multimodal data according to claim 4 is characterized in that: The multimodal joint learning module is optimized by contrastive loss function.

6. The traffic incident detection system based on multimodal data according to claim 5, characterized in that: The large language module includes an encoder and a decoder; The encoder encodes the common multimodal features through multi-head self-attention, and outputs the semantic prediction results of the sample events through the decoder after fully understanding the semantics of the common multimodal features; and then obtains the answer text sequence of the sample event based on the semantic prediction results.

7. The traffic incident detection system based on multimodal data according to claim 6, characterized in that: The large language module adopts the Transformer architecture, and its encoder includes an embedding layer, a position encoding layer, a multi-head attention layer, and a feedforward neural network layer; The common multimodal features are used to generate text embeddings through the embedding layer, and then the position encoding is added through the position encoding layer to obtain a text sequence with sequential information; the multi-head attention layer is used to calculate the correlation between text sequences, and the semantic features of multiple texts are processed in parallel to capture the global dependencies of text sequences; the feedforward neural network layer performs a nonlinear transformation on the output of the multi-head attention layer to obtain the encoding result.

8. The traffic incident detection system based on multimodal data according to claim 7, characterized in that: The decoder of the large language module uses an attention layer to align the encoding results of the encoder.

9. The traffic incident detection system based on multimodal data according to claim 8, characterized in that: The traffic event classification module performs feature classification based on the semantic prediction result to obtain a traffic event classification result.

10. The traffic incident detection system based on multimodal data according to claim 9, characterized in that: The traffic event classification module uses multiple fully connected layers to integrate and classify the semantic prediction results, and uses the Softmax function to calculate the confidence of each category. The confidence calculation is normalized to generate the probability distribution of each event category, and the classification results with high confidence are output as the current traffic event classification results.

Images