Real-time vehicle collision prediction method based on multi-modal depth fusion and time sequence modeling

Abstract

The invention provides a real-time vehicle collision prediction method based on multi-modal deep fusion and time sequence modeling, and belongs to the cross technical field of computer vision, deep learning, intelligent traffic systems and the like, and the technical scheme is as follows: step 1: video acquisition and frame sequence construction; 2, multi-modal feature extraction is carried out on each frame of image to extract the following four types of features in parallel, and all networks load pre-training weights; 3, feature preprocessing and space alignment are carried out; 4, multi-modal feature enhancement and fusion are carried out; 5, high-level semantic coding is carried out; step 6, time sequence modeling; step 7, collision prediction output; and step 8, loss function design and model training. According to the invention, the accuracy, robustness and real-time performance of collision prediction are improved.

Description

Technical Field

[0001] This invention relates to the interdisciplinary fields of computer vision, deep learning and intelligent transportation systems, and in particular to a real-time vehicle collision prediction method based on multimodal deep fusion and temporal modeling. Background Technology

[0002] With the rapid development of autonomous driving technology and intelligent transportation systems, the autonomous perception and risk warning capabilities of vehicles in dynamic environments have become key factors in ensuring road safety. Especially in the context of increasingly complex urban traffic, accurately identifying potential collision risks and providing early warnings has become a core technical challenge in the design of autonomous driving systems. Traditional collision prediction systems mostly rely on radar, lidar, or a single vision sensor, typically using rule-based collision determination algorithms or simple trajectory extrapolation methods for risk assessment. While these methods are structurally simple, they generally suffer from poor adaptability to dynamic environments, weak target recognition capabilities, and high response latency, making it difficult to meet the dual requirements of real-time performance and accuracy in complex traffic scenarios.

[0003] In recent years, with the widespread application of deep learning and computer vision technologies, research on collision prediction based on image data has made some progress. Some methods attempt to extract static or motion features from RGB video sequences using convolutional neural networks, and further model collision risks through temporal models. However, single-modal data often fails to fully characterize the semantic and geometric features of traffic scenes, and is easily affected by occlusion, lighting changes, or multi-target interference, leading to unstable prediction results or low accuracy. Although optical flow estimation can provide some motion information, it lacks semantic recognition capabilities; depth estimation, while supplementing three-dimensional geometric information, is inherently deficient in scale estimation due to the uncertainty of monocular input; and while object detection and semantic segmentation models can identify key targets, they lack structured interaction and cannot be modeled in a unified manner.

[0004] Existing technologies largely rely on a single visual modality, making it difficult to comprehensively and robustly perceive complex dynamic traffic scenes. Predictive performance significantly degrades when faced with changes in lighting, inclement weather, or target occlusion. Furthermore, existing multimodal fusion strategies often employ simple feature concatenation or weighting, failing to fully exploit the deep structural information within each modality's features and the complex collaborative relationships between modalities, resulting in insufficient feature fusion and low information utilization. Simultaneously, in terms of temporal modeling, traditional structures such as recurrent neural networks (RNNs) have limited ability to capture long-distance temporal dependencies, making it difficult to accurately model the long-term evolution trend of collision risk.

[0005] Meanwhile, some research has begun to explore the introduction of the Transformer architecture into visual perception systems to handle temporal dependencies and enhance the model's ability to perceive dynamic scene evolution. However, existing methods remain imperfect in multimodal feature fusion strategies, struggling to achieve effective alignment and joint modeling of features from different sources. This is particularly problematic in real-time applications, where issues such as high model inference latency and rigid fusion mechanisms persist. Therefore, constructing a collision prediction system that can efficiently extract multi-source visual features, achieve deep fusion and temporal modeling, and meet high accuracy and real-time requirements remains a critical technical challenge in this field. Summary of the Invention

[0006] The purpose of this invention is to provide a real-time vehicle collision prediction method based on multimodal deep fusion and temporal modeling. This method significantly improves prediction accuracy, robustness, and early warning lead time by designing a system capable of deeply extracting and enhancing multimodal visual features, achieving efficient and adaptive feature fusion, and utilizing powerful temporal modeling capabilities to capture the dynamic evolution of the scene. The method integrates multiple visual perception technologies, including optical flow estimation, depth estimation, semantic segmentation, and object detection. By constructing a hybrid architecture of convolutional neural networks and Transformers, it performs real-time and accurate prediction of potential collision events in traffic scenarios. This method can be widely applied in Advanced Driver Assistance Systems (ADAS), autonomous driving systems, and intelligent traffic monitoring systems to achieve core functions such as forward collision warning, traffic risk perception, and driving decision assistance.

[0007] The inventive concept of this invention is as follows: This invention proposes a real-time vehicle collision prediction method based on multimodal deep fusion and temporal modeling, the system architecture of which is as follows: Figure 2The diagram shows an end-to-end deep learning model. This architecture follows a modular design, decomposing the complex prediction task into five core stages: data input, parallel multimodal feature extraction, feature enhancement and fusion, spatiotemporal semantic modeling, and multi-task prediction output. Specifically, the system first constructs a temporal image frame sequence from the video input and feeds it into a parallel multimodal feature extraction front-end. This front-end consists of four independent deep learning models, responsible for extracting optical flow, depth, semantic segmentation, and object-level features, respectively. These heterogeneous feature streams, after being deeply refined by their respective dedicated feature enhancement networks (CDC+LCFI modules), are efficiently integrated into a unified multimodal feature representation by a multi-head attention fusion module. Subsequently, this fused feature is fed into a spatiotemporal semantic modeling back-end composed of a deep convolutional network (ResNeXt101) and a Transformer encoder for high-level abstraction and long-term temporal dependency mining. Ultimately, the spatiotemporally modeled features are distributed to multiple independent prediction heads, which simultaneously output a series of comprehensive risk assessment indicators such as collision probability, collision type, expected collision time (TTC), and risk map in a multi-task learning manner.

[0008] To achieve the aforementioned objectives, the present invention employs the following technical solution: a real-time vehicle collision prediction method based on multimodal deep fusion and temporal modeling, comprising the following steps:

[0009] Step 1: Video Acquisition and Frame Sequence Construction

[0010] The system connects to vehicle-mounted or traffic monitoring cameras, sampling the real-time video stream at 30fps and converting it into a continuous sequence of image frames. To reduce redundancy and computational overhead, the system selects one frame every M frames as a "keyframe," constructing a frame sequence window with a duration of 1-2 seconds. The length of each time window is set to T (e.g., T=8), resulting in an input frame sequence denoted as { , ,..., The frame images are preprocessed uniformly (e.g., resized to 288×512 and normalized to the ImageNet mean) before being sent to the next stage.

[0011] Step 2: Multimodal Feature Extraction

[0012] For each frame of the image, the following four types of features are extracted in parallel, and all networks can load pre-trained weights:

[0013] 2.1) Optical flow extraction: Using the RAFT model, the current frame is input. With the previous frame Output the optical flow map corresponding to the current frame. The size is , representing pixel-level displacements in the horizontal and vertical directions, respectively. Optical flow features can be formally represented as... ,in These represent the displacement components of the pixel in the horizontal and vertical directions, respectively.

[0014] 2.2) Depth estimation: Using the Depth Anything V2 network, input a single frame image. Output the depth map of the current frame. The size is This network uses a ViT backbone to predict dense, scale-stable depth maps, which can be used to estimate target spacing and TTC. To avoid scale differences between different videos, the depth map is usually normalized to... ,in It is a very small constant to ensure numerical stability.

[0015] 2.3) Semantic segmentation: Using the SegFormer-B2 network, the input is a single-frame image. Output semantic feature map This network, pre-trained on the Cityscapes dataset, can recognize 19 categories of traffic semantic information, including vehicles, pedestrians, roads, lane lines, and non-motorized vehicles. The output features are projected onto a 16-dimensional unified feature space using a 1×1 convolution + BN + ReLU. This mapping process can be represented as: ,in Finally, semantic features are obtained.

[0016] The size is This feature serves as input to the multimodal fusion module, used to build a comprehensive scene semantic understanding.

[0017] 2.4) Target Detection: Using the DETR target detector for... Perform object detection and output bounding box coordinates, class labels, and confidence scores. To be consistent with other modalities, the detection results can be encoded as a target heatmap or anchor-free format. ,size .

[0018] Step 3: Feature Preprocessing and Spatial Alignment

[0019] Since the output feature maps of each modality have different sizes and number of channels, a unified size is required to ensure the feasibility of subsequent fusion operations.

[0020] 3.1) All feature maps

[0021] ( , , , All were adjusted to a uniform resolution using bilinear interpolation, for example, H′=240, W′=135;

[0022] 3.2) Use convolutional layers (kernel=1, stride=1) to adjust the number of channels in each modality to a uniform dimension C, such as C=64;

[0023] 3.3) For each frame I t The processing output is: All of them The tensor.

[0024] Step 4: Multimodal Feature Enhancement and Fusion

[0025] After completing spatial alignment, in order to further improve the structural expressiveness of modal features and the semantic consistency of channels, this invention introduces a CDC precoding module and an LCFI channel enhancement module before the attention fusion module to perform multi-scale perception and semantic interaction enhancement on each modal feature.

[0026] 4.1) The CDC (Cascaded Dilated Convolutions) precoding module is designed based on a cascaded structure of multiple convolutional kernels with different dilation rates. For each modality feature map, multiple 3×3 convolutional layers are used, with dilation rates of 1, 2, 4, etc., and then channel fusion is performed after cascading. This module can expand the receptive field without increasing the number of additional parameters, extract local structural features at multiple scales, and adapt to the complex appearance and motion patterns of vehicles, pedestrians, etc. at different scales. Specifically, CDC first applies multi-scale dilated convolutions (such as dilation rates of 1, 2, 4) to the input features, and then concatenates the results and fuses them through 1×1 convolutions to obtain multi-scale contextual features.

[0027] 4.2) The LCFI (Lightweight Channel-wise Feature Interaction) module follows the CDC module, utilizing a channel covariance attention mechanism to model the contextual relevance between channels. This module first performs channel normalization on the input features, then calculates the covariance matrix between each channel, and generates dynamic weights between channels through self-attention or nonlinear mapping, thereby achieving information flow and semantic enhancement along the channel dimension. Specifically, the features are flattened by channel, the covariance matrix between channels is calculated, and then the attention weight matrix between channels is obtained through Softmax. Finally, the original features are weighted to achieve information interaction between channels.

[0028] 4.3) For each modality, the CDC and LCFI modules are concatenated to form a "feature enhancement sub-network," whose output is used to replace the original modality feature input and is uniformly fed into the multi-head attention fusion module. This module adopts a Transformer structure, using the global scene semantic vector as the query and the enhanced features of each modality as the key and value. It calculates the attention weights and performs the fusion operation to obtain the multi-modality fused feature map. The size is .

[0029] Step 5: High-level semantic encoding

[0030] Multimodal feature map fusion The input is a deep semantic encoding network, from which global abstract representations are extracted. In this stage, the classic ResNeXt101 network is used to achieve high-level spatial modeling and abstract representation of the fused features through a deep residual structure. After local enhancement by the CDC + LCFI module, the encoder further integrates global contextual information, providing a stable and discriminative semantic representation for subsequent temporal modeling modules. The fused semantic features output in this stage are denoted as... Its size is ,in and For spatial dimensions, This is for the channel dimension. To further adapt to subsequent time-series modeling modules, [the following will be implemented / implemented]. After performing global average pooling, the frame-level semantic feature vector can be represented as:

[0031]

[0032] Compressed in spatial dimension to a length of a one-dimensional vector As frame-level semantic representation features, they are input into the Transformer encoder and participate in the multi-frame temporal modeling process.

[0033] Step 6: Timing Modeling

[0034] To capture the dynamic evolution of vehicle collision risk over time, this invention introduces a temporal modeling module based on the Transformer architecture after feature fusion encoding. This module performs deep temporal modeling on the fused semantic feature sequence of consecutive frames. The module includes the following specific sub-steps:

[0035] 6.1) Feature Sequence Construction: Perform fusion feature extraction and high-level semantic coding operations sequentially on T frames of video images to obtain a frame-level fusion semantic feature set. Each for 3D feature map. Global average pooling is used to divide each... Compressed into a D-dimensional vector , forming a time series .

[0036] 6.2) Position Encoding Addition: To preserve temporal order information, this invention adds position encoding to the frame feature sequence. Position encoding can employ sine / cosine position encoding or learnable position embedding, integrating time... Mapped to vector ,and By adding elements one by one, we obtain the position-aware feature sequence. .

[0037] 6.3) Transformer Encoder Modeling: The sequence with added positional encoding is input into a multi-layer Transformer encoder structure. Each encoder layer includes a multi-head self-attention mechanism, a feedforward fully connected network, residual connections, and layer normalization. Each Transformer layer contains multi-head attention and a feedforward network, with residual connections and layer normalization added between the two parts. Long-range inter-frame dependencies are modeled through multi-layer stacking, extracting temporal contextual feature representations. .

[0038] 6.4) Timing Aggregation and Output:

[0039] After completing temporal modeling, the system processes the temporal features of each frame independently to achieve continuous prediction and visualization. Specifically, for the feature sequence output by the Transformer... Preserve the output vector of each frame. The values ​​are input into the collision prediction module to generate a frame-level collision probability sequence:

[0040] The aforementioned probability sequence will be used to generate a "collision probability-time" curve in real time, enabling continuous monitoring and trend presentation of vehicle collision risk. This method is particularly suitable for sliding window prediction and system-embedded visualization terminals.

[0041] In certain static prediction or offline analysis scenarios, it is also possible to Perform temporal aggregation, such as through average pooling, max pooling, or by introducing a [CLS] class identifier vector mechanism, to obtain a global representation vector. As a semantic summary of the entire video segment, it is used for tasks such as unified event-level collision type determination and TTC estimation.

[0042] Step 7: Collision Prediction Output

[0043] After completing the temporal modeling, the obtained context-enhanced features will be input into the multi-task collision prediction module. This module is based on a multi-head output structure and performs the following sub-tasks:

[0044] 7.1) Collision probability prediction

[0045] The temporal feature sequence output by the Transformer encoder Each frame represents Input to the accident probability prediction branch, output the collision probability for each frame. The predicted probability is obtained by the Sigmoid function, i.e.

[0046] The prediction results for all frames constitute a collision risk time series. It is used to generate an "accident probability-time" curve, enabling real-time visualization of collision risk in video sequences.

[0047] 7.2) Collision Type Classification

[0048] Through another fully connected network Mapped to a predefined collision type space (e.g., frontal collision, rear-end collision, side collision, no collision), outputting multi-class classification results. The probability distribution is normalized using the Softmax function, and the category corresponding to the highest probability value is selected as the collision type prediction result.

[0049] 7.3) Expected Time-To-Collapse (TTC)

[0050] Use regression branch pairs A linear transformation is performed to output the estimated time-to-collision (TTC) between the vehicle and the target, which is a positive real number in seconds. This prediction helps in enabling early braking or decision-making intervention. To ensure that the predicted value is always positive, the regression output of TTC can be constrained using the softplus or ReLU functions.

[0051] 7.4) Risk Spatial Positioning

[0052] Use the spatial prediction branch to extract the features of each frame output by the Transformer. Reverse mapping to image space, then restoring it through upsampling and convolution. A heatmap or bounding box probability map is used to mark the regions where collisions are most likely to occur in the current frame, and the output is a risk region map. .

[0053] The module structure can be adjusted to change the prediction dimensions or output format according to actual deployment needs, and supports multi-frame prediction and real-time sliding window updates.

[0054] Step 8: Loss Function Design and Model Training

[0055] To achieve multi-objective collaborative optimization and enhance model robustness, this invention employs a multi-task joint loss function during the training phase, specifically including the following sub-items:

[0056] 8.1) Collision probability prediction loss ( )

[0057] A binary cross-entropy loss function is used to measure the deviation between the predicted probability and the true label.

[0058] 8.2) Collision type classification loss ( )

[0059] Multi-class cross-entropy loss is used to supervise collision type learning and improve classification accuracy.

[0060] 8.3) TTC Time Regression Loss ( )

[0061] Using a smoothed L1 loss function or an L2 loss function, the difference between the predicted time and the actual collision time is penalized, encouraging the model to stably predict the time interval.

[0062] 8.4) Spatial location loss ( )

[0063] For heatmap-type outputs, pixel-level MSE loss is used; for bounding box-type outputs, IoU or GIoU loss is used to optimize spatial positioning accuracy.

[0064] 8.5) Temporal consistency loss ( )

[0065] Introducing an inter-frame prediction smoothing regularization term limits drastic changes in output between adjacent frames, improving the consistency and stability of continuous video prediction.

[0066] 8.6) Total Loss Function Form

[0067]

[0068] in These are adjustable hyperparameters used to balance the optimization importance of each subtask.

[0069] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0070] 1. This invention proposes a video vehicle collision prediction method based on multimodal fusion and temporal modeling. In autonomous driving and traffic safety monitoring scenarios, it can effectively fuse information from multiple modalities in the video (such as optical flow, depth, segmentation, and detection), and achieve more accurate, stable, and visualized collision prediction through structural optimization.

[0071] 2. This invention, based on traditional multimodal fusion, introduces CDC (Continuously Dilated Convolution) and LCFI (Local-Contextual Channel Interaction) modules to perform multi-scale structural extraction and channel context enhancement for each modality feature. By achieving more refined structural representation and global modeling at the spatial feature level, the fusion quality is improved, and the system's adaptability to complex dynamic scenes (such as high-speed lane changes, sudden braking, occlusion, etc.) is enhanced, thereby significantly improving the accuracy of collision prediction.

[0072] 3. This invention employs a Transformer-based temporal modeling structure, utilizing a multi-head attention mechanism to capture long-term dependencies between consecutive frames and uncover potential risk evolution trends. Compared to traditional RNN or LSTM methods, this structure possesses stronger feature representation capabilities in the temporal dimension, enabling it to model factors such as changes in target approach speed and behavioral trends, effectively improving prediction stability and generalization ability, and reducing false positives and false negatives.

[0073] 4. This invention supports frame-by-frame prediction of accident probabilities and outputs a complete "collision probability-time" change curve. This output can not only be used for real-time risk warning, but also facilitates reference by the driving decision module, or for locating risk change points in system backtracking analysis. It has good human-machine readability and visualization effect, and is particularly suitable for deployment in ADAS or intelligent transportation platforms.

[0074] 5. The method of this invention features a flexible and modular design, supporting different modal feature configurations, different temporal window lengths, and different prediction target dimensions, adapting to various pre-installed or post-installed intelligent video processing platforms. Its CDC+LCFI module and Transformer structure can be pruned or replaced according to computing resources, achieving efficient deployment from the cloud to the edge. It supports real-time sliding window prediction or offline batch processing analysis, possessing good application portability and scalability.

[0075] 6. Compared with traditional methods that rely solely on a single modality, static features, or inefficient temporal structures, this invention fully utilizes multi-source information, cross-modal enhancement, and temporal context modeling. It still performs well under conditions such as occlusion, lighting changes, and high scene complexity, improving the system's ability to detect potential collisions in unstructured environments and effectively compensating for the shortcomings of existing video analysis methods in terms of robustness and visual interpretation. Attached Figure Description

[0076] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.

[0077] Figure 1 This is a schematic diagram of the overall process of the method of the present invention.

[0078] Figure 2 This is a system architecture diagram of the end-to-end prediction model of the present invention.

[0079] Figure 3 This is a visualization of the multimodal feature extraction and collision probability prediction results of the present invention in a dangerous scenario. Detailed Implementation

[0080] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. Of course, the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0081] Example 1

[0082] See Figure 1 and Figure 3 This embodiment provides a technical solution: a real-time vehicle collision prediction method based on multimodal deep fusion and temporal modeling, comprising the following steps:

[0083] Step 1: Video Acquisition and Frame Sequence Construction

[0084] The system connects to vehicle-mounted or traffic monitoring cameras, samples the real-time video stream at a frame rate of 30fps, and converts it into a continuous image frame sequence. To reduce redundancy and computational overhead, the system selects one frame every M frames as a "keyframe," constructing a frame sequence window with a duration of 1-2 seconds. The length of each time window is set to T (e.g., T=8), and the resulting input frame sequence is denoted as { , ,..., After uniform preprocessing (such as resizing to 288×512 and normalizing to the ImageNet mean), the frame images are sent to the next stage.

[0085] Step 2: Multimodal Feature Extraction

[0086] For each frame of the image, the following four types of features are extracted in parallel, and all networks can load pre-trained weights:

[0087] 2.1) Optical flow extraction: Using the RAFT model, the current frame is input. With the previous frame Output the optical flow map corresponding to the current frame. The size is , representing pixel-level displacements in the horizontal and vertical directions, respectively. Optical flow features can be formally represented as... ,in These represent the displacement components of the pixel in the horizontal and vertical directions, respectively.

[0088] 2.2) Depth estimation: Using the Depth Anything V2 network, input a single frame image. Output the depth map of the current frame. The size is This network employs a ViT backbone to predict dense, scale-stable depth maps, which can be used to estimate target spacing and TTC. To avoid scale differences between different videos, the depth map is typically normalized to... ,in It is a very small constant to ensure numerical stability.

[0089] 2.3) Semantic segmentation: Using the SegFormer-B2 network, the input is a single-frame image. Output semantic feature map This network, pre-trained on the Cityscapes dataset, can recognize 19 categories of traffic semantic information, including vehicles, pedestrians, roads, lane lines, and non-motorized vehicles. The output features are projected onto a 16-dimensional unified feature space using a 1×1 convolution + BN + ReLU. This mapping process can be represented as: ,in Finally, semantic features are obtained.

[0090] The size is This feature serves as input to the multimodal fusion module, used to build a comprehensive scene semantic understanding.

[0091] 2.4) Target Detection: Using the DETR target detector for... Perform object detection, and output includes bounding box coordinates, class labels, and confidence scores. To be consistent with other modalities, the detection results can be encoded as an object heatmap or anchor-free format. ,size .

[0092] Step 3: Feature Preprocessing and Spatial Alignment

[0093] Since the output feature maps of each modality have different sizes and number of channels, a unified size is required to ensure the feasibility of subsequent fusion operations.

[0094] 3.1) All feature maps

[0095] ( , , , All were adjusted to a uniform resolution using bilinear interpolation, for example, H′=240, W′=135.

[0096] 3.2) Use convolutional layers (kernel=1, stride=1) to adjust the number of channels in each modality to a uniform dimension C (e.g., C=64).

[0097] 3.3) For each frame I t The processing output is: All of them The tensor.

[0098] Step 4: Multimodal Feature Enhancement and Fusion

[0099] After completing spatial alignment, in order to further improve the structural expressiveness of modal features and the semantic consistency of channels, this invention introduces a CDC precoding module and an LCFI channel enhancement module before the attention fusion module to perform multi-scale perception and semantic interaction enhancement on each modal feature.

[0100] 4.1) The CDC (Cascaded Dilated Convolutions) precoding module is designed based on a cascaded structure of multiple convolutional kernels with different dilation rates. For each modality feature map, multiple 3×3 convolutional layers are used, with dilation rates of 1, 2, 4, etc., and then channel fusion is performed after cascading. This module can expand the receptive field without increasing the number of additional parameters, extract local structural features at multiple scales, and adapt to the complex appearance and motion patterns of vehicles, pedestrians, etc. at different scales. Specifically, CDC first applies multi-scale dilated convolutions (such as dilation rates of 1, 2, 4) to the input features, and then concatenates the results and fuses them through 1×1 convolutions to obtain multi-scale contextual features.

[0101] 4.2) The LCFI (Lightweight Channel-wise Feature Interaction) module follows the CDC module, utilizing a channel covariance attention mechanism to model the contextual relevance between channels. This module first performs channel normalization on the input features, then calculates the covariance matrix between each channel, and generates dynamic weights between channels through self-attention or nonlinear mapping, thereby achieving information flow and semantic enhancement along the channel dimension. Specifically, the features are flattened by channel, the covariance matrix between channels is calculated, and then the attention weight matrix between channels is obtained through Softmax. Finally, the original features are weighted to achieve information interaction between channels.

[0102] 4.3) For each modality, the CDC and LCFI modules are concatenated to form a "feature enhancement sub-network," whose output is used to replace the original modality feature input and is uniformly fed into the multi-head attention fusion module. This module adopts a Transformer structure, using the global scene semantic vector as the query and the enhanced features of each modality as the key and value. It calculates the attention weights and performs the fusion operation to obtain the multi-modality fused feature map. The size is .

[0103] Step 5: High-level semantic encoding

[0104] Multimodal feature map fusion The input is a deep semantic encoding network, from which global abstract representations are extracted. In this stage, the classic ResNeXt101 network is used to achieve high-level spatial modeling and abstract representation of the fused features through a deep residual structure. After local enhancement by the CDC + LCFI module, the encoder further integrates global contextual information, providing a stable and discriminative semantic representation for subsequent temporal modeling modules. The fused semantic features output in this stage are denoted as... Its size is ,in and For spatial dimensions, This is for the channel dimension. To further adapt to subsequent time-series modeling modules, [the following will be implemented / implemented]. After performing global average pooling, the frame-level semantic feature vector can be represented as:

[0105]

[0106] Compressed in spatial dimension to a length of a one-dimensional vector As frame-level semantic representation features, they are input into the Transformer encoder and participate in the multi-frame temporal modeling process.

[0107] Step 6: Timing Modeling

[0108] To capture the dynamic evolution of vehicle collision risk over time, this invention introduces a temporal modeling module based on the Transformer architecture after feature fusion encoding. This module performs deep temporal modeling on the fused semantic feature sequence of consecutive frames. The module includes the following specific sub-steps:

[0109] 6.1) Feature Sequence Construction: Perform fusion feature extraction and high-level semantic coding operations sequentially on T frames of video images to obtain a frame-level fusion semantic feature set. Each for 3D feature map. Global average pooling is used to divide each... Compressed into a D-dimensional vector , forming a time series .

[0110] 6.2) Position Encoding Addition: To preserve temporal order information, this invention adds position encoding to the frame feature sequence. Position encoding can employ sine / cosine position encoding or learnable position embedding, integrating time... Mapped to vector ,and By adding elements one by one, we obtain the position-aware feature sequence. .

[0111] 6.3) Transformer Encoder Modeling: The sequence with added positional encoding is input into a multi-layer Transformer encoder structure. Each encoder layer includes a multi-head self-attention mechanism, a feedforward fully connected network, residual connections, and layer normalization. Each Transformer layer contains multi-head attention and a feedforward network, with residual connections and layer normalization added between the two parts. Long-range inter-frame dependencies are modeled through multi-layer stacking, extracting temporal contextual feature representations. .

[0112] 6.4) Timing Aggregation and Output:

[0113] After completing temporal modeling, the system processes the temporal features of each frame independently to achieve continuous prediction and visualization. Specifically, for the feature sequence output by the Transformer... Preserve the output vector of each frame. The values ​​are input into the collision prediction module to generate a frame-level collision probability sequence:

[0114] The aforementioned probability sequence will be used to generate a "collision probability-time" curve in real time, enabling continuous monitoring and trend presentation of vehicle collision risk. This method is particularly suitable for sliding window prediction and system-embedded visualization terminals.

[0115] In certain static prediction or offline analysis scenarios, it is also possible to Perform temporal aggregation, such as through average pooling, max pooling, or by introducing a [CLS] class identifier vector mechanism, to obtain a global representation vector. As a semantic summary of the entire video segment, it is used for tasks such as unified event-level collision type determination and TTC estimation.

[0116] Step 7: Collision Prediction Output

[0117] After completing the temporal modeling, the obtained context-enhanced features will be input into the multi-task collision prediction module. This module is based on a multi-head output structure and performs the following sub-tasks:

[0118] 7.1) Collision probability prediction

[0119] The temporal feature sequence output by the Transformer encoder Each frame represents Input to the accident probability prediction branch, output the collision probability for each frame. The predicted probability is obtained by the Sigmoid function, i.e.

[0120] The prediction results for all frames constitute a collision risk time series. It is used to generate an "accident probability-time" curve, enabling real-time visualization of collision risk in video sequences.

[0121] 7.2) Collision Type Classification

[0122] Through another fully connected network Mapped to a predefined collision type space (e.g., frontal collision, rear-end collision, side collision, no collision), outputting multi-class classification results. The probability distribution is normalized using the Softmax function, and the category corresponding to the highest probability value is selected as the collision type prediction result.

[0123] 7.3) Expected Time-To-Collapse (TTC)

[0124] Use regression branch pairs A linear transformation is performed to output the estimated time-to-collision (TTC) between the vehicle and the target, which is a positive real number in seconds. This prediction helps in enabling early braking or decision-making intervention. To ensure that the predicted value is always positive, the regression output of TTC can be constrained using the softplus or ReLU functions.

[0125] 7.4) Risk Spatial Positioning

[0126] Use the spatial prediction branch to extract the features of each frame output by the Transformer. Reverse mapping to image space, then restoring it through upsampling and convolution. A heatmap or bounding box probability map is used to mark the regions where collisions are most likely to occur in the current frame, and the output is a risk region map. .

[0127] The module structure can be adjusted to change the prediction dimensions or output format according to actual deployment needs, and supports multi-frame prediction and real-time sliding window updates.

[0128] Step 8: Loss Function Design and Model Training

[0129] To achieve multi-objective collaborative optimization and enhance model robustness, this invention employs a multi-task joint loss function during the training phase, specifically including the following sub-items:

[0130] 8.1) Collision probability prediction loss ( )

[0131] A binary cross-entropy loss function is used to measure the deviation between the predicted probability and the true label.

[0132] 8.2) Collision type classification loss ( )

[0133] Multi-class cross-entropy loss is used to supervise collision type learning and improve classification accuracy.

[0134] 8.3) TTC Time Regression Loss ( )

[0135] Using a smoothed L1 loss function or an L2 loss function, the difference between the predicted time and the actual collision time is penalized, encouraging the model to stably predict the time interval.

[0136] 8.4) Spatial location loss ( )

[0137] For heatmap-type outputs, pixel-level MSE loss is used; for bounding box-type outputs, IoU or GIoU loss is used to optimize spatial positioning accuracy.

[0138] 8.5) Temporal consistency loss ( )

[0139] Introducing an inter-frame prediction smoothing regularization term limits drastic changes in output between adjacent frames, improving the consistency and stability of continuous video prediction.

[0140] 8.6) Total Loss Function Form

[0141]

[0142] in These are adjustable hyperparameters used to balance the optimization importance of each subtask.

[0143] Example 2

[0144] Performance comparison and validation with baseline methods and ablation models:

[0145] To verify the effectiveness and advancement of the method described in this invention (hereinafter referred to as "this method"), a comparative experiment was conducted on the publicly available large-scale autonomous driving dataset Waymo Open Dataset. We selected the F1 score for collision prediction, the mean absolute error (MAE) of the predicted time to collision (TTC), and the accuracy of collision type classification as the core evaluation metrics.

[0146] The comparison methods include:

[0147] 1. Single-modal (RGB only): Baseline model that uses only RGB frames of video as input.

[0148] 2. Early Concat: The basic fusion strategy is to simply concatenate the four modal features extracted in Example 1 along the channel dimension and then feed them into the subsequent network.

[0149] 3. This method (ablation version): The CDC and LCFI feature enhancement modules in step 4 of Example 1 were removed to verify its effectiveness.

[0150] 4. This method (complete solution): adopts the complete technical solution described in Example 1.

[0151] Experimental results

[0152] The performance comparison of each model on the test set is shown in Table 1 below:

[0153] Table 1: Performance Comparison of Different Methods

[0154]

[0155] Conclusion Analysis

[0156] From the comparison data in Table 1, we can conclude that:

[0157] 1. The complete scheme of this method significantly outperforms other comparative methods in all key indicators, demonstrating the advanced nature of its overall design.

[0158] 2. Compared to the simple “early fusion” strategy, our method (ablation version) has achieved significant performance improvement through attention-based deep fusion, demonstrating the importance of a refined fusion mechanism.

[0159] 3. Compared with the ablation version, the complete solution improved the F1 score by 2.6% and reduced the TTC error by 25%, which fully demonstrates that the CDC multi-scale perception module and the LCFI channel interaction module play a key role in improving feature quality and thus enhancing the final prediction accuracy.

[0160] The comparative experimental results of this embodiment show that the technical solution proposed in this invention has significant advantages in prediction accuracy and multi-dimensional task processing, and is the preferred solution for achieving high-precision real-time vehicle collision prediction.

[0161] Example 3

[0162] Robustness comparison verification in complex simulation scenarios

[0163] To further verify the stability and reliability of this method under adverse weather or lighting conditions, this embodiment uses the CARLA simulation platform to generate simulation datasets containing three typical challenging scenarios: nighttime, heavy rain, and dense fog, and compares them with the better-performing "early fusion (Concat)" scheme.

[0164] Experimental results

[0165] The performance comparison of each model under different simulation scenarios is shown in Table 2 below, where the values ​​in parentheses represent the performance degradation or error increase compared to the normal weather baseline.

[0166] Table 2: Performance Comparison in Complex Simulation Scenarios

[0167]

[0168] Conclusion Analysis

[0169] The simulation results in Table 2 show that:

[0170] 1. In all test scenarios, the performance of this method is comprehensively superior to the baseline method.

[0171] 2. While the performance of all models deteriorates as the environment worsens, the performance degradation of our method is significantly smaller. For example, in nighttime scenes, the F1 score of the baseline method drops by 7.4 percentage points, while our method only drops by 4.6 percentage points.

[0172] 3. This result strongly demonstrates that, through deep fusion and enhancement of multimodal features, the present invention can more effectively utilize complementary information from other modalities (such as depth and optical flow) when the information quality of one modality (such as RGB image) is severely degraded, thereby suppressing the overall performance degradation and exhibiting stronger environmental robustness.

[0173] Therefore, the conclusion of this embodiment is that the technical solution of the present invention not only has excellent performance in standard environments, but also has high stability and reliability in complex and ever-changing scenarios, and has strong practical application value.

[0174] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A real-time vehicle collision prediction method based on multimodal deep fusion and temporal modeling, characterized in that, Includes the following steps: Step 1: Video Acquisition and Frame Sequence Construction The system connects to vehicle-mounted or traffic monitoring cameras, samples the real-time video stream at a frame rate of 30fps, and converts it into a continuous image frame sequence. The system selects one frame every M frames as a "keyframe," constructing a frame sequence window with a duration of 1-2 seconds. The length of each time window is set to T, resulting in the input frame sequence denoted as { , , ..., After unified preprocessing, the frame images are sent to the next stage; Step 2: Multimodal Feature Extraction For each frame of the image, the following four types of features are extracted in parallel, and all networks are loaded with pre-trained weights: Step 3: Feature Preprocessing and Spatial Alignment Since the output feature maps of each modality have different sizes and number of channels, a uniform size is used to ensure the feasibility of subsequent fusion operations; Step 4: Multimodal Feature Enhancement and Fusion After spatial alignment is completed, a CDC precoding module and an LCFI channel enhancement module are introduced before the attention fusion module to perform multi-scale perception and semantic interaction enhancement on each modality feature, respectively. Step 5: High-level semantic encoding Multimodal feature map fusion The input is a deep semantic encoding network, which extracts a global abstract representation. At this stage, the classic ResNeXt101 network is used, employing a deep residual structure to achieve high-level spatial modeling and abstract representation of the fused features. After local enhancement by the CDC + LCFI module, this encoder further integrates global contextual information, providing a stable and discriminative semantic representation for subsequent temporal modeling modules. The fused semantic features output at this stage are denoted as... Its size is ,in and For spatial dimensions, For the channel dimension, After performing global average pooling, the frame-level semantic feature vector is represented as follows: ； Compressed in spatial dimension to a length of a one-dimensional vector As frame-level semantic representation features, they are input into the Transformer encoder and participate in the multi-frame temporal modeling process; Step 6: Timing Modeling After the fusion feature encoding, a temporal modeling module based on the Transformer structure is introduced to perform deep modeling of the fusion semantic feature sequence of consecutive frames in the temporal dimension; Step 7: Collision Prediction Output After completing the temporal modeling, the obtained context-enhanced features will be input into the multi-task collision prediction module, which is based on a multi-head output structure. Step 8: Loss Function Design and Model Training A multi-task joint loss function is used during the training phase.

2. The real-time vehicle collision prediction method based on multimodal deep fusion and temporal modeling according to claim 1, characterized in that, Step 2 includes the following steps: 2.1) Optical flow extraction: Using the RAFT model, the current frame is input. With the previous frame Output the optical flow map corresponding to the current frame. The size is , representing pixel-level displacements in the horizontal and vertical directions respectively, and the optical flow feature is formally represented as: ,in These are the horizontal and vertical displacement components of the pixel, respectively. 2.2) Depth estimation: Using the Depth Anything V2 network, input a single frame image. Output the depth map of the current frame. The size is The network employs a ViT backbone to predict densely scale-stable depth maps, which are used to estimate target spacing and TTC. The depth map is normalized to... ,in It is a very small constant to ensure numerical stability; 2.3) Semantic segmentation: Using the SegFormer-B2 network, the input is a single-frame image. Output semantic feature map The network is pre-trained on the Cityscapes dataset and identifies semantic information about vehicles, pedestrians, roads, lane lines, and non-motorized vehicles. The output features are projected onto a 16-dimensional unified feature space through 1×1 convolution + BN + ReLU. This mapping process is represented as follows: ,in Finally, semantic features are obtained. The size is This feature serves as input to the multimodal fusion module, used to construct a comprehensive scene semantic understanding; 2.4) Target Detection: Using the DETR target detector for... Perform object detection, and output the bounding box coordinates, class labels, and confidence scores. The detection results are encoded as a target heatmap or anchor-free format. ,size ; Step 3: Feature Preprocessing and Spatial Alignment Since the output feature maps of each modality have different sizes and number of channels, a unified size is used to ensure the feasibility of subsequent fusion operations.

3. The real-time vehicle collision prediction method based on multimodal deep fusion and temporal modeling according to claim 1, characterized in that, Step 3 includes the following steps: 3.1) All feature maps ( , , , All were adjusted to a uniform resolution using bilinear interpolation. 3.2) Use convolutional layers (kernel=1, stride=1) to adjust the number of channels in each modality to a uniform dimension C; 3.3) For each frame I t The processing output is: All of them tensor; Step 4: Multimodal Feature Enhancement and Fusion After spatial alignment is completed, a CDC precoding module and an LCFI channel enhancement module are introduced before the attention fusion module to perform multi-scale perception and semantic interaction enhancement on each modality feature, respectively.

4. The real-time vehicle collision prediction method based on multimodal deep fusion and temporal modeling according to claim 1, characterized in that, Step 4 includes the following steps: 4.1) The design of the CDC precoding module is based on a concatenated structure of multiple convolutional kernels with different dilation rates. For each modality feature map, multiple 3×3 convolutional layers are used with dilation rates of 1, 2, and 4. After concatenation, channel fusion is performed. This module expands the receptive field without increasing the number of additional parameters, extracts local structural features at multiple scales, and adapts to the complex appearance and motion patterns of vehicles and pedestrians at different scales. CDC first applies multi-scale dilated convolution to the input features, then concatenates the results and fuses them through 1×1 convolution to obtain multi-scale contextual features. 4.2) The LCFI channel enhancement module is applied after the CDC module. It uses the channel covariance attention mechanism to model the contextual correlation between channels. This module first performs channel normalization on the input features, then calculates the covariance matrix between each channel, and generates dynamic weights between channels through self-attention or nonlinear mapping. The features are flattened by channel, the covariance matrix between channels is calculated, and then the attention weight matrix between channels is obtained through Softmax. Finally, the original features are weighted to realize information interaction between channels. 4.3) For each modality, the CDC and LCFI modules are concatenated to form a "feature enhancement sub-network," whose output is used to replace the original modality feature input and is uniformly fed into the multi-head attention fusion module. This module adopts a Transformer structure, using the global scene semantic vector as the query and the enhanced features of each modality as the key and value, to calculate the attention weights and perform the fusion operation to obtain the multi-modality fused feature map. The size is .

5. The real-time vehicle collision prediction method based on multimodal deep fusion and temporal modeling according to claim 1, characterized in that, Step 6 includes the following steps: 6.1) Feature Sequence Construction: Perform fusion feature extraction and high-level semantic coding operations sequentially on T frames of video images to obtain a frame-level fusion semantic feature set. Each for The feature map is divided into 3D features, and global average pooling is used to divide each feature map into 3D features. Compressed into a D-dimensional vector , forming a time series ; 6.2) Adding location encoding: Add positional encoding to the frame feature sequence. The positional encoding can be either sine or cosine positional encoding or learnable positional embedding, and time... Mapped to vector ,and By adding elements one by one, we obtain the position-aware feature sequence. ; 6.3) Transformer encoder modeling: The sequence with added positional encoding is input into a multi-layer Transformer encoder structure. Each encoder layer includes a multi-head self-attention mechanism, a feedforward fully connected network, residual connections, and layer normalization. Each Transformer layer contains multi-head attention and a feedforward network, with residual connections and layer normalization added between the two parts. By stacking multiple layers, long-range inter-frame dependencies are modeled, and temporal contextual feature representations are extracted. ; 6.4) Timing Aggregation and Output: After completing the temporal modeling, the system processes the temporal features of each frame independently to achieve continuous prediction and visualization. This is for the feature sequence output by the Transformer. Preserve the output vector of each frame. The values ​​are input into the collision prediction module to generate a frame-level collision probability sequence: ; In certain static prediction or offline analysis scenarios, Temporal aggregation is performed, and a global representation vector is obtained through average pooling, max pooling, or the introduction of a [CLS] class identifier vector mechanism. As a semantic summary of the entire video segment, it is used to unify event-level collision type judgment and TTC estimation tasks.

6. The real-time vehicle collision prediction method based on multimodal deep fusion and temporal modeling according to claim 1, characterized in that, Step 7 includes the following steps: 7.1) Collision probability prediction The temporal feature sequence output by the Transformer encoder Each frame represents Input to the accident probability prediction branch, output the collision probability for each frame. The predicted probability is obtained by the Sigmoid function, i.e. The prediction results of all frames constitute a collision risk time series. It is used to generate "accident probability-time" curves, enabling real-time visualization of collision risks in video sequences; 7.2) Collision Type Classification Through another fully connected network Mapped to a predefined collision type space, outputting multi-class classification results. The probability distribution is normalized using the Softmax function, and the category corresponding to the highest probability value is selected as the collision type prediction result. 7.3) Expected collision time to return to TTC Use regression branch pairs A linear transformation is performed to output the predicted time of collision (TTC) between the vehicle and the target, which is a positive real number in seconds. This prediction helps to enable early braking or decision intervention. To ensure that the predicted value is always positive, the regression output of TTC is constrained by the softplus or ReLU function. 7.4) Risk Spatial Positioning Use the spatial prediction branch to extract the features of each frame output by the Transformer. Reverse mapping to image space, then restoring it through upsampling and convolution. A heatmap or bounding box probability map is used to mark the collision locations in the current frame, and the output is a risk area map. ; Step 8: Loss Function Design and Model Training A multi-task joint loss function is used during the training phase, including the following sub-items.

7. The real-time vehicle collision prediction method based on multimodal deep fusion and temporal modeling according to claim 1, characterized in that, Step 8 includes the following steps: 8.1) Collision probability prediction loss ( ); The binary cross-entropy loss function is used to measure the deviation between the predicted probability and the true label; 8.2) Collision type classification loss ( ); Multi-class cross-entropy loss is used for supervised learning of collision types; 8.3) TTC Time Regression Loss ( ); Using a smoothed L1 loss function or an L2 loss function, the difference between the predicted time and the actual collision time is penalized to encourage the model to make stable predictions of the time interval. 8.4) Spatial location loss ( ); For heatmap-type outputs, pixel-level MSE loss is used; for bounding box-type outputs, IoU or GIoU loss is used to optimize spatial positioning accuracy. 8.5) Temporal consistency loss ( ); Introduce an inter-frame prediction smoothing regularization term to limit drastic changes in output between adjacent frames; 8.6) Total Loss Function Form ； in These are adjustable hyperparameters used to balance the optimization importance of each subtask.

Images