Traffic incident detection method based on deep spatiotemporal memory and interactive network

Abstract

This invention discloses a traffic event detection method based on deep spatiotemporal memory and interactive networks. First, a spatiotemporal information perception network is designed, which can effectively capture the core semantics from small to large spatial regions, as well as short-term features in the temporal domain, from input video frames, improving the feature extraction capability of video frames and enabling more accurate classification of traffic events. Second, a temporal feature learning network is proposed to address the problem that the spatiotemporal information perception network fails to fully capture long-term temporal information when processing video frames. This method, by strengthening the learning of temporal features, can obtain more complete contextual temporal information features, improving the accurate localization of the occurrence and termination times of anomalies.

Description

Technical Field

[0001] This invention belongs to the field of intelligent video processing, and in particular relates to a traffic incident detection method based on deep spatiotemporal memory and interactive networks. Background Technology

[0002] Autonomous vehicles still face significant challenges in accident detection and prediction, a problem that has garnered widespread attention in both industry and academia. In recent years, the widespread adoption of dashboard cameras has made them an important tool for recording accident scenes. This unique advantage not only enhances the ability to record accident scene data but also provides richer sample data for training deep learning models. Models based on dashcams will become more reliable, thereby significantly improving the decision-making capabilities of autonomous vehicles in accident scenarios.

[0003] However, the characteristics of video captured by dashcams differ significantly from those of data from highway or traffic light surveillance cameras. Dashcams record traffic video in a horizontal view, where the camera and surrounding objects are in motion, increasing the complexity of traffic incident detection, especially when classifying objects approaching the dashcam and traffic participants. Therefore, identifying traffic incidents (including anomalies involving vehicles and pedestrians) in natural driving videos captured by in-vehicle cameras is particularly important. This task is applicable not only to road safety warnings, autonomous driving, and traffic flow control, but also to effectively promoting pedestrian protection. Although many researchers have focused on anomaly detection methods under static monitoring in recent years, these methods often struggle to achieve ideal traffic incident detection performance due to the dynamic and complex background of driving videos.

[0004] Models applied to traffic incident detection in dashcams typically employ a hybrid deep learning framework, combining convolutional neural networks (CNNs) with recurrent neural networks (RNNs) to identify potential hazards in dashcam-recorded videos. These methods utilize CNNs for feature extraction and object localization, with the extracted spatial features then fed into the RNN to capture temporal patterns and understand the evolution of these spatial features over time. For example, some studies use Mask R-CNN for semantic segmentation, followed by a CNN-LSTM network to classify dangerous lane changes based on mask-based frame coverage. However, unguided deep learning models may produce spurious patterns, and CNN models also have limitations in extracting unobservable contextual relationships.

[0005] Current research on combining spatiotemporal information from traffic videos requires further investigation. Therefore, designing a model that can simultaneously consider the differences in motion changes of dashcams and the characteristics of target scale changes in the video, and combining spatial features and short- and long-term temporal information in video frames to quickly detect and accurately locate the start time of traffic events remains an urgent problem to be solved. Summary of the Invention

[0006] Objective: This invention aims to provide a traffic event detection method based on deep spatiotemporal memory and interactive networks. First, a spatiotemporal information perception network is designed, which can effectively capture the core semantics from small to large spatial regions, as well as short-term features in the temporal domain, from input video frames, improving the feature extraction capability of video frames and enabling more accurate classification of traffic events. Second, a temporal feature learning network is proposed to address the problem that the spatiotemporal information perception network fails to fully capture long-term temporal information when processing video frames. This method, by strengthening the learning of temporal features, can obtain more complete contextual temporal information features, improving the accurate localization of the occurrence and termination times of anomalies.

[0007] Technical Solution: This invention provides a traffic incident detection method based on deep spatiotemporal memory and interactive networks. It constructs a detection model combining a spatiotemporal information perception network and a temporal feature learning network, using video frames as input to locate traffic accidents and predict their start time. The method includes the following steps:

[0008] Step 1: Construct a spatiotemporal information sensing network, using video frames as input, and introduce MarphFC in the spatial dimension. s (Spatial Domain Fully Connected Layer) Mines spatial semantic information and introduces MorphFC in the temporal channel. t (Temporal domain fully connected layer) captures the relationship between the input video features and the changes over time, and remaps the features in the temporal dimension to the spatial dimension, thereby extracting spatial and temporal information features in the video frame;

[0009] Step 2: Construct a temporal feature learning network to enhance the learning ability of temporal information features in video frames and to comprehensively model the temporal information in video frames;

[0010] Step 3: Train the detection model using weighted cross-entropy loss.

[0011] Furthermore, step 1 specifically involves introducing MarphFC into the MFCMLP (Fully Connected Spatiotemporal Multilayer Perceptron) module. s (Spatial domain fully connected layer) to hierarchically extend the receiver domain of FC and enable it to operate from small areas to large areas; the MarphFC referenced s(Spatial domain fully connected layer) processes each video frame independently in the horizontal and vertical paths; taking the horizontal path as an example, given an input video frame X∈R HW×C It is projected into a token sequence, first divided along the horizontal direction X, with the block length set to L, thus obtaining

[0012] X i ∈R L×C (1)

[0013] Where i∈{1,…,HW / L}, to reduce computational cost, each X is divided along the channel dimension. i Divide into multiple groups, each with D channels, to obtain segmented blocks, each block being...

[0014] X i k ∈R LD (2)

[0015] Where k∈{1,…,C / D}, each data block is flattened into a 1D vector, and an FC weight matrix W∈R is applied. LD ×LD Each data block is transformed, i.e., Y is...

[0016]

[0017] After feature transformation, all blocks Y i k Remodeled into the original dimension Y∈R H×W×C The same applies to the vertical direction, except that the label sequence is split along the vertical direction; the FC layer is applied to process each label individually so that the groups can communicate along the channel dimension; finally, the horizontal, vertical and channel features are summed element by element to obtain the output result; as the network goes deeper, the block length L will increase layer by layer, so that the FC filter can gradually discover more core semantics in the spatial region from small to large.

[0018] When processing traffic videos, it is necessary to process not only the information in the horizontal and vertical path dimensions of continuous video frames, but also the information in the temporal dimension. Since the MLP in VST typically operates independently at each location, meaning that the feature vector at each location undergoes a linear transformation independently through a fully connected layer, it cannot directly capture the correlation and temporal dependencies between different locations. Therefore, this invention also introduces MorphFC into MFCMLP. t (Temporal fully connected layer) This module transforms the input feature map in the temporal dimension by flattening, fully connecting, and then restoring it to a high-dimensional space. This processing method can be used to capture the relationship between the input video features and the changes over time, and remap these temporal features to the spatial dimension.

[0019] Furthermore, the introduction of MorphFC in MFCMLP t (Temporal fully connected layer) This module transforms the input feature map in the temporal dimension, performing flattening, fully connected operations, and then restoring it to a high-dimensional space. Specifically:

[0020] Given an input video segment labeled X∈R H×W×T×C First, X is divided into several groups along the channel dimension to reduce computational cost. Each group has D channels, resulting in...

[0021] X k ∈R H×W×T×D (4)

[0022] Where k∈{1,…,C / D}, for each spatial location s, the features of all frames are concatenated into a block. Where s∈{1,…,HW};

[0023] Referencing the FC matrix W∈R TD×TD After transforming the time characteristics, we get:

[0024]

[0025] All blocks Y s k ∈R TD Resize back to the original tokens dimension and output Y∈R H×W×T×C In this way, the FC filter can easily aggregate token relationships within a block along the time dimension, providing VST with stronger time-dimensional modeling capabilities, while also enhancing the MarphFC... s The original input and output of the (spatial domain fully connected layer) are connected by residual connections to increase the stability of model training.

[0026] Furthermore, step 2 specifically involves: In the input time series, let the input time series be... Where T represents the time step, after a 1×1 convolution transformation, we get:

[0027]

[0028] Where W1 represents the weight parameter, which is the weight matrix of the above linear transformation, and b1 represents the bias in the above linear transformation.

[0029] The parallel processing part of dilated convolution mainly consists of two parallel convolution processing blocks. Each convolution block contains two dilated convolutional layers. For each dilated convolutional layer, the output of the l-th layer is represented as:

[0030] X(l) =DilatedConv(X (l-1) dilation=d l (7)

[0031] Where, d l This refers to the expansion rate of the l-th layer;

[0032] Let the output of the first dilated convolutional block be X1, and the output of the second dilated convolutional block be X2. The specific processing procedure for each block is as follows:

[0033] X1 = DilatedConv1(W1Z + b1) (8)

[0034] X2=DilatedConv2(W2Z+b2) (9)

[0035] Where W2 is the weight parameter in the dilated convolution operation, and b2 represents the bias in the second dilated convolution layer.

[0036] The information obtained after residual join and subsequent processing is as follows:

[0037] X = X1 + X2 (10)

[0038] Then, after a series of nonlinear activation and regularization processes, we obtain:

[0039] X norm =WeightNorm(X) (11)

[0040] X relu =Relu(X) (12)

[0041] X dropout =Dropout(X) (13)

[0042] The final output is:

[0043] Y=W3Dropout(Relu(WeightNorm(X1+X2)))+b3 (14)

[0044] Where W3 represents the weight matrix of the final output layer, and b3 represents the bias of the final output layer.

[0045] Furthermore, step 3 specifically involves:

[0046] For each video frame F[t], the final output anomaly score S[t]∈[0,1], where 0 indicates no anomaly and 1 indicates an anomaly in the frame; to assign higher weights to the anomaly classes that reflect the data distribution, a weighted cross-entropy loss is chosen to train the model, and for each weight ω i The formula used is as follows:

[0047] ω i =e / e i (15)

[0048] Where e is the total number of examples in the dataset, e i It is the number of examples of category i.

[0049] The present invention also discloses a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method of the present invention.

[0050] The present invention also discloses a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implements the steps of the method of the present invention.

[0051] The present invention also discloses a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the method of the present invention.

[0052] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages:

[0053] This invention designs a deep spatiotemporal memory and interaction network. This network extracts spatiotemporal information from traffic videos in two different stages, from local to global perspectives, to obtain a richer real-time description of traffic events in driver-centric traffic videos without speculating on the future. Specifically, in the spatiotemporal information perception network, MarphFC is introduced in the spatial dimension. s (Spatial domain fully connected layer) to mine spatial semantic information, while MorphFC is used on the temporal channel. t (Temporal fully connected layer) to capture the changes in video features over time. These temporal features are remapped to the spatial dimension, solving the tedious operations of expanding, reshaping, and collapsing video frames using the original sliding window in the Video Swing Transformer (VST), while MarphFC... s The (spatial domain fully connected layer) uses residual connections between the original input and output to increase the stability of model training, thereby more effectively extracting spatial and short-term temporal information from video frames. In the temporal feature learning network, since the first stage mainly extracts short-term temporal information, a dilated convolutional dual-branch residual structure is used to extract long-term temporal contextual information features in order to better locate the start time of traffic accidents. Attached Figure Description

[0054] Figure 1 This is a diagram illustrating the overall framework of the deep spatiotemporal memory and interaction network of this invention.

[0055] Figure 2 This is a diagram of the spatiotemporal information perception network architecture.

[0056] Figure 3 MarphFC in spatial dimension s (Spatial Domain Fully Connected Layer) Diagram.

[0057] Figure 4 For MarphFC acting on the time dimension t picture.

[0058] Figure 5 A learning network for temporal features.

[0059] Figure 6 This is a diagram illustrating the principle of dilated convolution.

[0060] Figure 7 A visual comparison of traffic incident detection examples on the DoTA dataset. Detailed Implementation

[0061] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0062] This invention proposes a traffic incident detection method based on deep spatiotemporal memory and interactive networks, comprising the following steps:

[0063] This invention addresses the challenges of traffic event detection in a driver's first-person perspective video. Due to distance variations, obstacles appear to differ in size, leading to variations in the size of the target traffic event to be identified, thus hindering effective event recognition and classification. A traffic event detection method based on deep spatiotemporal memory and interactive networks is proposed, with the overall block diagram shown below. Figure 1 As shown, the model mainly consists of two parts: a spatiotemporal information perception network and a temporal feature learning network. In the spatiotemporal information perception network, MarphFC is introduced in the spatial dimension. s (Spatial Domain Fully Connected Layer) Mines spatial semantic information and introduces MorphFC in the temporal channel. t(Temporal fully connected layer) captures the relationship between the input video features and time, and remaps these temporal features to the spatial dimension, thereby better extracting spatial and short-term temporal information features from video frames. In the temporal feature learning network, since the first stage of processing extracts short-term temporal information features, to better locate the time of traffic accidents, this network structure utilizes a dilated convolutional dual-branch residual structure to better extract long-term temporal information features. Finally, based on deep spatiotemporal memory and interactive networks, the extracted local and global spatiotemporal information features are combined to more accurately locate the time of traffic events and identify the type of traffic events.

[0064] To better handle the spatiotemporal features in traffic videos, this invention proposes a spatiotemporal information perception network that can effectively capture spatial and temporal features in video frames. Specifically, after preprocessing the input video frames, the network divides the resulting token sequence into blocks to process different scales, and gradually captures spatial and temporal information from local to global by progressively expanding the block length. This processing method can expand from local to global information in the spatial domain, while also progressively capturing information features in the temporal domain. Through this multi-scale spatiotemporal perception, the network can more comprehensively extract key features from video frames, thereby achieving better detection results in subsequent traffic video anomaly detection by classifying abnormal events and accurately locating abnormal times. The overall framework is as follows: Figure 2 (a) shows the overall architecture of the spatiotemporal information sensing network, and (b) shows the schematic diagram of the spatiotemporal fully connected multilayer sensor in the spatiotemporal information sensing network.

[0065] In traffic video processing, extracting core spatiotemporal semantic information is crucial for the identification and classification of traffic accidents within the video. However, current typical CNNs and MLPs only focus on modeling local or global information, thus failing to effectively capture the local-to-global information within a video sequence. To address this issue, this invention designs the MFCMLP module, which first introduces MarphFC... s (Spatial domain fully connected layer) This layer can extend the receiver domain of a fully connected (FC) system, enabling it to operate from small areas to large areas. (The MarphFC referenced here...) s (The spatial domain fully connected layer) processes each frame of video independently in the horizontal and vertical paths. As shown in the figure below, this invention mainly takes a horizontal block as an example (e.g., Figure 3 (Middle green block).

[0066] Specifically, given an input video frame X∈R HW×CIt is projected into a token sequence. The present invention first divides X along the horizontal direction, setting the block length to L, thereby obtaining...

[0067] X i ∈R L×C (1)

[0068] Where i∈{1,…,HW / L}. To reduce computational cost, this invention also divides each X along the channel dimension. i The system is divided into multiple groups, each with D channels. Therefore, segmented blocks are obtained, each block being...

[0069] X i k ∈R LD (2)

[0070] Where k∈{1,…,C / D}. Next, each data block is flattened into a 1D vector, and an FC weight matrix W∈R is applied. LD×LD Each data block is transformed, i.e., Y is...

[0071]

[0072] After feature transformation, all blocks Y i k Remodeled into the original dimension Y∈R H×W×C Vertical direction ( Figure 3 The same applies to the yellow blocks (in the diagram), except that the marker sequence is split vertically. To enable communication between groups along the channel dimension, this invention also applies an FC layer to process each marker individually. Finally, the horizontal, vertical, and channel features are summed element-wise to obtain the output. As the network deepens, the block length L increases layer by layer, allowing the FC filter to gradually discover more core semantics in spatial regions from small to large.

[0073] When processing traffic videos, it is necessary to process not only the information in the horizontal and vertical path dimensions of continuous video frames, but also the information in the temporal dimension. Since the MLP in VST typically operates independently per location, meaning that the feature vector at each location undergoes a linear transformation independently through a fully connected layer, it cannot directly capture the correlation and temporal dependencies between different locations. Therefore, this invention also introduces MorphFC into MFCMLP. t (Temporal fully connected layer) This module transforms the input feature map in the temporal dimension, performing flattening, fully connected operations, and then restoring it to a high-dimensional space. This processing method can be used to capture the temporal changes of input video features and remap these temporal features back to the spatial dimension. Its principle and network architecture are as follows: Figure 4As shown.

[0074] Suppose that, given an input video segment labeled X∈R H×W×T×C First, X is divided into several groups (each group has D channels) along the channel dimension to reduce computational cost, resulting in...

[0075] X k ∈R H×W×T×D (4)

[0076] Where k∈{1,…,C / D}. For each spatial location s, this invention concatenates the features of all frames into a single block. Where s∈{1,…,HW}.

[0077] Next, this invention references the FC matrix W∈R TD×TD After transforming the time characteristics, we get:

[0078]

[0079] Finally, the present invention will combine all blocks Y s k ∈R TD Resize back to the original tokens dimension and output Y∈R H ×W×T×C In this way, the FC filter can easily aggregate token relationships within a block along the time dimension, providing VST with stronger time-dimensional modeling capabilities, while also enhancing the MarphFC... s The original input and output of the (spatial domain fully connected layer) are connected by residual connections to increase the stability of model training.

[0080] Spatiotemporal information perception networks can learn dependencies on input frames by aggregating time stamps at each spatial location. However, numerous local and global temporal information exists between video frames that need to be processed to obtain more complete temporal features, thereby more accurately pinpointing the time of anomalies. This invention proposes a temporal feature learning network (as follows). Figure 5 As shown in the figure, it enhances the ability to learn temporal features from the processed feature map.

[0081] exist Figure 5 In this invention, the 1×1 convolution operation only considers temporal information from neighboring frames. This operation can effectively capture short-term dependencies between video frames and extract local temporal features. Secondly, this invention uses parallel processing of dilated convolutions with a dilation rate of 3 in the temporal domain. This parameter allows the same-sized convolution kernel to obtain a larger receptive field without increasing computational complexity. Figure 6(Where (a) is the initial input convolutional kernel setting the receptive field through dilation factor d; (b) is the extraction of features through dilated convolution, expansion of the receptive field, and generation of intermediate layer output; (c) is the further extraction of high-level features and the final output used for subsequent tasks.) As shown, this invention designs a temporal feature learning network composed of two cascaded structures to supplement the temporal information missed at intervals in the input time series under dilated convolution with a dilation rate of 3. This design allows for the capture of features over a longer time range even with smaller convolutional kernels, thus processing global temporal information. This approach can aggregate information from more distant time frames without losing local details.

[0082] exist Figure 5 In the input time series, assuming the input time series is... Where T represents the time step, after a 1×1 convolution transformation, we get:

[0083]

[0084] The parallel processing part of dilated convolution mainly consists of two parallel convolution processing blocks. Each convolution block contains two dilated convolutional layers. For each dilated convolutional layer, the output of the l-th layer can be represented as:

[0085] X (l) =DilatedConv(X (l-1) dilation=d l (7)

[0086] Where, d l This refers to the expansion rate of the l-th layer.

[0087] Assuming the output of the first dilated convolutional block is X1 and the output of the second dilated convolutional block is X2, the specific processing procedure for each block is as follows:

[0088] X1 = DilatedConv1(W1Z + b1) (8)

[0089] X2=DilatedConv2(W2Z+b2) (9)

[0090] The information obtained after residual join and subsequent processing is as follows:

[0091] X = X1 + X2 (10)

[0092] Then, after a series of nonlinear activation and regularization processes, we obtain:

[0093] X norm =WeightNorm(X) (11)

[0094] X relu =Relu(X) (12)

[0095] X dropout =Dropout(X) (13)

[0096] The final output is:

[0097] Y=W3Dropout(Relu(WeightNorm(X1+X2)))+b3 (14)

[0098] The temporal feature learning network proposed in this invention processes local and global temporal information in videos through dilated convolution and causal convolution, and achieves comprehensive modeling of temporal information in video frames through multi-branch and multi-scale feature fusion.

[0099] For each video frame F[t], the final output anomaly score S[t]∈[0,1], where 0 represents no anomaly and 1 represents an anomaly in the frame. To assign higher weights to anomaly classes that reflect the data distribution, this invention chooses weighted cross-entropy loss to train the model. For each weight ω i The formula used in this invention is as follows:

[0100] ω i =e / e i (15)

[0101] Where e is the total number of examples in the dataset, e i It is the number of examples of category i.

[0102] Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0103] Furthermore, the effectiveness of the method proposed in this invention is verified through simulation experiments.

[0104] Simulation Experiment 1:

[0105] The model proposed in this invention was trained and evaluated on an Nvidia GeForce RTX 3090 server, and experiments were conducted using PyTorch. Secondly, to make the model training more stable, this invention used the SDG optimizer for training, with a batch size of 4, a learning rate of 0.00005, a momentum of 0.9, and a video segment length of 8. For network training, the input video frame size was set to 640×480, and the dataset was configured according to the DoTA dataset, with a training cycle of 100 epochs. This invention uses weighted cross-entropy loss for training to address the data imbalance problem in the DoTA dataset, where ω... n =0.3 and ω a =0.7 is assigned to the normal and abnormal classes respectively, and follows ω. i =e / e i equation.

[0106] Table 1. AUC (%) of different methods on the DoTA dataset

[0107]

[0108] This invention compares the proposed traffic event detection method based on deep spatiotemporal memory and interactive networks with existing anomaly detection methods, including LSTM, ConvAE, ConvLSTMAE, TAD, TAD+ML, FC, Encoder-Decoder, AnoPred, FOL-Ensemble, Movad, TRN, and STFE. Among these, FC is a video-level detection method; TAD, TAD+ML, and Ensemble are localization-based segmented detection methods; and ConvAE, ConvLSTMAE, AnoPred, LSTM, Encoder-Decoder, Movad, TRN, and STFE are frame-level detection methods. Table 1 shows the AUC results of different methods on the DoTA dataset.

[0109] As can be observed from Table 1, the proposed method achieves an AUC of 80.9%, outperforming all other methods. Compared to the video-level detection method FC, FC has an AUC of 61.7%, indicating lower accuracy. This is because FC processes the entire video rather than fine-grained segments or frames, thus limiting its ability to capture details. Localization-based segmented methods such as TAD and TAD+ML have AUCs of 69.2% and 69.7%, respectively, showing a slight improvement over video-level methods, as segmented detection can more accurately pinpoint the time period of an event. However, since these methods still process at the segment level, they may miss some important details. Frame-level detection methods, including LSTM, ConvAE, ConvLSTMAE, AnoPred, Encoder-Decoder, Movad, TRN, and STFE, capture more detailed temporal information through frame-by-frame analysis, thus generally outperforming video-level and segmented methods. STFE performs particularly well in frame-level detection, with an AUC of 79.3%. Therefore, the method proposed in this invention significantly outperforms other methods in terms of AUC performance. This model extracts local-to-global spatiotemporal information from traffic videos using two different networks, thereby more accurately detecting the type of traffic event and locating the start time of the traffic accident. Simulation Experiment 2:

[0110] As shown in Table 2, different methods exhibit significant differences in AUC performance across various traffic accident classification tasks. Compared to other methods, the method proposed in this invention achieves the best AUC score in the vast majority of traffic accident categories. Methods such as AnoPred, FOL-STD, and FOL-Ensemble show fluctuating performance across different categories. Especially in complex traffic accident types (e.g., the OO category), these methods generally have lower AUC scores. The method proposed in this invention also achieves high performance in such complex accidents, further demonstrating the model's robustness. Compared to STFE, the method proposed in this invention performs better in most scenarios (e.g., ST, LA, OC, TC, VP, VO, OO, and UK), especially showing significant advantages in the ST and OO traffic event detection tasks. Its detection performance in the AH category is slightly lower, possibly due to small target obstacles in driving videos or sudden flying debris; the method proposed in this invention has not yet fully handled the detection of these types of traffic events, but overall, it has a superior performance. In Table 2, the method proposed in this invention demonstrates optimal performance across categories ST, AH, LA, OC, TC, and VP, with a particularly significant lead in ST, LA, OC, and VP categories. As shown in Table 3, in the classification of non-self-abnormal traffic accidents, the method proposed in this invention exhibits excellent performance in categories *ST, *AH, *LA, *OC, *TC, *VP, *VO, *OO, and *UK. Overall, based on deep spatiotemporal memory and interactive networks, the method proposed in this invention achieves the highest AUC in almost all traffic accident categories, particularly demonstrating strong classification accuracy in key categories such as ST, LA, OC, TC, VP, VO, OO, and UK. The method proposed in this invention achieves the highest AUC values ​​in most categories and exhibits good robustness and generalization ability.

[0111] Table 2. Comparison of AUC (%) of different methods in self-detection of abnormal traffic events

[0112] model ST AH LA OC TC VP VO OO UK AnoPred 69.9 73.6 75.2 69.7 73.5 63.3 N / A N / A N / A FOL-STD 67.3 77.4 71.1 68.6 69.2 65.1 N / A N / A N / A FOL-Ensemble 73.3 81.2 74.0 73.4 75.1 70.1 N / A N / A N / A Movad 75.1 71.8 70.3 72.2 73.5 70.7 73.2 69.8 / 73.0 65.1 STFE (SOTA) 75.2 84.5 72.1 77.3 72.8 71.9 N / A N / A N / A Ours 86.3 84.1 82.6 80.3 84.6 78.6 77.8 85.1 / 89.2 72.9

[0113] Table 3. Comparison of AUC (%) of different methods in the detection of non-self-generated abnormal traffic events.

[0114] model *ST *AH *LA *OC *TC *VP *VO *OO *UK AnoPred 70.9 62.6 60.1 65.6 65.4 64.9 64.2 57.8 N / A FOL-STD 75.1 69.8 68.1 74.1 72.0 69.7 63.8 69.2 N / A FOL-Ensemble 77.5 69.8 68.1 76.7 73.9 71.2 65.2 69.6 N / A Movad 62.8 62.2 62.8 61.6 62.9 61.9 63.2 63.1 / 67.8 57.8 STFE (SOTA) 80.6 65.6 69.9 76.5 74.2 N / A 75.6 70.5 N / A Ours 79.5 78.5 78.4 80.2 80.4 75.4 77.7 76.1 / 82.2 69.6

[0115] Simulation Experiment 3:

[0116] From below Figure 7(1) It can be seen that when the car is driving normally on the road, the anomaly score tends to 0. When the car deviates from its course on the road, the anomaly score rises significantly. When a collision occurs, the anomaly score approaches 1. This indicates that the method proposed in this invention can detect anomalies and pinpoint the start and end times of the anomalies. Figure 7 In this invention, visualization examples of traffic event detection with self-abnormal categories and non-abnormal self categories are selected and compared with the visualization results of SOTA. Figure 7 In the abnormal region shown in (1), the anomaly score of the SOTA method rises slowly and then drops rapidly after briefly holding at the peak, exhibiting a certain degree of volatility. In contrast, the anomaly score of the method proposed in this invention reaches 1 more quickly within the abnormal region and remains stable, with less overall volatility. Figure 7 In (2), the anomaly score obtained by the SOTA method fluctuates significantly within the anomaly region, containing multiple small peaks, reflecting the instability of the detection. In contrast, the anomaly score obtained by the method proposed in this invention remains relatively stable within the anomaly region, although there are some fluctuations, it exhibits a generally stable high value. Figure 7 In (3), the outlier score obtained by the SOTA method shows a gradual upward trend within the outlier region, but rapidly decreases after approaching 1, accompanied by significant fluctuations. In contrast, the outlier score obtained by our method rapidly reaches 1 within the outlier region and remains stable without significant fluctuations. Figure 7 In (4), the output curve of the SOTA method fluctuates significantly within the abnormal region and fails to reach a stable high value, indicating instability in the detection of this type of traffic event. In contrast, the anomaly score obtained by the method proposed in this invention remains stable within the abnormal region, close to 1, and exhibits almost no significant fluctuations. In summary, the two-stage spatiotemporal information fusion method demonstrates a smoother anomaly score curve in all four given scenarios, exhibiting higher accuracy and stability in traffic event detection and classification.

[0117] Simulation Experiment 4:

[0118] To better verify the effectiveness of the two-stage spatiotemporal information combined network on the DOTA dataset, ablation experiments were conducted as shown in Table 4. First, when using the spatiotemporal information perception network alone, the model's AUC was 74.17% and its accuracy was 73.04%. Second, when using only the temporal feature learning network, the model's AUC was 72.08% and its accuracy was 71.33%. When both the spatiotemporal information perception network and the temporal feature learning network were included, the model's AUC was 80.9% and its accuracy was 76.05%, indicating the highest performance based on the deep spatiotemporal memory and interaction network. This demonstrates that the model proposed in this invention, through the combined action of the spatiotemporal information perception network and the temporal feature learning network, can extract more effective spatiotemporal information features from video frames, capture the spatiotemporal dependencies between different video frames, and thus better classify and locate the start time of traffic events.

[0119] Table 4 Ablation experiments on the DoTA dataset

[0120] Spatiotemporal information sensing network Temporal Feature Learning Network AUC(%) Accuracy (%) w w / o 74.17 73.04 w / o w 72.08 71.33 w w 80.9 76.05

Claims

1. A traffic incident detection method based on deep spatiotemporal memory and interactive networks, characterized in that, A detection model combining a spatiotemporal information perception network and a temporal feature learning network is constructed. Using video frames as input, it can locate traffic accidents and predict their start time. The model includes the following steps: Step 1: Construct a spatiotemporal information sensing network, using video frames as input, and introduce a spatial domain fully connected layer, MarphFC, in the spatial dimension. s Mining spatial semantic information and introducing a temporally fully connected layer MorphFC in the temporal channel. t The system captures the relationship between the input video features and time, and remaps the temporal features to the spatial dimension, thereby extracting spatial and temporal information features from the video frames. Step 2: Construct a temporal feature learning network to enhance the learning ability of temporal information features in video frames and to comprehensively model the temporal information in video frames; Step 2 specifically involves: Let the input time series be... ,in T Indicates the time step, after which... After the convolution transformation, we get: （6） in, W 1 represents the weight parameter, which is the weight matrix of the linear transformation described above. b 1 represents the bias in the above linear transformation; The parallel processing part of dilated convolution includes two parallel convolution processing blocks, each containing two dilated convolutional layers. For each dilated convolutional layer, the... l The output of the layer is represented as: （7） in, It refers to the first l The expansion rate of the layer; Let the output of the first dilated convolution block be The output of the second dilated convolutional block is The specific processing procedure for each block is as follows: （8） （9） in, W 2 represents the weight parameters in the dilated convolution operation. b 2 represents the bias in the second dilated convolutional layer; The information obtained after residual join and subsequent processing is as follows: （10） Then, after a series of nonlinear activation and regularization processes, we obtain: （11） （12） （13） The final output is: （14） in, W 3 represents the weight matrix of the final output layer. b 3 indicates the bias of the final output layer; Step 3: Train the detection model using weighted cross-entropy loss.

2. The traffic event detection method based on deep spatiotemporal memory and interactive networks according to claim 1, characterized in that, Step 1 specifically involves introducing the spatial domain fully connected layer MarphFC into the MFCMLP module. s This is used to extend the receiving domain of the FC in layers, enabling it to operate from small areas to large areas; the spatial domain fully connected layer MarphFC is referenced. s Process each frame of video independently in both horizontal and vertical paths; Taking a horizontal path as an example, given an input video frame It is projected into a token sequence, first along the horizontal direction... X Perform segmentation, setting the block length to... L Thus obtain （1） in, To reduce computational costs, each channel is divided along the channel dimension. X i Divided into multiple groups, each group has D Each channel is used to obtain segmented blocks, and each block is... （2） in, Flatten each data block into 1 D Vectors, and apply FC weight matrix Transform each data block, that is Y for （3） After feature transformation, all blocks Remodeled into the original dimension The same principle applies to the vertical direction, except that the marker sequence is split vertically; application FC Each tag is processed individually in layers, enabling communication between groups along the channel dimension; finally, the horizontal, vertical, and channel features are summed element-wise to obtain the output; as the network deepens, the block length... L It will increase in layers, thus FC The filter gradually discovers more core semantics by expanding the spatial regions from small to large. Introducing MorphFC, a time-domain fully connected layer, in MFCMLP t This module transforms the input feature map in the time dimension by flattening, performing fully connected operations, and then restoring it to a high-dimensional space.

3. The traffic event detection method based on deep spatiotemporal memory and interactive networks according to claim 2, characterized in that, The introduction of a time-domain fully connected layer, MorphFC, in MFCMLP t This module transforms the input feature map in the time dimension, performing flattening, fully connected operations, and then restoring it to a high-dimensional space. Specifically: Given an input video segment tag First, along the channel dimension X Divide into several groups to reduce computational costs, each group D One channel, obtain （4） in, For each spatial location s, the features of all frames are concatenated into a block. ,in ; Reference FC matrix After transforming the time characteristics, we get: （5） All blocks Resize back to the original tokens dimension and output. .

4. The traffic event detection method based on deep spatiotemporal memory and interactive networks according to claim 1, characterized in that, Step 3 specifically involves: For each video frame F [ t The final output is the anomaly score. S [ t ]∈[0,1], where 0 represents no anomaly and 1 represents an anomaly in the frame; to assign higher weights to the anomaly classes that reflect the data distribution, a weighted cross-entropy loss is chosen to train the model, and for each weight The formula used is as follows: （15） in, It is the total number of examples in the dataset. It is a category i The number of examples.

5. A computer device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method of claim 1.

6. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method of claim 1.

7. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method of claim 1.

Images