Traffic accident detection method and system integrating temporal convolution and self-attention mechanism

Abstract

The present invention discloses a method and system for detecting traffic accidents by integrating temporal convolution and self-attention mechanism. A U-shaped fully convolutional neural network (U-Net) that combines a multi-head attention mechanism with temporal convolution is proposed. The prediction network based on temporal convolution and a multi-head attention mechanism network with strong feature expression ability are combined to predict the next frame by learning the normal mode of the existing video frame, and judge whether a traffic accident occurs according to the difference between the predicted frame and the real frame. Given continuous t frames I1,I2,...,I t To predict the t+1th frame #imgabs0#, in order to make the predicted frame #imgabs1# consistent with the real frame I t+1 Closer, constrain the appearance features (intensity and gradient), add generative adversarial loss to make the predicted frames more realistic. For vehicle driving video data, dynamic graphs are introduced to enhance the spatiotemporal constraints in intensity and gradient, which improves the accuracy of car accident event detection.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a traffic accident event detection and system integrating temporal convolution and self-attention mechanisms. Background Art

[0002] Rapid detection of traffic accidents can reduce property losses and casualties caused by untimely rescue. In the past two decades, traffic monitoring has spread to every corner of urban streets, providing a large source of video data for visual-based accident detection. How to effectively use the existing monitoring data for accident detection is very critical. In recent years, with the development of deep learning technology, computer vision technology based on convolutional neural networks has developed rapidly. Computer vision technology based on deep learning does not require manual extraction of image / video features. All features are obtained by the model through a large amount of data learning. Therefore, computer vision technology based on deep learning is more accurate than traditional image processing methods that extract features manually.

[0003] At present, some works judge samples with large reconstruction errors as abnormal events. Due to the limited representation ability of artificially designed features, the robustness is poor in complex scenarios. In addition, other methods usually make judgments based on end-to-end anomaly scores, and judge input samples with anomaly scores higher than the threshold as abnormal. Therefore, the threshold for abnormal judgment is not easy to determine, which is easy to cause false alarms or missed detections. In addition, most of the time series modeling in traditional traffic accident detection is carried out using recurrent neural networks (RNN) and their variants. Recurrent neural networks (RNN) generally have difficulty in preserving information for a long time, and there are also problems of gradient disappearance and gradient explosion, so the prediction effect is not very good. The emergence of Long Short-Term Memory Networks (LSTM) and gated recurrent unit (GRU) has solved the long-term dependence and gradient disappearance problems of RNN. They can selectively remember or forget information through gating mechanisms, but their receptive field is not wide enough, which limits the performance of traffic accident detection based on Long Short-Term Memory Networks (LSTM). Summary of the invention

[0004] The purpose of the present invention is to provide a method and system for detecting traffic accidents by integrating temporal convolution and self-attention mechanism, so as to solve the above-mentioned problems existing in the prior art.

[0005] In a first aspect, an embodiment of the present invention provides a method for detecting a traffic accident event by integrating temporal convolution with a self-attention mechanism, comprising:

[0006] Obtaining a plurality of video frames to be tested; the plurality of video frames to be tested are images of a plurality of consecutive frames in a video to be predicted; the video frame to be predicted represents a next frame image of a predicted current frame image;

[0007] Based on the video frame to be tested, a first coding feature map, a second coding feature map, a third coding feature map and a fourth coding feature map are obtained through an encoder; a plurality of corresponding first coding feature maps, a plurality of corresponding second coding feature maps, a plurality of corresponding third coding feature maps and a plurality of corresponding fourth coding feature maps are obtained from a plurality of video frames to be tested;

[0008] Based on the plurality of fourth encoded feature maps, a temporal feature map is obtained through a temporal convolutional network; the temporal feature map is a feature map with time information;

[0009] Based on the time feature map, an attention feature map is obtained through a multi-head attention mechanism;

[0010] Based on the plurality of corresponding first encoding feature maps, the plurality of corresponding second encoding feature maps, the plurality of corresponding third encoding feature maps and the attention feature map, a decoder is used to fuse global and local information to obtain a predicted image; the predicted image includes the occurrence of the traffic accident;

[0011] The encoder, temporal convolutional network, multi-head attention mechanism and decoder constitute a U-shaped fully convolutional neural network:

[0012] The output of the encoder is the input of the temporal convolutional network, and the feature map output by the temporal convolutional network is linearly mapped to a sequence of fixed length as the input of the multi-head attention mechanism; the output of the attention mechanism encoder is the input of the decoder.

[0013] Among them, the fixed length dimension is determined by the setting of the multi-head attention mechanism.

[0014] Optionally, the temporal feature map is obtained through a temporal convolutional network based on the plurality of fourth encoded feature maps; the temporal feature map is a feature map with temporal information, including:

[0015] The temporal convolutional network includes a causal convolutional network, a dilated convolution and a residual connection;

[0016] Obtaining a feature map time point; the feature map time point is a time point of the video frame to be tested corresponding to the fourth encoded feature map;

[0017] Input the plurality of fourth encoded feature maps into the causal convolutional network according to the feature map time points from far to near to obtain a time feature map

[0018] Multiple inputs of a causal convolutional network correspond to one output, and one input corresponds to one fourth encoding feature map;

[0019] The causal convolution network uses a 2×2 convolution kernel to perform causal convolution, and convolves the data from far to near in sequence until it reaches the top layer;

[0020] The dilated convolution increases the distance of input time steps in the causal convolutional network;

[0021] Residual connections are used between each layer of the causal convolutional network.

[0022] Optionally, obtaining an attention feature map based on the time feature map through a multi-head attention mechanism includes:

[0023] Linearly mapping the time feature graph into a sequence of fixed length to obtain a time feature sequence;

[0024] Adding position embedding to the time feature sequence; one position corresponds to one time point;

[0025] The time feature sequence with added position embedding is input into the multi-head attention mechanism to obtain the attention feature map.

[0026] Optionally, the multi-head attention mechanism includes multiple multi-head self-attention blocks and multi-layer perceptron blocks;

[0027] Each multi-head self-attention block and multi-layer perceptron block is preceded and followed by layer normalization and residual connections.

[0028] Optionally, the training method of the U-shaped fully convolutional neural network includes:

[0029] Obtaining a training set; the training set includes a plurality of training images and a plurality of real frame images;

[0030] Inputting the training image into a U-shaped fully convolutional neural network to obtain a predicted frame image;

[0031] Based on the predicted frame image and the corresponding real frame image, calculating the loss through the objective function to obtain the total generation loss;

[0032] The objective function is specifically calculated in the following manner:

[0033]

[0034] in, is the total generation loss, is the strength loss, is the gradient loss, is the generator loss;int is the weight parameter of the intensity loss, λ gd is the weight parameter of the gradient loss, λ adv is the weight parameter of the generator loss, Represents the predicted frame image, I t+1 Represents the real frame image.

[0035] Optionally, the strength loss is specifically obtained by the following calculation method:

[0036]

[0037] in, Represents the predicted frame image, I t+1 Represents a real frame image;

[0038] The gradient loss is specifically obtained by the following calculation method:

[0039]

[0040] in, is the strength loss, is the gradient loss, is the generator loss, Indicates the pixel with abscissa i and ordinate j in the predicted frame image, where I t+1 (i, j) represents the pixel point with horizontal coordinate i and vertical coordinate j in the real frame image, RMSE is the root mean square error, || ||2 is the l2 norm, i represents the horizontal coordinate, and j represents the vertical coordinate.

[0041] Optionally, the real frame image and the predicted frame image are input into a generative adversarial network to train the generative adversarial network;

[0042] The generative adversarial network is a generative adversarial network trained by generator loss and discriminant loss;

[0043] The generator loss of the generative adversarial network is specifically obtained by the following calculation method:

[0044]

[0045] Among them, the Represents the value of the predicted frame image through the discriminator in the generative adversarial network, It indicates the pixel with horizontal coordinate i and vertical coordinate j in the predicted frame image. RMSE is the root mean square error, i represents the horizontal coordinate and j represents the vertical coordinate.

[0046] The discriminator loss of the generative adversarial network is specifically obtained by the following calculation method:

[0047]

[0048] Wherein, the D(I t+1 (i,j)) represents the value of the real frame image passing through the discriminator, Represents the value of the predicted frame image through the discriminator, Indicates the pixel with abscissa i and ordinate j in the predicted frame image, where I t+1 (i, j) represents the pixel point with horizontal coordinate i and vertical coordinate j in the real frame image. RMSE is the root mean square error, i represents the horizontal coordinate, and j represents the vertical coordinate.

[0049] Optionally, the obtaining a predicted image by fusing global and local information through a decoder based on the plurality of corresponding first encoding feature maps, the plurality of corresponding second encoding feature maps, the plurality of corresponding third encoding feature maps and the attention feature map includes:

[0050] Adjusting the attention feature map to the same size as the fourth encoding feature map to obtain an adjusted attention feature map;

[0051] Upsampling the adjusted attention feature map and fusing multiple third encoded feature maps to obtain a third upsampled feature map;

[0052] Upsampling the third upsampled feature map and fusing multiple second encoded feature maps to obtain a second upsampled feature map;

[0053] Upsampling the second upsampled feature map and fusing multiple first encoded feature maps to obtain a first upsampled feature map;

[0054] Decoding the first up-sampled feature map to obtain a predicted image;

[0055] The upsampling is deconvolved through a 4x4 convolution kernel.

[0056] Optionally, obtaining a first coding feature map, a second coding feature map, a third coding feature map, and a fourth coding feature map through an encoder based on the video frame to be tested includes:

[0057] Encoding the video frame to be tested to obtain a first encoding feature map;

[0058] Downsampling the first encoding feature map to obtain a second encoding feature map;

[0059] Downsampling the second encoding feature map to obtain a third encoding feature map;

[0060] Downsampling the third encoding feature map to obtain a fourth encoding feature map;

[0061] The downsampling is deconvolved with a 4x4 deconvolution kernel.

[0062] In a second aspect, an embodiment of the present invention provides a traffic accident event detection system integrating temporal convolution and self-attention mechanism, including:

[0063] Acquisition module: obtain multiple video frames to be tested; the multiple video frames to be tested are multiple consecutive frame images in the video to be predicted; the video frame to be predicted represents the next frame image predicted by the current frame image;

[0064] Coding module: Based on the video frame to be tested, a first coding feature map, a second coding feature map, a third coding feature map and a fourth coding feature map are obtained through an encoder; a plurality of video frames to be tested obtain a plurality of corresponding first coding feature maps, a plurality of corresponding second coding feature maps, a plurality of corresponding third coding feature maps and a plurality of corresponding fourth coding feature maps;

[0065] Temporal convolution module: based on multiple fourth encoding feature maps, a temporal feature map is obtained through a temporal convolution network; the temporal feature map is a feature map with time information;

[0066] Multi-head attention mechanism module: based on the time feature map, an attention feature map is obtained through a multi-head attention mechanism;

[0067] Decoding module: Based on the plurality of corresponding first coding feature maps, the plurality of corresponding second coding feature maps, the plurality of corresponding third coding feature maps and the attention feature map, a decoder is used to fuse the global and local information to obtain a predicted image; the predicted image includes the occurrence of the traffic accident;

[0068] The encoder, temporal convolutional network, multi-head attention mechanism and decoder constitute a U-shaped fully convolutional neural network:

[0069] The output of the encoder is the input of the temporal convolutional network, and the feature map output by the temporal convolutional network is linearly mapped to a sequence of fixed length as the input of the multi-head attention mechanism; the output of the attention mechanism encoder is the input of the decoder.

[0070] Compared with the prior art, the embodiments of the present invention achieve the following beneficial effects:

[0071] The embodiment of the present invention also provides a method and system for detecting a traffic accident event by integrating temporal convolution and self-attention mechanism, the method comprising: obtaining a plurality of video frames to be tested. The plurality of video frames to be tested are images of a plurality of continuous frames in a video to be predicted. The video frame to be predicted indicates the next frame image of the predicted current frame image. Based on the video frame to be tested, a first coding feature map, a second coding feature map, a third coding feature map and a fourth coding feature map are obtained through an encoder. A plurality of corresponding first coding feature maps, a plurality of corresponding second coding feature maps, a plurality of corresponding third coding feature maps and a plurality of corresponding fourth coding feature maps are obtained from the plurality of video frames to be tested. Based on the plurality of fourth coding feature maps, a temporal feature map is obtained through a temporal convolution network. The temporal feature map is a feature map with temporal information. Based on the temporal feature map, an attention feature map is obtained through a multi-head attention mechanism. Based on the plurality of corresponding first coding feature maps, a plurality of corresponding second coding feature maps, a plurality of corresponding third coding feature maps and an attention feature map, a decoder is used to fuse global and local information to obtain a predicted image. The predicted image contains the occurrence of the traffic accident. The encoder, temporal convolutional network, multi-head attention mechanism and decoder constitute a U-shaped fully convolutional neural network: the output of the encoder is the input of the temporal convolutional network, the feature map output by the temporal convolutional network is linearly mapped to a sequence of fixed length as the input of the multi-head attention mechanism. The output of the attention mechanism encoder is the input of the decoder.

[0072] This paper proposes a U-shaped fully convolutional neural network (U-Net) that combines a multi-head attention mechanism with temporal convolution. The prediction network is combined with a multi-head attention mechanism network with strong feature expression ability. The next frame is predicted by learning the normal mode of the existing video frame, and the abnormality is judged based on the difference between the predicted frame and the real frame, so as to judge whether a car accident has occurred. t To predict the t+1th frame In order to make the prediction frame With the real frame I t+1 Closer, constrain the appearance features (intensity and gradient), and add generative adversarial loss to make the predicted frames more realistic. For periodic vehicle driving datasets, dynamic graphs are introduced to enhance the spatiotemporal constraints on intensity and gradient. A temporal convolution module and a multi-head attention mechanism module are connected to the deepest layer of the encoder, which captures richer video feature information by obtaining the timeline before and after information of the wide-area receptive field and encoding global spatial information. Using the area under the curve (Area Under Curve, AUC) as the evaluation indicator, a performance comparison was carried out on a real car accident dataset, and the obtained area under the curve (Area Under Curve, AUC) value was 0.688, indicating that due to the current algorithm. By adopting the above scheme, the accuracy of car accident event detection is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 It is a flow chart of a method for detecting traffic accidents that integrates temporal convolution and self-attention mechanisms, provided in an embodiment of the present invention.

[0074] Figure 2 It is a schematic diagram of a U-shaped fully convolutional neural network in a traffic accident detection method that integrates temporal convolution and self-attention mechanism provided in an embodiment of the present invention.

[0075] Figure 3 It is a structural schematic diagram of a multi-head attention mechanism in a traffic accident event detection method that integrates temporal convolution and self-attention mechanism provided by an embodiment of the present invention.

[0076] Figure 4 It is a schematic diagram of a causal convolutional network in a traffic accident detection method that integrates temporal convolution and self-attention mechanism provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0077] The present invention will be described in detail below in conjunction with the accompanying drawings.

[0078] Example 1

[0079] like Figure 1 As shown, an embodiment of the present invention provides a method for detecting traffic accidents by integrating temporal convolution and self-attention mechanism, the method comprising:

[0080] S101: Obtain multiple video frames to be tested. The multiple video frames to be tested are multiple consecutive frames in a video to be predicted. The video frames to be predicted represent a video of an image of the next frame to be predicted.

[0081] Among them, given the continuous t frames I1,I2,...,I t To predict the t+1th frame

[0082] S102: Based on the video frame to be tested, a first coding feature map, a second coding feature map, a third coding feature map and a fourth coding feature map are obtained through an encoder. A plurality of video frames to be tested obtain a plurality of corresponding first coding feature maps, a plurality of corresponding second coding feature maps, a plurality of corresponding third coding feature maps and a plurality of corresponding fourth coding feature maps.

[0083] Among them, the attention feature map is adjusted to the same size as the fourth encoding feature map to obtain an adjusted attention feature map; the adjusted attention feature map is upsampled and fused with multiple third encoding feature maps to obtain a third upsampled feature map; the third upsampled feature map is upsampled and fused with multiple second encoding feature maps to obtain a second upsampled feature map; the second upsampled feature map is upsampled and fused with multiple first encoding feature maps to obtain a first upsampled feature map; the first upsampled feature map is decoded to obtain a predicted image; the upsampling is deconvolved through a 4x4 convolution kernel.

[0084] In this embodiment, the encoder is composed of 4 convolution layers and 3 pooling layers. The size of the convolution kernel is set to 4×4, and the maximum pooling kernel size is 3×3. Therefore, in this embodiment, one video frame to be tested obtains 4 encoded feature maps.

[0085] Among them, the first coding feature map, the second coding feature map, the third coding feature map and the fourth coding feature map are feature maps of different sizes.

[0086] S103: Based on the plurality of fourth encoded feature maps, a temporal feature map is obtained through a temporal convolutional network. The temporal feature map is a feature map with time information.

[0087] S104: Based on the time feature map, an attention feature map is obtained through a multi-head attention mechanism.

[0088] Among them, a temporal convolution module and a multi-head attention mechanism module are connected to the deepest layer of the encoder.

[0089] S105: Based on the plurality of corresponding first encoding feature maps, the plurality of corresponding second encoding feature maps, the plurality of corresponding third encoding feature maps and the attention feature map, a decoder is used to fuse global and local information to obtain a predicted image. The predicted image includes the occurrence of the traffic accident.

[0090] The encoder, temporal convolutional network, multi-head attention mechanism and decoder constitute a U-shaped fully convolutional neural network:

[0091] The structural diagram of the U-shaped fully convolutional neural network is as follows: Figure 2 shown.

[0092] The output of the encoder is the input of the temporal convolutional network, and the feature map output by the temporal convolutional network is linearly mapped to a sequence of fixed length as the input of the multi-head attention mechanism. The output of the attention mechanism encoder is the input of the decoder.

[0093] Optionally, the fourth encoded feature map is used to obtain a temporal feature map through a temporal convolutional network. The temporal feature map is a feature map with time information, including:

[0094] The temporal convolutional network includes a causal convolutional network, dilated convolutions and residual connections.

[0095] Obtain a feature map time point, wherein the feature map time point is a time point of the video frame to be tested corresponding to the fourth encoded feature map.

[0096] The multiple fourth encoded feature maps are input into the causal convolutional network according to the feature map time points from far to near to obtain a time feature map.

[0097] Multiple inputs of a causal convolutional network correspond to one output, and one input corresponds to one fourth encoded feature map.

[0098] The causal convolutional network uses a 2×2 convolution kernel for causal convolution, and convolves the data in sequence from far to near from the current time point until it reaches the top layer.

[0099] The structural diagram of the causal convolutional network is as follows: Figure 4 shown.

[0100] Among them, the top layer represents the output layer of the last convolutional layer corresponding to the most recent time point.

[0101] The dilated convolution increases the distance of input time steps in the causal convolutional network.

[0102] Among them, dilated convolution is used to expand the receptive field.

[0103] Residual connections are used between each layer of the causal convolutional network.

[0104] Among them, the temporal convolutional network contains three basic modules, namely causal convolution, dilated convolution and residual connection. Based on the temporal convolutional network, causal convolution limits the ordinary convolution operation to one direction, and the output at time t can only come from the input at time t-1 and before. Its structure is as follows Figure 2 As shown. In the input, each position corresponds to a historical data. The historical encoded feature data is input into the network, and the convolution operation is performed with the 2×2 convolution kernel. It moves from left to right in each layer, and finally passes it to the top layer to get the result of causal convolution. There are some defects in causal convolution. In order to capture longer historical information, the number of network layers must also be increased, which is likely to cause problems such as gradient disappearance and gradient explosion. This problem can be solved by introducing dilated convolution on the basis of causal convolution. Dilated convolution can effectively make the network run on a coarser scale. This is similar to pooling or strided convolution, and it can have a very large receptive field with only a few layers.

[0105] Through the above method, causal convolution is used. From the overall point of view, the output of the network depends only on historical data and will not leak future features. Dilated convolution is used to expand the receptive field. To prevent gradient explosion, residual is used.

[0106] Optionally, obtaining an attention feature map based on the time feature map through a multi-head attention mechanism includes:

[0107] The time feature graph is linearly mapped into a sequence of fixed length to obtain a time feature sequence.

[0108] The time feature sequence is embedded with a position, where one position corresponds to one time point.

[0109] Among them, after linearly mapping the deepest feature map of the U-Net encoder to a fixed-length sequence, position embedding is added and input into the multi-head attention mechanism. Each position is numbered, and each number corresponds to the feature of a time point. By combining the position vector and the feature vector of a time point, certain position information is introduced into the image information of each time point.

[0110] Among them, the fixed length dimension is determined by the setting of the multi-head attention mechanism.

[0111] The time feature sequence with added position embedding is input into the multi-head attention mechanism to obtain the attention feature map.

[0112] Among them, the structural diagram of the multi-head attention mechanism is as follows Figure 3 shown.

[0113] The multi-head attention mechanism consists of N layers of multi-head self-attention blocks and multi-layer perceptron (MLP) blocks. The feature expression ability of the network is improved by modeling long-distance dependencies.

[0114] Through the above method, causal convolution is used and a multi-head attention mechanism is introduced to improve the predictive ability of the model.

[0115] Optionally, the multi-head attention mechanism includes multiple multi-head self-attention blocks and multi-layer perceptron blocks.

[0116] Each multi-head self-attention block and multi-layer perceptron block is preceded and followed by layer normalization and residual connections.

[0117] Among them, layer normalization (layer norm).

[0118] Optionally, the training method of the U-shaped fully convolutional neural network includes:

[0119] A training set is obtained, wherein the training set includes a plurality of training images and a plurality of real frame images.

[0120] The training image is input into a U-shaped fully convolutional neural network to obtain a predicted frame image.

[0121] Based on the predicted frame image and the corresponding real frame image, the loss is calculated through the objective function to obtain the total generation loss.

[0122] The objective function is specifically calculated in the following manner:

[0123]

[0124] in, is the total generation loss, is the strength loss, is the gradient loss, is the generator loss. int is the weight parameter of the intensity loss, λ gd is the weight parameter of the gradient loss, λ adv is the weight parameter of the generator loss, Represents the predicted frame image, I t+1 Represents the real frame image.

[0125] Among them, in the training stage, in order to obtain a predicted frame that is more similar to a real frame, the present invention adopts intensity constraints, gradient constraints and generative adversarial constraints.

[0126] Through the above method, the anomaly is judged based on the difference between the predicted frame and the real frame. The soft threshold function is used to improve the prediction ability of the model.

[0127] Optionally, the strength loss is specifically obtained by the following calculation method:

[0128]

[0129] in, Represents the predicted frame image, I t+1 Represents the real frame image.

[0130] The gradient loss is specifically obtained by the following calculation method:

[0131]

[0132] in, is the strength loss, is the gradient loss, is the generator loss, Indicates the pixel with abscissa i and ordinate j in the predicted frame image, where I t+1(i, j) represents the pixel point with horizontal coordinate i and vertical coordinate j in the real frame image, RMSE is the root mean square error, || ||2 is the l2 norm, i represents the horizontal coordinate, and j represents the vertical coordinate.

[0133] Through the above method, a dynamic graph is introduced to enforce the spatiotemporal constraints in terms of intensity and gradient.

[0134] Optionally, the real frame image and the predicted frame image are input into a generative adversarial network to train the generative adversarial network.

[0135] The generative adversarial network is a generative adversarial network trained by generator loss and discriminant loss.

[0136] The generator loss of the generative adversarial network is specifically obtained by the following calculation method:

[0137]

[0138] Among them, the Represents the value of the predicted frame image through the discriminator in the generative adversarial network, It represents the pixel point with horizontal coordinate i and vertical coordinate j in the predicted frame image. RMSE is the root mean square error, i represents the horizontal coordinate, and j represents the vertical coordinate.

[0139] The discriminator loss of the generative adversarial network is specifically obtained by the following calculation method:

[0140]

[0141] Wherein, the D(I t+1 (i,j)) represents the value of the real frame image passing through the discriminator, Represents the value of the predicted frame image through the discriminator, Indicates the pixel with abscissa i and ordinate j in the predicted frame image, where I t+1 (i, j) represents the pixel point with horizontal coordinate i and vertical coordinate j in the real frame image. RMSE is the root mean square error, i represents the horizontal coordinate, and j represents the vertical coordinate.

[0142] Optionally, the obtaining a predicted image by fusing global and local information through a decoder based on the plurality of corresponding first encoding feature maps, the plurality of corresponding second encoding feature maps, the plurality of corresponding third encoding feature maps and the attention feature map includes:

[0143] The attention feature map is adjusted to the same size as the fourth encoding feature map to obtain an adjusted attention feature map.

[0144] Among them, the hidden features obtained by the multi-head attention mechanism module are adjusted to the original size of the U-shaped fully convolutional neural network (U-Net) encoder.

[0145] The adjusted attention feature map is upsampled and then fused with multiple third encoded feature maps to obtain a third upsampled feature map.

[0146] The third up-sampled feature map is up-sampled and fused with multiple second encoded feature maps to obtain a second up-sampled feature map.

[0147] The second up-sampled feature map is up-sampled and fused with multiple first encoded feature maps to obtain a first up-sampled feature map.

[0148] The first up-sampled feature map is decoded to obtain a predicted image.

[0149] The upsampling is deconvolved through a 4x4 convolution kernel.

[0150] The convolution kernel size of the encoder is set to 4×4 each time, and the maximum pooling kernel size is set to 3×3.

[0151] The adjusted attention feature map is upsampled and then fused with a coding feature map of the same size to obtain a first fused feature map. The first fused feature map is upsampled multiple times by the above method and then fused with a coding feature map of the same size. It is worthwhile that the size of the fused feature map is the same as the video frame to obtain a predicted image.

[0152] The decoder upsamples the encoded features and fuses them with the low-level features of the same resolution in the encoder, combining global spatial information with local detail information to capture richer feature information.

[0153] Through the above method, the attention feature map is upsampled and fused with the low-level features of the same resolution in the encoder to combine the global spatial information with the local detail information.

[0154] Optionally, obtaining a first coding feature map, a second coding feature map, a third coding feature map, and a fourth coding feature map through an encoder based on the video frame to be tested includes:

[0155] The video frame to be tested is encoded to obtain a first encoding feature map.

[0156] The first encoding feature map is downsampled to obtain a second encoding feature map.

[0157] The second encoding feature map is downsampled to obtain a third encoding feature map.

[0158] The third encoding feature map is downsampled to obtain a fourth encoding feature map.

[0159] The downsampling is deconvolved with a 4x4 deconvolution kernel.

[0160] The deconvolution kernel size of the decoder is set to 4×4 each time, and the maximum pooling kernel size is set to 3×3.

[0161] The method for detecting a traffic accident event by integrating temporal convolution and self-attention mechanism provided by the embodiment of the present invention has a significantly improved area under the curve (AUC) evaluation index compared with ViT (Vision Transformer, ViT) and RUVA. Specifically, the evaluation index of the method for detecting a traffic accident event by integrating temporal convolution and self-attention mechanism provided by the embodiment of the present invention reaches 0.688, while ViT and RUVA are 0.632 and 0.639 respectively. The details are shown in Table 1 below:

[0162] Table 1

[0163] method AUC This program 0.688 ViT 0.632 RUVA 0.639

[0164] Through the above solution, the accuracy and reliability of traffic accident detection are improved.

[0165] Example 2

[0166] Based on the above-mentioned method for detecting a traffic accident event by integrating temporal convolution and self-attention mechanism, an embodiment of the present invention further provides a system for detecting a traffic accident event by integrating temporal convolution and self-attention mechanism, wherein the system includes an acquisition module, an encoding module, a temporal convolution module, a multi-head attention mechanism module and a decoding module:

[0167] The acquisition module is used to obtain a plurality of video frames to be tested; the plurality of video frames to be tested are a plurality of continuous frames in the video to be predicted; the video frame to be predicted represents a video of an image of the next frame to be predicted;

[0168] The encoding module is used to input the multiple video frames to be tested into the encoder to obtain multiple encoding feature maps; the encoding feature map is the feature map after the multiple video frames to be tested are convolved through the convolution kernel of the encoder each time; the number of convolutions corresponds to the number of feature maps; obtain a fourth encoding feature map; the fourth encoding feature map is the feature map of the last convolution among the multiple encoding feature maps;

[0169] The time convolution module is used to obtain a time feature map through a time convolution network based on the fourth encoded feature map; the time feature map is a feature map with time information;

[0170] The multi-head attention mechanism module is used to obtain an attention feature map through a multi-head attention mechanism based on the time feature map;

[0171] The decoding module is used to obtain a predicted image by fusing global and local information through a decoder based on the multiple encoded feature maps and attention feature maps.

[0172] The algorithm and display provided herein are not inherently related to any particular computer, virtual system or other device. Various general purpose systems can also be used together with the teachings based on this. According to the above description, it is obvious that the structure required for constructing such systems. In addition, the present invention is not directed to any specific programming language either. It should be understood that various programming languages ​​can be utilized to realize the content of the present invention described herein, and the description of the above specific languages ​​is for disclosing the best mode of the present invention.

[0173] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0174] Similarly, it should be understood that in order to streamline the present disclosure and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the present invention, various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting the intention that the claimed invention requires more features than those explicitly recited in each claim. More specifically, as reflected in the claims below, inventive aspects lie in less than all the features of the individual embodiments disclosed above. Therefore, the claims that follow the specific embodiment are hereby expressly incorporated into the specific embodiment, with each claim itself serving as a separate embodiment of the present invention.

[0175] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition they may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0176] In addition, those skilled in the art will appreciate that, although some embodiments herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present invention and form different embodiments. For example, in the claims below, any one of the claimed embodiments may be used in any combination.

[0177] The various component embodiments of the present invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or digital signal processor (DSP) may be used in practice to implement some or all of the functions of some or all of the components in the apparatus according to an embodiment of the present invention. The present invention may also be implemented as a device or device program (e.g., a computer program and a computer program product) for executing part or all of the methods described herein. Such a program implementing the present invention may be stored on a computer-readable medium, or may have the form of one or more signals. Such a signal may be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0178] It should be noted that the above embodiments illustrate the present invention rather than limit it, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be construed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising a number of different elements and by means of a suitably programmed computer. In a unit claim enumerating a number of devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc., does not indicate any order. These words may be interpreted as names.

Claims

1. A method for detecting traffic accidents by integrating temporal convolution and self-attention mechanism, characterized in that: include: Obtaining a plurality of video frames to be tested; the plurality of video frames to be tested are images of a plurality of consecutive frames in the video to be predicted; the video frames to be tested represent the next frame image predicted for the current frame image; Based on the video frame to be tested, obtaining a first coding feature map, a second coding feature map, a third coding feature map and a fourth coding feature map through an encoder; A plurality of video frames to be tested obtain a plurality of corresponding first coding feature maps, a plurality of corresponding second coding feature maps, a plurality of corresponding third coding feature maps, and a plurality of corresponding fourth coding feature maps; Based on the plurality of fourth encoded feature maps, a temporal feature map is obtained through a temporal convolutional network; the temporal feature map is a feature map with time information; Based on the time feature map, an attention feature map is obtained through a multi-head attention mechanism; Based on the plurality of corresponding first encoding feature maps, the plurality of corresponding second encoding feature maps, the plurality of corresponding third encoding feature maps and the attention feature map, a decoder is used to fuse global and local information to obtain a predicted image; the predicted image includes the occurrence of the traffic accident; The encoder, temporal convolutional network, multi-head attention mechanism and decoder constitute a U-shaped fully convolutional neural network: The output of the encoder is the input of the temporal convolutional network, and the feature map output by the temporal convolutional network is linearly mapped to a sequence of fixed length as the input of the multi-head attention mechanism; the output of the attention mechanism encoder is the input of the decoder; Based on the plurality of fourth encoding feature maps, a temporal feature map is obtained through a temporal convolutional network; The time feature graph is a feature graph with time information, including: The temporal convolutional network includes a causal convolutional network, a dilated convolution and a residual connection; Obtaining a feature map time point; the feature map time point is a time point of the video frame to be tested corresponding to the fourth encoded feature map; Input the plurality of fourth encoded feature maps into the causal convolutional network according to the feature map time points from far to near, to obtain a time feature map; Multiple inputs of a causal convolutional network correspond to one output, and one input corresponds to one fourth encoding feature map; The causal convolution network uses a 2×2 convolution kernel to perform causal convolution, and convolves the data from far to near in sequence until it reaches the top layer; The dilated convolution increases the distance of input time steps in the causal convolutional network; Residual connections are used between each layer of the causal convolutional network; The attention feature map is obtained based on the time feature map through a multi-head attention mechanism, including: Linearly mapping the time feature graph into a sequence of fixed length to obtain a time feature sequence; Adding position embedding to the time feature sequence; one position corresponds to one time point; wherein, after linearly mapping the deepest feature map of the U-Net encoder to a sequence of fixed length, adding position embedding and inputting it into the multi-head attention mechanism; numbering each position, each number corresponding to a feature of a time point, by combining the position vector and the feature vector of a time point, the position information is introduced into the image information of each time point; wherein, the fixed length is determined by the setting of the multi-head attention mechanism; The time feature sequence with added position embedding is input into the multi-head attention mechanism to obtain the attention feature map.

2. According to claim 1, a method for detecting traffic accidents by integrating temporal convolution and self-attention mechanism is characterized in that: The multi-head attention mechanism includes multiple multi-head self-attention blocks and multi-layer perceptron blocks; Each multi-head self-attention block and multi-layer perceptron block is preceded and followed by layer normalization and residual connections.

3. According to claim 1, a method for detecting traffic accidents by integrating temporal convolution and self-attention mechanism is characterized in that: The training method of the U-shaped fully convolutional neural network comprises: Obtaining a training set; the training set includes a plurality of training images and a plurality of real frame images; Inputting the training image into a U-shaped fully convolutional neural network to obtain a predicted frame image; Based on the predicted frame image and the corresponding real frame image, calculating the loss through the objective function to obtain the total generation loss; The objective function is specifically calculated in the following manner: Among them, L G is the total generation loss, is the strength loss, is the gradient loss, is the generator loss; int is the weight parameter of the intensity loss, λ gd is the weight parameter of the gradient loss, λ adv is the weight parameter of the generator loss, Represents the training prediction frame image, I t+1 Represents a real frame image.

4. The method for detecting traffic accidents by integrating temporal convolution and self-attention mechanism according to claim 3 is characterized in that: The strength loss is specifically obtained by the following calculation method: in, Represents the predicted frame image, I t+1 Represents a real frame image; The gradient loss is specifically obtained by the following calculation method: in, is the strength loss, is the gradient loss, is the generator loss, represents the pixel with abscissa i and ordinate j in the training prediction frame image, where I t+1 (i, j) represents the pixel point with horizontal coordinate i and vertical coordinate j in the labeled real frame image, RMSE is the root mean square error, || ||2 is the l2 norm, i represents the horizontal coordinate, and j represents the vertical coordinate.

5. According to claim 3, a method for detecting traffic accidents by integrating temporal convolution and self-attention mechanism is characterized in that: Input the real frame image and the predicted frame image into the generative adversarial network to train the generative adversarial network; The generative adversarial network is a generative adversarial network trained by generator loss and discriminant loss; The generator loss of the generative adversarial network is specifically obtained by the following calculation method: Among them, the Represents the value of the predicted frame image through the discriminator in the generative adversarial network, It indicates the pixel with horizontal coordinate i and vertical coordinate j in the predicted frame image. RMSE is the root mean square error, i represents the horizontal coordinate and j represents the vertical coordinate. The discriminator loss of the generative adversarial network is specifically obtained by the following calculation method: Wherein, the D(I t+1 (i,j)) represents the value of the real frame image passing through the discriminator, Represents the value of the predicted frame image through the discriminator, Indicates the pixel with abscissa i and ordinate j in the predicted frame image, where I t+1 (i, j) represents the pixel point with horizontal coordinate i and vertical coordinate j in the real frame image. RMSE is the root mean square error, i represents the horizontal coordinate, and j represents the vertical coordinate.

6. The method for detecting traffic accidents by integrating temporal convolution and self-attention mechanism according to claim 1, characterized in that: The method of obtaining a predicted image by fusing global and local information through a decoder based on the plurality of corresponding first encoding feature maps, the plurality of corresponding second encoding feature maps, the plurality of corresponding third encoding feature maps and the attention feature map comprises: Adjusting the attention feature map to the same size as the fourth encoding feature map to obtain an adjusted attention feature map; Upsampling the adjusted attention feature map and fusing multiple third encoded feature maps to obtain a third upsampled feature map; Upsampling the third upsampled feature map and fusing multiple second encoded feature maps to obtain a second upsampled feature map; Upsampling the second upsampled feature map and fusing multiple first encoded feature maps to obtain a first upsampled feature map; Decoding the first up-sampled feature map to obtain a predicted image; The upsampling is deconvolved through a 4x4 convolution kernel.

7. The method for detecting traffic accidents by integrating temporal convolution and self-attention mechanism according to claim 1, characterized in that: The method of obtaining a first coding feature map, a second coding feature map, a third coding feature map and a fourth coding feature map based on the video frame to be tested through an encoder includes: Encoding the video frame to be tested to obtain a first encoding feature map; Downsampling the first encoding feature map to obtain a second encoding feature map; Downsampling the second encoding feature map to obtain a third encoding feature map; Downsampling the third encoding feature map to obtain a fourth encoding feature map; The downsampling is deconvolved with a 4x4 deconvolution kernel.

8. A traffic accident detection system integrating temporal convolution and self-attention mechanism, characterized in that: include: Acquisition module: obtain multiple video frames to be tested; the multiple video frames to be tested are multiple consecutive frame images in the video to be predicted; the video frames to be tested represent the next frame image predicted by the current frame image; Coding module: Based on the video frame to be tested, obtaining a first coding feature map, a second coding feature map, a third coding feature map and a fourth coding feature map through an encoder; A plurality of video frames to be tested obtain a plurality of corresponding first coding feature maps, a plurality of corresponding second coding feature maps, a plurality of corresponding third coding feature maps, and a plurality of corresponding fourth coding feature maps; Temporal convolution module: based on multiple fourth encoding feature maps, a temporal feature map is obtained through a temporal convolution network; the temporal feature map is a feature map with time information; Multi-head attention mechanism module: based on the time feature map, an attention feature map is obtained through a multi-head attention mechanism; Decoding module: Based on the plurality of corresponding first coding feature maps, the plurality of corresponding second coding feature maps, the plurality of corresponding third coding feature maps and the attention feature map, a decoder is used to fuse the global and local information to obtain a predicted image; the predicted image includes the occurrence of the traffic accident; The encoder, temporal convolutional network, multi-head attention mechanism and decoder constitute a U-shaped fully convolutional neural network: The output of the encoder is the input of the temporal convolutional network, and the feature map output by the temporal convolutional network is linearly mapped to a sequence of fixed length as the input of the multi-head attention mechanism; the output of the attention mechanism encoder is the input of the decoder; Based on the plurality of fourth encoding feature maps, a temporal feature map is obtained through a temporal convolutional network; The time feature graph is a feature graph with time information, including: The temporal convolutional network includes a causal convolutional network, a dilated convolution and a residual connection; Obtaining a feature map time point; the feature map time point is a time point of the video frame to be tested corresponding to the fourth encoded feature map; Input the plurality of fourth encoded feature maps into the causal convolutional network according to the feature map time points from far to near, to obtain a time feature map; Multiple inputs of a causal convolutional network correspond to one output, and one input corresponds to one fourth encoding feature map; The causal convolution network uses a 2×2 convolution kernel to perform causal convolution, and convolves the data from far to near in sequence until it reaches the top layer; The dilated convolution increases the distance of input time steps in the causal convolutional network; Residual connections are used between each layer of the causal convolutional network; The attention feature map is obtained based on the time feature map through a multi-head attention mechanism, including: Linearly mapping the time feature graph into a sequence of fixed length to obtain a time feature sequence; Adding position embedding to the time feature sequence; one position corresponds to one time point; wherein, after linearly mapping the deepest feature map of the U-Net encoder to a sequence of fixed length, adding position embedding and inputting it into the multi-head attention mechanism; numbering each position, each number corresponding to a feature of a time point, by combining the position vector and the feature vector of a time point, the position information is introduced into the image information of each time point; wherein, the fixed length is determined by the setting of the multi-head attention mechanism; The time feature sequence with added position embedding is input into the multi-head attention mechanism to obtain the attention feature map.

Images