Traffic video anomaly detection method based on multi-scale bottleneck transformer

Abstract

The application discloses a traffic video anomaly detection method based on a multi-scale bottleneck Transformer. The application establishes a neural network model based on a multi-scale bottleneck Transformer multi-modal fusion module, including a plurality of single-modal encoders, a multi-modal fusion module, a global encoder and an anomaly detection head. The multi-modal fusion module uses a reduced number of bottleneck tokens to gradually compress information and fuse each pair of modal features, and uses a weighting scheme based on the bottleneck tokens to highlight more important fusion features. In addition, the application proposes a temporal consistency contrast loss function to semantically align the pair of fused features. Experimental results prove that the application achieves advanced performance on the largest video anomaly detection dataset XD-Violence.

Description

Technical Field

[0001] This invention relates to a traffic video anomaly detection method in the field of computer vision technology, and in particular to a multimodal traffic video anomaly detection method based on Multiscale Bottleneck Transformer (MSBT). Background Technology

[0002] Traffic video anomaly detection aims to identify abnormal behaviors or events in traffic video streams using video analytics. With the development of artificial intelligence, the field of computer vision has also seen many significant breakthroughs, and using computer vision technology to detect traffic video anomalies is a very promising method. In the pursuit of effective multimodal traffic video anomaly detection, three key problems need to be addressed:

[0003] 1) Information redundancy: Each modality contains redundant information, which may introduce unwanted semantic biases.

[0004] 2) Modal imbalance: The information content within one modality may far exceed that of another modality. Treating each modality equally may lead to a decrease in detection performance.

[0005] 3) Modal asynchrony: Even if the signals are synchronized, there may be time inconsistencies between different modes.

[0006] With the availability of multimodal data, there has been increasing attention focused on anomaly detection in traffic videos. Many works have dedicated themselves to fusing audio and visual modalities, leading to the development of various fusion methods, such as connection-based and cross-attention-based techniques. While the attention mechanisms used in these works implicitly address the information redundancy problem, few works have explicitly addressed the inherent issues of information redundancy, modal imbalance, and modal desynchronization in their design. Summary of the Invention

[0007] To address the problems existing in the background technology, this invention provides a multimodal traffic video anomaly detection method based on multi-scale bottleneck Transformer. It performs feature fusion between modalities through a multimodal fusion module based on multi-scale bottleneck Transformer, adopts a bottleneck token-based weighting strategy to highlight more important fused features, and utilizes consistency contrast loss to semantically align paired fused features.

[0008] The technical solution adopted in this invention is:

[0009] I. A Multimodal Traffic Video Anomaly Detection Method Based on Multi-Scale Bottleneck Transformer

[0010] 1) Obtain a multimodal traffic video dataset;

[0011] 2) Construct a neural network model based on the multi-scale bottleneck Transformer multi-modal fusion module. After training the neural network model based on the multi-scale bottleneck Transformer multi-modal fusion module using a multi-modal traffic video dataset, a multi-modal traffic video anomaly detection network model is obtained.

[0012] 3) Input the multimodal traffic video data to be detected into the multimodal traffic video anomaly detection network model, and the model outputs the traffic video anomaly detection results.

[0013] Each of the single-mode encoders includes a backbone network, a projection layer, and a single-mode Transformer layer connected in sequence.

[0014] In step 2), the neural network model based on the multi-scale bottleneck Transformer multimodal fusion module includes a single-modal encoder, a multimodal fusion module, a global encoder, and an anomaly detection head. Data of the same modality in the multimodal traffic video dataset is input into the corresponding single-modal encoder. Multiple single-modal encoders are connected to the multimodal fusion module. The multimodal fusion module is connected to the anomaly detection head after passing through the global encoder. The output of the anomaly detection head is used as the output of the neural network model.

[0015] The multimodal fusion module includes a connected multi-scale bottleneck Transformer module and a weight fusion module. Multiple single-modal features input into the multimodal fusion module are combined in pairs to form multiple modal pairs. Each modal pair is input into the multi-scale bottleneck Transformer module and then outputs the corresponding bimodal fusion feature. Then, the weight fusion module fuses all the bimodal fusion features to obtain the multimodal fusion feature, which is used as the output of the multimodal fusion module.

[0016] The multi-scale bottleneck Transformer module includes multiple sequentially connected multi-scale bottleneck Transformer layers. Each multi-scale bottleneck Transformer layer includes a first Transformer layer and a third Transformer layer. The first single-modal feature and the first bottleneck token in the input of each multi-scale bottleneck Transformer layer are concatenated on a channel before being input into the first Transformer layer. The output of the first Transformer layer is then split on a channel to obtain the second bottleneck token and the first single-modal feature of the next multi-scale bottleneck Transformer layer. The second single-modal feature and the second bottleneck token in the input of each multi-scale bottleneck Transformer layer are concatenated on a channel before being input into the second Transformer layer. The second Transformer layer outputs the second single-modal feature of the next multi-scale bottleneck Transformer layer. The second bottleneck token is also input into the third Transformer layer along with the learnable bottleneck token. The third Transformer layer outputs the first bottleneck token of the next multi-scale bottleneck Transformer layer.

[0017] In the weight fusion module, the second bottleneck token output by the last multi-scale bottleneck Transformer layer corresponding to each bimodal fusion feature is sequentially input into L. w In the Transformer layer and the regression layer, the regression layer outputs the weights corresponding to the bimodal fusion feature. Then, based on the weights of each bimodal fusion feature, all bimodal fusion features are weighted and concatenated along the channel dimension to obtain the multimodal fusion feature.

[0018] In step 2), during the training of the neural network model based on the multi-scale bottleneck Transformer multimodal fusion module, the loss function includes a temporal consistency comparison loss, as shown in the following formula:

[0019]

[0020] in, N represents the time consistency comparison loss value. F This represents the number of modal pairs, and T represents the number of time segments. Let φ(·,·) represent the modal set, φ(·,·) represent the calculation of cosine similarity, and τ represent the temperature hyperparameter. This represents the fusion feature from mode a to mode b in segment t. This represents the fusion feature from mode c to mode d in segment k.

[0021] The modal information in the multimodal traffic video dataset includes RGB video stream, optical stream, and audio stream.

[0022] When the input modal information is RGB video stream and optical stream, the backbone network adopts the I3D model; when the input modal information is audio modality, the backbone network adopts the VGGish model.

[0023] II. A computer device

[0024] The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the method.

[0025] III. A computer-readable storage medium

[0026] The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method.

[0027] IV. A computer program product

[0028] The product includes a computer program / instructions that, when executed by a processor, implement the steps of the method.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] 1. This invention proposes a fusion module based on a multi-scale bottleneck Transformer (i.e., a multi-modal fusion module), which uses a reduced number of bottleneck tokens to progressively compress information and fuse each pair of modal features, and utilizes a weighting scheme based on bottleneck tokens to highlight more important fusion features, effectively solving the problems of information redundancy and modal imbalance.

[0031] 2. This invention proposes a temporal consistency contrast loss, which effectively addresses the problem of modal asynchrony by semantically aligning and fusing paired features at the same moment in the video. Attached Figure Description

[0032] Figure 1 This is a flowchart of a method according to an embodiment of the present invention.

[0033] Figure 2 This is a schematic diagram of the network model in an embodiment of the present invention.

[0034] Figure 3 This is a schematic diagram of the multimodal mixing process in an embodiment of the present invention. Detailed Implementation

[0035] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0036] The embodiments and specific implementation processes of the present invention are as follows:

[0037] like Figure 1 As shown, the present invention includes the following steps:

[0038] 1) Construct a multimodal traffic video dataset; the modal information in the multimodal traffic video dataset includes RGB video stream, optical stream, and audio stream.

[0039] In practice, cameras, microphones, optical flow sensors and other devices are used in various scene areas to collect video data, audio data and optical flow data, and then abnormal videos are labeled, and then the training set and test set are divided.

[0040] 2) Construct a neural network model based on the multi-scale bottleneck Transformer multi-modal fusion module. After training the neural network model based on the multi-scale bottleneck Transformer multi-modal fusion module using a multi-modal traffic video dataset, a multi-modal traffic video anomaly detection network model is obtained.

[0041] like Figure 2 As shown, the neural network model based on the multi-scale bottleneck Transformer multimodal fusion module includes a single-modal encoder, a multimodal fusion module, a global encoder, and an anomaly detection head. Data of the same modality in the multimodal traffic video dataset is input into a uniquely corresponding single-modal encoder, that is, the number of modalities is the same as the number of single-modal encoders. Multiple single-modal encoders are connected to the multimodal fusion module. The multimodal fusion module is connected to the anomaly detection head after passing through the global encoder. The output of the anomaly detection head is used as the output of the neural network model. The anomaly detection head is used to output the anomaly score of each frame of the video stream.

[0042] Each unimodal encoder consists of a backbone network, a projection layer, and a unimodal Transformer layer connected in sequence. The backbone network is used to tokenize the input information, and a linear projection layer obtains features of a uniform dimension across all modalities. D E The feature dimension is L. Finally, the original Transformer is used to aggregate the context information of each modality; that is, the embedded features are fed into a vector with L. U A single-modal Transformer with layers, where each layer contains a multi-head attention (MSA) and a feedforward network (FFN) module. The l-th layer Transformer is computed as follows: z l+1 =Transformer(z l The expression is as follows:

[0043]

[0044]

[0045] Here, LN(·) denotes layer normalization. The multi-head attention (MSA) module consists of multiple parallel attention mechanisms (called "heads"), each learning a representation of a different part of the input data. The feedforward network (FFN) module contains two linear layers and a ReLU activation function between the two layers. The backbone network and linear projection layers are modality-specific, while the single-modality Transformer is shared across all modalities.

[0046] When the input modal information is RGB video stream and optical flow, the backbone network adopts the I3D model; when the input modal information is audio modality, the backbone network adopts the VGGish model.

[0047] The multimodal fusion module is used to fuse features from two modalities. The multimodal fusion module includes a connected multiscale bottleneck Transformer (MSBT) module and a weight fusion module. Multiple single-modal features input into the multimodal fusion module are combined in pairs to form multiple modal pairs. Each modal pair is input into the multiscale bottleneck Transformer module and then outputs the corresponding bimodal fusion feature. Then, the weight fusion module fuses all the bimodal fusion features to obtain the multimodal fusion feature, which is used as the output of the multimodal fusion module.

[0048] like Figure 2 and Figure 3 As shown, the multi-scale bottleneck Transformer module includes multiple sequentially connected multi-scale bottleneck Transformer layers. Each multi-scale bottleneck Transformer layer includes a first Transformer layer and a third Transformer layer. The first single-modal feature and the first bottleneck token in the input of each multi-scale bottleneck Transformer layer are concatenated on a channel before being input into the first Transformer layer. The output of the first Transformer layer is then split on a channel to obtain the second bottleneck token and the first single-modal feature of the next multi-scale bottleneck Transformer layer. The second single-modal feature and the second bottleneck token in the input of each multi-scale bottleneck Transformer layer are concatenated on a channel before being input into the second Transformer layer. The second Transformer layer outputs the second single-modal feature of the next multi-scale bottleneck Transformer layer. The second bottleneck token is also input into the third Transformer layer along with the learnable bottleneck token. The third Transformer layer outputs the first bottleneck token of the next multi-scale bottleneck Transformer layer.

[0049] In this embodiment, taking RGB video stream, optical stream, and audio stream as examples, the single-mode features obtained after the RGB video stream, optical stream, and audio stream are passed through the corresponding single-mode encoders are denoted as Z. R Z F Z A ; in Let T represent a segment T in the output of a single-modal Transformer layer, where T represents the number of segments. Given features Z of two different modalities a and b. a Z b , total L M Information about the MSBT hybrid mode a to mode b of layer l, for the l-th layer:

[0050]

[0051]

[0052] Here, [·||·] represents a join operation. This represents the first bottleneck token learned from layers l and input to layer l, while It was obtained through random initialization. Carrying information from previously compressed mode a. For the current layer, the first bottleneck token. Acting as an intermediary for transmitting information from mode a to mode b, it first re-aggregates the information from mode a, and then passes the information to mode b, thereby obtaining the fusion result. On the other hand, in order to enable the bottleneck tokens to function as a means of passing compressed information from the current layer to the next layer, an additional set of learnable bottleneck tokens is initialized in layer l. The number of tokens is reduced by half, that is To store more compressed information. The compressed information from layer l is passed to layer l+1 via a cross-attention-based Transformer:

[0053]

[0054] Attention is calculated in the cross-attention Transformer (x, y) using Attention(xW). Q ,yW K ,yW V ).

[0055] In the fusion modal feature Z a To Z b At that time, take The fusion result (i.e., the second single-modal feature output by the last multi-scale bottleneck Transformer layer) is denoted as Z. ab The entire process involves transmitting compressed information from mode a to mode b; that is, the fusion of the two modes is asymmetric. Therefore, after fusing all mode pairs, six paired mode fusion features are obtained, including Z. RF Z FR Z RA Z AR Z FA Z AF .

[0056] To obtain better fused features across all modalities, a bottleneck token-based weighting strategy is adopted. The second bottleneck token is used in the last multi-scale bottleneck Transformer layer. This can be used to measure important metrics for information transformation between different modalities. Therefore, this invention uses learned bottleneck tokens to weight the fusion features of all modality pairs to highlight more important features. In the weighted fusion module, the second bottleneck token output from the last multi-scale bottleneck Transformer layer corresponding to each bimodal fusion feature is sequentially input into L. w In the Transformer and regression layers, the regression layer outputs the weights corresponding to the bimodal fusion feature, as shown in the following formula:

[0057]

[0058] Among them, Z bt This represents the stack of final bottleneck tokens obtained from all pairwise modal fusions; L represents w Layer Transformer; This represents a regressor consisting of a three-layer multilayer perceptron;

[0059] Next, based on the weight of each bimodal fusion feature, all bimodal fusion features are weighted and concatenated along the channel dimension to obtain the multimodal fusion feature. The final weighted feature calculation formula is as follows:

[0060]

[0061] in, ∥ represents multimodal fusion features, and ∥ represents channel dimension splicing.

[0062] In the global encoder and anomaly score calculation stages, the multimodal fusion features generated by the multimodal fusion module... After a L GA global encoder composed of layers of Transformers is used to model the global context information of all modalities, and a regressor is further used to obtain anomaly scores:

[0063]

[0064] in, Ω(·) is the anomaly score of all segments in the video, and Ω(·) is a regressor implemented by a three-layer multilayer perceptron.

[0065] To ensure that features fused at the same time are semantically similar, this invention proposes Temporal Consistency Contrast (TCC) loss. During the training of the neural network model based on the multi-scale bottleneck Transformer multimodal fusion module, the loss function includes the Temporal Consistency Contrast loss, as shown in the following formula:

[0066]

[0067] in, N represents the time consistency comparison loss value. F This represents the number of modal pairs, and T represents the number of time segments. Let φ(·,·) represent the modal set, φ(·,·) represent the calculation of cosine similarity, and τ represent the temperature hyperparameter. This represents the fusion feature from mode a to mode b in segment t. This represents the fusion feature from mode c to mode d in segment k.

[0068] During training, the regressor is trained using the top-K MIL loss and feature regularization is performed using temporal consistency contrastive loss. To compute the MIL loss for a video, the average of the top K anomaly scores is obtained from the set of video anomaly scores, s. Γ k (S) represents the K highest anomaly scores in set s, and the MIL loss. The definition is as follows:

[0069]

[0070] Where y represents the label, which is 0 or 1.

[0071] The entire training loss is the top-K MIL loss. Loss compared to time consistency Combinations:

[0072]

[0073] Here, λ is a hyperparameter that balances the two types of losses.

[0074] 3) Input the multimodal traffic video data to be detected into the multimodal traffic video anomaly detection network model, and the model outputs the traffic video anomaly detection results.

[0075] To verify the effectiveness of this invention, it was validated on XD-Violence, the largest publicly available video anomaly detection dataset, and compared with state-of-the-art video anomaly detection methods. XD-Violence contains 4754 undressed videos, totaling 217 hours in length, covering six anomaly types, and providing RGB, optical flow, and audio streams for each video. The entire dataset was divided into a training set of 3954 videos with video-level labels and a test set of 800 videos with frame-level labels.

[0076] This invention uses frame-level average precision (AP) as the evaluation metric. To achieve optimal network performance, the parameter details are as follows: In the single-modal encoder, the dimension D of the encoded feature... E =128, attention heads are 4, number of unimodal Transformer layers L U =1. In MSBT, the number of MSBT layers L is used. M =5, the number of bottleneck tokens in the first layer In the weight calculation module of the multimodal fusion module, L is used. w =1. In the global encoder, L is used. G =4 Transformer layers to aggregate global context information. The temperature hyperparameter τ of the temporal consistency comparison loss is 0.5, and K=9 in the top-K MIL loss, balancing the top-K MIL loss. Loss compared to time consistency The hyperparameter λ = 0.1. Furthermore, the model was trained for 50 epochs using the SGD optimizer, with a batch size of 128 and a learning rate of 0.0005.

[0077] The experiment mainly consists of two parts. The first part is a comparative experiment between the method of this invention and the most advanced video anomaly detection methods. The second part is a controlled variable experiment of each module in the network of this invention to illustrate the effectiveness of each module.

[0078] Part 1: Comparative experiments with state-of-the-art video anomaly detection methods demonstrate the superior performance of the network in this invention. Table 1 shows the comparison results of the MSTB network model of this invention with other state-of-the-art methods on the XD-Violence dataset. The comparison method uses multimodal combinations as input, namely RGB+audio, RGB+optical flow, audio+optical flow, and RGB+audio+optical flow, to illustrate the experimental results of frame-level average precision (AP) on the XD-Violence dataset. * indicates the results obtained by training the same model under different multimodal input combinations. The comparison results show that, except for the RGB+audio input combination which ranks second among published or backup paper methods, the network in this invention achieves the best performance for all other multimodal input combinations.

[0079] Table 1 compares the performance metrics of this invention with those of published state-of-the-art (SOTA) methods on XD-Violence.

[0080]

[0081] Part Two: Controlled variable experiments for each module in this invention to illustrate the effectiveness of each module. Table 2 shows the frame-level average accuracy (AP) on the XD-Violence dataset for model variants with and without Multi-Scale Bottleneck Transformer (MSBT) and Temporal Consistency Contrast (TCC) loss under different multimodal input combinations. The comparison results show that the models using MSBT and TCC improve AP under all multimodal input combinations, proving the effectiveness of both.

[0082] Table 2 is a comparison of indicators (AP (%)) for whether MSBT and TCC are used.

[0083]

[0084] Table 3 shows the frame-level average precision (AP) performance of different model variants on the XD-Violence dataset. 'CT' indicates that the multi-scale bottleneck Transformer (MSBT) module uses a Cross-Transformer, 'T_W' indicates that the weight calculation of the multimodal fusion module uses a Transformer, and 'Weighting' indicates the use of a bottleneck token-based weighting strategy. The comparison results show that the model using the bottleneck token-based weighting strategy improves AP across all multimodal input combinations, demonstrating the effectiveness of the bottleneck token-based weighting strategy.

[0085] Table 3 compares the performance indicators (AP (%)) of different model variants.

[0086]

[0087]

[0088] Table 4 shows the frame-level average precision (AP) performance of model variants using a fixed number and a decreasing number of bottleneck tokens per layer in the Multi-Scale Bottleneck Transformer (MSBT) module on the XD-Violence dataset. The model's multimodal input combination is RGB + audio + optical flow. In the fixed number, 2, 4, 8, and 16 represent the number of tokens used in all layers; in the decreasing number per layer, 2, 4, 8, and 16 represent the number of tokens in the first layer. The comparison results show that the method using a decreasing number of bottleneck tokens per layer in this invention is more effective.

[0089] Table 4 compares the bottleneck token metrics (AP (%)) of fixed number and decreasing number per layer in MSBT.

[0090]

[0091] The above description is merely a specific embodiment of the present invention and is not intended to limit the present invention in any way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments. However, any simple modifications, equivalent changes, and modifications made to the above examples based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A traffic video anomaly detection method based on multi-scale bottleneck Transformer, characterized in that, Includes the following steps: 1) Obtain a multimodal traffic video dataset; 2) Construct a neural network model based on the multi-scale bottleneck Transformer multi-modal fusion module. After training the neural network model based on the multi-scale bottleneck Transformer multi-modal fusion module using a multi-modal traffic video dataset, a multi-modal traffic video anomaly detection network model is obtained. In step 2), the neural network model based on the multi-scale bottleneck Transformer multimodal fusion module includes a single-modal encoder, a multimodal fusion module, a global encoder, and an anomaly detection head. Data of the same modality in the multimodal traffic video dataset is input into the corresponding single-modal encoder. Multiple single-modal encoders are connected to the multimodal fusion module. The multimodal fusion module is connected to the anomaly detection head after passing through the global encoder. The output of the anomaly detection head is used as the output of the neural network model. The multimodal fusion module includes a connected multi-scale bottleneck Transformer module and a weight fusion module. Multiple single-modal features input into the multimodal fusion module are combined in pairs to form multiple modal pairs. Each modal pair is input into the multi-scale bottleneck Transformer module and then outputs the corresponding bimodal fusion feature. Then, the weight fusion module fuses all the bimodal fusion features to obtain the multimodal fusion feature, which is used as the output of the multimodal fusion module. The multi-scale bottleneck Transformer module includes multiple sequentially connected multi-scale bottleneck Transformer layers. Each multi-scale bottleneck Transformer layer includes a first Transformer layer and a third Transformer layer. The first single-modal feature and the first bottleneck token in the input of each multi-scale bottleneck Transformer layer are concatenated on a channel before being input into the first Transformer layer. The output of the first Transformer layer is then split on a channel to obtain the second bottleneck token and the first single-modal feature of the next multi-scale bottleneck Transformer layer. The second single-modal feature and the second bottleneck token in the input of each multi-scale bottleneck Transformer layer are concatenated on a channel before being input into the second Transformer layer. The second Transformer layer outputs the second single-modal feature of the next multi-scale bottleneck Transformer layer. The second bottleneck token is also input into the third Transformer layer along with the learnable bottleneck token. The third Transformer layer outputs the first bottleneck token of the next multi-scale bottleneck Transformer layer. 3) Input the multimodal traffic video data to be detected into the multimodal traffic video anomaly detection network model, and the model outputs the traffic video anomaly detection results.

2. The traffic video anomaly detection method based on multi-scale bottleneck Transformer according to claim 1, characterized in that, Each of the single-mode encoders includes a backbone network, a projection layer, and a single-mode Transformer layer connected in sequence.

3. The traffic video anomaly detection method based on multi-scale bottleneck Transformer according to claim 1, characterized in that, In the weight fusion module, the second bottleneck token output by the last multi-scale bottleneck Transformer layer corresponding to each bimodal fusion feature is sequentially input into the weight fusion module. In the Transformer layer and the regression layer, the regression layer outputs the weights corresponding to the bimodal fusion feature. Then, based on the weights of each bimodal fusion feature, all bimodal fusion features are weighted and concatenated along the channel dimension to obtain the multimodal fusion feature.

4. The traffic video anomaly detection method based on multi-scale bottleneck Transformer according to claim 1, characterized in that, In step 2), during the training of the neural network model based on the multi-scale bottleneck Transformer multimodal fusion module, the loss function includes a time consistency comparison loss, as shown in the following formula: in, For time consistency comparison loss value, Indicates the number of modal pairs. Indicates the number of time segments. Represents a set of modes. This indicates the calculation of cosine similarity. This indicates temperature hyperparameters. Representing fragments Fusion characteristics from mode a to mode b. Representing fragments Fusion characteristics from mode c to mode d.

5. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 4.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 4.

7. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method described in any one of claims 1 to 4.

Images