Video abnormal event identification method based on Re-TimeSFormer model

Abstract

The invention discloses a video abnormal event identification method based on a Re-TimeSFormer model. The method comprises the following steps: a step 1, a step 2, a step 3, a step 4, a step 5 and a step 6, the method comprises the following steps of: 1, preprocessing F video frames of which the original video sampling size is H * W; 2, converting the preprocessed video frame into an image block embedding vector # imgabs0 #, and sending the image block embedding vector # imgabs0 # into an abnormal event identification model; step 3, an abnormal event identification model based on Re-TimeSform is used for detecting whether an event set to be abnormal exists in the video or not; 4, calculating the difference between a predicted value and a real label by using a Focal Loss loss function, and carrying out parameter optimization to obtain a trained abnormal event recognition model; and 5, in a test stage of the abnormal event identification module, inputting a test video into the trained abnormal event identification model to obtain an identification result. According to the method, the capability of recognizing the low-level features can be improved while the capability of extracting the global context by the Transformers model is reserved, and the diversity of the visual features and the stability of the model are improved, so that the performance of a visual recognition task is enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of video anomaly detection, and in particular to a video abnormal event recognition method based on a Re-TimeSFormer model. Background Art

[0002] With the advancement of smart city public security information construction, video surveillance, as an important means of security, has been widely deployed and applied in major public places to detect abnormal situations as early as possible and take corresponding measures to protect the lives and property of the public and promote urban public security management. Relying on traditional manual screening and analysis of surveillance videos, usually after the event alarm, the surveillance video is reviewed again, which has a series of problems such as low efficiency and high cost. Therefore, how to process massive video data, detect events that threaten public safety in real time, and process and alarm them has become an urgent problem to be solved in modern intelligent monitoring systems.

[0003] Video anomaly detection uses computer vision algorithms to identify whether there are events that threaten public safety in surveillance videos, such as fighting, robbery, and vandalism. The intelligent monitoring system that integrates the abnormal event detection model can automatically detect and report abnormal events, allowing security personnel to quickly take countermeasures to avoid further deterioration of abnormal situations and provide the public with a safer and more harmonious environment. In addition, it can also reduce the workload of monitoring personnel and improve work efficiency. Therefore, building an efficient abnormal event detection model and realizing intelligent analysis of surveillance videos are of great significance to ensuring public safety and promoting urban safety development.

[0004] With the rapid development of computer vision technology, abnormal event detection in video content has become an important research topic in the current field of intelligent video analysis. In recent years, many experts and scholars at home and abroad have conducted in-depth research on abnormal event detection in videos. Among them, researchers have tried to apply Transformer to images and videos. The Vision Transformer (ViT) proposed by Dosovitskiy et al. successfully applied the Transformer of natural language processing to computer vision and performed image classification tasks. TimeSFormer applies ViT to video recognition by extending the self-attention mechanism from image space to spatiotemporal 3D volume and designing a divided space-time attention mechanism.

[0005] However, in the ViT architecture, the attention mechanism is designed to capture the global relationship between different feature patches. The features extracted from different image patches will lose their uniqueness as the network depth increases and become more and more similar, resulting in feature collapse and loss of feature diversity, leading to reduced visual recognition performance.

[0006] Therefore, how to avoid feature collapse and improve recognition performance is one of the challenges facing Transformer-based video abnormal event detection. Summary of the invention

[0007] In order to overcome the above technical problems, the purpose of the present invention is to provide a video abnormal event recognition method based on the Re-TimeSFormer (the Residual Attention Time-Space Transformer) model, which can improve the ability to recognize low-level features while retaining the ability of the Transformers model to extract global context, increase the diversity of visual features and model stability, thereby enhancing the performance of visual recognition tasks.

[0008] In order to achieve the above object, the technical solution adopted by the present invention is:

[0009] A video abnormal event recognition method based on the Re-TimeSFormer model comprises the following steps:

[0010] Step 1: Sample the original video into F video frames x of size H×W f (f=1, 2, ..., F) for preprocessing;

[0011] Step 2: Data processing: Convert preprocessed video frames into image patch embedding vectors Then it is sent to the abnormal event recognition model;

[0012] Step 3: Build an abnormal event recognition model based on Re-TimeSFormer. The abnormal event recognition model based on Re-TimeSFormer is used to detect whether there are events set as abnormal in the video;

[0013] Step 4: Use the Focal Loss function to calculate the difference between the predicted value and the true label, and use this to optimize the parameters to obtain a trained abnormal event recognition model;

[0014] Step 5: In the testing phase of the abnormal event recognition module, the test video is input into the trained abnormal event recognition model to obtain the recognition result.

[0015] The preprocessing in step 1 includes adjusting the pixels of the video frame and image enhancement, and the obtained preprocessed video frame sequence is used for subsequent information encoding and anomaly identification.

[0016] The data processing in step 2 is specifically as follows:

[0017] Each frame of F video frames is divided into N non-overlapping patches of size P×P, i.e., image blocks, where N = HW / P 2 , (H; W) is the resolution of each frame image;

[0018] Then flatten these patches into one-dimensional vectors Each vector is linearly mapped to an embedding vector through a learnable weight matrix E The calculation formula is as follows, and a learnable category vector is added at the first position of the sequence

[0019]

[0020] Where p = 1, ..., N; represents the sequence number of the patch; t = 1, ..., F; represents the index of different video frames, Represents the learnable position embedding added to encode the spatiotemporal position of each image patch.

[0021] The abnormal event recognition model based on Re-TimeSFormer in step 3 includes a spatiotemporal separation self-attention module, a residual attention module and a classification module;

[0022] Step 3.1: Embed the image block into a vector Send to the spatiotemporal separation self-attention module;

[0023] Step 3.2: Construct a spatiotemporal separation self-attention module, that is, cascade the temporal attention module and the spatial attention module, add normalization and residual connections to form a spatiotemporal attention layer, and then stack multiple spatiotemporal attention layers;

[0024] Step 3.3: Introduce the residual attention module based on the spatiotemporal separation attention module;

[0025] Step 3.4: Select the fully connected layer as the classification module.

[0026] The spatiotemporal separation attention module in step 3.2 specifically includes a temporal self-attention module and a spatial self-attention module, specifically:

[0027] The temporal self-attention module calculates by comparing with all image blocks at the same spatial position in other frames to obtain the temporal attention of the position, and then feeds the obtained result back to the spatial attention calculation; the spatial self-attention module simultaneously calculates the information of each frame at the same moment to obtain the spatial attention at that moment;

[0028] The spatiotemporal attention layer consists of Ψ encoding blocks, where According to the representation vector of the previous block encoding Compute the query / key / value vector for each patch:

[0029]

[0030]

[0031]

[0032] Where LN represents LayerNorm, D h =D / A represents the potential dimension of each attention head, Α represents the total number of attention heads; a=1,…,Α is the index on multiple attention heads, p represents the sequence number of the patch, and t represents the index of different video frames;

[0033] For spatiotemporal separation of attention, in each block Within, the temporal attention is first calculated by comparing each patch (p, t) with all image patches at the same spatial position in other frames, and the self-attention weight is calculated by the dot product:

[0034]

[0035] Where SM represents softmax; D h =D / A represents the potential dimension of each attention head, a=1,…,Α is the index on multiple attention heads; is the query vector, is the key vector, p represents the sequence number of the patch, and t represents the index of different video frames;

[0036] The value vector is calculated by using the self-attention coefficient of each attention head The weighted sum

[0037]

[0038] Where a=1,…,Α is the index of multiple attention heads; p represents the sequence number of the patch, and t represents the index of different video frames;

[0039] The concatenation of the vectors from all heads is then projected and passed through the MLP, using a residual connection after each operation to encode It is obtained by the following formula and then fed back to the spatial attention calculation:

[0040]

[0041] The spatial self-attention module calculates each frame of information at the same time to obtain the spatial attention at that moment, using the following formula:

[0042]

[0043] Where SM represents softmax; D h =D / A represents the potential dimension of each attention head, a=1,…,Α is the index on multiple attention heads; is the query vector, is the key vector, p represents the sequence number of the patch, and t represents the index of different video frames;

[0044] The value vector is then calculated by using the self-attention coefficient of each attention head The weighted sum

[0045]

[0046] Where a=1,…,Α is the index of multiple attention heads; p represents the sequence number of the patch, and t represents the index of different video frames;

[0047] The concatenation of the vectors from all heads is then projected and passed through the MLP, using a residual connection after each operation, to compute the block The final encoding of the patch at The specific calculation formula is as follows:

[0048]

[0049]

[0050] Among them, a=1,…,Α is the index on multiple attention heads; p represents the sequence number of the patch, and t represents the index of different video frames.

[0051] The residual attention module in step 3.3 is specifically:

[0052] Using skip connections between consecutive multi-head self-attention layers (the multi-head self-attention layer is a core component in the Transformer model, which uses multiple self-attention heads in parallel to capture different contextual information in the input sequence) allows attention to propagate and accumulate from shallow layers to deeper layers. The proposed residual attention mechanism is implemented by changing the calculation of the defined MHSA (Multi-Head Self-Attention) matrix:

[0053]

[0054]

[0055] Here, a learnable variable α∈[0, 1] is used to allow the network to autonomously determine how much attention to propagate between layers.

[0056] The classification module in step 3.4 is specifically:

[0057] From the final code Extract the category vector As the image representation y, on top of this representation, a multi-layer perceptron (MLP) with one hidden layer is added to get the predicted video category, that is, to determine whether the video is abnormal:

[0058]

[0059] The Focal Loss function in step 4 calculates the difference between the predicted value and the true label to optimize the parameters, which is implemented as follows:

[0060] Using the category vectors of F video frames Where t = 1, ..., F represents the index of different video frames, and the category score vector S = [s1, s2, ..., s k ], the calculation formula is as follows:

[0061]

[0062] Among them, K represents the number of categories, K = 2, that is, there are two categories of abnormal events in the video and no abnormal events;

[0063] Then according to the score s of the k = (1, 2, ... K) category i (i=1, 2, ..., K), calculate the probability p of the corresponding category i (i=1,2,…,K), the formula is as follows:

[0064]

[0065] During the test phase, the category with the highest probability is selected as the model's label for whether the video is abnormal;

[0066] Next, the category probability p i Input into the Focal Loss calculation model to measure the difference between the predicted value and the true label, and optimize the model parameters through loss feedback. The formula is as follows:

[0067] L focal = -α(1-p i ) γ log(p i )

[0068] Among them, α adjusts the weights between different categories according to the frequency of the category; γ is used to control the degree of weight distribution of difficult and easy samples.

[0069] Beneficial effects of the present invention:

[0070] This paper proposes an improved Re-TimeSFormer based on Vision Transformer, introduces a residual attention mechanism based on spatiotemporal separation attention, and uses the Focal loss function to measure the difference between the model prediction value and the true label to optimize the parameters.

[0071] Compared with the prior art, the technical solution proposed in the present invention achieves high efficiency and accuracy in video anomaly recognition. The attention residual learning technology is introduced in TimeSFormer, which integrates low-level information from shallow to deep network layers, establishes connections between features at different layers, avoids feature collapse, and improves recognition performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 The present invention is a flowchart of a method for identifying abnormal video events based on the Re-TimeSFormer model.

[0073] Figure 2 This is a structural diagram of the spatiotemporal separation of attention that introduces residual attention in an embodiment of the present invention. DETAILED DESCRIPTION

[0074] The present invention will be further described in detail below in conjunction with the accompanying drawings.

[0075] like Figure 1 FIG. 1 is a flowchart of a method for identifying abnormal video events based on a Re-TimeSFormer model according to an embodiment of the present invention, which specifically includes the following steps:

[0076] Step 1: Sample the original video into F video frames x of size H×W f (f=1, 2, ..., F), and perform preprocessing, including adjusting the pixels of the video frame and image enhancement.

[0077] Step 2: Data processing: converting sampled video frames into image patch embedding vectors Then it is sent to the abnormal event recognition model;

[0078] The details are as follows:

[0079] Each frame is decomposed into N non-overlapping image blocks, each block is of size P×P, where N = HW / P 2 , and then flatten these image blocks into vector x (p , t)(p=1,…,N;t=1,…,F), each block x is transformed into (p , t) Linear mapping to an embedding vector The calculation formula is as follows, and a learnable category vector is added at the first position of the sequence

[0080]

[0081] in, Represents the learnable position embedding added to encode the spatiotemporal position of each image patch.

[0082] Step 3: Construct an abnormal event recognition model based on Re-TimeSFormer; the TimeSFormer model mainly includes a spatiotemporal separation self-attention module and a classification module. The Re-TimeSFormer model is constructed by introducing residual attention based on the spatiotemporal separation attention module of TimeSFormer. Figure 2 : Introducing residual attention can improve the model's ability to recognize low-level features, thereby enhancing the performance of visual recognition tasks.

[0083] Step 3.1: Embed image patches into vectors Send to the spatiotemporal separation self-attention module;

[0084] Step 3.2: Construct a spatiotemporal separation attention module, that is, cascade the temporal attention module and the spatial attention module, add normalization and residual connections to form a spatiotemporal attention layer, and then stack multiple spatiotemporal attention layers;

[0085] The spatiotemporal separation attention module specifically includes a temporal self-attention module and a spatial self-attention module;

[0086] The temporal self-attention module calculates by comparing with all image blocks at the same spatial position in other frames to obtain the temporal attention of the position, and then feeds the obtained result back to the spatial attention calculation; the spatial self-attention module simultaneously calculates the information of each frame at the same moment to obtain the spatial attention at that moment;

[0087] The spatiotemporal attention layer consists of Ψ encoding blocks, where According to the representation vector of the previous block encoding Compute the query / key / value vector for each patch:

[0088]

[0089]

[0090]

[0091] Where LN represents LayerNorm, D h =D / A represents the potential dimension of each attention head, Α represents the total number of attention heads; a=1,…,Α is the index on multiple attention heads, p represents the sequence number of the patch, and t represents the index of different video frames;

[0092] For spatiotemporal separation of attention, in each block Within, the temporal attention is first calculated by comparing each patch (p, t) with all image patches at the same spatial position in other frames, and the self-attention weight is calculated by the dot product:

[0093]

[0094] Where SM represents the softmax activation function; D h =D / A represents the potential dimension of each attention head, a=1,…,Α is the index on multiple attention heads; is the query vector, is the key vector, p represents the sequence number of the patch, and t represents the index of different video frames;

[0095] The value vector is calculated by using the self-attention coefficient of each attention head The weighted sum

[0096]

[0097] Where a=1,…,Α is the index of multiple attention heads; p represents the sequence number of the patch, and t represents the index of different video frames;

[0098] The concatenation of the vectors from all heads is then projected and passed through the MLP, using a residual connection after each operation to encode It is obtained by the following formula and then fed back to the spatial attention calculation:

[0099]

[0100] The spatial self-attention module calculates each frame of information at the same time to obtain the spatial attention at that moment, using the following formula:

[0101]

[0102] Where SM represents softmax; D h =D / A represents the potential dimension of each attention head, a=1,…,Α is the index on multiple attention heads; is the query vector, is the key vector, p represents the sequence number of the patch, and t represents the index of different video frames;

[0103] The value vector is then calculated by using the self-attention coefficient of each attention head The weighted sum The calculation formula is as follows:

[0104]

[0105] Where a=1,…,Α is the index of multiple attention heads; p represents the sequence number of the patch, and t represents the index of different video frames;

[0106] The concatenation of the value vectors from all heads is then projected and passed through the MLP, using a residual connection after each operation, to compute the block The final encoding of the patch at The specific calculation formula is as follows:

[0107]

[0108]

[0109] Step 3.3: Introduce the residual attention mechanism based on the spatiotemporal separation attention module. Specifically, use jump connections between consecutive multi-head self-attention layers so that attention can be propagated and accumulated from shallow layers to deeper layers. The proposed residual attention mechanism is implemented by changing the calculation of the defined MHSA (Multi-Head Self-Attention) matrix:

[0110]

[0111]

[0112] Here, a learnable variable α∈[0, 1] is used to allow the network to autonomously determine how much attention to propagate between layers.

[0113] Step 3.4: Select the fully connected layer as the classification module, specifically: The final segment embedding is obtained from the final block of classification tags:

[0114]

[0115] On this basis, a hidden layer MLP is added to predict the final video class, that is, to determine whether the video is abnormal.

[0116] Step 4: Input the training set into the abnormal event recognition model, use the Focal Loss function to calculate the difference between the predicted value and the true label, and optimize the parameters to obtain the trained model. The specific implementation is as follows:

[0117] Using the category vectors of F video frames Get the category score vector S = [s1, s2, ..., s k ], the calculation formula is as follows:

[0118]

[0119] Among them, K represents the number of categories, K = 2, that is, there are two categories of abnormal events in the video and no abnormal events;

[0120] Then according to the score s of the k = (1, 2, ... K) category i (i=1, 2, ..., K), calculate the probability p of the corresponding category i (i=1,2,…,K), the formula is as follows:

[0121]

[0122] Next, the category probability p i Input into the Focal Loss calculation model to measure the difference between the predicted value and the true label, and optimize the model parameters through loss feedback. The formula is as follows:

[0123] L focal = -α(1-p i ) γ log(p i )

[0124] Among them, α adjusts the weights between different categories according to the frequency of the category; γ is used to control the degree of weight distribution of difficult and easy samples.

[0125] Step 5: Input the test set into the trained abnormal event recognition model, select the category with the highest probability as the model's label for whether the video is abnormal, and obtain the recognition result.

Claims

1. A video abnormal event recognition method based on the Re-TimeSFormer model, characterized in that: The steps include: Step 1: Sample the original video into F video frames x of size H×W f (f=1, 2, ..., F) for preprocessing; Step 2: Data processing: Convert preprocessed video frames into image patch embedding vectors After that, it is sent to the abnormal event recognition model; Step 3: Build an abnormal event recognition model based on Re-TimeSFormer. The abnormal event recognition model based on Re-TimeSFormer is used to detect whether there are events set as abnormal in the video; Step 4: Use the Focal Loss function to calculate the difference between the predicted value and the true label, and use this to optimize the parameters to obtain a trained abnormal event recognition model; Step 5: In the testing phase of the abnormal event recognition module, the test video is input into the trained abnormal event recognition model to obtain the recognition result.

2. According to the method for video abnormal event recognition based on the Re-TimeSFormer model in claim 1, it is characterized in that: The preprocessing in step 1 includes adjusting the pixels of the video frame and image enhancement, and the obtained preprocessed video frame sequence is used for subsequent information encoding and anomaly identification.

3. The video abnormal event recognition method based on the Re-TimeSFormer model according to claim 1 is characterized in that: The data processing in step 2 is specifically as follows: Each frame of F video frames is divided into N non-overlapping patches of size P×P, i.e., image blocks, where N = HW / P 2 , (H; W) is the resolution of each frame image; Then flatten the patch into a one-dimensional vector Each vector is linearly mapped to an embedding vector through a weight matrix, and a learnable category vector is added at the first position of the sequence; where p = 1, ..., N; represents the sequence number of the patch; t = 1, ..., F; represents the index of different video frames.

4. The video abnormal event recognition method based on the Re-TimeSFormer model according to claim 1 is characterized in that: The abnormal event recognition model based on Re-TimeSFormer in step 3 includes a spatiotemporal separation self-attention module, a residual attention module and a classification module; Step 3.1: Embed the image block into a vector Send to the spatiotemporal separation self-attention module; Step 3.2: Construct a spatiotemporal separation self-attention module, that is, cascade the temporal attention module and the spatial attention module, add normalization and residual connections to form a spatiotemporal attention layer, and then stack multiple spatiotemporal attention layers; Step 3.3: Introduce the residual attention module based on the spatiotemporal separation attention module; Step 3.4: Select the fully connected layer as the classification module.

5. The video abnormal event recognition method based on the Re-TimeSFormer model according to claim 4 is characterized in that: The spatiotemporal separation attention module in step 3.2 specifically includes a temporal self-attention module and a spatial self-attention module, specifically: The temporal self-attention module calculates by comparing with all image patches at the same spatial position in other frames, obtains the temporal attention of that position, and then feeds the result back to the spatial attention calculation; The spatial self-attention module simultaneously calculates each frame of information at the same moment to obtain the spatial attention at that moment; The spatiotemporal attention layer consists of Ψ encoding blocks, where According to the representation vector of the previous block encoding Compute the query / key / value vector for each patch: Where LN represents LayerNorm, D h =D / A represents the potential dimension of each attention head, Α represents the total number of attention heads; a=1,…,Α is the index on multiple attention heads, p represents the sequence number of the patch, and t represents the index of different video frames; For spatiotemporal separation of attention, in each block Within, the temporal attention is first calculated by comparing each patch (p, t) with all image patches at the same spatial position in other frames, and the self-attention weight is calculated by the dot product: Where SM represents softmax; D h =D / A represents the potential dimension of each attention head, a=1,…,Α is the index on multiple attention heads; is the query vector, is the key vector, p represents the sequence number of the patch, and t represents the index of different video frames; The value vector is calculated by using the self-attention coefficient of each attention head The weighted sum Where a=1,…,Α is the index of multiple attention heads; p represents the sequence number of the patch, and t represents the index of different video frames; The concatenation of the vectors from all heads is then projected and passed through the MLP, using a residual connection after each operation to encode It is obtained by the following formula and then fed back to the spatial attention calculation: The spatial self-attention module calculates each frame of information at the same time to obtain the spatial attention at that moment, using the following formula: Where SM represents softmax; D h =D / A represents the potential dimension of each attention head, a=1,…,Α is the index on multiple attention heads; is the query vector, is the key vector, p represents the sequence number of the patch, and t represents the index of different video frames; The value vector is then calculated by using the self-attention coefficient of each attention head The weighted sum Where a=1,…,Α is the index of multiple attention heads; p represents the sequence number of the patch, and t represents the index of different video frames; The concatenation of the vectors from all heads is then projected and passed through the MLP, using a residual connection after each operation, to compute the block The final encoding of the patch at The specific calculation formula is as follows: Among them, a=1,…,Α is the index on multiple attention heads; p represents the sequence number of the patch, and t represents the index of different video frames.

6. The video abnormal event recognition method based on the Re-TimeSFormer model according to claim 4 is characterized in that: The residual attention module in step 3.3 is specifically: Using skip connections between consecutive multi-head self-attention layers, attention can be propagated and accumulated from shallow layers to deeper layers. The proposed residual attention mechanism is implemented by changing the calculation of the defined MHSA (Multi-Head Self-Attention) matrix: Here, a learnable variable α∈[0, 1] is used to allow the network to autonomously determine how much attention to propagate between layers.

7. The video abnormal event recognition method based on the Re-TimeSFormer model according to claim 4 is characterized in that: The classification module in step 3.4 is specifically: From the final code Extract the category vector As the image representation y, on top of this representation, a multi-layer perceptron (MLP) with one hidden layer is added to get the predicted video category, that is, to determine whether the video is abnormal:

8. The video abnormal event recognition method based on the Re-TimeSFormer model according to claim 1 is characterized in that: The FocalLoss loss function in step 4 calculates the difference between the predicted value and the true label, and optimizes the parameters as follows: Using the category vectors of F video frames Where t = 1, ..., F represents the index of different video frames, and the category score vector S = [s1, s2, ..., s k ], the calculation formula is as follows: Among them, K represents the number of categories, K = 2, that is, there are two categories of abnormal events in the video and no abnormal events; Then according to the score s of the k = (1, 2, ... K) category i (i=1, 2, ..., K), calculate the probability p of the corresponding category i (i=1,2,…,K), the formula is as follows: During the test phase, the category with the highest probability is selected as the model's label for whether the video is abnormal; Next, the category probability p i Input into the Focal Loss calculation model to measure the difference between the predicted value and the true label, and optimize the model parameters through loss feedback. The formula is as follows: L focal =-α(1-p i ) γ log(p i ) Among them, α adjusts the weights between different categories according to the frequency of the category; γ is used to control the degree of weight distribution of difficult and easy samples.

Images