A video abnormal event identification method based on a Re-TimeSFormer model

Abstract

The application discloses a video abnormal event identification method based on a Re-TimeSFormer model, which comprises the following steps: step 1, preprocessing F video frames with a sampling size of HxW of an original video; step 2, converting the preprocessed video frames into image block embedding vectors and then inputting the image block embedding vectors into an abnormal event identification model; step 3, using the abnormal event identification model based on Re-TimeSFormer to detect whether an event set as abnormal exists in the video; step 4, using a Focal Loss loss function to calculate the difference between a predicted value and a real label, thereby optimizing parameters, and obtaining a trained abnormal event identification model; and step 5, in a test stage of the abnormal event identification module, inputting a test video into the trained abnormal event identification model to obtain an identification result. The application can retain the ability of the Transformers model to extract global context, improve the ability to identify low-level features, increase the diversity of visual features and the stability of the model, and thus enhance the performance of the visual recognition task.

Description

Technical Field

[0001] This invention relates to the field of video anomaly detection technology, and specifically to a video anomaly event recognition method based on the Re-TimeSFormer model. Background Technology

[0002] With the advancement of smart city public safety information technology construction, video surveillance, as an important security measure, is widely deployed and applied in various public places to detect anomalies as early as possible and take corresponding measures to protect the lives and property of the public and promote urban security management. Relying on traditional manual screening and analysis of surveillance videos, typically involving reviewing the footage only after an incident is reported, suffers from low efficiency and high costs. Therefore, how to process massive amounts of video data, detect events threatening public safety in real time, and then handle and trigger alarms has become a pressing problem to be solved in modern intelligent monitoring systems.

[0003] Video anomaly detection uses computer vision algorithms to identify whether surveillance videos contain events that threaten public safety, such as fights, robberies, and vandalism. Intelligent monitoring systems integrating anomaly detection models can automatically detect and report anomalies, enabling security personnel to take swift action to prevent further escalation and provide a safer and more harmonious environment for the public. Furthermore, it can reduce the workload of monitoring personnel and improve work efficiency. Therefore, building efficient anomaly detection models and achieving intelligent analysis of surveillance videos plays a vital role in ensuring public safety and promoting safe urban development.

[0004] With the rapid development of computer vision technology, anomaly detection in video content has become an important research topic in the field of intelligent video analysis. In recent years, many domestic and international experts and scholars have conducted in-depth research on anomaly detection in videos. Among these efforts, researchers have attempted to apply Transformers to images and videos. Dosovitskiy et al.'s Vision Transformer (ViT) successfully applied the Transformer from natural language processing to computer vision, performing image classification tasks. TimeSFormer extends the self-attention mechanism from image space to spatiotemporal 3D volume, designing a divided space-time attention mechanism to apply ViT to video recognition.

[0005] However, in the ViT architecture, the attention mechanism aims to capture the global relationship between different feature patches. Features extracted from different image patches lose their uniqueness as the network depth increases and become more and more similar, resulting in feature collapse, loss of feature diversity, and reduced performance of visual recognition.

[0006] Therefore, avoiding feature collapse and improving recognition performance is one of the challenges faced by Transformer video anomaly detection. Summary of the Invention

[0007] To overcome the above technical problems, the present invention aims to provide a video anomaly event recognition method based on the Re-TimeSFormer (the Residual Attention Time-Space Transformer) model, which can improve the ability to recognize low-level features while retaining the ability of the Transformers model to extract global context, increase the diversity of visual features and model stability, thereby enhancing the performance of visual recognition tasks.

[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0009] A video anomaly event identification method based on the Re-TimeSFormer model includes the following steps;

[0010] Step 1: Sample F video frames of size H×W from the original video. f (f = 1, 2, ..., F) are preprocessed;

[0011] Step 2: Data Processing: Convert the preprocessed video frames into image patch embedding vectors. The data is then fed into the abnormal event recognition model;

[0012] Step 3: Construct an anomaly event recognition model based on Re-TimeSFormer. The anomaly event recognition model based on Re-TimeSFormer is used to detect whether there are events set as anomalies in the video.

[0013] Step 4: Calculate the difference between the predicted value and the true label using the Focal Loss function, and optimize the parameters accordingly to obtain the trained anomaly recognition model;

[0014] Step 5: During the testing phase of the abnormal event recognition module, input the test video into the trained abnormal event recognition model to obtain the recognition results.

[0015] The preprocessing in step 1 includes adjusting the pixels of the video frames and image enhancement. The resulting preprocessed video frame sequence is used for subsequent information encoding and anomaly detection.

[0016] The data processing in step 2 is specifically as follows:

[0017] Each of the F video frames is divided into N non-overlapping patches of size P×P, i.e., image blocks, where N = HW / P. 2 (H; W) represents the resolution of each frame of the image;

[0018] Then flatten these patches into one-dimensional vectors Each vector is linearly mapped to an embedding vector using a learnable weight matrix E. The calculation formula is as follows, and a learnable category vector is added to the first position of the sequence.

[0019]

[0020] Where p = 1, ..., N; represents the patch number; t = 1, ..., F; represents the index of a different video frame. The learnable location embedding is added to encode the spatiotemporal location of each image patch.

[0021] The abnormal event recognition model based on Re-TimeSFormer in step 3 includes a spatiotemporal separation self-attention module, a residual attention module, and a classification module;

[0022] Step 3.1: Embed the image patch into a vector. Feed into the spatiotemporal separation self-attention module;

[0023] Step 3.2: Construct a spatiotemporal separation self-attention module, that is, cascade the temporal attention module and the spatial attention module, add normalization and residual connections to form a spatiotemporal attention layer, and then stack multiple spatiotemporal attention layers;

[0024] Step 3.3: Introduce a residual attention module based on the spatiotemporal separation attention module;

[0025] Step 3.4: Select a fully connected layer as the classification module.

[0026] The spatiotemporal separation attention module in step 3.2 specifically includes a temporal self-attention module and a spatial self-attention module, as follows:

[0027] The temporal self-attention module calculates the temporal attention at a given location by comparing it with all image blocks at the same spatial location in other frames, and then feeds the result back to the spatial attention calculation. The spatial self-attention module simultaneously calculates the spatial attention at that moment by analyzing the information of each frame at the same instant.

[0028] The spatiotemporal attention layer consists of Ψ coding blocks, in each block Inside, based on the representation vector encoded in the previous block Calculate the query / key / value vector for each patch:

[0029]

[0030]

[0031]

[0032] Where LN represents LayerNorm, D h =D / A represents the potential dimension of each attention head, A represents the total number of attention heads; a = 1, ..., A is the index of multiple attention heads, p represents the patch number, and t represents the index of different video frames;

[0033] For spatiotemporal separation attention, in each block Within the framework, temporal attention is first computed by comparing each block (p, t) with all image blocks at the same spatial location in other frames. The self-attention weights are calculated via dot products.

[0034]

[0035] Where SM represents softmax; D h =D / A represents the potential dimension of each attention head, where a = 1, ..., A are indices of multiple attention heads; For query vector, This is a key vector, where p represents the patch number and t represents the index of a different video frame;

[0036] The value vector is calculated using the self-attention coefficient of each attention head. weighted sum

[0037]

[0038] Where a = 1, ..., A are indices on multiple attention heads; p represents the patch number, and t represents the index of a different video frame;

[0039] Then, the concatenation of vectors from all heads is projected and passed through an MLP, with residual connections used after each operation for encoding. The following formula is used to obtain the spatial attention calculation:

[0040]

[0041] The spatial self-attention module simultaneously calculates the spatial attention for each frame at the same moment, using the following formula:

[0042]

[0043] Where SM represents softmax; D h =D / A represents the potential dimension of each attention head, where a = 1, ..., A are indices of multiple attention heads; For query vector, This is a key vector, where p represents the patch number and t represents the index of a different video frame;

[0044] The value vector is then calculated using the self-attention coefficient of each attention head. weighted sum

[0045]

[0046] Where a = 1, ..., A are indices on multiple attention heads; p represents the patch number, and t represents the index of a different video frame;

[0047] Then, the concatenation of vectors from all heads is projected and passed through an MLP, with residual connections used after each operation to compute blocks. The final encoding of the patch at that location The specific calculation formula is as follows:

[0048]

[0049]

[0050] Where a = 1, ..., A are indices on multiple attention heads; p represents the patch number, and t represents the index of different video frames.

[0051] The residual attention module in step 3.3 is specifically as follows:

[0052] By using skip connections between consecutive multi-head self-attention layers (a core component of the Transformer model that uses multiple self-attention heads in parallel to capture different contextual information in the input sequence), attention can propagate and accumulate from shallow to deeper layers. The proposed residual attention mechanism is achieved by modifying the calculation of the defined MHSA (Multi-Head Self-Attention) matrix.

[0053]

[0054]

[0055] The learnable variable α∈[0,1] is used to allow the network to autonomously determine how much attention to propagate between layers.

[0056] The classification module in step 3.4 specifically includes:

[0057] From the final encoding Extracting the category vector As an image representation y, a multilayer perceptron (MLP) with one hidden layer is added on top of this representation to obtain the predicted video category, i.e., to determine whether the video contains anomalies:

[0058]

[0059] Step 4 uses the Focal Loss function to calculate the difference between the predicted value and the true label, and then optimizes the parameters accordingly, as follows:

[0060] Using the class vectors of each of the F video frames Where t = 1, ..., F represents the index of different video frames, resulting in the category score vector S = [s1, s2, ..., s]. k The calculation formula is as follows:

[0061]

[0062] Where K represents the number of categories, K=2, that is, there are two categories in the video: those with abnormal events and those without.

[0063] Then, based on the score s of the kth category (1, 2, ..., K), i (i = 1, 2, ..., K), calculate the probability p of the corresponding category. i (i = 1, 2, ..., K), the formula is as follows:

[0064]

[0065] During the testing phase, the category with the highest probability is selected as the model's label for whether the video contains anomalies.

[0066] Next, the class probability p i The input is fed into the Focal Loss calculation model to measure the difference between the predicted value and the true label, and the model parameters are optimized through the feedback of the loss, as shown in the following formula:

[0067] L focal =-α(1-p i ) γ log(p i )

[0068] Here, α adjusts the weights between different categories based on the frequency of the categories; γ is used to control the degree of weight allocation between easy and difficult samples.

[0069] The beneficial effects of this invention are:

[0070] This invention proposes an improved Re-TimeSFormer based on Vision Transformer, introducing a residual attention mechanism on the basis of spatiotemporal separation attention. In addition, it uses the Focal loss function to measure the difference between the model's predicted values ​​and the true labels, thereby optimizing the parameters.

[0071] Compared with existing technologies, the technical solution proposed in this invention achieves high efficiency and accuracy in video anomaly recognition. It introduces attention residual learning technology into TimeSFormer, integrating low-level information from shallow to deep network layers to establish connections between features at different layers, avoiding feature collapse and improving recognition performance. Attached Figure Description

[0072] Figure 1 This is a flowchart of a video anomaly event recognition method based on the Re-TimeSFormer model, according to an embodiment of the present invention.

[0073] Figure 2 This is a structural diagram of spatiotemporal separation attention that incorporates residual attention in an embodiment of the present invention. Detailed Implementation

[0074] The present invention will now be described in further detail with reference to the accompanying drawings.

[0075] like Figure 1 The diagram shown is a flowchart of a video anomaly event recognition method based on the Re-TimeSFormer model according to an embodiment of the present invention, which specifically includes the following steps:

[0076] Step 1: Sample F video frames of size H×W from the original video. f (f = 1, 2, ..., F), and perform preprocessing, including adjusting the pixels of video frames and image enhancement.

[0077] Step 2: Data Processing: Convert the sampled video frames into image patch embedding vectors. The data is then fed into the abnormal event recognition model;

[0078] Specifically as follows:

[0079] Each frame is decomposed into N non-overlapping image blocks, each block being P×P in size, where N = HW / P. 2 Then flatten these image patches into vector x (p , t)(p = 1, ..., N; t = 1, ..., F), each block x is weighted by a learnable weight matrix E. (p , t) Linear mapping to an embedded vector The calculation formula is as follows, and a learnable category vector is added to the first position of the sequence.

[0080]

[0081] in, The learnable location embedding is added to encode the spatiotemporal location of each image patch.

[0082] Step 3: Construct an anomaly event recognition model based on Re-TimeSFormer; the TimeSFormer model mainly includes a spatiotemporal separation self-attention module and a classification module. The Re-TimeSFormer model is constructed by introducing residual attention on top of the spatiotemporal separation attention module of TimeSFormer. (See attached diagram.) Figure 2 Introducing residual attention can improve the model's ability to recognize low-level features, thereby enhancing the performance of visual recognition tasks.

[0083] Step 3.1: Embed the image patch into the vector Feed into the spatiotemporal separation self-attention module;

[0084] Step 3.2: Construct a spatiotemporal separation attention module, that is, cascade the temporal attention module and the spatial attention module, add normalization and residual connections to form a spatiotemporal attention layer, and then stack multiple spatiotemporal attention layers;

[0085] The aforementioned spatiotemporal separation attention module specifically includes a temporal self-attention module and a spatial self-attention module;

[0086] The temporal self-attention module calculates the temporal attention at a given location by comparing it with all image blocks at the same spatial location in other frames, and then feeds the result back to the spatial attention calculation. The spatial self-attention module simultaneously calculates the spatial attention at that moment by analyzing the information of each frame at the same instant.

[0087] The spatiotemporal attention layer consists of Ψ coding blocks, in each block Inside, based on the representation vector encoded in the previous block Calculate the query / key / value vector for each patch:

[0088]

[0089]

[0090]

[0091] Where LN represents LayerNorm, D h =D / A represents the potential dimension of each attention head, A represents the total number of attention heads; a = 1, ..., A is the index of multiple attention heads, p represents the patch number, and t represents the index of different video frames;

[0092] For spatiotemporal separation attention, in each block Within the framework, temporal attention is first computed by comparing each block (p, t) with all image blocks at the same spatial location in other frames. The self-attention weights are calculated via dot products.

[0093]

[0094] Where SM represents the softmax activation function; D h =D / A represents the potential dimension of each attention head, where a = 1, ..., A are indices of multiple attention heads; For query vector, This is a key vector, where p represents the patch number and t represents the index of a different video frame;

[0095] The value vector is calculated using the self-attention coefficient of each attention head. weighted sum

[0096]

[0097] Where a = 1, ..., A are indices on multiple attention heads; p represents the patch number, and t represents the index of a different video frame;

[0098] Then, the concatenation of vectors from all heads is projected and passed through an MLP, with residual connections used after each operation for encoding. The following formula is used to obtain the spatial attention calculation:

[0099]

[0100] The spatial self-attention module simultaneously calculates the spatial attention for each frame at the same moment, using the following formula:

[0101]

[0102] Where SM represents softmax; D h =D / A represents the potential dimension of each attention head, where a = 1, ..., A are indices of multiple attention heads; For query vector, This is a key vector, where p represents the patch number and t represents the index of a different video frame;

[0103] The value vector is then calculated using the self-attention coefficient of each attention head. weighted sum The calculation formula is as follows:

[0104]

[0105] Where a = 1, ..., A are indices on multiple attention heads; p represents the patch number, and t represents the index of a different video frame;

[0106] Then, the concatenation of value vectors from all heads is projected and passed through an MLP, with residual connections used after each operation to compute blocks. The final code of the patch at that location The specific calculation formula is as follows:

[0107]

[0108]

[0109] Step 3.3: Based on the spatiotemporal separation attention module, a residual attention mechanism is introduced. Specifically, skip connections are used between consecutive multi-head self-attention layers, allowing attention to propagate and accumulate from shallow layers to deeper layers. The proposed residual attention mechanism is achieved by changing the calculation of the defined MHSA (Multi-Head Self-Attention) matrix.

[0110]

[0111]

[0112] The learnable variable α∈[0,1] is used to allow the network to autonomously determine how much attention to propagate between layers.

[0113] Step 3.4: Select a fully connected layer as the classification module. Specifically, the final fragment embedding is obtained from the final block of the classification label.

[0114]

[0115] Based on this, a hidden layer MLP is added to predict the final video class, that is, to determine whether there are any anomalies in the video.

[0116] Step 4: Input the training set into the anomaly event recognition model, use the Focal Loss function to calculate the difference between the predicted values ​​and the true labels, and optimize the parameters accordingly to obtain the trained model. The specific implementation is as follows:

[0117] Using the class vectors of each of the F video frames The category score vector S = [s1, s2, ..., s] is obtained. k The calculation formula is as follows:

[0118]

[0119] Where K represents the number of categories, K=2, that is, there are two categories in the video: those with abnormal events and those without.

[0120] Then, based on the score s of the kth category (1, 2, ..., K), i (i = 1, 2, ..., K), calculate the probability p of the corresponding category. i (i = 1, 2, ..., K), the formula is as follows:

[0121]

[0122] Next, the class probability p i The input is fed into the Focal Loss calculation model to measure the difference between the predicted value and the true label, and the model parameters are optimized through the feedback of the loss, as shown in the following formula:

[0123] L focal =-α(1-p i ) γ log(p i )

[0124] Here, α adjusts the weights between different categories based on the frequency of the categories; γ is used to control the degree of weight allocation between easy and difficult samples.

[0125] Step 5: Input the test set into the trained anomaly event recognition model, select the category with the highest probability as the model's label for whether the video has an anomaly, and obtain the recognition result.

Claims

1. A video anomaly event recognition method based on the Re-TimeSFormer model, characterized in that, Includes the following steps; Step 1: Sample the original video to a size of [size missing]. W's F video frames Preprocessing is required; Step 2: Data Processing: Convert the preprocessed video frames into image patch embedding vectors. Then, it is fed into the abnormal event recognition model; Step 3: Construct an anomaly event recognition model based on Re-TimeSFormer. The anomaly event recognition model based on Re-TimeSFormer is used to detect whether there are events set as anomalies in the video. Step 4: Calculate the difference between the predicted value and the true label using the Focal Loss function, and optimize the parameters accordingly to obtain the trained anomaly recognition model; Step 5: During the testing phase of the abnormal event recognition module, input the test video into the trained abnormal event recognition model to obtain the recognition results; The abnormal event recognition model based on Re-TimeSFormer in step 3 includes a spatiotemporal separation self-attention module, a residual attention module, and a classification module; Step 3.1: Embed the image patch into a vector. Feed into the spatiotemporal separation self-attention module; Step 3.2: Construct a spatiotemporal separation self-attention module, cascade the temporal attention module and the spatial attention module, and add normalization and residual connections to form a spatiotemporal attention layer, and then stack multiple spatiotemporal attention layers; The temporal self-attention module calculates the temporal attention at a given location by comparing it with all image blocks at the same spatial location in other frames, and then feeds the result back to the spatial attention calculation. The spatial self-attention module simultaneously calculates the spatial attention at that moment by analyzing the information of each frame at the same instant. Step 3.3: Introduce a residual attention module based on the spatiotemporal separation attention module; By using skip connections between consecutive multi-head self-attention layers, attention can propagate and accumulate from shallow layers to deeper layers. The proposed residual attention mechanism is achieved by modifying the calculation of the defined MHSA matrix: Among them, learnable variables [0, 1] is used to allow the network to autonomously determine how much attention to propagate between layers; where SM represents softmax; A represents the total number of attention heads; It is an index on multiple attention heads; Representing the latent dimensions of each attention head; For query vector, The key vector; Indicates the patch number. Indicates the index of different video frames; Step 3.4: Select a fully connected layer as the classification module.

2. The video anomaly event recognition method based on the Re-TimeSFormer model according to claim 1, characterized in that, The preprocessing in step 1 includes adjusting the pixels of the video frames and image enhancement. The resulting preprocessed video frame sequence is used for subsequent information encoding and anomaly detection.

3. The video anomaly event recognition method based on the Re-TimeSFormer model according to claim 2, characterized in that, The data processing in step 2 is specifically as follows: Divide each of the F video frames into N elements. Non-overlapping patches of varying sizes, i.e., image patches, where (H;W) represents the resolution of each frame of the image; The patch is then flattened into a one-dimensional vector. Each vector is linearly mapped to an image patch embedding vector using a weight matrix. And add a learnable category vector at the first position of the sequence. ; ; indicates the patch number; ; represents the index of different video frames.

4. The video anomaly event recognition method based on the Re-TimeSFormer model according to claim 3, characterized in that, The spatiotemporal separation attention module in step 3.2 specifically includes a temporal self-attention module and a spatial self-attention module, as follows: Spatiotemporal attention layer It consists of coded blocks, and each block is composed of coded blocks. Inside, based on the representation vector encoded in the previous block Calculate the query / key / value vector for each patch: Where LN represents LayerNorm; For spatiotemporal separation attention, in each block Inside, first by each block Temporal attention is computed by comparing all image patches at the same spatial location in other frames, and the self-attention weights are calculated via dot products: The value vector is calculated using the self-attention coefficient of each attention head. weighted sum : Then, the concatenation of vectors from all heads is projected and passed through an MLP, with residual connections used after each operation for encoding. The following formula is used to obtain the spatial attention calculation: The spatial self-attention module simultaneously calculates the spatial attention for each frame at the same moment, using the following formula: The value vector is then calculated using the self-attention coefficient of each attention head. weighted sum : Then, the concatenation of vectors from all heads is projected and passed through an MLP, with residual connections used after each operation to compute blocks. The final encoding of the patch at that location The specific calculation formula is as follows: 。 5. The video anomaly event recognition method based on the Re-TimeSFormer model according to claim 4, characterized in that, The classification module in step 3.4 specifically includes: From the final encoding Extracting the category vector As the image representation y, a multilayer perceptron (MLP) with one hidden layer is added on top of this representation to obtain the predicted video category, i.e., to determine whether the video has anomalies: 。 6. The video anomaly event recognition method based on the Re-TimeSFormer model according to claim 5, characterized in that, Step 4 uses the Focal Loss function to calculate the difference between the predicted value and the true label, and then optimizes the parameters accordingly, as follows: Using the class vectors of each of the F video frames ,in Representing the indices of different video frames, we obtain the category score vector. The calculation formula is as follows: in, Indicates the number of categories. The video can be categorized into two types: those containing abnormal events and those not containing abnormal events. Then according to the first Scores for each category Calculate the probability of the corresponding category. The formula is shown below: During the testing phase, the category with the highest probability is selected as the model's label for whether the video contains anomalies. Next, the class probabilities The input is fed into the Focal Loss calculation model to measure the difference between the predicted value and the true label, and the model parameters are optimized through the feedback of the loss, as shown in the following formula: in, Adjust the weights between different categories; Used to control the degree of weight allocation between easy and difficult samples.

Images