Future frame anomaly detection method based on meta learning and time-space relationship
By introducing a meta-learning module and a spatiotemporal relationship attention mechanism in video anomaly detection, the problems of overfitting and neglecting of spatiotemporal relationships in the prior art are solved, and the accuracy of anomaly detection and the adaptability of the model to dynamic changes in spatiotemporality are improved.
Patent Information
- Application Number
- CN202411225619.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-03
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2044-09-03
AI Technical Summary
Existing video anomaly detection methods rely on a large amount of normal data to train, which can easily lead to overfitting and ignore spatiotemporal relationships, resulting in the model being insensitive to abnormal objects.
The meta-learning module and the attention mechanism that can learn space-time relationships are introduced, normal dynamic features are learned through the meta-learning module and automatic encoder encoding is enriched, and future frame prediction is carried out in combination with the space-time relationship attention mechanism to enhance the model's adaptability to space-time dynamic changes.
The accuracy of abnormal detection is improved, overfitting is avoided, the model's ability to detect different scenarios and abnormal behaviors is enhanced, and the accuracy of future frame prediction is significantly improved.
Smart Images

Figure CN119091356A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of intelligent video processing, and in particular relates to a future frame anomaly detection method based on meta-learning and spatiotemporal relationship. Background Art
[0002] With the popularity of surveillance equipment and people's widespread concern about social public safety, video anomaly detection, that is, identifying abnormal events or behaviors in video sequences, has gradually become a research hotspot in the field of computer vision. Autoencoder is a popular video anomaly detection method. Researchers usually use AE to model the normal pattern of historical frames and reconstruct the current frame or predict the upcoming frame. Since the model is trained only with normal data, the prediction error of abnormal input is expected to be higher than the prediction error of normal corresponding input. Previously, many methods were based on this assumption for anomaly detection. However, this assumption does not always hold. This makes video anomaly detection still extremely challenging: on the one hand, existing methods rely on a large amount of normal training data to model shared normal patterns. These models are prone to the dilemma of "overgeneralization". Due to the powerful representation ability of convolutional neural networks, all video frames can be well predicted, whether they are normal or abnormal. It is worth noting that a potential limitation of the encoder-decoder structure is that the decoder only focuses on predicting the correspondence between the visual features of future frames and input video frames when decoding, while the same target object in a normal sample may appear in multiple scenes with similar but not identical contextual information, which may cause abnormal samples to be well predicted. In video anomaly detection, Yiwei Lu, Frank Yu, Mahesh Kumar Krishna Reddy, and Yang Wang. Few-shot scene-adaptive anomaly detection. In ECCV, 2020. Following the optimization-based meta-learning method Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML, 2017. 4, 6 and applying it to training scene adaptive anomaly detection models; Lv, H.; Chen, C.; Cui, Z.; Xu, C.; Li, Y.; Yang, J. Learning Normal Dynamics in Videos with MetaPrototype Network. In Proceedings of the 2021 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), virtual, 19–25 June 2021; pp.15420–15429. A dynamic prototype unit is proposed to dynamically encode normal samples into prototypes in real time without additional memory cost.Zhang S, Song X, Wang C, et al. Object-meta and MSGAE-MP: Multi-dimensional video anomaly detection [J]. IET Computer Vision, 2022. Object meta-frames are used instead of video frames to strengthen the multi-dimensional information carried by the input. Inspired by these works, a meta-learning module is introduced in AE to overcome the limitations of the autoencoder. First, the encoder encodes the prior knowledge of the current task and stores it in the prototype network of the meta-learning module. Then, the prototype is obtained as a set of feature encodings to form a meta-learning normal behavior pool, and the complex changes between feature encodings are used to reason about new events to determine whether they are abnormal. This module directly learns and matches the encoded features, avoiding the chain clustering of high-dimensional feature space in the low-dimensional mapping process that leads to errors in the determination of the actual distance between features; at the same time, the meta-learning prototype network can continuously store and update the feature encoding. Finally, the output encoding of the meta-learning module is passed through the remaining AE layers and then predicted in the prediction network for subsequent frames. The introduction of the meta-learning module alleviates the prediction of the same target object in normal samples in different contexts.
[0003] On the other hand, most of the surveillance videos are static, and the normal patterns that appear in different scenes are also different. The spatiotemporal relationship between the environment and the background is very important to determine whether the behavior is abnormal. For example, when a person runs on the playground, it should be normal behavior, but the same behavior in the office may be regarded as abnormal behavior. Liu W, Luo WX, Lian DZ, Gao SH (2018) Future frame prediction for anomaly detection--a new baseline. Processing of the IEEE Conf Comput Vis Pattern Recognit: 6536-6545 proposed a novel anomaly detection method to learn normal sample features through the U-Net network architecture, but this method focuses on learning the appearance features of video frames and ignores an important clue, that is, when using optical flow to constrain the motion features between adjacent video frames, optical flow usually assumes that the pixels of the image move consistently in time within a small local area. However, in complex scenes, the motion of objects may be non-uniform or non-local, which may cause optical flow to deviate when capturing motion, which makes them insensitive to some abnormal objects. With the popularity of memory-enhanced deep autoencoders, the Memory project is used to memorize the typical distribution features of normal data and use them to constrain the expressive power of convolutional neural networks. This memory module can guide the feature extraction process well, but the features of Memory are only spatial features at a certain stage of the network, and there is no attempt to guide the memory of features related to the time dimension. Memory can only provide limited help for the model to distinguish between normal and abnormal. In text work, video anomaly detection is performed by learning the spatiotemporal relationship between objects. Combining spatial features and temporal features helps to understand the video more comprehensively.
[0004] Based on the above analysis, it is necessary to provide a U-Net based on meta-learning and spatiotemporal relationship learning for future frame prediction framework. In order to fully understand the spatiotemporal dimension of the video, an attention module that can learn spatiotemporal relationships is introduced into the U-Net network to learn the spatiotemporal relationship between objects in the input continuous video frames, so that the neural network can better adapt to the spatiotemporal dynamic changes of the video sequence, and the prediction network based on U-Net has better prediction effect. Summary of the invention
[0005] In response to the technical problems in the above-mentioned prior art, the present invention proposes a future frame anomaly detection method based on meta-learning and spatiotemporal relationships, so as to solve the problems that the automatic encoder only focuses on the correspondence between the visual features of the predicted frame and the input video frame during decoding, and ignores the time domain information, resulting in the model being insensitive to some abnormal objects, and improves the accuracy of anomaly detection.
[0006] In order to achieve the above-mentioned purpose, the present invention adopts the following technical solutions:
[0007] The future frame anomaly detection method based on meta-learning and spatiotemporal relationship includes the following steps:
[0008] Step 1: Based on the autoencoder, a continuous video frame I1, I2, ..., I t As input, we stack all these frames on the channel and use them to predict future frames.
[0009] Step 2: Feed the input frame through the selected AE code into the meta-learning module, which is trained to learn and compress the normal dynamics of real-time sequence information as multiple normal prototypes and enrich the input AE code with the normal dynamic information;
[0010] Step 3: Run the output code through the remaining AE layers to process the normal behavior for subsequent frame prediction.
[0011] Step 4: To make the prediction frame Closer to I t+1 , while minimizing their distance in terms of intensity and gradient, in order to maintain the temporal consistency between adjacent frames, an attention mechanism that can learn spatiotemporal relations is introduced to constrain the real frame I t+1 and predicted frame
[0012] Step 5: The difference between the predicted future frame and its normal frame determines whether the event is normal or abnormal.
[0013] Further, as a preferred technical solution of the present invention, in the meta-learning module, the normal behavior feature set is defined as:
[0014]
[0015] Where m represents the maximum number of normal behavior features contained in the meta-learning module Z; represents the type of the i-th meta-learned normal behavior label, θ i Represents the value of the normal behavior label of the ith meta-learning. For the sample points inside the support set Z, the encoding formula f is used φ To generate a prototype representation for each category:
[0016]
[0017] Here the encoding formula f φIt can be any information extraction method. In this paper, the AE encoding is used. The meta-learning module is trained to learn the normal dynamics of real-time sequence information as multiple prototypes and enrich the input AE encoding with normal dynamic information. Extract the tth input encoding map x(t) = f from AE φ (x),x(t)∈R h×w×c , which can be regarded as a c-dimensional N = w × h vector, assigning a normal weight to each pixel position represents f φ for:
[0018]
[0019] f φ It can be called the normal behavior pool, that is, the prototype pool. Inputting a continuous video frame can represent the set as
[0020] Further, as a preferred technical solution of the present invention, similarly, the encoding of the query set is also generated for the query set. In the query process, the input encoding vector from the AE encoding map is used as a query to retrieve relevant terms in the prototype pool to reconstruct the normal encoding f φ (x), encode the query set as f φ ′(x):
[0021]
[0022] in, Denoted as n-th encoding vector and the mth prototype item The correlation score between .
[0023] Finally, the probability p of the current sample belonging to each category is calculated w (y=k|x), the softmax calculation method is used here:
[0024]
[0025] The key idea of introducing the meta-learning module is to enrich the AE encoding with normal information to enhance the prediction of the normal part of the video frame while suppressing the abnormal part. The output encoding of the meta-learning module is passed through the remaining AE layers and then used in the prediction network for subsequent frame prediction.
[0026] Further, as a preferred technical solution of the present invention, future frame prediction: by modifying U-Net for future frame prediction, an attention module is introduced in the U-Net network to learn the spatiotemporal relationship of the input image, so that the prediction network based on U-Net has better prediction effect. Specifically, for every two convolutional layers, the output resolution is retained. Therefore, cropping and resizing operations are no longer required when adding shortcuts. The kernel size of all convolutional layers, deconvolutional layers, and maximum pooling layers is set to 3. In the U-Net prediction network, the input and output have the same resolution. The attention mechanism module will be introduced in the next section.
[0027] First, the network input image size is set to a (256, 256) grayscale image, and then after two-by-two 3*3 convolutions, the dimension becomes (64, 256, 256). After three downsamplings, the image dimension is (512, 32, 32). After the downsampling part is implemented, the attention mechanism module is used to repeat the upsampling three times until the image is downsampled to (64, 256, 256), obtaining the input and output with the same resolution.
[0028] Following the work of future frame prediction, intensity and gradient differences are used to make the prediction close to its true situation. The intensity constraint ensures the similarity of all pixels in RGB space. Specifically, minimize the predicted frame in intensity space. The L2 distance between it and its true I:
[0029]
[0030] Gradient constraints are used to sharpen the generated image:
[0031]
[0032] Where i, j represents the spatial index of the video frame.
[0033] Attention mechanism based on spatiotemporal relations: A new attention mechanism is designed to capture the spatiotemporal relations between objects in video frames. The right side of the channel represents the attention process of the spatial relations between objects, while the left side of the channel focuses on the temporal relations. Finally, the original input is jointly weighted by the spatial and temporal attention channels and used as the output of the module to achieve the purpose of learning the spatiotemporal relations between objects.
[0034] In the attention mechanism that can learn spatiotemporal relationships, it is possible to extract and aggregate dimensional features separately by combining spatial and temporal information to obtain spatiotemporal relationship information. The specific operation is to input the features of H dimensions First, it is encoded through a one-to-one one-dimensional feature encoding operation, which uses two pooling kernels with a spatial size of H; (H×1) and (W×1) average pooling encoding is performed on each channel, in the horizontal direction and vertical direction respectively, and encoded in the vertical direction. The output of the Cth channel in the H dimension is as follows:
[0035]
[0036] The same applies to encoding in the horizontal direction:
[0037]
[0038] Through the spatial feature information extraction operation, the generated channels are cascaded, and the feature Y of the video frame sequence at time t is extracted from each channel feature t It can be expressed as:
[0039]
[0040] Among them, σ represents the normalization operation, F represents the convolution operation, and F 3×3 、F 1×1 , represents the size of the convolution kernel in the convolution operation, and C / r represents the reduction ratio in the control channel number change.
[0041] The outputs of the three transformations are aggregated along each of the three dimensions to generate a set of spatiotemporal-aware feature maps. The three transformations capture the long-range temporal dependencies and spatial relationships in the feature space, which enables the network to more accurately obtain the spatiotemporal relationships between objects. The spatial relationships and temporal dependencies between objects are used to design an attention mechanism to highlight the areas prone to anomalies in video anomaly detection. c h (h) and Y c w (w) is connected and the output is obtained after the convolution operation:
[0042] Y hw =δ(F 1×1 [Y h ,Y w ]) (11)
[0043] Among them, δ represents a nonlinear activation function, which gives Y hw Split into Y h ∈R C / r×H and Y w ∈R C / r×W Two tensors of the same dimension. h and Y w The final tensor g is obtained by Sigmoid activation function respectively. h and gw .
[0044] For the tensor of the time channel attention part, for Y t After processing with nonlinear activation function, we get:
[0045] g t =relu(Y t (X)) (12)
[0046] Tensor g t represents the weight of regional features that are sensitive to time changes. Finally, the output of the spatiotemporal relationship attention module Y is as follows:
[0047]
[0048] Since the input and output have the same resolution, the spatiotemporal relationship attention mechanism module can be learned and then inserted into the U-net network to help it make better future frame predictions without changing the input and output structure.
[0049] Further, as a preferred technical solution of the present invention, the objective function: all these constraints on appearance, motion and adversarial training are combined into the objective function, and the following objective functions are obtained, which support prototype learning representing the normal behavior of the target, feature reconstruction of enhanced coding, and future frame prediction.
[0050] First, the overall loss function ψ is composed of the feature reconstruction term ψ under the attention mechanism cd and frame prediction term ψ pre These two items are balanced by weights λ1 and λ2 as follows:
[0051] ψ=λ1ψ pre +λ2ψ cd (14)
[0052] The frame prediction loss is expressed as the input video real frame y t and future frames The prediction network outputs the Euclidean distance between predicted frames:
[0053]
[0054] The feature reconstruction loss under the attention mechanism is expressed as the learned common prototype has information diversity and feature representativeness, that is, high quality and diversity, and low redundancy. It has two terms ψ c and ψ d , respectively for these two properties, can be written as:
[0055] ψ cd =μ1ψ c +μ2ψ d (16)
[0056] Where μ1 and μ2 are weights. Feature compactness loss ψ c Used to reconstruct normal encoding using compact prototypes, it can query the most similar mode closer to the normal behavior pool, enhance the representativeness of each mode, and minimize the L2 norm between the query and the corresponding mode, that is, it measures the average Euclidean distance between the input encoding vector and its most relevant prototype:
[0057]
[0058] Where k is the query The index of the closest pattern in the pool of normal behavior patterns, expressed as:
[0059]
[0060] Further promote the diversity among prototype projects. The diversity term ψ d Expressed as
[0061]
[0062] Among them, p, p′ represent different prototypes, and α is the expected distance of the pattern in the normal behavior pattern pool. Using the above two loss functions ψ c and ψ d ,The future frame prediction model can focus on the spatiotemporal relationships among objects to better predict normal future frames.
[0063] Further as a preferred technical solution of the present invention, anomaly score: Assuming that normal events can be well predicted. Therefore, the difference between the predicted frame and its true value can be used to predict anomalies. Peak signal-to-noise ratio (PSNR) is a method of image quality assessment.
[0064]
[0065] in, is the maximum value representing the color of an image point
[0066] After calculating the PSNR of each frame of each test video, the PSNR of all frames in the test video are normalized to the range [0,1] and the regular score for each frame is calculated using the following equation:
[0067]
[0068] Among them, Score all(t) is the regularity score, which corresponds to the normality of each frame in the video, and plays an indicative role in the confidence of the detection result. A threshold can be set to distinguish normal frames from abnormal frames. When the corresponding value is lower than the threshold, it means that an abnormal event has occurred; when the corresponding value is higher than the threshold, it means that no abnormal event has occurred.
[0069] The future frame anomaly detection method based on meta-learning and spatiotemporal relationship described in the present invention adopts the above technical solution and has the following technical effects compared with the prior art:
[0070] (1) The method of the present invention proposes a meta-learning module. The model-based meta-learning method enables the model to learn common features from multiple tasks, and the system can learn more discriminative and generalized feature representations from the data. In different monitoring scenarios and abnormal behaviors, the model can also accurately detect anomalies without overfitting or underfitting.
[0071] (2) The method of the present invention introduces a meta-learning module into the autoencoder, and utilizes the learning characteristics of the meta-learning module to extract, save and update the features extracted by the autoencoder. At the same time, the importance of the features extracted from the input video frame is automatically obtained through learning, and important weights are assigned to features that deserve more attention. This helps to improve the utilization of key features in the input stage of the future frame prediction network.
[0072] (3) The attention module in the method of the present invention enables the model to pay attention to the spatiotemporal relationship between objects. It is introduced into the future frame prediction network, and the three channels of the attention module are used to extract and fuse the features of the spatiotemporal and spatial dimensions of the input video frame, and learn the regularity of the spatial feature space and the motion feature space, so that the future frame prediction network can better predict the next frame, which is conducive to detecting abnormal events. BRIEF DESCRIPTION OF THE DRAWINGS
[0073] Figure 1 Schematic diagram of future frame prediction framework that introduces meta-learning into the present invention;
[0074] Figure 2 Schematic diagram of the meta-learning module of the present invention;
[0075] Figure 3 It is a schematic diagram of the U-net prediction network of the present invention;
[0076] Figure 4 Schematic diagram of the attention mechanism capable of learning spatiotemporal relationships of the present invention;
[0077] Figure 5 A schematic diagram of a positive / abnormal sample of the present invention;
[0078] Figure 6 The real frames and thermal maps for detecting normal events and abnormal events of the present invention;
[0079] Figure 7 is the anomaly score of video #02, #06 from the UCSDPed2 dataset of the present invention;
[0080] Figure 8 is the anomaly score of videos #04 and #13 from the CHUKAvenue dataset of the present invention;
[0081] Fig. 9 is the anomaly score of videos #01_0133 and #05_0017 from the ShanghaiTech dataset of the present invention;
[0082] Fig.10 A comparison chart of the introduced meta-learning module of the present invention and the UCSDPed2 basic model;
[0083] Fig.11 A comparison diagram of the introduced meta-learning module of the present invention and the CHUKAvenue basic model;
[0084] Fig.12 This is a comparison chart between the meta-learning module introduced in the present invention and the ShanghaiTech basic model. DETAILED DESCRIPTION
[0085] The present invention is further explained below in detail with reference to the accompanying drawings so that those skilled in the art can more deeply understand the present invention and be able to implement it. However, the following reference examples are only used to explain the present invention and are not intended to limit the present invention.
[0086] The present invention proposes a method for detecting abnormal events in videos based on meta-learning and spatiotemporal relationships, which mainly realizes unsupervised tasks by predicting frames from future frames of videos. In order to better predict future frames of videos, most existing works only consider appearance constraints by adding intensity loss, gradient loss or adversarial training loss. However, appearance constraints alone cannot guarantee the generation of realistic video frames. Only obvious clues of target objects are considered, while the importance of motion constraints for video anomaly detection is ignored. In addition to spatial information, temporal information is also an important feature of videos. Therefore, an attention mechanism that can learn spatiotemporal relationships is added to the objective function to ensure the motion consistency of normal target events in the training set. In addition, many existing works use clustering discriminant algorithms to cluster the extracted features to detect whether there are anomalies in video frames. However, it ignores the influence of feature distance determination in the process of clustering chain feature mapping from high dimensions to low dimensions. Considering the potential limitations of the encoder-decoder structure and the strong generalization of neural networks, meta-learning module example features are introduced in the autoencoder to learn normal behavior patterns. It is worth noting that abnormal events may be caused by appearance or motion, while the future frame prediction scheme introduced by the present invention utilizes the appearance and motion constraints of normal events. Therefore, this video prediction with appearance and motion constraints is more consistent with video anomaly detection. Abnormal events can be easily identified by comparing the predicted frame results with normal events in the real frame. The future frame prediction framework introduced by meta-learning is as follows Figure 1 shown.
[0087] The method of the present invention comprises the following steps: Step 1: Based on the autoencoder, a continuous video frame I1, I2, ..., I t As input, we stack all these frames on the channel and use them to predict future frames. Step 2: Feed the input frame through the selected AE code into the meta-learning module, which is trained to learn and compress the normal dynamics of real-time sequence information as multiple normal prototypes and enrich the input AE code with normal dynamic information; Step 3: Run the output code through the remaining AE layers to process the normal behavior for subsequent frame prediction Step 4: To make the prediction frame Closer to I t+1 , while minimizing their distance in terms of intensity and gradient, in order to maintain the temporal consistency between adjacent frames, an attention mechanism that can learn spatiotemporal relations is introduced to constrain the real frame I t+1 and predicted frame Step 5: The difference between the predicted future frame and its normal frame determines whether the event is normal or abnormal.
[0088] Meta-learning module: In the meta-learning module, as follows Figure 2As shown, the query set images [m1,m2…,m n ] is marked as the label of the nearest normal behavior prototype, and the label set is
[0089] The prototype network works by extracting features from the image and then calculating a prototype for each class. To do this, the average of the image embeddings is calculated for each class. Once the prototype is obtained, classification is performed by simply calculating the Euclidean distance from the image embedding to the prototype, which results in a meta-learned normal behavior pool that stores the extracted features in the meta-learned normal pool and updates it.
[0090] In the meta-learning module, the set of normal behavior features is defined as:
[0091]
[0092] Where m represents the maximum number of normal behavior features contained in the meta-learning module Z; represents the type of the i-th meta-learned normal behavior label, θ i Represents the value of the normal behavior label of the ith meta-learning. For the sample points inside the support set Z, the encoding formula f is used φ To generate a prototype representation for each category:
[0093]
[0094] Here the encoding formula f φ It can be any information extraction method. The AE coding used in the present invention, the meta-learning module is trained to learn the normal dynamics of real-time sequence information as multiple prototypes, and enrich the input AE coding with normal dynamic information. Extract the tth input coding map x(t)=f from AE φ (x),x(t)∈R h×w×c , which can be regarded as a c-dimensional N = w × h vector, assigning a normal weight to each pixel position represents f φ for:
[0095]
[0096] f φ It can be called the normal behavior pool, that is, the prototype pool. Inputting a continuous video frame can represent the set as
[0097] Similarly, the encoding of the query set is also generated for the query set. During the query process, the input encoding vector from the AE encoding graph is used as a query to retrieve relevant terms in the prototype pool to reconstruct the normal encoding f φ (x), encode the query set as fφ ′(x):
[0098]
[0099] in, Denoted as n-th encoding vector and the mth prototype item The correlation score between .
[0100] Finally, the probability p of the current sample belonging to each category is calculated w (y=k|x), the softmax calculation method is used here:
[0101]
[0102] The key idea of introducing the meta-learning module is to enrich the AE encoding with normal information to enhance the prediction of the normal part of the video frame while suppressing the abnormal part. The output encoding of the meta-learning module is passed through the remaining AE layers and then used in the prediction network for subsequent frame prediction.
[0103] Future frame prediction: By modifying U-Net for future frame prediction, an attention module is introduced into the U-Net network to learn the spatiotemporal relationship of the input image, so that the prediction network based on U-Net has better prediction effect. Specifically, for every two convolutional layers, the output resolution is retained. Therefore, cropping and resizing operations are no longer required when adding shortcuts. The details of this network are as follows Figure 3 As shown in the figure. The kernel size of all convolutional layers, deconvolutional layers, and max pooling layers is set to 3. In the U-Net prediction network, the input and output have the same resolution. The attention mechanism module will be introduced in the next section.
[0104] First, the network input image size is set to a (256, 256) grayscale image, and then after two-by-two 3*3 convolutions, the dimension becomes (64, 256, 256). After three downsamplings, the image dimension is (512, 32, 32). After the downsampling part is implemented, the attention mechanism module is used to repeat the upsampling three times until the image is downsampled to (64, 256, 256), obtaining the input and output with the same resolution.
[0105] Following the work of future frame prediction, intensity and gradient differences are used to make the prediction close to its true situation. The intensity constraint ensures the similarity of all pixels in RGB space. Specifically, minimize the predicted frame in intensity space. The L2 distance between it and its true I:
[0106]
[0107] Gradient constraints are used to sharpen the generated image:
[0108]
[0109] Where i, j represents the spatial index of the video frame.
[0110] Attention Mechanism Based on Spatiotemporal Relations: A new attention mechanism is designed to capture the spatiotemporal relations between objects in video frames. Figure 4 The structure of the attention module is shown. In the figure, the right side of the channel represents the attention process of the spatial relationship between objects, while the left side of the channel focuses on the temporal relationship. Finally, the original input is jointly weighted by the spatial and temporal attention channels and used as the output of the module to achieve the purpose of learning the spatiotemporal relationship of objects.
[0111] In the attention mechanism that can learn spatiotemporal relationships, it is possible to extract and aggregate dimensional features separately by combining spatial and temporal information to obtain spatiotemporal relationship information. The specific operation is to input the features of H dimensions First, it is encoded through a one-to-one one-dimensional feature encoding operation, which uses two pooling kernels with a spatial size of H; (H×1) and (W×1) average pooling encoding is performed on each channel, in the horizontal direction and vertical direction respectively, and encoded in the vertical direction. The output of the Cth channel in the H dimension is as follows:
[0112]
[0113] The same applies to encoding in the horizontal direction:
[0114]
[0115] Through the spatial feature information extraction operation, the generated channels are cascaded, and the feature Y of the video frame sequence at time t is extracted from each channel feature t It can be expressed as:
[0116]
[0117] Among them, σ represents the normalization operation, F represents the convolution operation, and F 3×3 、F 1×1 , represents the size of the convolution kernel in the convolution operation, and C / r represents the reduction ratio in the control channel number change.
[0118] The outputs of the three transformations are aggregated along each of the three dimensions to generate a set of spatiotemporal-aware feature maps. The three transformations capture the long-range temporal dependencies and spatial relationships in the feature space, which enables the network to more accurately obtain the spatiotemporal relationships between objects. The spatial relationships and temporal dependencies between objects are used to design an attention mechanism to highlight the areas prone to anomalies in video anomaly detection. Connecting the two channels, i.e. and Connect and get the output after convolution operation:
[0119] Y hw =δ(F 1×1 [Y h ,Y w ]) (11)
[0120] Among them, δ represents a nonlinear activation function, which gives Y hw Split into Y h ∈R C / r×H and Y w ∈R C / r×W Two tensors of the same dimension. h and Y w The final tensor g is obtained by Sigmoid activation function respectively. h and g w .
[0121] For the tensor of the time channel attention part, for Y t After processing with nonlinear activation function, we get:
[0122] g t =relu(Y t (X)) (12)
[0123] Tensor g t represents the weight of regional features that are sensitive to time changes. Finally, the output of the spatiotemporal relationship attention module Y is as follows:
[0124]
[0125] Since the input and output have the same resolution, the spatiotemporal relationship attention mechanism module can be learned and then inserted into the U-net network to help it make better future frame predictions without changing the input and output structure.
[0126] Objective Function: Combining all these constraints on appearance, motion, and adversarial training into the objective function yields the following objective function, which supports prototype learning that represents the normal behavior of the target, feature reconstruction for enhanced encoding, and future frame prediction.
[0127] First, the overall loss function ψ is composed of the feature reconstruction term ψ under the attention mechanism cd and frame prediction term ψ pre These two items are balanced by weights λ1 and λ2 as follows:
[0128] ψ=λ1ψ pre +λ2ψ cd (14)
[0129] The frame prediction loss is expressed as the input video real frame yt and future frames The prediction network outputs the Euclidean distance between predicted frames:
[0130]
[0131] The feature reconstruction loss under the attention mechanism is expressed as the learned common prototype has information diversity and feature representativeness, that is, high quality and diversity, and low redundancy. It has two terms ψ c and ψ d , respectively for these two properties, can be written as:
[0132] ψ cd =μ1ψ c +μ2ψ d (16)
[0133] Where μ1 and μ2 are weights. Feature compactness loss ψ c It is used to reconstruct the normal encoding using compact prototypes, which can query the most similar mode in the normal behavior pool closer, enhance the representativeness of each mode, and minimize the L2 norm between the query and the corresponding mode, i.e., it measures the average Euclidean distance between the input encoding vector and its most relevant prototype:
[0134]
[0135] Where k is the query The index of the closest pattern in the pool of normal behavior patterns, expressed as:
[0136]
[0137] Further promote the diversity among prototype projects. The diversity term ψ d Expressed as
[0138]
[0139] Among them, p, p′ represent different prototypes, and α is the expected distance of the pattern in the normal behavior pattern pool. Using the above two loss functions ψ c and ψ d ,The future frame prediction model can focus on the spatiotemporal relationships among objects to better predict normal future frames.
[0140] Anomaly score: It is assumed that normal events can be predicted well. Therefore, the difference between the predicted frame and its true value can be used to predict anomalies. Peak signal-to-noise ratio (PSNR) is a method of image quality assessment.
[0141]
[0142] in, is the maximum value representing the color of an image point
[0143] After calculating the PSNR of each frame of each test video, the PSNR of all frames in the test video are normalized to the range [0,1] and the regular score for each frame is calculated using the following equation:
[0144]
[0145] Among them, Score all (t) is the regularity score, which corresponds to the normality of each frame in the video, and plays an indicative role in the confidence of the detection result. A threshold can be set to distinguish normal frames from abnormal frames. When the corresponding value is lower than the threshold, it means that an abnormal event has occurred; when the corresponding value is higher than the threshold, it means that no abnormal event has occurred.
[0146] In the specific implementation, the designed model is trained and tested on three well-known video anomaly detection datasets: UCSDPed2, CUHKAvenue, and ShanghaiTech. In all datasets, the training set is used for model learning and determining the network structure, while the test set is only used to evaluate the performance of the trained model. Figure 5 Some examples of positive / abnormal samples are shown. The first row represents normal samples of Ped2, Avenue, and ShanghaiTech data. The second row represents the related abnormal samples, where the box is the abnormal area.
[0147] The UCSDPed2 subset includes 16 training videos and 12 test videos. The test videos contain 12 abnormal events with a resolution of 240×360, all of which are related to vehicles such as bicycles and cars. In the Avenue dataset, 14 events (i.e., walking in the wrong direction, running across the sidewalk, throwing paper, pushing a bicycle, etc.) are considered abnormal behaviors. In addition, it should be noted that the camera in the dataset has a shaking problem, while other datasets come from fixed cameras and the size of a person may change with the angle of the camera position. ShanghaiTech is one of the most challenging datasets for video anomaly detection to date. It consists of 13 scenes and various anomaly types (i.e., fighting, running, skating, and cycling), containing 130 abnormal events with different lighting conditions and camera angles.
[0148] To train the network, the input frame is resized to a resolution of 256×256, and the pixel intensity is normalized to the range of [-1,1]. During AE pre-training, the model is trained by the Adam optimizer, and the model is trained with an initial learning rate of 0.0001, a batchsize of 4, and a maximum epoch of 1000. For future frame prediction tasks, the present invention sets the initial learning rate to 0.0004 and gradually decays it using a cosine annealing strategy. The resolution of the encoded feature map is 256×256×128, and the training cycles on Ped2, Avenue, and Shanghai Tech are set to 60, 60, and 10, respectively. Set t=4 and predict the input of the 5th frame. During training: λ1 and λ2 are both set to 1, μ1 and μ2 are set to 0.99 and 0.01, respectively, and finally λ s Set to 1. In the experiments, the model is trained on a Nvidia GTX1080Ti GPU using the PyTorch framework.
[0149] The method proposed in the present invention includes three stages. In the first stage, the video block is provided as input to the prototype network of the meta-learning module and features are encoded; in the second stage, the encoded features are stored and updated by the meta-learning module; in the third stage, the spatial and temporal features of the video are more comprehensively understood through the learnable spatiotemporal attention module, the feature representation of new events is learned and feature matching is performed, and a set of anomaly scores are calculated to predict abnormal events.
[0150] The effectiveness of the method proposed in the present invention is verified through simulation experiments.
[0151] Using commonly used evaluation indicators, the receiver operating characteristic (ROC) is calculated by gradually changing the threshold of the regular score, and then the area under the characteristic curve (AUC) is accumulated as a scalar for performance evaluation. The relationship between the true positive rate (TPR) and the false positive rate (FPR) is plotted by changing the threshold of the anomaly score to obtain the ROC curve. The AUC value is mainly used to evaluate the robustness and effectiveness of the model. The higher the AUC value, the better the video anomaly detection performance of the model.
[0152] In order to more objectively illustrate the performance of the anomaly detection method in the present invention, Table 1 lists the methods based on different manual features (MAC), (GAS) and (Unmasking) and the latest deep learning-based methods (MPGT), (STemGAN), (zxVAD), (MGME) and (STMC) for comparison. At the same time, it is compared with the methods based on video prediction (Frame-Pred), (MemAE), (MNAD) and (MPN); based on spatiotemporal information or motion information (MPN), (Frame-Pred), (MPGT), (zxVAD), (STMC) and (MGME). The AUC of different methods is listed in Table 1.
[0153] Table 1 Comparison of AUC results of different methods. Bold fonts represent the best results, and underlined numbers represent suboptimal results.
[0154]
[0155]
[0156] As can be seen from Table 1, compared with these excellent deep feature-based methods, our method achieves excellent performance on the UCSDPed2, CUHKAvenue and ShanghaiTech datasets, and the proposed method obtains 97.63%, 98.41% and 75.43% detection accuracy respectively. The model is compared with many deep learning-based video anomaly detection methods and sorted by the order of article publication time. The comparison results are shown in Table 1. The performance of the compared methods is obtained from the original article or reproduced on our device. From Table 1, the following conclusions can be drawn: (1) The method proposed in this paper is effective on three commonly used datasets and achieves a relatively advanced level of 75.43% AUC on ShanghaiTech. (2) The model proposed in this paper can significantly improve the anomaly detection performance of the basic model. On Ped2, Avenue and ShanghaiTech, our method achieves an AUC performance improvement of 2.23%, 3.31% and 2.63% compared with the Frame-Pred baseline results. (3)MGME has the best detection performance on the UCSD Ped2 dataset, while MPN has the best performance on the Avenue dataset. Compared with MGME, MGME adopts multi-layer embedding of memory. Feature information is automatically filtered during the decoding process, and the features are enhanced by embedding prototype information in the memory to compensate for the information required for detection, resulting in better detection results. Compared with DPU, Dynamic prototype unit (DPU) can encode normal dynamics into prototypes in real time without additional memory costs. This improves the quality of the obtained video frame features and the accuracy of subsequent detection. Although Frame-Pred uses the same U-Net network for future frame prediction, the Frame-Pred method extracts appearance and optical flow motion information and lacks features in a temporal sense.
[0157] The attention mechanism that can learn spatiotemporal relationships enables the neural network to better obtain the complete temporal information and spatial information of the video sequence, which helps to obtain better video frame features, thereby obtaining higher accuracy. Especially on the ShanghaiTech dataset, the advantages of the model of the present invention are more prominent. Since the ShanghaiTech dataset has more diverse scenes and abnormal event types, the ShanghaiTech dataset is more significant compared with the other two datasets, and the attention mechanism that can learn spatiotemporal relationships is more conducive to feature extraction in large data sets. Compared with other datasets and models, our model has a better detection effect on this dataset.
[0158] The method of the present invention designs a deep neural network, which introduces a meta-learning module to learn and update the input video features, and generates codes with feature prototypes to help the future frame prediction network to make better predictions. At the same time, the attention module that can learn spatiotemporal relationships can help the future frame prediction network to refine rough predictions by extracting features in spatial and temporal dimensions, which is more conducive to extracting cognitive information. Therefore, its performance on the three data sets is slightly better.
[0159] Qualitative results: Figure 6 The prediction of normal and abnormal events by the method of the present invention is further visualized in FIG. Figure 6 As shown in Figure 2, some images containing normal and abnormal events and their heatmaps are shown. The heatmap is derived from the generated predicted frames as the difference with the target frame. It is obvious from the heatmap that the predicted frames with abnormal samples reconstructed by the model have large precision errors on abnormal objects or abnormal behaviors. It can also be seen from the heatmap that normal events can be reconstructed well with almost no obvious errors. This shows that the method can adequately reconstruct normal video frames, but cannot reconstruct abnormal video frames. Therefore, when abnormal events are detected, large reconstruction errors occur.
[0160] Qualitative analysis: To qualitatively analyze the anomaly detection performance of the future frame prediction network with meta-learning, we visualized anomaly detection examples under the future frame prediction network framework with meta-learning, such as Figure 7 , Figure 8 and Fig. 9 Anomaly curves of UCSD Ped2, CHUK Avenue, and ShanghaiTech in the video. The regularity score curve displays the anomaly scores of all frames in the video in order, which can more intuitively show the performance of the proposed method. In each sub-graph, the regularity score indicates the possibility of normality, and the shaded part in the video frame indicates the anomaly in the ground truth.
[0161] As can be seen from the figure, for normal frames, the anomaly score curve remains at a high value and is relatively stable. However, for abnormal frames, when there is an abnormal action, the anomaly score curve will drop significantly, producing a strong downward peak, and the prediction results are often distorted. Figure 7 It can be seen that the future frame prediction network introduced with meta-learning can correctly detect anomalies even in crowded scenes. Figure 8 , Fig. 9 As shown in the figure, when an anomaly occurs, the anomaly score curve will drop significantly and vibrate violently. When the abnormal target is more obvious, the anomaly score curve will drop to the lowest level.
[0162] Ablation Experiments: To analyze the role and impact of different components in the network, ablation experiments were conducted on three datasets, and the anomaly detection performance based on AUC was performed, as shown in Table 2. Table 2 Ablation experiments were conducted on UCSD Ped2, CUHK Avenue, and ShanghaiTech datasets. The model effect is measured by AUC. The bold font indicates the best effect.
[0163]
[0164] As can be seen from the table, before adding the meta-learning module, the prediction results of the learnable spatiotemporal relationship attention module greatly exceeded those of the original future frame prediction network. Taking Avenue as an example, the AUC increased from 84.35% to 85.97%, an increase of 1.62%. This is because the learnable spatiotemporal relationship attention module has good constraints on fast-moving objects such as running and riding bicycles, and can better learn the spatiotemporal relationship of objects, indicating the necessity of adding the learnable spatiotemporal relationship attention module as additional information to improve the performance of anomaly detection. After adding the meta-learning module, the AUC of Ped2, Avenue, and ShanghaiTech increased by 1.48% (from 96.15% to 97.63%), 2.44% (from 85.97% to 88.41%), and 1.91% (from 73.52% to 75.43%) respectively compared with before adding. On the one hand, it is because the meta-learning module converts the features of high-dimensional space into low-dimensional dynamics for storage. On the other hand, the continuous updating of contextual information by the meta-learning module helps the neural network to generalize abnormal behavior events in the video and reduces the prediction error of the model. This fully verifies the effectiveness of the future frame prediction network introduced with meta-learning in video anomaly detection and reflects better detection performance.
[0165] like Figure 10-12 As shown in Figure 2, the larger the area under the curve, the better the detection effect. Figure 10-12 The comparison effect of improving the performance model is also shown in Figure 1. It can be clearly observed from the figure that the area under the curve corresponding to the method proposed in the present invention is significantly larger than that of the original prediction method in all three data sets. This proves the effectiveness of the future frame prediction network introduced with meta-learning.
[0166] The specific implementation scheme described above further describes in detail the purpose, technical scheme and beneficial effects of the present invention. It should be understood that the above is only a specific implementation scheme of the present invention and is not intended to limit the scope of the present invention. Any equivalent changes and modifications made by any technician in the field without departing from the concept and principle of the present invention should fall within the scope of protection of the present invention.
Claims
1. A future frame anomaly detection method based on meta-learning and spatiotemporal relations, characterized in that: The following steps are involved: Step 1: Based on the autoencoder, a continuous video frame I1, I2, ..., I t As input, we stack all these frames on the channel and use them to predict future frames. Step 2: Feed the input frame through the selected AE code into the meta-learning module, which is trained to learn and compress the normal dynamics of real-time sequence information as multiple normal prototypes and enrich the input AE code with the normal dynamic information; Step 3: Run the output code through the remaining AE layers to process the normal behavior for subsequent frame prediction. Step 4: To make the prediction frame Closer to I t+1 , while minimizing their distance in terms of intensity and gradient, in order to maintain the temporal consistency between adjacent frames, an attention mechanism that can learn spatiotemporal relations is introduced to constrain the real frame I t+1 and predicted frame Step 5: The difference between the predicted future frame and its normal frame determines whether the event is normal or abnormal.
2. The future frame anomaly detection method based on meta-learning and spatiotemporal relationship according to claim 1, characterized in that: In the meta-learning module, the set of normal behavior features is defined as: Where m represents the maximum number of normal behavior features contained in the meta-learning module Z; represents the type of the i-th meta-learned normal behavior label, θ i represents the value of the normal behavior label of the ith meta-learning; for the sample points inside the support set Z, the encoding formula f is used φ To generate a prototype representation for each category: Among them, the encoding formula f φ is an arbitrary information extraction method, using AE encoding, the meta-learning module is trained to learn the normal dynamics of real-time sequence information as multiple prototypes, and enrich the input AE encoding with normal dynamic information; extract the tth input encoding map x(t) = f from AE φ (x),x(t)∈R h×w×c , treating it as a c-dimensional N = w × h vector, assigning a normal weight to each pixel position represents f φ for: f φ It is called the normal behavior pool, that is, the prototype pool; the input of a continuous video frame represents the set 3. The future frame anomaly detection method based on meta-learning and spatiotemporal relationship according to claim 2 is characterized in that: During the query process, the input encoding vector from the AE encoding graph is used as a query to retrieve relevant terms in the prototype pool to reconstruct the normal encoding f φ (x), encode the query set as f φ ′(x): in, Denoted as n-th encoding vector and the mth prototype item The correlation score between them; the probability p of the current sample belonging to each category is calculated w (y=k|x), using the softmax calculation method:
4. The future frame anomaly detection method based on meta-learning and spatiotemporal relationship according to claim 3 is characterized in that: Future frame prediction: U-Net is modified for future frame prediction. In the U-Net network, an attention module is introduced to learn the spatiotemporal relationship of the input image, so that the prediction network based on U-Net has better prediction effect; for every two convolutional layers, the output resolution is retained; therefore, cropping and resizing operations are no longer required when adding shortcuts; the kernel size of all convolutional layers, deconvolutional layers, and maximum pooling layers is set to 3; in the U-Net prediction network, the input and output have the same resolution; Attention mechanism, the network input image size is set to a (256, 256) grayscale image, and then after two-by-two 3*3 convolutions, the dimension becomes (64, 256, 256). After three downsampling, the image dimension is (512, 32, 32). After the downsampling part is implemented, the attention mechanism module is used to repeat the upsampling three times until the image is downsampled to (64, 256, 256), obtaining the input and output of the same resolution; Following the future frame prediction work, intensity and gradient differences are used to make the prediction close to its true situation; the intensity constraint ensures the similarity of all pixels in RGB space; Minimize the predicted frame in intensity space The L2 distance between it and its true I: Gradient constraints are used to sharpen the generated image: Where i, j represents the spatial index of the video frame; In the attention mechanism that can learn spatiotemporal relationships, it is possible to extract and aggregate dimensional features separately by combining spatial and temporal information to obtain spatiotemporal relationship information; for the features of the input H dimension First, it is encoded through a one-to-one one-dimensional feature encoding operation, which uses two pooling kernels with a spatial size of H; (H×1) and (W×1) average pooling encoding is performed on each channel, in the horizontal direction and vertical direction respectively, and encoded in the vertical direction. The output of the Cth channel in the H dimension is as follows: The same applies to encoding in the horizontal direction: Through the spatial feature information extraction operation, the generated channels are cascaded, and the feature Y of the video frame sequence at time t is extracted from each channel feature t It is expressed as: Among them, σ represents the normalization operation, F represents the convolution operation, and F 3×3 、F 1×1 , represents the size of the convolution kernel in the convolution operation, and C / r represents the reduction ratio in the change of the number of control channels; The outputs of the three transformations are aggregated along each of the three dimensions to generate a set of spatiotemporal-aware feature maps; the three transformations capture the long-range temporal dependencies and spatial relationships in the feature space, which enables the network to more accurately obtain the spatiotemporal relationships between objects; the spatial relationships and temporal dependencies between objects are used to design an attention mechanism to highlight the areas prone to anomalies in video anomaly detection; the two channels are connected, i.e. and Connect and get the output after convolution operation: Among them, δ represents a nonlinear activation function, which gives Y hw Split into Y h ∈R C / r×H and Y w ∈R C / r×W Two tensors of the same dimension; Y h and Y w The final tensor g is obtained by Sigmoid activation function respectively. h and g w ; For the tensor of the time channel attention part, for Y t After processing with nonlinear activation function, we get: g t =relu(Y t (X)) (12) Tensor g t Represents the regional feature weights that are sensitive to time changes; the output of the spatiotemporal relationship attention module Y is as follows:
5. The future frame anomaly detection method based on meta-learning and spatiotemporal relationship according to claim 4 is characterized in that: Objective function: All constraints on appearance, motion, and adversarial training are combined into the objective function, and the following objective functions are derived, which support prototype learning representing the normal behavior of the target, feature reconstruction for enhanced encoding, and prediction of future frames; First, the overall loss function ψ is composed of the feature reconstruction term ψ under the attention mechanism cd and frame prediction term ψ pre Composition, balanced by weights λ1 and λ2: ψ=λ1ψ pre +λ2ψ cd (14) The frame prediction loss is expressed as the input video ground-truth frame y t and future frames The prediction network outputs the Euclidean distance between predicted frames: The feature reconstruction loss under the attention mechanism is expressed as the learned common prototype has information diversity and feature representativeness, that is, high quality and diversity, and low redundancy, with two terms ψ c and ψ d , respectively, for these two properties, written as: ψ cd =μ1ψ c +m2ψ d (16) Among them, μ1 and μ2 are weights; feature compactness loss ψ c For reconstructing normal encodings using compact prototypes, queries are closer to the most similar patterns in the pool of normal behavior, enhancing the representativeness of individual patterns and minimizing the L2 norm between the query and the corresponding pattern, i.e., it measures the average Euclidean distance between the input encoding vector and its most relevant prototype: Where k is the query The index of the closest pattern in the pool of normal behavior patterns, expressed as: Promote diversity among prototype projects, the diversity term ψ d It is expressed as: Among them, p, p′ represent different prototypes, and α is the expected distance of the pattern in the normal behavior pattern pool.
6. The future frame anomaly detection method based on meta-learning and spatiotemporal relationship according to claim 5, characterized in that: Anomaly prediction is performed using the difference between the predicted frame and its true value: in, It is the maximum value representing the color of an image point; PSNR is the peak signal-to-noise ratio; After calculating the PSNR of each frame of each test video, the PSNR of all frames in the test video are normalized to the range [0,1] and the regular score for each frame is calculated using the following equation: Among them, Score all (t) is a regularity score, which corresponds to the normality of each frame in the video and indicates the confidence of the detection result. A threshold is set to distinguish normal frames from abnormal frames. When the corresponding value is lower than the threshold, it indicates that an abnormal event has occurred. When the corresponding value is higher than the threshold, it indicates that no abnormal event has occurred.
Citation Information
Patent Citations
Video abnormal event detection method based on gray adversarial perception network
CN115063742A
Abnormal event detection method of concerning time-space position relation of target
CN116630861A
Video anomaly detection network structure and method based on future frame prediction and attention mechanism
CN117541898A
Self-supervision video anomaly detection method combined with self-attention module
CN118262273A
Cited By
Daily scene millimeter wave sensing non-human target filtering method and system
CN121454515A
