Future frame anomaly detection method based on meta-learning and spatio-temporal relationship

Abstract

The present application belongs to the technical field of intelligent video processing, and particularly relates to a future frame anomaly detection method based on meta learning and space-time relationship. The present application method proposes a meta learning module, and a model-based meta learning method enables the model to learn general features from multiple tasks, so that the system can learn more discriminative and generalizable feature representations from data. In different monitoring scenes and abnormal behaviors, the model can also accurately perform anomaly detection without overfitting or underfitting. The present application method introduces the meta learning module into the autoencoder, uses the learning characteristics of the meta learning module to extract, save and update the features extracted by the autoencoder, automatically acquires the feature importance of the input video frame through learning, and assigns important weights to the features that are more worthy of attention, which helps to improve the utilization rate of key features in the input stage of the future frame prediction network.

Description

Technical Field

[0001] This invention belongs to the field of intelligent video processing technology, specifically relating to a future frame anomaly detection method based on meta-learning and spatiotemporal relationships. Background Technology

[0002] With the widespread adoption of surveillance equipment and increasing public concern for safety, video anomaly detection—identifying abnormal events or behaviors in video sequences—has gradually become a research hotspot in computer vision. Autoencoders (AEs) are a popular method for video anomaly detection. Researchers typically use AEs to model normal patterns in historical frames and then reconstruct the current frame or predict upcoming frames. Since the model is trained using only normal data, the prediction error for anomalous inputs is expected to be higher than the prediction error for normal corresponding inputs. Previously, many methods were based on this assumption for anomaly detection. However, this assumption is not always valid. This makes video anomaly detection still extremely challenging: on the one hand, existing methods rely on large amounts of normal training data to model shared normal patterns. These models are prone to the "overgeneralization" dilemma, as all video frames can be predicted well due to the powerful representational capabilities of convolutional neural networks, regardless of whether they are normal or anomalous. It is worth noting that a potential limitation of encoder-decoder structures is that the decoder focuses only on predicting the correspondence between the visual features of future frames and input video frames during decoding. The same target object in a normal sample may appear in multiple scenes with similar but different contextual information, potentially leading to anomalous samples being predicted well. In video anomaly detection, Yiwei Lu, Frank Yu, Mahesh Kumar Krishna Reddy, and Yang Wang. Few-shot scene-adaptive anomaly detection. In ECCV, 2020. Following an optimization-based meta-learning method, Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML, 2017, 4, 6, and applying it to train a scene-adaptive anomaly detection model; Lv, H.; Chen, C.; Cui, Z.; Xu, C.; Li, Y.; Yang, J. Learning Normal Dynamics in Videos with MetaPrototype Network. In Proceedings of the 2021 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), virtual, 19–25 June. 2021; pp.15420–15429. A dynamic prototyping unit is proposed that dynamically encodes normal samples into prototypes in real time without additional memory cost.Zhang S, Song X, Wang C, et al. Object-meta and MSGAE-MP: Multi-dimensional video anomaly detection[J]. IET ComputerVision, 2022. Using object meta-frames instead of video frames enhances the multi-dimensional information carried by the input. Inspired by these works, a meta-learning module is introduced into the autoencoder (AE) to overcome the limitations of the autoencoder. First, the prior knowledge of the current task is encoded by the encoder and stored in the prototype network of the meta-learning module. Then, the prototype is obtained as a set of feature encodings, forming a meta-learning normal behavior pool. The complex changes between feature encodings are used to infer whether new events have occurred. This module directly learns and matches the encoded features, avoiding the error in determining the actual distance between features caused by chain clustering in the high-dimensional feature space during low-dimensional mapping. Simultaneously, the meta-learning prototype network can continuously store and update the feature encodings. Finally, the output encoding of the meta-learning module is passed through the remaining AE layers and then used in the prediction network for subsequent frame prediction. The introduction of the meta-learning module alleviates the prediction of the same target object in different contexts in normal samples.

[0003] On the other hand, most of the surveillance video is static, and the normal patterns appearing in different scenes are also different. The spatiotemporal relationship between the environment and background is very important for determining whether behavior is abnormal. For example, when a person runs on a playground, it should be normal behavior, but performing the same behavior in an office may be considered abnormal behavior. Liu W, Luo WX, Lian DZ, Gao SH (2018) Future frame prediction for anomaly detection--a new baseline. Processing of the IEEE ConfComput Vis Pattern Recognit:6536-6545 proposed a novel anomaly detection method that learns normal sample features through a U-Net network architecture. However, this method focuses on learning the apparent features of video frames and ignores an important clue, namely, when constraining the motion features between adjacent video frames with optical flow, optical flow usually assumes that the pixels of the image move in time in a small local area. However, in complex scenes, the motion of objects may be non-uniform or non-local, which may cause the optical flow to deviate when capturing motion, making it insensitive to some abnormal objects. With the increasing popularity of memory-enhanced deep autoencoders, the Memory module is used to memorize typical distribution features of normally distributed data and leverage them to constrain the expressive power of convolutional neural networks. This memory module can effectively guide the feature extraction process; however, the features in the Memory module are only spatial features at a certain stage of the network and do not attempt to guide the memorization of features related to the temporal dimension. Therefore, the Memory module can only provide limited assistance to the model in distinguishing between normal and abnormal data. In text-based work, video anomaly detection is performed by learning the spatiotemporal relationships between objects. Combining spatial and temporal features helps to achieve a more comprehensive understanding of the video.

[0004] Based on the above analysis, it is necessary to provide a U-Net framework for future frame prediction based on meta-learning and spatiotemporal relationship learning. In order to fully understand the spatiotemporal dimension of video, an attention module that can learn spatiotemporal relationships is introduced into the U-Net network to learn the spatiotemporal relationships between objects in consecutive input video frames, so that the neural network can better adapt to the spatiotemporal dynamic changes of video sequences and the prediction network based on U-Net can have better prediction performance. Summary of the Invention

[0005] This invention addresses the technical problems in the prior art by proposing a future frame anomaly detection method based on meta-learning and spatiotemporal relationships. This method aims to solve the problems that autoencoders focus only on the correspondence between the visual features of the predicted frame and the input video frame during decoding, and that ignoring temporal information leads to the model being insensitive to some abnormal objects, thereby improving the anomaly detection accuracy.

[0006] To achieve the above-mentioned objectives, the present invention adopts the following technical solution:

[0007] A future frame anomaly detection method based on meta-learning and spatiotemporal relationships includes the following steps:

[0008] Step 1: Based on an autoencoder, process a continuous video frame sequence I1, I2, ..., I... t As input, all these frames are stacked on a channel and used to predict future frames.

[0009] Step 2: Feed the input frame into the meta-learning module through the selected AE code. The meta-learning module is trained to learn and compress the normal dynamics of real-time sequence information as multiple normal prototypes and enrich the input AE code with normal dynamic information.

[0010] Step 3: The output encoding is run through the remaining AE layers, processing normal behavior for use in subsequent frame prediction.

[0011] Step 4: To predict the frame Closer to I t+1 To minimize the distance between them in terms of intensity and gradient, and to maintain temporal consistency between adjacent frames, an attention mechanism with learnable spatiotemporal relationships is introduced to constrain the real frame I. t+1 and predicted frames

[0012] Step 5: The difference between the predicted future frame and its normal frame determines whether the event is normal or abnormal.

[0013] Furthermore, as a preferred embodiment of the present invention, in the meta-learning module, the set of normal behavior features is defined as:

[0014]

[0015] Where m represents the maximum number of normal behavioral features contained in the meta-learning module Z; θ represents the type of the normal behavior label learned by the i-th element. i This represents the value of the label for the i-th element learning normal behavior. For sample points within the support set Z, the encoding formula f is used. φ To generate a prototype representation for each category:

[0016]

[0017] The encoding formula f here φAny information extraction method can be used; this paper employs AE encoding. The meta-learning module is trained to learn the normal dynamics of real-time sequence information as multiple prototypes, and enriches the input AE encoding with this normal dynamic information. The t-th input encoded image x(t) = f is extracted from the AE. φ (x),x(t)∈R h×w×c It can be viewed as a c-dimensional N = w × h vector, with normal weights assigned to each pixel position. f φ for:

[0018]

[0019] f φ This can be called the normal behavior pool, or prototype pool. A continuous sequence of video frames can be represented as a set.

[0020] Furthermore, as a preferred embodiment of the present invention, similarly, an encoding of the query set is also generated for the query set. During the query process, the input encoding vector from the AE encoding graph... It is used as a query to retrieve relevant items from the prototype pool in order to reconstruct the normal code f. φ (x), encode the query set as f φ ′(x):

[0021]

[0022] in, This is represented as the nth encoded vector. and the m-th prototype item The correlation scores between them.

[0023] Finally, the probability p of the current sample belonging to each category is calculated. w (y = k|x), here we use the softmax calculation method:

[0024]

[0025] The key idea behind introducing the meta-learning module is to enrich the AE (Advanced Effect) encoding with normal information to enhance the prediction of normal parts of video frames while suppressing abnormal parts. The output encoding of the meta-learning module is then passed through the remaining AE layers and used in the prediction network for subsequent frame prediction.

[0026] Further, as a preferred technical solution of the present invention, future frame prediction: By modifying U-Net for future frame prediction, an attention module is introduced into the U-Net network to learn the spatiotemporal relationships of the input image, resulting in better prediction performance for the U-Net-based prediction network. Specifically, the output resolution is preserved for every two convolutional layers. Therefore, cropping and resizing operations are no longer required when adding shortcuts. The kernel size of all convolutional layers, deconvolutional layers, and max-pooling layers is set to 3. In the U-Net prediction network, the input and output have the same resolution. The attention mechanism module will be described in the following section.

[0027] First, the input image size is set to a grayscale image of (256, 256). Then, after pairwise 3*3 convolutions, the image becomes (64, 256, 256). After three downsampling operations, the image dimension becomes (512, 32, 32). After the downsampling is completed, the upsampling is repeated three times through the attention mechanism module until the image is downsampled to (64, 256, 256), thus obtaining input and output with the same resolution.

[0028] Following the future frame prediction process, intensity and gradient difference are used to make the prediction approximate the true value. Intensity constraints guarantee the similarity of all pixels in the RGB space. Specifically, the predicted frame is minimized in the intensity space. L2 distance between it and its true I:

[0029]

[0030] Gradient constraints are used to sharpen the generated image:

[0031]

[0032] Where i,j represent the spatial indices of the video frames.

[0033] A novel attention mechanism based on spatiotemporal relationships was designed to capture the spatiotemporal relationships between objects in video frames. The right side of the channel represents the attention process for spatial relationships between objects, while the left side focuses on temporal relationships. Finally, the original input is jointly weighted by the spatial and temporal attention channels and used as the module's output to learn the spatiotemporal relationships of objects.

[0034] In attention mechanisms that enable learning spatiotemporal relationships, spatiotemporal relationship information can be obtained by combining spatial and temporal information to extract and aggregate dimensional features separately. Specifically, this involves processing the input H-dimensional features... First, encoding is performed using a one-to-one dimensional feature encoding operation. This operation utilizes two pooling kernels of size H; (H×1) and (W×1) are used to perform average pooling encoding on each channel, in the horizontal and vertical directions respectively. Encoding is performed in the vertical direction, and the output of the C-th channel in the H dimension is as follows:

[0035]

[0036] The same applies to horizontal encoding:

[0037]

[0038] By extracting spatial feature information, the generated channels are concatenated, and the feature Y of the video frame sequence at time t is extracted from the features of each channel. t It can be represented as:

[0039]

[0040] Where σ represents the normalization operation, F represents the convolution operation, and F 3×3 F 1×1 , where represents the kernel size in the convolution operation, and C / r represents the scaling ratio that controls the change in the number of channels.

[0041] The outputs of the three transformations are aggregated along each of the three dimensions to generate a set of spatiotemporally aware feature maps. These three transformations capture long-range temporal dependencies and spatial relationships in the feature space, enabling the network to more accurately capture the spatiotemporal relationships between objects. An attention mechanism is designed using the spatial relationships and temporal dependencies between objects to highlight regions prone to anomalies in video anomaly detection. Two channels are concatenated, i.e., Y... c h (h) and Y c w (w) are connected, and the output is obtained after convolution operation:

[0042] Y hw =δ(F 1×1 [Y h ,Y w (11)

[0043] Where δ represents the nonlinear activation function, thus obtaining Y hw Divide into Y h ∈R C / r×H and Y w ∈R C / r×W Two tensors with the same dimension. Let Y... h and Y w The final tensor g is obtained by passing it through the Sigmoid activation function. h and gw .

[0044] For the tensor of the attention component of the time channel, for Y t After processing with a nonlinear activation function, we get:

[0045] g t =relu(Y t (X)) (12)

[0046] Tensor g t This represents the weights of regional features that are sensitive to time changes. Finally, the output of the spatiotemporal attention module Y is as follows:

[0047]

[0048] Since the input and output have the same resolution, the spatiotemporal attention mechanism module can be inserted into the U-net network to help it make better predictions of future frames without changing the input and output structure.

[0049] Furthermore, as a preferred embodiment of the present invention, the objective function incorporates all these constraints regarding appearance, motion, and adversarial training into the objective function, resulting in the following objective functions that support prototype learning representing the normal behavior of the target, feature reconstruction of augmented encoding, and prediction of future frames.

[0050] First, the overall loss function ψ is composed of the feature reconstruction term ψ under the attention mechanism. cd And frame prediction term ψ pre Composition. These two items are balanced by weights λ1 and λ2 as follows:

[0051] ψ=λ1ψ pre +λ2ψ cd (14)

[0052] Frame prediction loss is represented as the sum of the real frames y in the input video. t and future frames The Euclidean distance between the predicted frames output by the prediction network:

[0053]

[0054] The feature reconstruction loss under the attention mechanism is represented by the fact that the learned prototypes possess information diversity and feature representativeness, i.e., high quality and diversity, and low redundancy. It has two terms ψ. c and ψ d For these two properties respectively, it can be written as:

[0055] ψ cd =μ1ψ c +μ2ψ d (16)

[0056] Where μ1 and μ2 are weights. Feature compactness loss ψ c Used to reconstruct normal coding using compact prototypes, it queries patterns that are closer to the most similar patterns in the pool of normal behavior, enhances the representativeness of each pattern, and minimizes the L2 norm between the query and the corresponding pattern. That is, it measures the average Euclidean distance between the input coding vector and its most relevant prototype.

[0057]

[0058] Where k is the query The index of the closest pattern in the normal behavior pattern pool is represented as:

[0059]

[0060] Further promote diversity among prototype projects. Diversity term ψ d Represented as

[0061]

[0062] Where p and p′ represent different prototypes, and α is the expected distance of patterns in the normal behavior pattern pool. Using the two loss functions ψ mentioned above... c and ψ d The future frame prediction model can focus on the spatiotemporal relationships between objects in order to better predict normal future frames.

[0063] As a further preferred embodiment of the present invention, anomaly scoring assumes that normal events can be predicted well. Therefore, anomaly prediction can be performed using the difference between the predicted frame and its true value. Peak signal-to-noise ratio (PSNR) is a method for image quality assessment.

[0064]

[0065] in, It is the maximum value representing the color of a point in the image.

[0066] After calculating the PSNR for each frame of each test video, the PSNR of all frames in the test video is normalized to the range [0,1], and the regular score for each frame is calculated using the following equation:

[0067]

[0068] Among them, Score all(t) is the regularity score, corresponding to the normality of each frame in the video, and it serves as an indicator of the confidence level of the detection results. A threshold can be set to distinguish between normal and abnormal frames. When the corresponding value is below the threshold, it indicates that an abnormal event has occurred; when the corresponding value is above the threshold, it indicates that no abnormal event has occurred.

[0069] The future frame anomaly detection method based on meta-learning and spatiotemporal relationships described in this invention, compared with existing technologies, has the following technical advantages:

[0070] (1) The method of this invention proposes a meta-learning module. The model-based meta-learning method enables the model to learn general features from multiple tasks, and the system can learn more discriminative and generalizable feature representations from the data. The model can also accurately detect anomalies in different monitoring scenarios and abnormal behaviors without overfitting or underfitting.

[0071] (2) The method of the present invention introduces the meta-learning module into the autoencoder. By utilizing the learning characteristics of the meta-learning module, the autoencoder extracts, saves and updates the features extracted, while automatically acquiring the importance of the features of the input video frame through learning, and assigning important weights to the more noteworthy features. This helps to improve the utilization rate of key features in the future frame prediction network input stage.

[0072] (3) The attention module in the method of this invention enables the model to focus on the spatiotemporal relationship between objects. It is then introduced into the future frame prediction network. The three channels of the attention module are used to extract and fuse features of the spatiotemporal and spatial dimensions of the input video frame, and to learn the regularity of the spatial feature space and motion feature space, so that the future frame prediction network can better predict the next frame, which is beneficial for detecting abnormal events. Attached Figure Description

[0073] Figure 1 A schematic diagram illustrating the future frame prediction framework for introducing meta-learning in this invention;

[0074] Figure 2 This is a schematic diagram of the meta-learning module of the present invention;

[0075] Figure 3 This is a schematic diagram of the U-net prediction network of the present invention;

[0076] Figure 4 This is a schematic diagram of the attention mechanism for learnable spatiotemporal relationships in this invention;

[0077] Figure 5 This is a schematic diagram of positive / abnormal samples for the present invention;

[0078] Figure 6 This invention provides real frames and heatmaps for detecting normal and abnormal events.

[0079] Figure 7 The outlier scores for videos #02 and #06 from the UCSDPed2 dataset of this invention;

[0080] Figure 8 The outlier scores for videos #04 and #13 from the CHUKAvenue dataset of this invention;

[0081] Figure 9 The outlier scores of videos #01_0133 and #05_0017 from the ShanghaiTech dataset are for this invention.

[0082] Figure 10 This is a comparison diagram between the meta-learning module introduced in this invention and the UCSDPed2 basic model;

[0083] Figure 11 This is a comparison diagram between the meta-learning module introduced in this invention and the CHUKAvenue basic model;

[0084] Figure 12 This is a comparison diagram between the meta-learning module introduced in this invention and the ShanghaiTech basic model. Detailed Implementation

[0085] The present invention will be further explained in detail below with reference to the accompanying drawings, so that those skilled in the art can better understand and implement the present invention. However, the following examples are only used to explain the present invention and are not intended to limit the present invention.

[0086] This invention proposes a video anomaly detection method based on meta-learning and spatiotemporal relationships, primarily achieving an unsupervised task by predicting future video frames. To better predict future frames, most existing works only consider appearance constraints by incorporating intensity loss, gradient loss, or adversarial training loss. However, appearance constraints alone cannot guarantee the generation of realistic video frames. They only consider obvious cues of target objects, neglecting the importance of motion constraints for video anomaly detection. Besides spatial information, temporal information is also a crucial feature of videos. Therefore, an attention mechanism with learnable spatiotemporal relationships is added to the objective function to ensure the consistency of motion in normal target events within the training set. Furthermore, many existing works use clustering discriminative algorithms to cluster extracted features to detect anomalies in video frames. However, this ignores the impact of the high-dimensional to low-dimensional clustering chain feature mapping process on feature distance determination. Considering the potential limitations of encoder-decoder structures and the strong generalization of neural networks, a meta-learning module is introduced into the autoencoder to learn normal behavior patterns using example features. Notably, anomalies can be caused by appearance or motion, and this invention's meta-learning future frame prediction scheme utilizes both appearance and motion constraints of normal events. Therefore, this video prediction with appearance and motion constraints is more consistent with video anomaly detection. Anomalies can be easily identified by comparing predicted frame results with normal events in real frames. A future frame prediction framework incorporating meta-learning is then introduced, such as... Figure 1 As shown.

[0087] The method of this invention includes the following steps: Step 1: Based on an autoencoder, a continuous video frame I1, I2, ..., I... t As input, all these frames are stacked on a channel and used to predict future frames. Step 2: Feed the input frame forward through the selected AE encoding into the meta-learning module. The meta-learning module is trained to learn and compress normal dynamics of real-time sequence information as multiple normal prototypes, and enrich the input AE encoding with normal dynamic information; Step 3: Run the output encoding through the remaining AE layers to process normal behavior for subsequent frame prediction. Step 4: To predict the frame Closer to I t+1 To minimize the distance between them in terms of intensity and gradient, and to maintain temporal consistency between adjacent frames, an attention mechanism with learnable spatiotemporal relationships is introduced to constrain the real frame I. t+1 and predicted frames Step 5: The difference between the predicted future frame and its normal frame determines whether the event is normal or abnormal.

[0088] Meta-learning module: In the meta-learning module, as follows: Figure 2As shown, the query set image [m1,m2,…,m] will be... n The label is the closest normal behavior prototype, and the label set is...

[0089] The prototype network processes data by extracting features from the image and then calculating a prototype for each category. To do this, the average image embedding is calculated for each class. Once the prototype is obtained, classification can be performed simply by calculating the Euclidean distance between the image embedding and the prototype. This results in a meta-learning normal behavior pool, which stores the extracted features and updates them as needed.

[0090] In the meta-learning module, the set of normal behavior features is defined as:

[0091]

[0092] Where m represents the maximum number of normal behavioral features contained in the meta-learning module Z; θ represents the type of the normal behavior label learned by the i-th element. i This represents the value of the label for the i-th element learning normal behavior. For sample points within the support set Z, the encoding formula f is used. φ To generate a prototype representation for each category:

[0093]

[0094] The encoding formula f here φ Any information extraction method can be used. This invention employs AE encoding, where a meta-learning module is trained to learn the normal dynamics of real-time sequence information as multiple prototypes, and enriches the input AE encoding with this normal dynamic information. The t-th input encoded image x(t) = f is extracted from the AE. φ (x),x(t)∈R h×w×c It can be viewed as a c-dimensional N = w × h vector, with normal weights assigned to each pixel position. f φ for:

[0095]

[0096] f φ This can be called the normal behavior pool, or prototype pool. A continuous sequence of video frames can be represented as a set.

[0097] Similarly, an encoding for the query set is also generated. During the query process, the input encoding vector comes from the AE encoding graph. It is used as a query to retrieve relevant items from the prototype pool in order to reconstruct the normal code f. φ (x), encode the query set as fφ ′(x):

[0098]

[0099] in, This is represented as the nth encoded vector. and the m-th prototype item The correlation scores between them.

[0100] Finally, the probability p of the current sample belonging to each category is calculated. w (y = k|x), here we use the softmax calculation method:

[0101]

[0102] The key idea behind introducing the meta-learning module is to enrich the AE (Advanced Effect) encoding with normal information to enhance the prediction of normal parts of video frames while suppressing abnormal parts. The output encoding of the meta-learning module is then passed through the remaining AE layers and used in the prediction network for subsequent frame prediction.

[0103] Future Frame Prediction: By modifying U-Net for future frame prediction, an attention module is introduced into the U-Net network to learn the spatiotemporal relationships of the input image, resulting in better prediction performance for U-Net-based prediction networks. Specifically, the output resolution is preserved for every two convolutional layers. Therefore, cropping and resizing operations are no longer required when adding shortcuts. Details of this network are as follows... Figure 3 As shown. The kernel size for all convolutional, deconvolutional, and max-pooling layers is set to 3. In the U-Net prediction network, the input and output have the same resolution. The attention mechanism module will be introduced in the next section.

[0104] First, the input image size is set to a grayscale image of (256, 256). Then, after pairwise 3*3 convolutions, the image becomes (64, 256, 256). After three downsampling operations, the image dimension becomes (512, 32, 32). After the downsampling is completed, the upsampling is repeated three times through the attention mechanism module until the image is downsampled to (64, 256, 256), thus obtaining input and output with the same resolution.

[0105] Following the future frame prediction process, intensity and gradient difference are used to make the prediction approximate the true value. Intensity constraints guarantee the similarity of all pixels in the RGB space. Specifically, the predicted frame is minimized in the intensity space. L2 distance between it and its true I:

[0106]

[0107] Gradient constraints are used to sharpen the generated image:

[0108]

[0109] Where i,j represent the spatial indices of the video frames.

[0110] Attention Mechanism Based on Spatiotemporal Relationships: A novel attention mechanism is designed to capture the spatiotemporal relationships between objects in video frames. Figure 4 The structure of the attention module is shown. In the diagram, the right side of the channel represents the attention process for spatial relationships between objects, while the left side focuses on temporal relationships. Finally, the original input is jointly weighted by the spatial and temporal attention channels and used as the module's output to achieve the goal of learning the spatiotemporal relationships of objects.

[0111] In attention mechanisms that enable learning spatiotemporal relationships, spatiotemporal relationship information can be obtained by combining spatial and temporal information to extract and aggregate dimensional features separately. Specifically, this involves processing the input H-dimensional features... First, encoding is performed using a one-to-one dimensional feature encoding operation. This operation utilizes two pooling kernels of size H; (H×1) and (W×1) are used to perform average pooling encoding on each channel, in the horizontal and vertical directions respectively. Encoding is performed in the vertical direction, and the output of the C-th channel in the H dimension is as follows:

[0112]

[0113] The same applies to horizontal encoding:

[0114]

[0115] By extracting spatial feature information, the generated channels are concatenated, and the feature Y of the video frame sequence at time t is extracted from the features of each channel. t It can be represented as:

[0116]

[0117] Where σ represents the normalization operation, F represents the convolution operation, and F 3×3 F 1×1 , where represents the kernel size in the convolution operation, and C / r represents the scaling ratio that controls the change in the number of channels.

[0118] The outputs of the three transformations are aggregated along each of the three dimensions to generate a set of spatiotemporally aware feature maps. These three transformations capture long-range temporal dependencies and spatial relationships in the feature space, enabling the network to more accurately capture the spatiotemporal relationships between objects. An attention mechanism is designed using the spatial relationships and temporal dependencies between objects to highlight regions prone to anomalies in video anomaly detection. Two channels are concatenated, i.e. and After connecting the components and performing convolution operations, the output is obtained:

[0119] Y hw =δ(F 1×1 [Y h ,Y w (11)

[0120] Where δ represents the nonlinear activation function, thus obtaining Y hw Divide into Y h ∈R C / r×H and Y w ∈R C / r×W Two tensors with the same dimension. Let Y... h and Y w The final tensor g is obtained by passing it through the Sigmoid activation function. h and g w .

[0121] For the tensor of the attention component of the time channel, for Y t After processing with a nonlinear activation function, we get:

[0122] g t =relu(Y t (X)) (12)

[0123] Tensor g t This represents the weights of regional features that are sensitive to time changes. Finally, the output of the spatiotemporal attention module Y is as follows:

[0124]

[0125] Since the input and output have the same resolution, the spatiotemporal attention mechanism module can be inserted into the U-net network to help it make better predictions of future frames without changing the input and output structure.

[0126] Objective function: Combining all these constraints on appearance, motion, and adversarial training into the objective function, we arrive at the following objective functions that support prototype learning representing the normal behavior of the target, feature reconstruction of augmented encoding, and prediction of future frames.

[0127] First, the overall loss function ψ is composed of the feature reconstruction term ψ under the attention mechanism. cd And frame prediction term ψ pre Composition. These two items are balanced by weights λ1 and λ2 as follows:

[0128] ψ=λ1ψ pre +λ2ψ cd (14)

[0129] Frame prediction loss is represented as the sum of the real frames y in the input video.t and future frames The Euclidean distance between the predicted frames output by the prediction network:

[0130]

[0131] The feature reconstruction loss under the attention mechanism is represented by the fact that the learned prototypes possess information diversity and feature representativeness, i.e., high quality and diversity, and low redundancy. It has two terms ψ. c and ψ d For these two properties respectively, it can be written as:

[0132] ψ cd =μ1ψ c +μ2ψ d (16)

[0133] Where μ1 and μ2 are weights. Feature compactness loss ψ c This is used to reconstruct normal encodings using compact prototypes. It queries patterns that are closer to the most similar patterns in the pool of normal behavior, enhances the representativeness of each pattern, and minimizes the L2 norm between the query and the corresponding pattern. It measures the mean Euclidean distance between the input encoding vector and its most relevant prototype:

[0134]

[0135] Where k is the query The index of the closest pattern in the normal behavior pattern pool is represented as:

[0136]

[0137] Further promote diversity among prototype projects. Diversity term ψ d Represented as

[0138]

[0139] Where p and p′ represent different prototypes, and α is the expected distance of patterns in the normal behavior pattern pool. Using the two loss functions ψ mentioned above... c and ψ d The future frame prediction model can focus on the spatiotemporal relationships between objects in order to better predict normal future frames.

[0140] Anomaly score: This assumes that normal events can be predicted well. Therefore, anomaly prediction can be performed using the difference between the predicted frame and its true value. Peak signal-to-noise ratio (PSNR) is a method for image quality assessment.

[0141]

[0142] in, It is the maximum value representing the color of a point in the image.

[0143] After calculating the PSNR for each frame of each test video, the PSNR of all frames in the test video is normalized to the range [0,1], and the regular score for each frame is calculated using the following equation:

[0144]

[0145] Among them, Score all (t) is the regularity score, corresponding to the normality of each frame in the video, and it serves as an indicator of the confidence level of the detection results. A threshold can be set to distinguish between normal and abnormal frames. When the corresponding value is below the threshold, it indicates that an abnormal event has occurred; when the corresponding value is above the threshold, it indicates that no abnormal event has occurred.

[0146] In practice, the designed model was trained and tested on three well-known video anomaly detection datasets: UCSDPed2, CUHKAvenue, and ShanghaiTech. In all datasets, the training set was used for model learning and determining the network architecture, while the test set was used solely to evaluate the performance of the trained model. Figure 5 This section displays some examples of positive and anomalous samples. The first row represents normal samples from Ped2, Avenue, and ShanghaiTech data, respectively. The second row represents the relevant anomalous samples, with the anomalous areas indicated by boxes.

[0147] The UCSDPed2 subset includes 16 training videos and 12 test videos. The test videos contain 12 types of anomalous events at a resolution of 240×360, all related to vehicles such as bicycles and cars. In the Avenue dataset, 14 events (i.e., walking in the wrong direction, running across the sidewalk, throwing paper, pushing a bicycle, etc.) are considered anomalous behaviors. It's also worth noting that the cameras in this dataset suffer from camera shake, whereas other datasets use fixed cameras, and the size of people may vary depending on the camera's position and angle. ShanghaiTech is one of the most challenging datasets for video anomaly detection to date. It consists of 13 scenes and various anomaly types (i.e., fighting, running, skating, and cycling), containing 130 anomalous events with different lighting conditions and camera angles.

[0148] To train the network, the input frame size was adjusted to a resolution of 256×256, and the pixel intensity was normalized to the range [-1, 1]. During AE pre-training, the model was trained using the Adam optimizer with an initial learning rate of 0.0001, a batch size of 4, and a maximum epoch of 1000. For future frame prediction tasks, the initial learning rate was set to 0.0004, and a cosine annealing strategy was used to gradually decay the learning rate. The encoded feature map resolution was 256×256×128, and the training epochs on Ped2, Avenue, and Shanghai Tech were set to 60, 60, and 10, respectively. t=4 was set to predict the input of the 5th frame. During training: λ1 and λ2 were both set to 1, μ1 and μ2 were set to 0.99 and 0.01, respectively, and finally λ s Set to 1. In the experiment, the model was trained using the PyTorch framework on an Nvidia GTX1080Ti GPU.

[0149] The method proposed in this invention comprises three stages. In the first stage, video blocks are provided as input to the prototype network of the meta-learning module and their features are encoded. In the second stage, the meta-learning module stores and updates the encoded features. In the third stage, a learnable spatiotemporal attention module is used to gain a more comprehensive understanding of the video's spatial and temporal features, learn the feature representation of new events, perform feature matching, and calculate a set of anomaly scores to predict anomalous events.

[0150] The effectiveness of the method proposed in this invention was verified through simulation experiments.

[0151] Using common evaluation metrics, the Receiver Operating Characteristic (ROC) is calculated by gradually changing the threshold of the anomaly score. The Area Under the Receiver Operating Characteristic Curve (AUC) is then accumulated as a scalar for performance evaluation. The relationship between the True Positive Rate (TPR) and False Positive Rate (FPR) is plotted by changing the anomaly score threshold, resulting in the ROC curve. The AUC value is primarily used to evaluate the robustness and effectiveness of the model; a higher AUC value indicates better video anomaly detection performance.

[0152] To more objectively illustrate the performance of the anomaly detection method in this invention, Table 1 lists comparisons between methods based on different handcrafted features (MAC), (GAS), and (Unmasking), and state-of-the-art deep learning-based methods (MPGT), (STemGAN), (zxVAD), (MGME), and (STMC). It also compares the methods with those based on video prediction (Frame-Pred), (MemAE), (MNAD), and (MPN); and those based on spatiotemporal or motion information (MPN), (Frame-Pred), (MPGT), (zxVAD), (STMC), and (MGME). The AUC of the different methods is listed in Table 1.

[0153] Table 1 compares the AUC results of different methods. Bold text indicates the best results, and underlined numbers indicate the second-best results.

[0154]

[0155]

[0156] As shown in Table 1, compared with these excellent deep feature-based methods, our method achieves superior performance on the UCSDPed2, CUHKAvenue, and ShanghaiTech datasets, with detection accuracies of 97.63%, 98.41%, and 75.43%, respectively. The model was compared with many deep learning-based video anomaly detection methods, sorted chronologically by publication date. The comparison results are shown in Table 1, with the performance of the compared methods obtained from the original papers or by reproducing the results on our devices. From Table 1, the following conclusions can be drawn: (1) The proposed method is effective on three commonly used datasets and achieves a state-of-the-art AUC of 75.43% on ShanghaiTech. (2) The proposed model can significantly improve the anomaly detection performance of the base model. On Ped2, Avenue, and ShanghaiTech, our method achieves AUC performance improvements of 2.23%, 3.31%, and 2.63% compared to the Frame-Pred baseline results, respectively. (3) MGME achieved the best detection performance on the UCSD Ped2 dataset, while MPN achieved the best performance on the Avenue dataset. Compared to MGME, MGME employs multi-layer embedding in memory. Feature information is automatically filtered during decoding, and prototype information is embedded in memory to enhance features, compensating for the information required for detection and resulting in better detection performance. Compared to DPU, the Dynamic Prototype Unit (DPU) can encode normal dynamics into prototypes in real time without additional memory costs. This improves the quality of the obtained video frame features and the accuracy of subsequent detection. Although Frame-Pred uses the same U-Net network for future frame prediction, the Frame-Pred method extracts appearance and optical flow motion information, lacking temporal features.

[0157] The learnable spatiotemporal attention mechanism enables neural networks to better acquire complete temporal and spatial information of video sequences, facilitating the extraction of better video frame features and thus achieving higher accuracy. This advantage is particularly pronounced on the ShanghaiTech dataset. Due to the greater diversity of scenes and anomalous event types in the ShanghaiTech dataset, it is more significant compared to the other two datasets, and the learnable spatiotemporal attention mechanism is more conducive to feature extraction from large datasets. Compared to other datasets and models, our model demonstrates better detection performance on this dataset.

[0158] The method of this invention designs a deep neural network that introduces a meta-learning module to learn and update the features of the input video, generating an encoding with feature prototypes to help the future frame prediction network make better predictions. At the same time, the attention module that can learn spatiotemporal relationships can help the future frame prediction network refine the coarse prediction by extracting features in spatial and temporal dimensions, which is more conducive to extracting cognitive information. Therefore, its performance on the three datasets is slightly better.

[0159] Qualitative results: In Figure 6 The method of this invention is further visualized for predicting normal and abnormal events. For example... Figure 6 The image shows images containing normal and abnormal events, along with their heatmaps. The heatmaps are derived from the differences between the generated predicted frames and the target frames. It is clear from the heatmaps that the model's reconstructed predicted frames with abnormal samples exhibit significant accuracy errors regarding abnormal objects or behaviors. Conversely, the heatmaps also show that normal events are reconstructed very well with almost no obvious errors. This indicates that the method can adequately reconstruct normal video frames but not abnormal ones. Therefore, a large reconstruction error occurs when abnormal events are detected.

[0160] Qualitative Analysis: This section introduces meta-learning into the anomaly detection performance of the future frame prediction network to provide a qualitative analysis. An example of anomaly detection within the meta-learning future frame prediction network framework is visualized, such as... Figure 7 , Figure 8 and Figure 9 The anomaly curves for UCSD Ped2, CHUK Avenue, and ShanghaiTech are shown. The regularity score curve displays the anomaly scores of all frames in the video sequentially, providing a more intuitive view of the performance of the proposed method. In each subplot, the regularity score represents the probability of normality, and the shaded areas in the video frames represent anomalies in the ground truth.

[0161] As shown in the figure, for normal frames, the anomaly scoring curve remains at a relatively high and stable value. However, for anomalous frames, when abnormal actions occur, the anomaly scoring curve drops significantly, producing a strong downward peak, and the prediction results are often quite distorted. Figure 7 It can be seen that the future frame prediction network incorporating meta-learning can correctly detect anomalies even in crowded scenes. For example... Figure 8 , Figure 9 As shown, when an anomaly occurs, the anomaly score curve will drop significantly and fluctuate violently. When the abnormal target is more obvious, the anomaly score curve will drop to its lowest point.

[0162] Ablation experiments: To analyze the roles and effects of different components in the network, ablation experiments were conducted on three datasets, and the anomaly detection performance based on AUC was evaluated, as shown in Table 2. Table 2 shows the ablation experiments conducted on the UCSD Ped2, CUHK Avenue, and ShanghaiTech datasets. The model effect was measured by AUC, with bold text indicating the best performance.

[0163]

[0164] As shown in the table, before adding the meta-learning module, the prediction results of adding the learnable spatiotemporal relationship attention module significantly exceeded those of the original future frame prediction network. Taking Avenue as an example, the AUC increased from 84.35% to 85.97%, an improvement of 1.62%. This is because the learnable spatiotemporal relationship attention module has a good constraint on fast-moving objects such as running and cycling, and can better learn the spatiotemporal relationships of objects, demonstrating the necessity of adding the learnable spatiotemporal relationship attention module as additional information to improve anomaly detection performance. After adding the meta-learning module, the AUC of Ped2, Avenue, and ShanghaiTech increased by 1.48% (from 96.15% to 97.63%), 2.44% (from 85.97% to 88.41%), and 1.91% (from 73.52% to 75.43%) compared to before adding the module, respectively. This is partly because the meta-learning module converts high-dimensional features into low-dimensional dynamic features for storage. On the other hand, the continuous updating of contextual information by the meta-learning module helps the neural network generalize to abnormal behavior events in videos, reducing the model's prediction error. This fully verifies the effectiveness of the future frame prediction network with meta-learning in video anomaly detection, demonstrating better detection performance.

[0165] like Figure 10-12 As shown, the larger the area under the curve, the better the detection effect. Figure 10-12 The figure also demonstrates the comparative effect of the improved performance model. It is clearly observed from the figure that the area under the curve corresponding to the proposed method is significantly larger than that of the original prediction method in all three datasets. This proves the effectiveness of the future frame prediction network incorporating meta-learning.

[0166] The specific implementation schemes described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific implementation schemes of the present invention and are not intended to limit the scope of the present invention. Any equivalent changes and modifications made by those skilled in the art without departing from the concept and principles of the present invention should fall within the scope of protection of the present invention.

Claims

1. A future frame anomaly detection method based on meta-learning and spatiotemporal relationships, characterized in that, Includes the following steps: Step 1: Based on an autoencoder, extract a continuous video frame... As input, all these frames are stacked on a channel and used to predict future frames. ; Step 2: Feed the input frame into the meta-learning module through the selected AE code. The meta-learning module is trained to learn and compress the normal dynamics of real-time sequence information as multiple normal prototypes and enrich the input AE code with normal dynamic information. Step 3: The output encoding is run through the remaining AE layers, processing normal behavior for use in subsequent frame prediction. ; Step 4: To predict the frame Closer To minimize the distance between them in terms of intensity and gradient, and to maintain temporal consistency between adjacent frames, an attention mechanism with learnable spatiotemporal relationships is introduced to constrain real frames. and predicted frames ; Step 5: The difference between the predicted future frame and its normal frame determines whether the event is normal or abnormal; Future frame prediction: By modifying U-Net for future frame prediction, an attention module is introduced into the U-Net network to learn the spatiotemporal relationships of the input image, resulting in better prediction performance for the U-Net-based prediction network. Output resolution is preserved for every two convolutional layers; therefore, cropping and resizing operations are no longer required when adding shortcuts. The kernel size of all convolutional, deconvolutional, and max-pooling layers is set to 3. In the U-Net prediction network, input and output have the same resolution. The attention mechanism sets the input image size to a grayscale image of size (256, 256), then performs pairwise 3*3 convolutions to obtain an image of dimension (64, 256, 256). After three downsampling operations, the image dimension becomes (512, 32, 32). After the downsampling is completed, the attention mechanism module repeats the upsampling three times until the image is downsampled to (64, 256, 256), thus obtaining input and output with the same resolution. Following the prediction of future frames, intensity and gradient difference are used to make the prediction closer to the truth; intensity constraints guarantee the similarity of all pixels in the RGB space; Minimize the predicted frame in the intensity space Rather than reality L2 distance between them: (6) Gradient constraints are used to sharpen the generated image: (7) in Indicates the spatial index of a video frame; In attention mechanisms that enable learning spatiotemporal relationships, spatiotemporal relationship information can be obtained by combining spatial and temporal information to extract and aggregate dimensional features separately; for input... Dimensional features First, encoding is performed through a one-to-one one-dimensional feature encoding operation, which utilizes two spatial units of size 1. Pooled kernel; and Average pooling encoding is performed on each channel, both horizontally and vertically, with encoding done in the vertical direction. Dimension 1 The channel output is as follows: (8) The same applies to horizontal encoding: (9) By extracting spatial feature information, the generated channels are concatenated, and features are extracted from each channel. Features of a video frame sequence at a given time Represented as: (10) in, This indicates a normalization operation. This represents the convolution operation. , , represents the kernel size in the convolution operation. This indicates the reduction ratio of the control channel number change; The outputs of the three transformations are aggregated along each of the three dimensions to generate a set of spatiotemporally aware feature maps; these three transformations capture long-range temporal dependencies and spatial relationships in the feature space, enabling the network to more accurately obtain the spatiotemporal relationships between objects; an attention mechanism is designed using the spatial relationships and temporal dependencies between objects to highlight regions prone to anomalies in video anomaly detection; the two channels are then connected, i.e. and After connecting the components and performing convolution operations, the output is obtained: (11) in, Let represent the nonlinear activation function, from which we obtain Divided into and Two tensors with the same dimension; and The final tensors are obtained by passing them through the Sigmoid activation function. and ; For the tensor of the attention part of the time channel, for After processing with a nonlinear activation function, we get: (12) tensor The weights of regional features that are sensitive to time changes are represented; the output of the spatiotemporal attention module Y is as follows: (13)。 2. The future frame anomaly detection method based on meta-learning and spatiotemporal relationships according to claim 1, characterized in that, In the meta-learning module, the set of normal behavior features is defined as: (1) in, Representation Meta-Learning Module The maximum number of normal behavioral characteristics included; Indicates the first Types of labels for normal behavior learned by individuals Indicates the first The value of the label for normal learning behavior of each element; for the support set Internal sample points, using encoding formula To generate a prototype representation for each category: (2) Among them, the coding formula It employs any information extraction method, using AE encoding. The meta-learning module is trained to learn the normal dynamics of real-time sequence information as multiple prototypes, and enriches the input AE encoding with normal dynamic information; the t-th input encoding image is extracted from the AE. Consider it as c-dimensional Vectors, assigning normal weights to each pixel location ,express for: (3) This is called the normal behavior pool, or prototype pool; the input is a continuous video frame representing the set. .

3. The future frame anomaly detection method based on meta-learning and spatiotemporal relationships according to claim 2, characterized in that, During the query process, the input encoding vector comes from the AE encoding map. It is used as a query to retrieve relevant items in the prototype pool to reconstruct the normal code. Encode the query set as : (4) in, This is represented as the nth encoded vector. and the m-th prototype item The correlation scores between them; It calculates the probability that the current sample belongs to each category. The calculation method used is softmax. (5)。 4. The future frame anomaly detection method based on meta-learning and spatiotemporal relationships according to claim 3, characterized in that, Objective function: All constraints on appearance, motion, and adversarial training are incorporated into the objective function, resulting in the following objective functions that support prototype learning representing the normal behavior of the target, feature reconstruction of augmented encoding, and prediction of future frames; First, the overall loss function Feature reconstruction terms under attention mechanism and frame prediction terms Composition, through weights The equilibrium is: (14) Frame prediction loss is represented by the true frames of the input video. and future frames The Euclidean distance between the predicted frames output by the prediction network: (15) The feature reconstruction loss under the attention mechanism is represented by the fact that the learned ordinary prototypes have information diversity and feature representativeness, that is, high quality and diversity, and low redundancy, with two terms. and For these two properties respectively, it can be written as: (16) in, and Weights; Feature compactness loss It is used to reconstruct normal encodings using compact prototypes, queries that are closer to the most similar patterns in the normal behavior pool, enhance the representativeness of each pattern, and minimize the L2 norm between the query and the corresponding pattern. That is, it measures the average Euclidean distance between the input encoding vector and its most relevant prototype. (17) in, It is a query The index of the closest pattern in the normal behavior pattern pool is represented as: (18) Promote diversity among prototype projects, diversity items Represented as: (19) in, They represent different prototypes. It is the expected distance of patterns in the normal behavior pattern pool.

5. The future frame anomaly detection method based on meta-learning and spatiotemporal relationships according to claim 4, characterized in that, Anomaly prediction is performed by utilizing the difference between the predicted frame and its true value. (20) in, It represents the maximum value of the color of an image point; PSNR is the peak signal-to-noise ratio. After calculating the PSNR for each frame of each test video, the PSNR of all frames in the test video is normalized to the range [0,1], and the regular score for each frame is calculated using the following equation: (21) in, It is a rule-based score, corresponding to the normality of each frame in the video, which serves as an indicator of the confidence level of the detection results; a threshold is set to distinguish between normal frames and abnormal frames. When the corresponding value is lower than the threshold, it indicates that an abnormal event has occurred; when the corresponding value is higher than the threshold, it indicates that no abnormal event has occurred.

Images