Video anomaly detection method based on optical flow reconstruction and variation prediction of enhanced memory

Abstract

The invention discloses a video anomaly detection method based on optical flow reconstruction and variation prediction of enhanced memory. The method comprises the following steps: acquiring a video to be detected; preprocessing a to-be-detected video to obtain a foreground object space-time cube and optical flow features; and inputting the space-time cube and the optical flow features into a pre-trained video anomaly detection model based on enhanced memory optical flow reconstruction and variational prediction to obtain an anomaly detection result. The video anomaly detection model comprises an optical flow reconstruction module which reconstructs optical flow features based on a convolution attention mechanism, a multi-level memory enhancement module and a feature enhancement bridge to obtain a reconstruction result; the condition prediction module is used for predicting a future video frame based on the reconstruction result and the space-time cube to obtain a predicted video frame; and the anomaly detection module is used for comparing the predicted video frame with the real video frame to obtain an anomaly detection result. According to the method, the accuracy of the prediction result is improved, and seamless integration of the dynamic motion mode characteristics in the prediction process is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of video detection, and in particular relates to a video anomaly detection method based on optical flow reconstruction and variational prediction based on enhanced memory. Background Art

[0002] In recent years, with the rapid development of science and technology, video surveillance systems have been widely used in many fields, such as public security, intelligent transportation, industrial monitoring, etc. However, traditional video surveillance systems mainly rely on manual processing of surveillance videos, but the data collected by cameras is usually massive, the workload required is very large, and it is difficult to quickly and effectively discover and handle abnormal situations. Therefore, the research on video anomaly detection technology has important practical significance. The anomaly mentioned in video anomaly detection (VAD) refers to the motion pattern that does not conform to normal events, such as running fast on a road where pedestrians walk slowly, riding a bicycle on a crosswalk, throwing objects, and walking against the flow. Most of these behaviors are some emergencies, or group events such as fights, traffic accidents, etc. The task of video anomaly detection is to detect video clips that do not conform to normal events. However, due to the relatively vague definition of abnormal events and the low frequency of occurrence, anomaly detection in videos is relatively difficult. At the same time, there are interferences from various factors such as video clarity, whether objects are blocked, lighting, and motion posture, which further increases the difficulty of anomaly detection, resulting in video anomaly detection is still a challenging task.

[0003] Due to the many difficulties in video anomaly detection, the current supervised learning method is difficult to detect effectively. Most method models use unsupervised methods. Specifically, by using videos that only contain normal behaviors to train the model, the model uses the distribution of normal events to detect videos containing abnormal behaviors, while the abnormal behaviors in the test video cannot be represented and are judged as abnormal. At this stage, most common models use a single reconstruction model or prediction model, with an autoencoder as the carrier, adhering to the assumption that deep network models can reconstruct or predict normal frames well. Reconstruction-based methods reconstruct the current frame by learning part of the past frames, and prediction-based methods predict the next frame by learning part of the past frames.

[0004] With the rapid development of hardware equipment and data processing technology, people have discovered the powerful advantages of deep learning methods in processing complex computer vision tasks. Deep learning models have been widely used in many tasks, such as target detection, pose estimation, target tracking, etc. In the task of video anomaly detection, people pay attention to the fact that the occurrence of anomalies is often accompanied by changes in motion state, and the optical flow feature is to find the "flowing part" in the video and describe the intensity of the action through the difference between the two frames before and after the video. Some existing models learn video frame features and optical flow features at the same time for better detection. The optical flow feature focuses on the event flow of the video, and the abnormal behavior in the video also has spatial characteristics. Therefore, some methods use a two-stream network for anomaly detection, including spatial stream and temporal stream. The former learns the spatial structure of the input frame, while the latter uses the optical flow between adjacent frames. By combining spatial and temporal information, the two-stream network can capture abnormal behaviors in the video more comprehensively. However, this method often only focuses on the spatial structure of the input frame and the motion information of the video frame, ignoring the correlation between appearance information and motion information. In addition, in order to eliminate some unnecessary noise interference in the video frame and focus on the foreground area, people introduced the attention mechanism into video anomaly detection. It can significantly improve the system's attention to key information. The attention mechanism allows the model to automatically and selectively focus on key frames or frame segments in the video, such as the area or time period where abnormal events occur, so as to more accurately locate and identify abnormal events. However, the self-attention mechanism captures the relationship between any two elements in the input sequence and is good at processing global dependencies. It has poor processing capabilities for local detail features and lacks the capture of detail information. Due to the loss of detailed information during the encoder downsampling process, the output image is usually blurred. In order to expand the prediction error gap between normal samples and abnormal samples and avoid unnecessary loss of spatial information during the encoding process, people usually use jump connections40 to improve it so that the model produces smaller prediction errors for normal data in the VAD task. However, people often ignore the semantic gap between encoder and decoder features caused by jump connections. Summary of the invention

[0005] In order to solve the above problems existing in the prior art, the present invention provides a video anomaly detection method based on optical flow reconstruction and variational prediction with enhanced memory. The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0006] The present invention provides a video anomaly detection method based on optical flow reconstruction and variational prediction with enhanced memory, the method comprising:

[0007] Get the video to be detected;

[0008] Preprocessing the video to be detected to obtain space-time cube and optical flow features corresponding to the foreground object in the video to be detected;

[0009] Inputting the spatiotemporal cube and optical flow features into a pre-trained video anomaly detection model based on enhanced memory optical flow reconstruction and variational prediction to obtain anomaly detection results;

[0010] Among them, the video anomaly detection model based on enhanced memory optical flow reconstruction and variational prediction includes: an optical flow reconstruction module, which is used to reconstruct optical flow features based on convolutional attention mechanism, multi-level memory enhancement module and feature enhancement bridging to obtain corresponding reconstruction results; a conditional prediction module, which is used to predict future video frames based on the reconstruction results and the corresponding space-time cube to obtain predicted video frames; an anomaly detection module, which is used to compare the predicted video frames with the real video frames to obtain anomaly detection results.

[0011] Beneficial effects of the present invention:

[0012] In the scheme provided by the present invention, when the optical flow reconstruction module is proposed to process the optical flow features, by fusing the convolutional attention mechanism and the multi-level memory enhancement module, the model's ability to capture subtle changes and long-term dependencies can be effectively improved, and efficient reuse of video features is achieved. The model's perception of contextual information is enhanced, thereby accurately extracting key features and retaining detail information, and significantly improving the model's sensitivity to abnormal video detection. At the same time, through the feature enhancement bridge through the jump connection method, the features extracted from the encoder are passed to the decoder after nonlinear processing, compensating for the semantic gap between the encoder and the decoder. This design not only accelerates the flow of information, but also enables the decoder to make full use of the rich memory content in the decoder when generating prediction results, that is, retaining key spatial information, and reducing the semantic differences in feature fusion through nonlinear transformation, thereby further improving the accuracy of the prediction results, detail fidelity and model performance. On this basis, the output of the optical flow reconstruction module is integrated into the conditional prediction module as conditional information, realizing the seamless integration of dynamic motion pattern features in the prediction process. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 A schematic diagram of the steps of a video anomaly detection method based on optical flow reconstruction and variational prediction based on enhanced memory provided by an embodiment of the present invention;

[0014] Figure 2 A schematic diagram of the structure of a video anomaly detection model based on optical flow reconstruction and variational prediction based on enhanced memory provided by an embodiment of the present invention;

[0015] Figure 3 A schematic diagram of the training steps of a video anomaly detection model based on optical flow reconstruction and variational prediction based on enhanced memory provided by an embodiment of the present invention;

[0016] Figure 4 A schematic diagram of the structure of a convolutional attention mechanism provided by an embodiment of the present invention;

[0017] Figure 5 A schematic diagram of the structure of a memory enhancement module provided by an embodiment of the present invention;

[0018] Figure 6 A schematic diagram of a structure of a feature enhanced bridging provided by an embodiment of the present invention;

[0019] Figure 7a-7d An example diagram of a normal and abnormal video frame provided by an embodiment of the present invention;

[0020] Figure 8a-8b An ROC curve diagram of a video anomaly detection method based on optical flow reconstruction and variational prediction based on enhanced memory provided by an embodiment of the present invention and an existing method on a data set;

[0021] Fig. 9 A visualization result diagram of the AUC values ​​corresponding to different numbers of convolutional attention mechanisms for a video anomaly detection method based on optical flow reconstruction and variational prediction based on enhanced memory provided by an embodiment of the present invention;

[0022] Fig.10 A visualization result diagram of the AUC values ​​corresponding to different numbers of convolutional attention mechanisms for a video anomaly detection method based on optical flow reconstruction and variational prediction based on enhanced memory provided by an embodiment of the present invention;

[0023] Fig.11 An example of anomaly score curve and a schematic diagram of representative video frames obtained by a video anomaly detection method based on optical flow reconstruction and variational prediction based on enhanced memory provided by an embodiment of the present invention in the UCSD Ped dataset;

[0024] Fig.12 An example of anomaly score curve and a schematic diagram of representative video frames obtained in the CUHK Avenue dataset by a video anomaly detection method based on optical flow reconstruction and variational prediction based on enhanced memory provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The present invention is further described in detail below with reference to specific embodiments, but the embodiments of the present invention are not limited thereto.

[0026] The embodiment of the present invention provides a video anomaly detection method based on optical flow reconstruction and variational prediction with enhanced memory, such as Figure 1 As shown, this may include:

[0027] S1, obtain the video to be detected;

[0028] It is understandable that the video requiring abnormality detection may be used as a video to be detected for subsequent processing.

[0029] S2, preprocessing the video to be detected to obtain the spatiotemporal cube and optical flow features corresponding to the foreground object in the video to be detected, which may include:

[0030] S21, using a pre-trained regional convolutional neural network module RCNN to process the video to be detected, and obtain a foreground object corresponding to each video frame in the video to be detected;

[0031] S22, for each video frame, using the RoI bounding box to identify the foreground object in the video frame, and using the optical flow estimation network FlowNet2.0 to extract the optical flow of the foreground object in the video frame; in the optical flow, using the RoI bounding box to identify the foreground object;

[0032] S23, for each foreground object, construct a space-time cube and optical flow features corresponding to the foreground object; the space-time cube includes information identifying the foreground object in the RoI bounding box at the same position in consecutive t-frame video frames; the optical flow features include the optical flow of the foreground object in the RoI bounding box at the same position in consecutive t-frame video frames.

[0033] It can be understood that in the data preprocessing stage, the embodiment of the present invention uses a pre-trained regional convolutional neural network module (RCNN, Region-based Convolutional Neural Networks) to extract the foreground objects in the data set based on the video frames in the data set. Each foreground object is identified by an accurate RoI bounding box. At the same time, the optical flow estimation network FlowNet2.0 is used to extract the optical flow features of all video frames, and these optical flow features also use RoI bounding boxes to define foreground objects. In order to unify the processing, the spatial dimensions of all foreground objects are adjusted to have the same size value.

[0034] For each extracted foreground object, a space-time cube STC corresponding to the foreground object is constructed. The space-time cube not only contains the information of the current video frame, but also incorporates the content of the bounding box of the same position of the j consecutive frames before the current video frame, where t=j+1, j can be equal to 4. Exemplarily, if only one object frame is extracted in the i-th frame, the coordinate information of the object frame is used to crop the same area from the (i-4)th frame, the (i-3)th frame, the (i-2)th frame and the (i-1)th frame, thereby generating a space-time cube STC with a shape of [5,3,H,W]; where 5 means that the STC contains 5 parts and 3 channels, H represents the height, and W represents the width, and both are 32 pixels. It can be understood that the same processing method can be used for optical flow information. This processing method not only considers the spatial information of the foreground object, but also incorporates the changes in the time dimension, which helps to more comprehensively analyze the dynamic characteristics of the foreground object. By introducing the space-time cube, the time dependency is enhanced.

[0035] S3, input the spatiotemporal cube and optical flow features into the pre-trained video anomaly detection model based on enhanced memory optical flow reconstruction and variational prediction to obtain the anomaly detection result.

[0036] For a diagram of the training steps for the video anomaly detection model, see Figure 2 , please refer to the structural diagram of the video anomaly detection model Figure 3 In order to facilitate the understanding of the video anomaly detection model proposed in the embodiment of the present invention, the following will be combined with Figure 2 and Figure 3 The structure and training process of the video anomaly detection model are introduced in detail.

[0037] Video anomaly detection model based on optical flow reconstruction and variational prediction based on enhanced memory, such as Figure 3 As shown, it may include: an optical flow reconstruction module, which is used to reconstruct the optical flow features based on the convolutional attention mechanism, the multi-level memory enhancement module and the feature enhancement bridge to obtain the corresponding reconstruction results; a conditional prediction module, which is used to predict the future video frames based on the reconstruction results and the corresponding space-time cubes to obtain the predicted video frames; an anomaly detection module, which is used to compare the predicted video frames with the real video frames to obtain the anomaly detection results.

[0038] Specifically, the optical flow reconstruction module is a TSTMA (Transformer-Enhanced Spatio-Temporal Memory Autoencoder) model, such as Figure 3 The upper middle part of TSTMA can include:

[0039] The input convolutional layer, the first reconstruction end encoder, the second reconstruction end encoder, the third reconstruction end encoder, the Transformer-based convolutional attention mechanism, the first level memory enhancement module, the first reconstruction end decoder, the second level memory enhancement module, the second reconstruction end decoder, the third level memory enhancement module and the third reconstruction end decoder are connected in sequence;

[0040] The second reconstruction end encoder is connected to the second reconstruction end decoder via a feature enhancement bridge, and the third reconstruction end encoder is connected to the first reconstruction end decoder via a feature enhancement bridge.

[0041] It can be understood that adding a transformer-based convolutional attention mechanism after the encoder enables it to capture the spatiotemporal dependencies between video frames and enhance the model's ability to represent key features and abnormal events in the video, thereby improving the accuracy of anomaly detection; at the same time, it helps to retain more detailed information, enabling the model to better capture subtle changes in the video, and also forms a feature enhancement bridge through convolutional layers and residual connections, thereby reducing the feature semantic differences between the encoder and decoder caused by simple jump connections, enabling the model to learn more effectively.

[0042] Specifically, the conditional prediction module is a CVAE (Conditional Variational Autoencoder) model, such as Figure 3 The Conditional-VAE in the lower middle part can include:

[0043] Encoder E θ , Encoder and decoder D δ .

[0044] Understandably, from Figure 3 As can be seen from the figure, the two encoders and decoders adopt a three-layer structure respectively.

[0045] The training process of the video anomaly detection model based on enhanced memory optical flow reconstruction and variational prediction, such as Figure 3 As shown, this may include:

[0046] S01, using an optical flow reconstruction module to reconstruct the optical flow features obtained after preprocessing the video data set to obtain a corresponding reconstruction result.

[0047] For S01, this may include:

[0048] S011, using the input convolutional layer to expand the initial number of channels of the optical flow feature Flow 1:t to obtain the expanded feature.

[0049] Specifically, in the optical flow reconstruction module, a four-layer structure model is designed for the optical flow features obtained in the data preprocessing stage as follows:

[0050] For the p-th video frame,

[0051] Among them, H, W, and C represent the height, width, and number of channels corresponding to the optical flow features of the input model, respectively. Represents X i To reconstruct a frame of partial optical flow features.

[0052] The optical flow feature first passes through the input convolution layer, which is composed of two stacked convolution blocks, and is used to expand the initial number of channels of the video frame features from 2 to 32 to obtain the expanded features.

[0053] In the specific implementation, the convolution block can contain a three-layer structure: a 2D convolution layer for feature extraction, a BatchNorm2D layer for normalizing feature distribution to accelerate training, and a ReLU activation layer for introducing nonlinearity.

[0054] It is understandable that in the convolution layer, downsampling can be achieved by setting a larger step size. A larger step size can reduce the size of the output feature map, thereby achieving the effect of downsampling. This downsampling method not only reduces the size of the feature map, but also helps to extract more abstract features, reduce the amount of calculation, and speed up the training of the model.

[0055] S012, the expanded features are processed in sequence by a first reconstruction end encoder, a second reconstruction end encoder, and a third reconstruction end encoder to obtain encoded features.

[0056] Specifically, the expanded features are sequentially passed through the first reconstruction end encoder, the second reconstruction end encoder and the third reconstruction end encoder. Each encoder includes: a downsampling layer, a convolution layer, a normalization layer and an activation layer to increase the receptive field of the feature map and gradually increase the number of channels to 256, thereby generating coded features.

[0057] Among them, h, w, and c are the height, width, and number of channels of the downsampled feature map, respectively.

[0058] S013, use the convolutional attention mechanism to process the encoded features and obtain key features.

[0059] After obtaining the encoded features, the convolutional attention mechanism is introduced to extract and utilize key information without changing the number of channels of the features. The input and output of the convolutional attention mechanism have the same dimensional size, which can be expressed as

[0060] The convolutional attention mechanism can be a convolutional attention mechanism based on Transformer. Transformer is a neural network architecture based on the self-attention mechanism, which initially achieved remarkable success in the field of natural language processing. The embodiment of the present invention combines the self-attention mechanism of Transformer with the convolutional neural network CNN to form a powerful feature extractor that can capture richer contextual information when processing image or video data. This combination enables the model to focus on important features both locally and globally, thereby improving the ability to capture details and key features.

[0061] For a schematic diagram of the convolutional attention mechanism, see Figure 4 .from Figure 4 As can be seen in , the convolutional attention mechanism consists of N layers, each of which contains two core sub-layers. The first core sub-layer focuses on the multi-head convolutional attention mechanism, aiming to capture the complex dependencies between features; it is followed by a 2-D convolutional feed-forward network layer to further refine and transform feature information. A residual connection is used between the two core sub-layers, and group normalization is added to improve the stability and convergence speed of the model. To promote residual connections, the channel dimension of all sub-layers in the convolutional attention mechanism is equal to the number of channels between the encoder and the decoder d = 256.

[0062] Understandably, given the nature of optical flow features —— Focusing mainly on the relative motion relationship between pixels or feature points, the embodiment of the present invention introduces position encoding in the convolutional attention mechanism. Position encoding provides spatial context information for the model, enabling the model to more accurately locate the position of feature points or pixels in the image. This is particularly important when processing complex video data, helping the model capture more sophisticated motion patterns and enhance its expressiveness.

[0063] The embodiment of the present invention introduces position encoding in the convolutional attention mechanism. However, unlike the transformer that directly adds position encoding to the input embedding, the embodiment of the present invention chooses to add position encoding to the query map Q and key map K in each attention layer. This processing method can make the position encoding have the same dimension as the feature map, so the addition operation can be performed directly without additional conversion or adjustment.

[0064] Specifically, the convolutional attention mechanism is calculated through feature maps, mapping the query mapping Q and a set of key-value mappings KV into one output. This mapping process is implemented through a convolutional neural network, which enables the model to capture the relationship between features in a local area. After a set of {Q, K, V} is given, the query mapping Q and the key mapping K are concatenated, and an attention map is generated through an attention network M. The attention network can be a CNN containing 2D convolution and LeakyReLU activation function. The attention network generates all attention maps after t iterations. Subsequently, the concatenation operation can be performed on the dim=3 dimension, and the SoftMax operation can be used along the third dimension to obtain the normalized attention weights, and the attention map and the value mapping V are summed to obtain the key features of the final output.

[0065] In order to make full use of the information in different feature spaces, the embodiment of the present invention may adopt a multi-head attention mechanism, allowing the model to learn and integrate information in different representation subspaces, thereby further enhancing the expressiveness and robustness of the model.

[0066] S014, using the first level memory enhancement module to store the key features.

[0067] S015, the stored key features are processed in sequence by the first reconstruction end decoder, the second level memory enhancement module, the second reconstruction end decoder, the third level memory enhancement module and the third reconstruction end decoder to obtain the corresponding reconstruction result Flow_rec1:t.

[0068] Specifically, after obtaining the key features, the upsampling layers in the three reconstruction end decoders gradually reduce the number of channels from 256 to 32 to restore the resolution of the feature map. Finally, the output convolution layer restores the number of channels from 32 to the initial 2 channels, thereby obtaining the final reconstruction result Flow_rec1:t of the optical flow feature.

[0069] The downsampling layer and upsampling layer are implemented by 2D convolution and 2D deconvolution, respectively, to control the size and number of channels of the feature map.

[0070] The efficient and highly targeted optical flow reconstruction module constructed in the embodiment of the present invention can effectively extract and utilize key information to achieve high-quality reconstruction of optical flow features.

[0071] The function of the memory enhancement module is similar to the memory function in the human brain, which can store and recall past information for use in current or future decisions. Using a multi-level memory enhancement module to further process the features generated by the convolutional attention mechanism can effectively help the model capture and remember the details in the video, thereby more accurately identifying abnormal events. The feature map of the encoder is passed to the decoder through a connection so that the high-level and low-level feature information can be fused. There may be a large semantic difference between the low-level features of the encoder and the high-level features of the decoder when they are merged. This difference may lead to incompatibility in the direct feature fusion process, thereby interfering with the learning process and affecting the model performance. In order to solve this problem, an embodiment of the present invention proposes a method for feature enhancement bridging. The method introduces a convolutional layer with a residual connection in the connection, and the features propagated by the encoder are compensated by further processing of the features through nonlinear transformation in the decoder stage. It aims to reduce the semantic difference between the encoder and decoder features, thereby improving the performance of the model.

[0072] A schematic diagram of the structure of a memory enhancement module provided by an embodiment of the present invention is shown in FIG. Figure 5 In the specific design of the memory enhancement module, each memory enhancement module is designed as a matrix containing N real-valued vectors of fixed dimension C. Understandably, is a symbol for representing a set, such as Figure 5 These matrices explicitly record the prototype normal patterns during training. The memory is defined as content-addressable memory, and the attention weights are determined by calculating the similarity between the memory items and the query, and then the weighted sum of similar memory vectors is used to represent the features of the input. For a schematic diagram of the structure of the feature enhancement bridge, see Figure 6 In the implementation of feature enhancement bridging, some convolutional layers are added along the skip connection to mitigate the difference between encoder-decoder features. Additional nonlinear transformations are performed on the features propagated from the encoder stage to cooperate with further processing in the decoder stage. Instead of using conventional convolutional layers, the embodiment of the present invention introduces residual connections, which helps to simplify the learning process. In a specific implementation, a 3×3 filter can be used for convolution operations, and a 1×1 filter can be used in the residual connection.

[0073] By designing a memory enhancement module and introducing a feature enhancement bridging method, the performance of the video anomaly detection task can be further improved, enabling the model to more accurately capture and remember important information in the video, thereby more reliably identifying abnormal events.

[0074] S02, the optical flow reconstruction module obtains an optical flow reconstruction loss function based on the reconstruction result, and trains the optical flow reconstruction module using the optical flow reconstruction loss function to obtain a trained optical flow reconstruction module.

[0075] In order to train the TSTMA model, it is necessary to optimize the model by combining the optical flow reconstruction loss function consisting of the reconstruction loss and the entropy loss. The expression of the optical flow reconstruction loss function is as follows:

[0076] L TSTMA =λ recon L recon +λ ent L ent ;

[0077] Among them, L recon represents the reconstruction loss, L ent represents entropy loss, λ recon represents the weight corresponding to the reconstruction loss, λ ent Represents the weight corresponding to the entropy loss.

[0078] The reconstruction loss ensures that the model can accurately reconstruct the input data, while the entropy loss helps the model make more certain matching decisions on the slots of each memory module. First, define the reconstruction loss. Given the input data y and the reconstruction result The reconstruction loss can be expressed as the l2 distance between the input data and the reconstruction result.

[0079] The expression of reconstruction loss is as follows:

[0080]

[0081] in, represents the l2 distance, y represents the optical flow feature before reconstruction, Represents the corresponding reconstruction result.

[0082] Define entropy loss. For each memory module in the multi-level memory enhancement module, calculate the entropy of the slot matching probability and add up the entropies of all memory modules. Let O be the number of memory modules, p i,j is the probability of matching the j-th slot in the i-th memory bank. The entropy loss is expressed as follows:

[0083]

[0084] in, Represents the sum of the matching probabilities of all slots in the i-th memory module. Finally, the reconstruction loss and entropy loss are combined to obtain the optical flow reconstruction loss function L TSTMA In the expression of the optical flow reconstruction loss function, λ recon and λ entIt can be a balancing factor used to adjust the weights of reconstruction loss and entropy loss in the optical flow reconstruction loss function. By minimizing the optical flow reconstruction loss function, the TSTMA model can be trained to simultaneously optimize the reconstruction quality and memory matching decisions. It can be understood that the optical flow reconstruction module generates a reconstructed optical flow field based on the input data. This reconstructed optical flow field is an estimate of the true optical flow field. The loss function is a function that evaluates the difference between the model output and the true target. Once the loss function is defined, the optical flow reconstruction module can be trained by minimizing this loss function. For example, an optimization algorithm (such as gradient descent) is used to adjust the parameters of the model to reduce the difference between the model output and the true target.

[0085] After multiple iterations of training, when the optical flow reconstruction loss function reaches a smaller value (or converges to a certain stable state), the optical flow reconstruction module can be considered to have been trained. At this point, the model can more accurately estimate the optical flow field in the input image sequence.

[0086] S03, using the conditional prediction module to predict future video frames based on the reconstruction result and the corresponding space-time cube to obtain predicted video frames.

[0087] For S03, this may include:

[0088] Using encoder E θ Process the reconstruction result Flow_rec1:t to obtain the encoding result E θ (Flow_rec 1：t );

[0089] Based on the encoding result E θ (Flow_rec 1：t ) to obtain the prior distribution p(zFlow_rec 1：t );

[0090] Using Encoders Reconstruct the result Flow_rec1: t and the corresponding space-time cube Frame 1：t Perform the merge process to obtain the output feature E θ (Frame 1：t ,Flow_rec 1：t );

[0091] Based on the output feature E θ (Frame 1：t ,Flow_rec 1：t ) to obtain the posterior distribution q(zFrame 1：t ,Flow_rec 1：t );

[0092] For the posterior distribution q(zFrame1：t ,Flow_rec 1：t ) to obtain the sampling result SamplingZ, and compare the sampling result SamplingZ with the prior distribution p(zFlow_rec 1：t ) is input to the decoder D δ , get the predicted video frame Frame_pre t+1 .

[0093] It can be understood that the CVAE model, as the core of the conditional prediction module, integrates the characteristics of dynamic motion patterns into the prediction process by incorporating the output of the optical flow reconstruction module as conditional information. Figure 2 As shown in the following part, the encoder of the CVAE model first captures and compresses the key information of the input video frame layer by layer. The decoding stage combines these extracted features with the optical flow condition information to finely restore the details of the video frame and achieve accurate classification at the pixel level. In this way, the conditional prediction module can output the predicted video frame.

[0094] When constructing the conditional prediction module, two encoders E θ , and a decoder D δ The reconstruction result generated by the optical flow reconstruction module is used as the conditional input of CVAE. Specifically, the reconstruction result Flow_rec1:t is input to the encoder E θ Get the encoding result E θ (Flow_rec 1：t ), and obtain the prior distribution p(z|Flow_rec 1：t ) as the prior conditional distribution of the conditional prediction module. At the same time, the space-time cube is merged with the corresponding reconstruction result Flow_rec1:t and input to the encoder Get the output feature E θ (Frame 1：t ,Flow_rec 1：t ), forming the posterior distribution q(z|Frame 1：t ,Flow_rec 1：t ). In the posterior distribution q(z|Frame 1：t ,Flow_rec 1：t ) to obtain SamplingZ, and then combine SamplingZ with the prior distribution p(z|Flow_rec 1：t ) is input to the decoder D δ , get the predicted video frame Frame_pre t+1 .

[0095] To further improve prediction performance, special skip connections are introduced between the encoder and decoder. These links not only directly transfer feature information, but also form feature enhancement bridges through convolutional layers and residual connections, aiming to reduce the semantic difference of features between the encoder and decoder, thereby helping the model generate prediction frames more accurately.

[0096] As the core of the conditional prediction module, CVAE successfully integrates the features of dynamic motion patterns into the prediction process by incorporating the output of the optical flow reconstruction module as conditional information, and explores the correlation between appearance information and motion information. In the prediction stage, the encoder of CVAE captures and compresses the key information of the input video frame layer by layer. In the decoding stage, these extracted features are combined with the optical flow conditional information to accurately restore the details of the video frame and achieve accurate classification at the pixel level. In order to further optimize the prediction results, the present invention introduces a jump connection mechanism. These connections directly cross the boundary between the encoder and the decoder, and directly pass the key feature maps in the encoding process to the decoder, effectively avoiding the unnecessary loss of spatial information in the encoding process, and ensuring the high fidelity of the predicted image in details and edge information. In order to avoid the large semantic difference between the low-level features of the encoder and the high-level features of the decoder when merging, resulting in incompatibility in the direct feature fusion process, thereby interfering with the learning process and affecting the model performance. The present invention introduces a convolutional layer with a residual connection in the connection, and further processes the features propagated by the encoder through nonlinear transformation in the decoder stage to compensate. This is intended to reduce the semantic difference between the features of the encoder and the decoder, while enabling the decoder to make full use of the rich memory information in the encoder, thereby improving the performance of the model. Through this model, the optical flow reconstruction module expands the distinction between normal and abnormal, while the conditional prediction module is able to output predicted video frames.

[0097] S04, the conditional prediction module obtains a prediction loss function based on the predicted video frame, and trains the conditional prediction module using the prediction loss function to obtain a trained conditional prediction module.

[0098] Specifically, the expression of the prediction loss function is as follows:

[0099] L total =αL CVAE +βL gd ;

[0100] Among them, L CVAE represents the CVAE loss, L gd represents the gradient loss, and α and β represent the balancing factors.

[0101] When training the CVAE model, a prediction loss function is defined that contains two parts of the loss function, one of which is the CVAE loss, which measures the difference between the predicted frame generated by the model and the real frame.

[0102] The expression of CVAE loss is as follows:

[0103]

[0104] KL stands for KL divergence, which is used to measure the difference between two distributions (posterior distribution and prior distribution). By minimizing this difference, the model can learn a representation of the latent variable z that is more consistent with the data prior, q(z|x 1：t ,y 1：t ) represents the posterior distribution, which can be specifically expressed as given the previous t frames of video x 1：t And the optical flow result y 1：t The posterior distribution is used to generate the latent variable z, p(z|y 1：t ) represents the prior distribution, which means that only given the optical flow result y 1：t The distribution of the latent variable z under the condition of x t+1 Indicates that the next frame that the model needs to predict is the real future frame. Indicates the next frame generated by the decoder, which is a predicted future frame. Represents the l2 distance, which is used to measure the pixel-level difference between the real frame and the predicted frame. The goal is to make the two as close as possible, thereby reducing the difference between the generated frame and the real frame.

[0105] The other part is the gradient loss, which measures the difference in gradient between the predicted frame and the real frame. It is specifically defined as the sum of the squares of the gradient differences at each spatial pixel position in the image.

[0106] The expression of gradient loss is as follows:

[0107]

[0108] Among them, i, j represents the spatial pixel position in the image, X i,j represents the real frame at the spatial pixel position i,j, represents the predicted frame at spatial pixel position i,j.

[0109] In the prediction loss function, α and β represent balance factors, which are used to adjust the weights of the two parts of the loss in the prediction loss function. By minimizing the prediction loss function, the CVAE model can be trained to simultaneously optimize the accuracy of the predicted frame and the consistency of the gradient.

[0110] S05, the anomaly detection module compares the predicted video frame and the real video frame based on the Euclidean distance to obtain an anomaly detection result.

[0111] Specifically, the core of anomaly detection is to quantify the deviation between the model output and the expected result. This deviation can be measured by calculating the error or the difference in probability distribution, and further converted into an anomaly score. The anomaly score directly reflects the degree of deviation of the data point from the normal behavior pattern, and is an important basis for judging whether it is an anomaly.

[0112] During the detection process, L2 distance (i.e. Euclidean distance) is used as a metric to evaluate the difference between the output of the optical flow reconstruction module and the conditional prediction module and the true value. L2 distance can intuitively represent the straight-line distance between two points and is an effective way to measure the accuracy of model prediction or reconstruction. The greater the difference, the farther the model's prediction or reconstruction result is from the true value. Therefore, the higher the anomaly score, the greater the possibility that the data point is judged as an outlier.

[0113] Specifically, for the optical flow reconstruction module, by calculating the L2 distance between the reconstructed output and the original input, an anomaly score reflecting the reconstruction quality is obtained. If the L2 distance is large, it means that the reconstruction effect is not good and the input data is likely to be abnormal. For the conditional prediction module, the L2 distance is used to calculate the difference between the predicted result and the true value, thereby obtaining an anomaly score reflecting the prediction accuracy. A large L2 distance indicates that there is a significant difference between the predicted value and the true value, which may mean that the current data point or state is abnormal.

[0114] In order to more comprehensively evaluate the degree of abnormality, the reconstruction error and the prediction error of the predicted video frame Combined to form the final video anomaly detection score. This method combines the anomaly scores of the two modules in a weighted sum manner, taking into account both reconstruction information and prediction information, thereby improving the accuracy and reliability of anomaly detection. The weights are determined based on the mean and standard deviation of the reconstruction error and prediction error of all training samples to ensure that the two modules have an appropriate contribution to the final anomaly detection score.

[0115] The final anomaly detection scores are as follows:

[0116]

[0117] Among them, μ r represents the mean value of the reconstruction error of the training samples, σ r represents the standard deviation of the reconstruction error of the training samples, μ p represents the mean of the prediction error of the training samples, σ p represents the standard deviation of the prediction error of the training sample, ω r and ω p Represent the weights of the two scores respectively.

[0118] The optical flow reconstruction module proposed in the embodiment of the present invention can effectively improve the model's ability to capture subtle changes and long-term dependencies by fusing the convolutional attention mechanism and the multi-level memory enhancement module, realize the efficient reuse of video features, enhance the model's perception of contextual information, thereby accurately extracting key features and retaining detail information, and significantly improve the model's sensitivity to abnormal video detection. At the same time, through the feature enhancement bridge through the jump connection method, the features extracted from the encoder are passed to the decoder after nonlinear processing, compensating for the semantic gap between the encoder and the decoder. This design not only accelerates the flow of information, but also enables the decoder to make full use of the rich memory content in the decoder when generating prediction results, that is, retaining key spatial information, and reducing the semantic differences in feature fusion through nonlinear transformation, thereby further improving the accuracy of the prediction results, detail fidelity and model performance. On this basis, the output of the optical flow reconstruction module is integrated into the conditional prediction module as conditional information, realizing the seamless integration of dynamic motion pattern features in the prediction process.

[0119] In order to demonstrate the performance of the video anomaly detection method based on optical flow reconstruction and variational prediction based on enhanced memory proposed in the embodiment of the present invention, a specific simulation experiment is carried out below to demonstrate the beneficial effects brought by the embodiment of the present invention.

[0120] The embodiment of the present invention selected UCSD Ped2 dataset and CUHK Avenue dataset for extensive experiments to evaluate the performance of the video anomaly detection method EM-OFRP based on enhanced memory optical flow reconstruction and variational prediction proposed in the embodiment of the present invention. The EM-OFRP proposed in the embodiment of the present invention was compared with other existing VAD methods to verify the effectiveness of EM-OFRP.

[0121] The present invention uses two benchmark datasets, as shown in Table 1, including UCSD Ped2 and CUHKAvenue datasets. For specific examples of normal and abnormal video frames, see Figure 7a-7d .

[0122] As iconic resources in the field of VAD (video anomaly detection), the UCSD Ped1 and Ped2 datasets are widely collected from university campus environments, each focusing on two different outdoor scenes on campus. These datasets, with their practical and basic scene settings, reflect the important application value of VAD technology in the field of public safety. Specifically, the Ped1 dataset has a resolution of 158×238 and is not considered in the current experiment due to its relatively low quality. In contrast, the Ped2 dataset has a higher resolution of 240×360, contains 16 training videos (covering 2550 frames) and 12 test videos (covering 2010 frames), and focuses on distinguishing normal walking pedestrians from abnormal behaviors or objects such as cycling, skateboarding, and driving.

[0123] The CUHK Avenue dataset also originates from the university campus, but it differs from the UCSD series in abnormal events. The 47 abnormal events in the Avenue dataset cover appearance abnormalities (such as bags left on the grass) and motion abnormalities (such as throwing objects, abnormal walking direction). The dataset consists of 16 training videos (a total of 15,328 frames) and 21 test videos (a total of 15,324 frames). Each frame is presented in a high-definition resolution of 360×640 pixels, and additional detailed spatial annotations at the frame and pixel levels are provided, providing researchers with a richer and more detailed analysis dimension.

[0124] Table 1 Dataset table

[0125]

[0126] Figure 7a An example of a normal frame corresponding to the UCSD Ped2 dataset can be, Figure 7b An example of an abnormal frame corresponding to the UCSD Ped2 dataset can be: Figure 7c An example of a normal frame corresponding to the CUHK Avenue dataset can be, Figure 7d An example of anomaly frames corresponding to the CUHK Avenue dataset can be shown.

[0127] First, the corresponding foreground target frames are extracted from all video frames of the dataset through RCNN, and the foreground frames of the current frame and the previous 4 frames are combined to construct a space-time cube. All subsequent processing is based on this space-time cube, and the size of the space-time cube is adjusted to 32×32 pixels. During the test, there may be multiple objects in each frame, and the maximum anomaly score of all objects is selected as the final anomaly score.

[0128] The optical flow reconstruction module includes a four-layer structure, which may include: in_conv / out_conv, down1 / up1, down2 / up2, down3 / up3. Each basic convolution block may be composed of conv2d, Batchnorm2d and ReLU. The feature maps corresponding to each layer are (32, 32, 32), (16, 16, 64), (8, 8, 128) and (4, 4, 256). The conditional prediction module consists of two encoders and a decoder, which also has a four-layer structure. The feature maps corresponding to each layer are (32, 32, 64), (16, 16, 128), (8, 8, 128), (4, 4, 128). The feature enhancement bridge module is used between the encoder and the decoder to replace the ordinary jump connection. Therefore, the present invention applies some convolution operations to the feature maps propagated from the encoder stage to the decoder stage. It is assumed that the strength of the semantic gap between the encoder and decoder feature maps may decrease when moving deeper into the model. Therefore, the number of convolution blocks used on the feature enhancement bridge is gradually reduced. In particular, 2 feature enhancement bridges and 1 convolutional block are used in the optical flow reconstruction module, while 3, 2, and 1 convolutional blocks are used in the conditional prediction module. In addition, in order to consider the number of feature maps in the encoder-decoder, 64, 128 filters and 64, 128, and 128 filters are used in the feature enhancement bridge blocks, respectively.

[0129] The framework proposed in the embodiment of the present invention is trained using the Pytorch framework and the NVIDIA RTX 4080Ti GPU. Adam is used as the optimizer, and the initial learning rate is set to 2e-4, which decays by 0.8 after every 20 epochs. The capacity of the memory module of Ped2 and Avenue is set to 2000, and the batch size and epoch number are set to (128, 80). The hyperparameter λ of Ped2 and Avenue recon , ent , α, β are set to 1, 2e -4 , 1, 1, the error fusion weight ω of Ped2 and Avenue r and ω p Set them to (0.2, 0.8) and (0.7, 0.3) respectively.

[0130] In video anomaly detection, there are usually frame-level and pixel-level evaluation criteria. Frame-level evaluation means that if a frame contains at least one abnormal pixel, the frame is detected normally. Pixel-level evaluation means that if at least 40% of the pixels in a frame are abnormal, the frame is considered to be detected normally. This paper uses frame-level evaluation criteria and uses the area under the receiver operator characteristic (ROC) curve (AUC) to evaluate model performance. The EM-OFRP proposed in this paper distinguishes whether a frame is normal or abnormal based on the anomaly score. Normal frames usually have lower anomaly scores, while abnormal frames have higher anomaly scores. The performance comparison results of the two datasets are shown in Table 2.

[0131] Table 2 Comparison results

[0132]

[0133] For the ROC curve of the optical flow reconstruction and variational prediction video anomaly detection method provided by the embodiment of the present invention and the existing method on the data set, please refer to Figure 8a-8b . Figure 8a The following is a ROC curve on the UCSD Ped2 dataset. It can be seen from the figure that EM-OFRP performs excellently, with an average frame-level AUC of 98.5%, significantly better than other baseline methods, including MemAE, AMMC-Net, SSAGAN, and HF2-VAD. This significant advantage is mainly due to the Transformer-based convolutional attention module in the EM-OFRP framework, which effectively enhances the framework's ability to learn optical flow motion features. By using optical flow reconstruction as a condition, EM-OFRP can assist the prediction module in learning features more accurately. The anomaly score is calculated by combining the prediction error and the reconstruction error to obtain an accurate prediction result. Therefore, EM-OFRP significantly enlarges the difference between normal frames and abnormal frames, thereby greatly improving the detection performance of the model.

[0134] Figure 8bThe figure is a ROC curve on the CUHK Avenue dataset. It can be seen from the figure that the framework proposed in the embodiment of the present invention is comprehensively compared with the most advanced VAD method. The framework proposed in the embodiment of the present invention performs well in the test, with a detection accuracy of up to 91.2%, surpassing most existing methods. Compared with the two-stream-based methods (such as VEC and AMMC-Net), the present invention has made significant progress, which fully verifies the efficiency of the strategy of exploring the correlation between appearance and motion in the unsupervised VAD. In particular, compared with the memory-based methods (such as MemAE, HF2-VAD, Le Wang and ZhiyuanNing), the method provided by the embodiment of the present invention also shows significant advantages, which highlights that multi-level jump connections to avoid the loss of spatial information in the encoding process are very critical in abnormal behavior detection. At the same time, the features transmitted by the encoder are nonlinearly transformed and compensated, narrowing the semantic gap between the encoding and decoding features, ensuring the compatibility of the two in the fusion process, thereby significantly improving the overall performance of the model. By accurately capturing abnormal motion features, efficient abnormal behavior detection is achieved.

[0135] The following will systematically carry out a number of experimental analyses to evaluate the effects of each sub-module of the model and comprehensively evaluate the performance of the model and the impact of key parameter settings. Specifically, it includes: analyzing the contribution of each module in the overall model to the final performance through ablation experiments to provide empirical evidence for model optimization; verifying the impact of the number of layers of the Transformer-based convolutional attention mechanism and the number of multi-head attention heads on the model performance.

[0136] Ablation experiment analysis

[0137] The impact of each component in EM-OFRP on the overall performance was studied in depth, and the detailed results are summarized in Table 3. The motion information of the video frame is effectively learned through the carefully designed optical flow reconstruction module. In this module, the memory enhancement module, convolutional attention mechanism and feature enhancement bridge FEBM are gradually integrated. The addition of these components enables the model to directly transfer shallow features to deep layers, realizing feature reuse and enhancement. Such a design helps to retain more detailed information, enabling the model to more effectively utilize historical information, accurately capture and memorize key information in the video, and thus improve the accuracy of abnormal event recognition.

[0138] Table 3 Ablation results

[0139]

[0140] First, when only the memory enhancement module is used, the model achieves an AUC of 94.9% on the UCSD Ped2 dataset. The convolutional attention mechanism is further integrated, which improves the performance by 1.2%. On this basis, the performance is further improved to 96.6% by embedding the feature enhancement bridge FEBM in the connection of multiple memory modules.

[0141] After learning the optical flow motion information through the optical flow reconstruction module, the reconstructed optical flow is input as a condition into the CVAE prediction module to build a hybrid model that integrates reconstruction and prediction. This design significantly improves the performance of anomaly detection by widening the gap between abnormal and normal behaviors, making abnormal events easier to detect. In addition, ablation studies were also conducted on three key components in the hybrid model, and ultimately achieved an excellent performance of 98.5% on the UCSD Ped2 dataset and an accuracy of 91.2% on the Avenue dataset.

[0142] These significant performance improvements fully demonstrate the indispensability of the various components proposed in the embodiments of the present invention in the EM-OFRP model. They significantly improve the model's discrimination ability and verify the effectiveness and feasibility of the prediction and reconstruction hybrid model strategy.

[0143] Convolutional Attention Mechanism Layer Experiment

[0144] In the task of video anomaly detection, we deeply explored the convolutional attention mechanism based on Transformer and created a multi-layered attention mechanism. It can finely extract and combine key features in the video at multiple levels. Unlike the transformer, the embodiment of the present invention combines the convolution operation's keen capture of local spatiotemporal dependencies with the self-attention mechanism's precise grasp of long-distance dependencies, enabling the model to fully understand and analyze video data.

[0145] When the number of layers of the convolutional attention mechanism is too small, the model can quickly capture the local spatiotemporal features in the video, but it is limited by the expressive power of the shallow structure and it is difficult to fully abstract and integrate these features to form a deep understanding of the global and long-term dependencies. This will cause the model to easily miss key information or misjudge complex scenes during anomaly detection, thereby reducing the accuracy of detection. On the contrary, if the number of layers of the convolutional attention mechanism is too large, although the model can gradually abstract higher-level feature representations through a deeper hierarchical structure, this process is also accompanied by a significant increase in computational complexity and an increase in the risk of overfitting. Too many layers may cause the model to learn noise in the data or details specific to the training set, rather than features with strong generalization capabilities, and thus perform poorly on new or changing data. In addition, excessive depth may also cause gradient vanishing or gradient exploding problems, further affecting the training effect and convergence speed of the model.

[0146] As the number of layers increases, the model not only gradually abstracts low-level features into higher-level representations, but also effectively filters irrelevant noise in the process and highlights the key information in the video. This hierarchical abstraction process greatly improves the efficiency and accuracy of the model in anomaly detection tasks. At the same time, higher-level representations also provide richer and more accurate information for subsequent anomaly detection tasks.

[0147] When selecting the number of layers in the multi-layer structure, the computational efficiency and model performance are fully balanced. Through comparative experiments on two datasets, the detailed experimental results are shown in Table 4 with different number of layers of the convolutional attention mechanism.

[0148] Table 4 Convolutional attention mechanism different layer settings

[0149]

[0150] As can be seen from Table 4, the Trans-L-5 model achieves an AUC of 98.513% on the Ped2 dataset and an AUC of 91.217% on the Avenue dataset. This result shows that the appropriate number of layers can ensure that the model is not too simple to fully capture complex patterns, nor too complex to cause overfitting and increase computational costs.

[0151] As the number of layers increases, the model performance gradually improves, but increasing the number of layers without limit does not bring about sustained performance gains. On the contrary, when the number of layers is too many, the model complexity increases sharply, which can easily lead to overfitting and increased computational costs, which in turn reduces the accuracy of anomaly detection (98.323%, 91.196%). Therefore, when selecting the number of model layers, it is necessary to weigh the characteristics of the specific task and data set to ensure the efficiency and accuracy of the model in practical applications. For a visualization of the AUC values ​​corresponding to different layers of the convolutional attention mechanism in a video anomaly detection method based on optical flow reconstruction and variational prediction based on enhanced memory provided by an embodiment of the present invention, please refer to Fig. 9 .from Fig. 9 It can be seen intuitively that Trans-L-5 has achieved a balance between performance and computational complexity.

[0152] Experiment on setting multiple heads of convolutional attention mechanism

[0153] In the convolutional attention mechanism, the number of heads H of the multi-head convolutional attention is a key hyperparameter that has a significant impact on the accuracy and performance of the model. The multi-head mechanism allows the model to focus on different subspaces of the input data in parallel, thereby capturing richer and more diverse feature relationships. When the number of heads H of the convolutional attention module is too small, the model is limited by a small number of parallel attention heads and may not be able to fully explore and utilize all the information of the input data, resulting in insufficient representation of complex features, which in turn affects the model's anomaly detection accuracy. In this case, the model is prone to ignore some critical but subtle abnormal patterns, reducing its generalization ability and robustness.

[0154] However, as the number of heads H increases, although the model can capture more diverse feature information, it also brings about a sharp increase in computational complexity. Too many heads not only increase the number of model parameters and the amount of computation, but may also cause the model to learn too many details and noise during training, thus causing overfitting problems. In the case of overfitting, although the model performs well on the training set, its performance will drop significantly on new or unseen data, and it will lose its generalization ability.

[0155] In order to find the optimal number of heads H for the convolutional attention module, a series of comparative experiments were conducted on two benchmark datasets, Ped2 and Avenue. The detailed experimental results are shown in Table 5.

[0156] Table 5 Head number setting table of convolutional attention module

[0157]

[0158] The comparative experiments on the Ped2 dataset show that the performance of Trans-H-4 is 0.644 higher than that of Trans-H-1, but the continuous increase in the number of heads will lead to a decrease in performance. The performance of Trans-H-6 is 0.883 lower than that of Trans-H-4. The comparative experiments on the Avenue dataset show that the performance of Trans-H-4 is 0.553 higher than that of Trans-H-1, but the continuous increase in the number of heads will lead to a decrease in performance. The performance of Trans-H-6 is 0.482 lower than that of Trans-H-4.

[0159] The experimental results show that when H is set to 4, the model achieves the best detection effect on both data sets, with significant performance improvement compared to single-head attention (H=1). However, with the further increase in the number of heads (such as H=6), the model performance has declined, verifying the problems of overfitting and increased computational costs caused by too many heads. Therefore, based on the experimental results and performance analysis, Trans-H-4 was finally determined as the optimal multi-head convolutional attention module configuration. This decision has achieved good results in balancing model accuracy and computational efficiency, and provides an efficient and accurate solution for video anomaly detection tasks. For a visualization of the AUC values ​​corresponding to different numbers of heads in the convolutional attention mechanism in a video anomaly detection method based on optical flow reconstruction and variational prediction based on enhanced memory provided by an embodiment of the present invention, please refer to Fig.10 ,from Fig.10 It can be seen intuitively that Trans-H-4 achieves the best detection effect on the two datasets.

[0160] An example of anomaly score curve and a representative video frame diagram obtained by a video anomaly detection method based on optical flow reconstruction and variational prediction based on enhanced memory provided by an embodiment of the present invention on the UCSD Ped dataset can be found in Fig.11 ; An example of anomaly score curves and representative video frames obtained in the CUHK Avenue dataset, see Fig.12 .

[0161] from Fig.11 and Fig.12 It can be seen that EM-OFRP performs well on the UCSD Ped2 and Avenue video anomaly detection datasets. These charts and representative video frames not only reveal the model's deep understanding of video content, but also intuitively reflect its ability to efficiently identify and distinguish normal and abnormal events.

[0162] First, from Fig.11 As can be seen from the anomaly score curve of the UCSD Ped2 dataset, EM-OFRP demonstrates extremely high sensitivity and accuracy. In normal scenarios, such as pedestrians walking freely, the anomaly score curve of the model remains at a relatively stable and low level, which reflects the model's accurate grasp of daily activity patterns. However, once abnormal objects or behaviors appear, such as vehicles illegally entering the sidewalk or pedestrians suddenly running, the anomaly score curve will respond immediately and rise sharply, quickly capturing these abnormal events. This immediate and significant response proves the effectiveness of the model in detecting sudden anomalies.

[0163] Further observation Fig.12From the Avenue dataset example in Figure 1, we can see that EM-OFRP also performs well. This dataset covers more diverse scenarios and more complex abnormal behaviors, such as sudden running in a crowd and objects being dropped. In these cases, the model's anomaly score curve can also accurately follow the occurrence and development of abnormal events. When the abnormal behavior begins, the curve quickly climbs to a high level and remains at this level until the abnormal event ends. This continued high level shows the model's accurate judgment of the persistence of abnormal events. When the abnormal behavior ends, whether through the removal of the object or the cessation of the behavior, the anomaly score curve will quickly fall back to the normal level, indicating the model's keen perception of the end of the event.

[0164] In addition, the representative video frames corresponding to the anomaly score curves shown in the figure provide intuitive visual verification. These video frames clearly show the contrast between normal and abnormal events, further confirming the accuracy and reliability of EM-OFRP in distinguishing between the two. Whether it is the normal walking and sudden running of pedestrians, or the stillness and abnormal movement of objects, the model can accurately capture and make corresponding judgments.

[0165] The embodiment of the present invention provides a novel video anomaly detection method - a video anomaly detection method based on optical flow reconstruction and variational prediction with enhanced memory (EM-OFRP). EM-OFRP is an unsupervised video anomaly detection (VAD) method that combines reconstruction and prediction modules. Its uniqueness lies in the proposal of an enhanced memory method. The convolutional attention mechanism is used to guide the network to focus on the key features in the input image. By performing weighted operations on the feature map, the network's ability to extract and represent these features is improved, thereby accurately grasping the dynamic changes in the video. At the same time, the key features extracted by the attention module are integrated into the multi-level memory enhancement module. With the help of jump connections with feature enhancement bridges, the ability to capture and long-term memory of important information is enhanced, and the subtle and easily overlooked dynamic changes and long-term dependencies in the video are deeply excavated, further enhancing the ability of the memory module in capturing and memorizing key features of details. These key clues are crucial for accurately identifying abnormal behaviors and promoting the detection accuracy of the reconstruction model.

[0166] The optical flow features are processed in depth to better capture abnormal motion information, thereby achieving better reconstruction results. It is worth emphasizing that the reconstruction module and the prediction module are not simply merged or added, but the results of the reconstruction module are integrated into the prediction module as conditional input through the CVAE structure. In the jump connection of the encoder-decoder, convolution blocks and residual connections are introduced to effectively reduce the semantic differences in feature representation between the encoder and decoder. This design forms motion condition constraints on the prediction module, significantly amplifying the distance between abnormal behavior and normal behavior, making abnormal behavior easier to detect.

[0167] Through experiments on public datasets, it can be seen that the performance of EM-OFRP is better than most existing unsupervised VAD methods that use reconstruction or prediction alone, which fully demonstrates the effectiveness of EM-OFRP.

[0168] It should be noted that in the description of the present invention, it should be understood that the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0169] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A video anomaly detection method based on optical flow reconstruction and variational prediction with enhanced memory, characterized in that: include: Get the video to be detected; Preprocessing the video to be detected to obtain space-time cube and optical flow features corresponding to the foreground object in the video to be detected; Inputting the spatiotemporal cube and optical flow features into a pre-trained video anomaly detection model based on enhanced memory optical flow reconstruction and variational prediction to obtain anomaly detection results; Among them, the video anomaly detection model based on enhanced memory optical flow reconstruction and variational prediction includes: an optical flow reconstruction module, which is used to reconstruct optical flow features based on convolutional attention mechanism, multi-level memory enhancement module and feature enhancement bridging to obtain corresponding reconstruction results; a conditional prediction module, which is used to predict future video frames based on the reconstruction results and the corresponding space-time cube to obtain predicted video frames; an anomaly detection module, which is used to compare the predicted video frames with the real video frames to obtain anomaly detection results.

2. The video anomaly detection method based on optical flow reconstruction and variational prediction based on enhanced memory according to claim 1 is characterized in that: Preprocessing the video to be detected to obtain the space-time cube and optical flow features corresponding to the foreground object in the video to be detected includes: Processing the video to be detected by using a pre-trained regional convolutional neural network module RCNN to obtain a foreground object corresponding to each video frame in the video to be detected; For each video frame, the foreground object in the video frame is identified by using the RoI bounding box, and the optical flow of the foreground object in the video frame is extracted by using the optical flow estimation network FlowNet2.0; in the optical flow, the foreground object is identified by using the RoI bounding box; For each foreground object, a space-time cube and an optical flow feature corresponding to the foreground object are constructed; the space-time cube includes information identifying the foreground object in the RoI bounding box at the same position in consecutive t-frame video frames; the optical flow feature includes the optical flow of the foreground object in the RoI bounding box at the same position in consecutive t-frame video frames.

3. The video anomaly detection method based on optical flow reconstruction and variational prediction based on enhanced memory according to claim 1 is characterized in that: The optical flow reconstruction module is a TSTMA model, including: The input convolutional layer, the first reconstruction end encoder, the second reconstruction end encoder, the third reconstruction end encoder, the Transformer-based convolutional attention mechanism, the first level memory enhancement module, the first reconstruction end decoder, the second level memory enhancement module, the second reconstruction end decoder, the third level memory enhancement module and the third reconstruction end decoder are connected in sequence; The second reconstruction end encoder is connected to the second reconstruction end decoder via a feature enhancement bridge, and the third reconstruction end encoder is connected to the first reconstruction end decoder via a feature enhancement bridge.

4. The video anomaly detection method based on optical flow reconstruction and variational prediction based on enhanced memory according to claim 3 is characterized in that: The conditional prediction module is a CVAE model, including: Encoder E θ , Encoder and decoder D δ .

5. The video anomaly detection method based on optical flow reconstruction and variational prediction based on enhanced memory according to claim 4 is characterized in that: The training process of the optical flow reconstruction and variational prediction video anomaly detection model based on enhanced memory includes: Reconstructing the optical flow features obtained after preprocessing the video data set using the optical flow reconstruction module to obtain a corresponding reconstruction result; The optical flow reconstruction module obtains an optical flow reconstruction loss function based on the reconstruction result, and trains the optical flow reconstruction module using the optical flow reconstruction loss function to obtain a trained optical flow reconstruction module; Using the conditional prediction module to predict future video frames based on the reconstruction result and the corresponding space-time cube to obtain predicted video frames; The conditional prediction module obtains a prediction loss function based on the predicted video frame, and trains the conditional prediction module using the prediction loss function to obtain a trained conditional prediction module; The anomaly detection module compares the predicted video frame with the real video frame based on the Euclidean distance to obtain an anomaly detection result.

6. The video anomaly detection method based on optical flow reconstruction and variational prediction based on enhanced memory according to claim 5 is characterized in that: The optical flow reconstruction module is used to reconstruct the optical flow features obtained after preprocessing the video data set to obtain the corresponding reconstruction results, including: The input convolution layer is used to expand the initial number of channels of the optical flow feature Flow 1:t to obtain an expanded feature; The expanded features are processed by the first reconstruction end encoder, the second reconstruction end encoder and the third reconstruction end encoder in sequence to obtain encoded features; Using the convolutional attention mechanism to process the encoded features to obtain key features; Using the first level memory enhancement module to store the key features; The stored key features are processed in sequence by the first reconstruction end decoder, the second level memory enhancement module, the second reconstruction end decoder, the third level memory enhancement module and the third reconstruction end decoder to obtain the corresponding reconstruction result Flow_rec1:t.

7. The video anomaly detection method based on optical flow reconstruction and variational prediction based on enhanced memory according to claim 5, characterized in that: The expression of the optical flow reconstruction loss function is as follows: L TSTMA =λ recon L recon +λ ent L ent ; Among them, L recon represents the reconstruction loss, L ent represents entropy loss, λ recon represents the weight corresponding to the reconstruction loss, λ ent Represents the weight corresponding to the entropy loss.

8. The video anomaly detection method based on optical flow reconstruction and variational prediction based on enhanced memory according to claim 5, characterized in that: The conditional prediction module is used to predict future video frames based on the reconstruction result and the corresponding space-time cube to obtain predicted video frames, including: Using the encoder E θ Process the reconstruction result Flow_rec1:t to obtain the encoding result E θ (Flow_rec 1：t ); Based on the encoding result E θ (Flow_rec 1：t ) to obtain the prior distribution p(z|Flow_rec 1：t ); Using the encoder Reconstruct the result Flow_rec1: t and the corresponding space-time cube Frame 1：t Perform the merge process to obtain the output feature E θ (Frame 1：t ,Flow_rec 1：t ); Based on the output feature E θ (Frame 1：t ,Flow_rec 1：t ) to obtain the posterior distribution q(z|Frame 1：t ,Flow_rec 1：t ); For the posterior distribution q(z|Frame 1：t ,Flow_rec 1：t ) to obtain a sampling result SamplingZ, and compare the sampling result SamplingZ with the prior distribution p(z|Flow_rec 1：t ) is input to the decoder D δ , get the predicted video frame Frame_pre t+1 .

9. The video anomaly detection method based on optical flow reconstruction and variational prediction based on enhanced memory according to claim 5, characterized in that: The expression of the prediction loss function is as follows: L total =αL CVAE +βL gd ; Among them, L CVAE represents the CVAE loss, L gd represents the gradient loss, and α and β represent the balancing factors.

Images