Video anomaly detection method based on high and low frequency attention and double dynamic memory network

Abstract

The invention discloses a video anomaly detection method based on high and low frequency attention and a double dynamic memory network. The method comprises the following steps: acquiring a to-be-detected video; and inputting the to-be-detected video into a pre-trained video anomaly detection model based on the high and low frequency attention and the double dynamic memory network, and outputting an anomaly detection result. According to the invention, a to-be-detected video is processed by using a pre-trained video anomaly detection model, and in a feature learning module FL, different relevance between high-frequency features and low-frequency features in the video is captured through a separated attention mechanism, so that boundary fuzziness between a normal event and an abnormal event is effectively improved; in a feature enhancement module FA, a normal memory unit and an abnormal memory unit are utilized to better distinguish normal and abnormal modes. Meanwhile, dual-storage separation loss is introduced to learn more robust and differentiated features, efficient utilization of normal and abnormal video data is realized, and the video anomaly detection model provided by the invention has better detection precision.

Description

Technical Field

[0001] The present invention belongs to the technical field of video detection, and in particular relates to a video anomaly detection method based on high- and low-frequency attention and dual dynamic memory networks. Background Art

[0002] Anomaly detection in video surveillance has become a hot research area in computer vision due to its wide range of applications, such as public safety, traffic management, and behavior analysis. With the installation and popularization of surveillance cameras in various scenarios, people have an increasing demand for anomaly detection in surveillance videos. Traditional manual monitoring relies on manpower to observe and analyze surveillance videos, which has problems such as low efficiency, fatigue and error, and untimely response. With the development of deep learning, video anomaly detection (VAD) has gradually replaced traditional manual monitoring and become an indispensable and important technology in intelligent video surveillance. The application of VAD has greatly improved the efficiency and accuracy of monitoring, making it possible to discover and identify abnormal events in massive video data in real time. However, VAD is still a very challenging task due to the fuzzy boundaries between normal and abnormal samples in video anomaly detection, large differences within abnormal events, and the lack of annotated abnormal videos.

[0003] Existing solutions usually treat VAD as an unsupervised learning problem, that is, only normal videos are used in the training phase. These unsupervised methods build models by learning patterns in normal training videos. In the testing phase, any behavior that deviates from these normal patterns will be identified as abnormal. Unsupervised VAD alleviates the problems caused by the lack of annotated abnormal videos to a certain extent. However, this approach has two major flaws. On the one hand, it can only detect abnormal events in scenes contained in the training videos, which makes it difficult to build a large-scale VAD dataset to meet the needs of various abnormal event detection in the real world. On the other hand, due to the lack of access to abnormal training data, unsupervised methods are likely to misidentify unseen normal events as abnormalities, resulting in a high false alarm rate.

[0004] In response to the problems of unsupervised VAD, researchers proposed a weakly supervised video anomaly detection (WS-VAD) method to improve the technical difficulties of video anomaly detection. This method introduces abnormal videos in the training stage to improve VAD performance. WS-VAD usually uses video-level annotations to achieve frame-level anomaly detection. Compared with unsupervised VAD, WS-VAD has achieved significant improvements in detection accuracy and adaptability. It is worth mentioning that compared with fully supervised anomaly detection methods, weakly supervised methods reduce the need for large-scale anomaly video data with annotates, reduce the cost and difficulty of data labeling, and only require relatively light video-level annotation work. Since the model has seen some abnormal data during the training process, the weakly supervised method can reduce the situation where unseen normal events are mistakenly identified as abnormalities, thereby reducing the false alarm rate. However, the existing WSVAD method still has the following shortcomings:

[0005] Due to the diversity of behaviors and their multiple interpretations in different contexts, the same behavior may be considered normal or abnormal, and the scene dependencies and dynamic changes make the definition of behavior different in different scenes and times. Abnormal clips may not be different enough from normal clips, which makes the boundary between normal and abnormal events in the video fuzzy and difficult to distinguish between normal and abnormal clips. Video data contains rich visual information, and abnormal events are usually rarer than normal events, which leads to the imbalance problem of training data. When the model learns and generalizes abnormal patterns, it is difficult to make efficient use of normal and abnormal video data due to data imbalance and scarcity as well as complex data characteristics. Summary of the invention

[0006] In order to solve the above problems existing in the prior art, the present invention provides a video anomaly detection method based on high-low frequency attention and dual dynamic memory network. The technical problem to be solved by the present invention is achieved by the following technical solutions:

[0007] The present invention provides a video anomaly detection method based on high-low frequency attention and dual dynamic memory network, the method comprising:

[0008] Get the video to be detected;

[0009] Inputting the video to be detected into a pre-trained video anomaly detection model based on high- and low-frequency attention and dual dynamic memory networks, and outputting anomaly detection results;

[0010] Wherein, the video anomaly detection model includes:

[0011] A pre-trained deep model feature extractor is used to extract features from the video to be detected, and obtain the I3D features of the normal video and the I3D features of the abnormal video in the video to be detected; a feature learning module FL is used to process the I3D features of the normal video and the I3D features of the abnormal video based on a separated attention mechanism, and obtain high-frequency features and low-frequency features corresponding to the normal video and high-frequency features and low-frequency features corresponding to the abnormal video; attention calculation is performed on the high-frequency features and low-frequency features corresponding to the normal video, and feature fusion is performed on the calculated results to obtain the normal comprehensive features of the normal video; the high-frequency features corresponding to the abnormal video are processed by the attention calculation module FL, and ... The high-frequency features and low-frequency features are respectively calculated for attention, and the calculated results are subjected to feature fusion to obtain the abnormal comprehensive features of the abnormal video; the feature enhancement module FA is used to use a dual dynamic memory network to input the normal comprehensive features and the abnormal comprehensive features into the normal memory unit and the abnormal memory unit respectively for storage and learning, and obtain normal enhanced comprehensive features and abnormal enhanced comprehensive features respectively; a preset memory loss function is used to constrain the learning of the normal memory unit and the abnormal memory unit; the normal enhanced comprehensive features and the abnormal enhanced comprehensive features are input into a classifier to obtain an abnormal score, thereby obtaining an abnormal detection result.

[0012] In one embodiment of the present invention, the training process of the video anomaly detection model includes:

[0013] Using a pre-trained deep model feature extractor, feature extraction is performed on a training data set consisting of videos to be detected, to obtain I3D features of normal videos and I3D features of abnormal videos in the training data set;

[0014] The feature learning module FL is used to process the I3D features of the normal video and the I3D features of the abnormal video based on the separation attention mechanism to obtain high-frequency features and low-frequency features corresponding to the normal video and high-frequency features and low-frequency features corresponding to the abnormal video; the high-frequency features and low-frequency features corresponding to the normal video are respectively calculated for attention, and the calculated results are subjected to feature fusion to obtain normal comprehensive features of the normal video; the high-frequency features and low-frequency features corresponding to the abnormal video are respectively calculated for attention, and the calculated results are subjected to feature fusion to obtain abnormal comprehensive features of the abnormal video;

[0015] The normal comprehensive features and the abnormal comprehensive features are input into the normal memory unit and the abnormal memory unit for storage and learning respectively by using the feature enhancement module FA based on the dual dynamic memory network, so as to obtain the normal enhanced comprehensive features and the abnormal enhanced comprehensive features; the learning of the normal memory unit and the abnormal memory unit is constrained by using the preset memory loss function;

[0016] The normal enhanced comprehensive features and the abnormal enhanced comprehensive features are input into a classifier to obtain an abnormality score, thereby obtaining an abnormality detection result.

[0017] Beneficial effects of the present invention:

[0018] In the scheme provided by the present invention, a pre-trained video anomaly detection model based on high-low frequency attention and dual dynamic memory network is used to process the video to be detected, and a pre-trained deep model feature extractor is used to extract features to obtain the I3D features of the normal video and the I3D features of the abnormal video in the video to be detected; in the feature learning module FL for processing the I3D features of the normal video and the I3D features of the abnormal video, a separation attention mechanism for separating high-frequency features and low-frequency features is used to capture the different correlations of the local features corresponding to the high-frequency features and the global features corresponding to the low-frequency features in the video data, thereby enhancing the expression embedding and effectively improving the boundary fuzziness between normal events and abnormal events; in the feature enhancement module FA, normal memory units and abnormal memory units are used to better distinguish normal and abnormal modes. At the same time, a dual storage separation loss is introduced to learn more robust and discriminative features, thereby achieving efficient use of normal and abnormal video data and having better detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A schematic diagram of the steps of a video anomaly detection method based on high-low frequency attention and dual dynamic memory network provided by an embodiment of the present invention;

[0020] Figure 2 A schematic diagram of a framework of a video anomaly detection method based on high- and low-frequency attention and dual dynamic memory networks provided by an embodiment of the present invention;

[0021] Figure 3 A schematic diagram of the training steps of a video anomaly detection model based on high- and low-frequency attention and dual dynamic memory networks provided by an embodiment of the present invention;

[0022] Figure 4 A structural diagram of a feature learning module FL provided in an embodiment of the present invention;

[0023] Figure 5 A visualization diagram of a high- and low-frequency attention mechanism provided by an embodiment of the present invention;

[0024] Figure 6 A structural diagram of a feature enhancement module FA provided in an embodiment of the present invention;

[0025] Figure 7a-7f Examples of normal and abnormal video frames in the UCF-Crime and XD-Violence datasets;

[0026] Figure 8a-8c Qualitative results of anomaly detection performance corresponding to UCF-Crime and XD-Violence datasets. DETAILED DESCRIPTION

[0027] The present invention is further described in detail below with reference to specific embodiments, but the embodiments of the present invention are not limited thereto.

[0028] The embodiment of the present invention provides a video anomaly detection method based on high-low frequency attention and dual dynamic memory network, such as Figure 1 As shown, this may include:

[0029] S1, obtain the video to be detected;

[0030] It can be understood that the video to be detected is used as the video to be detected, and is subsequently input into a pre-trained video anomaly detection model based on high and low frequency attention and dual dynamic memory network, so as to obtain the corresponding anomaly detection result using the video anomaly detection model.

[0031] S2, inputs the video to be detected into the pre-trained video anomaly detection model based on high- and low-frequency attention and dual dynamic memory network, and outputs the anomaly detection result;

[0032] Among them, video anomaly detection models, such as Figure 2 As shown, this may include:

[0033] The pre-trained deep model feature extractor is used to extract features from the video to be detected, and obtain the I3D features of the normal video and the I3D features of the abnormal video in the video to be detected; the feature learning module FL is used to process the I3D features of the normal video and the I3D features of the abnormal video based on the separated attention mechanism, and obtain the high-frequency features and low-frequency features corresponding to the normal video and the high-frequency features and low-frequency features corresponding to the abnormal video; the attention calculation is performed on the high-frequency features and low-frequency features corresponding to the normal video, and the calculated results are subjected to feature fusion to obtain the normal comprehensive features of the normal video; the attention calculation is performed on the high-frequency features and low-frequency features corresponding to the abnormal video, and the calculated results are subjected to feature fusion to obtain the abnormal comprehensive features of the abnormal video; the feature enhancement module FA is used to use the dual dynamic memory network to input the normal comprehensive features and the abnormal comprehensive features into the normal memory unit and the abnormal memory unit respectively for storage and learning, and obtain the normal enhanced comprehensive features and the abnormal enhanced comprehensive features; the preset memory loss function is used to constrain the learning of the normal memory unit and the abnormal memory unit; the classifier is used to obtain the abnormal score according to the normal enhanced comprehensive features and the abnormal enhanced comprehensive features, so as to obtain the abnormal detection result.

[0034] Specifically, the training process of the video anomaly detection model based on high- and low-frequency attention and dual dynamic memory networks is as follows: Figure 3 As shown, this may include:

[0035] S01, using a pre-trained deep model feature extractor, extract features from a training data set consisting of videos to be detected, and obtain I3D features of normal videos and I3D features of abnormal videos in the training data set.

[0036] For S01, we can give a set of video training data containing normal and abnormal events (consisting of normal videos and abnormal videos) and their corresponding weak labels. Based on the multi-instance learning MIL setting, the normal video is defined as {V n}, and define the abnormal video as {V a}, a normal video contains only normal video clips, while an abnormal video contains at least one abnormal clip. It can be understood that the I3D feature, namely the Inflated 3D ConvNet, is a deep learning model for video behavior recognition. The I3D model effectively utilizes spatiotemporal features by extending the pre-trained 2D convolutional network to a 3D convolutional network, thereby improving the accuracy of video behavior recognition. Abnormal clips represent clips containing abnormal events. The deep model feature extractor is obtained by training the I3D deep model using the Kinetics dataset.

[0037] Specifically, a pre-trained backbone network is used for feature extraction. Specifically, the I3D deep model pre-trained on the Kinetics dataset is used as the backbone network to extract features and extract RGB features of 16 consecutive frames. The extracted video features can be expressed as Where N represents the number of extracted segments, and F represents the dimension of the extracted video features.

[0038] S02, using the feature learning module FL based on the separation attention mechanism, the I3D features of the normal video and the I3D features of the abnormal video are processed to obtain the high-frequency features and low-frequency features corresponding to the normal video and the high-frequency features and low-frequency features corresponding to the abnormal video; the attention calculation is performed on the high-frequency features and low-frequency features corresponding to the normal video respectively, and the calculated results are subjected to feature fusion to obtain the normal comprehensive features of the normal video; the attention calculation is performed on the high-frequency features and low-frequency features corresponding to the abnormal video respectively, and the calculated results are subjected to feature fusion to obtain the abnormal comprehensive features of the abnormal video.

[0039] The fuzzy boundary between normal events and abnormal events in videos greatly affects the accuracy and complexity of video anomaly detection. In order to improve the accuracy of video anomaly detection, the key is to expand the relative gap between individual classification features by learning and enhancing video features. In video anomaly detection, the high-frequency features and low-frequency features of the image each play an important role. High-frequency features can capture local fine details and are suitable for detecting rapidly changing abnormal events, while low-frequency features can focus on global structures and are suitable for identifying long-term macroscopic anomalies. By separating these two types of features, the detection accuracy can be significantly improved, noise interference can be reduced, feature expression capabilities can be enhanced, and different application scenarios and needs can be better adapted.

[0040] Specifically, as the network layer deepens, the system can gradually extract high-frequency and low-frequency features. The feature learning module FL used in this model can pay close attention to both high-frequency and low-frequency features of image features. High-frequency features can capture subtle changes and local details in the image, which are often the key to rapidly changing abnormal events. For example, events such as sudden actions or abnormal objects that appear in an instant can be quickly detected by capturing high-frequency features. Low-frequency features focus on the global structure of the image and long-term change trends, which is very useful for identifying long-standing macroscopic abnormal events. For example, for persistent abnormal activities or slowly developing abnormal phenomena in an area, low-frequency features can provide a more comprehensive perspective to capture these changes.

[0041] Separate attention mechanisms, such as Figure 4 As shown, it may include: N h Self-attention head, high-frequency processing module F-highAttention and low-frequency processing module F-low Attention, fusion module;

[0042] Among them, N h In the self-attention heads, (1-α)N h The self-attention heads are connected to the high-frequency processing module F-highAttention, αN h The self-attention heads are connected to the low-frequency processing module F-low Attention; α represents the preset split ratio;

[0043] The high frequency processing module and the low frequency processing module are both connected to the fusion module.

[0044] Specifically, S021, using the feature learning module FL based on the separated attention mechanism, processing the I3D features of the normal video and the I3D features of the abnormal video to obtain the high-frequency features and low-frequency features corresponding to the normal video and the high-frequency features and low-frequency features corresponding to the abnormal video, may include:

[0045] Using Nh Each self-attention head processes the I3D features of normal videos and the I3D features of abnormal videos, obtains the corresponding query matrix Q, key matrix K and value matrix V through linear transformation, and calculates the attention score independently in each self-attention head;

[0046] Scale and normalize the attention scores corresponding to each self-attention head to obtain the output matrix of each self-attention head;

[0047] The first formula is used to concatenate the output matrices of all self-attention heads to obtain the high-frequency features and low-frequency features corresponding to normal videos and the high-frequency features and low-frequency features corresponding to abnormal videos.

[0048] Understandably, a complete learning of normal and abnormal features is crucial for video anomaly detection (VAD). The Transformer structure is superior in processing long-range correlations between feature information. Transformer is built on the basis of multi-head self-attention and can capture the long-range relationship between tokens at different positions. Specifically, let is the input sequence of the self-attention head, where N represents the length of the input sequence and D represents the number of hidden dimensions of the input sequence.

[0049] Each self-attention head calculates the query matrix Q, key matrix K and value matrix V by performing a linear transformation on the input sequence, Q = XW q , K = XW k , V = XW v .

[0050] Among them, W q represents the learnable parameters corresponding to the query matrix Q, W k represents the learnable parameters corresponding to the key matrix K, W v represents the learnable parameters corresponding to the value matrix V, D h Represents the number of hidden dimensions of the head. The output of the self-attention head is the weighted sum of N value vectors, which is expressed as follows:

[0051]

[0052] Among them, Softmax represents the Softmax function, Q represents the query matrix Q, K represents the key matrix K, V represents the value matrix V, and T represents the transpose operation.

[0053] Specifically, the first formula is as follows: MSA(X)=concat[SA h (X)]W o ;

[0054] Among them, MSA(X) represents the result of concatenating the output matrices of all self-attention heads, X represents the input sequence of the self-attention head, S represents the size of the window in the input sequence, and SA h (X) represents the output matrix of the self-attention head, W o Denotes the learnable parameters of the self-attention head.

[0055] It can be understood that the workflow of the self-attention head in the feature learning module FL is as follows:

[0056] Initialization: First, initialize the necessary parameters, including the weights of the query, key, and value matrices, and the number of heads in the multi-head attention. These weights will be used in subsequent linear transformations.

[0057] Linear transformation: linear transformation of the input query, key and value matrices. These linear transformations are achieved by multiplying with the corresponding weight matrices. The transformed matrices will be used in subsequent multi-head attention calculations.

[0058] Split and Project: Split the linearly transformed query, key, and value matrices into multiple heads. Each head has its own query, key, and value matrices. Then, the attention scores are calculated independently in each head.

[0059] Scaling and Softmax: The attention scores of each head are scaled to avoid the problem of vanishing or exploding gradients. Then, the Softmax function is applied to normalize the attention scores so that the sum of the weights at each position is 1.

[0060] Weighted summation: Use the normalized attention weights to perform weighted summation on the value matrix to obtain the output matrix of each head.

[0061] Concatenation and linear transformation: Concatenate the output matrices of all heads together to form a large output matrix. Then, perform a linear transformation on this output matrix to get the final output.

[0062] By splicing the output matrices of all self-attention heads, we can obtain the high-frequency features and low-frequency features corresponding to normal videos and the high-frequency features and low-frequency features corresponding to abnormal videos.

[0063] Specifically, S022, respectively performing attention calculation on the high-frequency features and low-frequency features corresponding to the normal video, and performing feature fusion on the calculated results to obtain the normal comprehensive features of the normal video; respectively performing attention calculation on the high-frequency features and low-frequency features corresponding to the abnormal video, and performing feature fusion on the calculated results to obtain the abnormal comprehensive features of the abnormal video, may include:

[0064] The high-frequency processing module F-high Attention in the feature learning module FL uses convolution operations and high-frequency filters to perform attention calculations on the high-frequency features corresponding to normal videos and the high-frequency features corresponding to abnormal videos, and obtains the high-frequency calculation results corresponding to normal videos and the high-frequency calculation results corresponding to abnormal videos;

[0065] The low-frequency processing module F-low Attention in the feature learning module FL uses global average pooling operations and low-frequency filters to perform attention calculations on the low-frequency features corresponding to normal videos and abnormal videos, and obtains the low-frequency calculation results corresponding to normal videos and abnormal videos;

[0066] The high-frequency calculation results and low-frequency calculation results corresponding to the normal video are feature fused to obtain the normal comprehensive features of the normal video; the high-frequency calculation results and low-frequency calculation results corresponding to the abnormal video are feature fused to obtain the normal comprehensive features of the abnormal video.

[0067] It can be understood that in order to better utilize high-frequency and low-frequency features, the embodiment of the present invention adopts an attention mechanism that can separate high-frequency and low-frequency features, which can independently process and learn high-frequency and low-frequency features. In this way, the system can more accurately identify and distinguish normal events from abnormal events, thereby improving the accuracy and efficiency of detection. Specifically, first, multi-level features in the video, including high-frequency and low-frequency features, are extracted through a deep neural network. For high-frequency features, convolution operations and high-frequency filters are used to capture local fine details in the video. High-frequency features are very sensitive to rapidly changing abnormal events and help to detect sudden abnormalities in a timely manner. For low-frequency features, global average pooling operations and low-frequency filters are used to focus on the overall structure of the video and long-term change trends. Low-frequency features are very effective for long-term macroscopic abnormal event detection. A separate attention mechanism is used to perform attention calculations on high-frequency and low-frequency features respectively. The high-frequency processing module F-high Attention focuses on the saliency of local details, while the low-frequency processing module F-lowAttention focuses on the saliency of global structures.

[0068] See the visualization diagram of high and low frequency attention mechanism for details. Figure 5 ,from Figure 5 It can be seen that after the separation of the attention mechanism, the characteristics of high-frequency learning are as follows Figure 5 As shown in the upper middle part, the characteristics of low-frequency learning are as follows Figure 5As shown in the lower middle part. By adopting an attention mechanism that separates high frequencies and low frequencies, the features of normal and abnormal events in video anomaly detection can be learned more accurately, which can significantly improve the accuracy and robustness of detection. Such a mechanism can better adapt to various complex application scenarios and requirements, and provide more reliable technical support for fields such as video surveillance and behavior analysis. For high-frequency processing, since high frequencies encode local details of objects, applying global attention on feature maps may be redundant and computationally expensive. Therefore, the embodiment of the present invention adopts F high To capture fine-grained high frequencies with local window self-attention, the computational complexity can be significantly saved. Understandably, the non-overlapping window partitioning is adopted in the feature learning module FL, which reduces the hardware requirements. Understandably, F high It can represent convolution operations and high-frequency filters. The global attention to the multi-head self-attention mechanism helps capture low-frequency information. Directly applying the multi-head self-attention mechanism to high-resolution feature maps requires huge computational cost. For low-frequency processing, since average pooling is a low-pass filter, F low First, average pooling is applied to each window in the feature map to obtain the low-frequency signal of the input sequence as the average pooling. The average pooling feature map is projected into the key matrix K and the value matrix V; where S represents the size of the window in the input sequence. As you can understand, F low It can represent low-frequency filters and global average pooling operations. low The query matrix Q in derives from the original feature map corresponding to the input sequence, and standard attention is applied to capture the rich low-frequency information in the feature map. Due to the reduced space of K and V, F low In order to achieve higher efficiency, the embodiment of the present invention adopts the strategy of head allocation, which is high and F low Specifically, according to the preset split ratio α, N h self-attention heads are allocated, where (1-α)N h self-attention heads for F high , the rest αN h self-attention heads for F low Since the complexity of each attention is low, the framework proposed in the embodiment of the present invention ensures low complexity and high throughput on the GPU. When head allocation is adopted, the self-attention head can learn the parameter W o can be decomposed into two smaller matrices, which helps reduce the parameters of the model. The output of the feature learning module FL is the concatenation of each attention output, which is expressed as X FL =[F High (X i3d );FLow (X i3d )].

[0069] S03, using the feature enhancement module FA based on the dual dynamic memory network, inputting the normal comprehensive features and the abnormal comprehensive features into the normal memory unit and the abnormal memory unit for storage and learning respectively, and obtaining the normal enhanced comprehensive features and the abnormal enhanced comprehensive features respectively; using the preset memory loss function to constrain the learning of the normal memory unit and the abnormal memory unit.

[0070] Specifically, the feature enhancement module FA, such as Figure 6 As shown, it may include: a normal memory unit and an abnormal memory unit; wherein the normal memory unit is used to process and store the features of normal videos, and the abnormal memory unit is used to process and store the features of abnormal videos.

[0071] For the feature enhancement module FA, refer to Figure 6 As shown, the method of using a dual dynamic memory network is used to input the learned video features of normal and abnormal events containing high-frequency and low-frequency information into the normal memory unit and the abnormal memory unit respectively, and the dynamic learnable normal and abnormal memory library is used to store the corresponding templates. The correct template learning is encouraged by querying and reading operations, and the memory loss function is set to limit the normal memory unit to learn normal events and the abnormal memory unit to learn abnormal events, thereby better distinguishing normal and abnormal events. In this dual dynamic memory network method, the normal memory unit mainly stores the template of normal events, while the abnormal memory unit stores the template of abnormal events. By querying and reading these templates, the system can encourage and strengthen the correct template learning. When a new video feature is input, the system will judge whether the feature belongs to a normal event or an abnormal event based on the stored template, thereby enhancing the feature. In order to ensure that each memory unit only learns the event type corresponding to each of them, an embodiment of the present invention designs a preset memory loss function. The preset memory loss function plays a constraining role, so that the normal memory unit focuses on learning normal events, while the abnormal memory unit focuses on learning abnormal events, so that normal and abnormal events can be better distinguished.

[0072] Specifically, in the dual dynamic memory network, the normal memory unit is responsible for processing and storing the features of normal events, while the abnormal memory unit is responsible for processing and storing the features of abnormal events. Each memory unit has a dynamically learnable memory library for storing corresponding event templates. The normal memory library stores normal event templates, and the abnormal memory library stores abnormal event templates. When new video features are input, the system will find the best matching template from the memory library through query and read operations. This process can help the system learn and update templates, thereby improving the accuracy of detection. Through the design of this dual dynamic memory unit, the system can more efficiently distinguish and detect normal and abnormal events in the video, thereby improving the accuracy and reliability of video anomaly detection. Traditional methods use a single memory unit to store prototypes of normal data to achieve better scoring. However, a single unit is not enough to distinguish difficult samples. The embodiments of the present invention use learnable normal and abnormal memory libraries to store corresponding templates. Wherein, M represents the memory library number, and D represents the dimension of the output feature. The embodiment of the present invention applies query and read operations to encourage correct template learning, and by querying the memory library T, the similarity between features can be evaluated:

[0073]

[0074] Where σ represents sigmoid activation, G represents query score, The memory-enhanced feature generated by the read operation is denoted as T aug G can characterize whether the fragment is similar to the memory library. The embodiment of the present invention applies the top-K selection method along the second dimension to determine which fragments are most similar to the memory library:

[0075]

[0076] In the above formula, K<<M, G k；ind represents the index of the K fragments to be stored, The query score corresponding to the index of the fragment. Indicates the highest memory match score.

[0077] When embedding normal video Generated by query and read operations when sent to an abnormal memory unit and in, represents the abnormal memory unit score of the normal video. Similarly, when the video embedding passes through the normal memory unit, we can obtain and in, The score of a normal memory cell representing a normal video. The score of an abnormal query due to normal input Restricted to Normal query score Should be Similarly, when abnormal videos are embedded When input to the abnormal memory unit, we get and in, The abnormal memory unit score representing the abnormal video. Due to the sparseness of abnormalities, the abnormal memory library will only activate a few clips. The embodiment of the present invention can obtain and in, Represents the normal memory unit score of the abnormal video. Different from the normal input, due to the weak label of the abnormality, the abnormal memory unit score of the normal video Not all are 1. and At least one element is equal to 1.

[0078] The preset memory loss function proposed in the embodiment of the present invention is as follows:

[0079] L=L cls +λ1L1+λ2L2+λ3L3+λ4L4;

[0080] Among them, L cls represents the BCE loss under video-level label supervision, λ1 represents the dual memory loss hyperparameter, L1 represents the dual memory loss, λ2 represents the triplet loss hyperparameter, L2 represents the triplet loss, λ3 represents the diversified regularization term hyperparameter, L3 represents the diversified regularization term, λ4 represents the magnitude distance loss hyperparameter, and L4 represents the magnitude distance loss.

[0081] For ease of understanding, the dual memory loss L1, triplet loss L2, diversified regularization term L3 and amplitude distance loss L4 are introduced below respectively.

[0082] Specifically, to train the dual memory unit, the dual memory loss L1 is defined to consist of four binary cross entropy (BCE) losses:

[0083] in, and yes The corresponding top-K selection method results are averaged along the first dimension. equal to 1. The first two terms of L1 restrict the normal memory unit to learn normal patterns, and the last two terms ensure that the abnormal memory unit learns abnormal patterns. Through L1, the memory unit can store homologous patterns. It can be understood that the label equal to 1 indicates that the input data belongs to the abnormal category, forcing the abnormal memory unit of the model to specifically learn the pattern of abnormal data, guiding the abnormal data to the abnormal memory unit, ensuring that the abnormal features are correctly captured, thereby improving the anomaly detection performance of the model.

[0084] Using query scores And triplet loss L2 distinguishes abnormal and normal feature embeddings.

[0085] L2=Triplet(f a ,f p ,f n );

[0086]

[0087] Among them, Triplet represents the triplet loss function, f a It's from X n The average features of the interest segment, which represents the main part of the normal video. The topK index selection of p represents the normal part in the abnormal video, f n Indicates the abnormal part of the abnormal video.

[0088] Amplitude feature learning is a method to extract features from the amplitude information of data for analysis and modeling in fields such as image processing and machine learning. Amplitude feature learning is applicable to the enhanced features T of memory output aug , the amplitude of the normal feature is lower than the amplitude of the abnormal feature E[||x a ||2]>>E[||x n ||2],x a and x n are the abnormal and normal segment features respectively. Therefore, the amplitude separability of the enhancement features between abnormal and normal is defined as d(a,n)=||m a ||2-||m n ||2. In addition, effective normal representation learning can capture the core variance and coverage variance introduced by noise. Learn the normal distribution so that the anomaly is out of the distribution and use the Gaussian distribution to constrain the potential normal representation z n ,Right now:

[0089] Among them, μ represents the mean of the normal feature, N represents the number of extracted fragments, and σ represents the variance. represents the i-th data feature, i represents the i-th data, and I represents the unit matrix. It can be understood that the unit matrix is ​​a matrix with all 1s on the diagonal and the remaining elements are 0. It is used to represent a special form of the Gaussian distribution covariance matrix. represents the square of the variance, Indicates that the covariance matrix is ​​a diagonal matrix, and each element on the diagonal is equal, each dimension of the distribution is independent and has the same variance. In order to ensure that the normal data sampling operation is differentiable, the embodiment of the present invention uses reparameterization to handle potential noise, and the data features Generated by random noise ε~N(0,1):

[0090] It can be seen from the above formula that when σ is too large, sample z is easily affected by σ and may be far from the true feature μ. In order to suppress the instability of σ, it is necessary to use the KLD (Kullback-Leibler) diversity regularization term:

[0091]

[0092] KL divergence, also known as Kullback-Leibler divergence, is a metric used to measure the difference between two probability distributions. KL divergence can measure the information gain between two probability distributions P and Q, or it can describe the additional information required to use probability distribution P given probability distribution Q. Regularization term is a constraint in machine learning models that is used to control the complexity of the model. It is added to the preset memory loss function in order to improve the generalization ability of the model and prevent overfitting by limiting the range of model parameters or reducing the numerical size of the parameters. Regularization terms can be divided into L1 regularization and L2 regularization, which respectively constrain the absolute value and square of the model parameters to achieve model regularization.

[0093] The anomaly is an OOD sample. For robustness, the present invention needs to widen the gap between the anomaly and the normal. The anomaly features are also fed to the mean encoder to obtain the embedding μ in the same space. a .

[0094] The embodiment of the present invention uses the magnitude distance loss L4 in the potential normal space to achieve the purpose of introducing the magnitude distance loss into the preset memory loss function.

[0095]

[0096] in, It is from T aug The average feature of the fragments. Through the topK index operation from Select the fragment in the and T aug get The obtained uncertainty learning feature and the output X of the feature learning module FL are combined FL Fusion is performed to obtain enhanced feature X FA The dual memory loss L1, triplet loss L2, diversified regularization term L3 and amplitude distance loss L4 are used to form part of the preset memory loss function. Enhanced feature X FA It can include normal enhancement comprehensive features and abnormal enhancement comprehensive features.

[0097] S04, inputting the normal enhanced comprehensive features and the abnormal enhanced comprehensive features into the classifier CLS to obtain an abnormality score, thereby obtaining an abnormality detection result.

[0098] After obtaining the enhanced features after fusion enhancement, they are input into the classifier, and the classifier outputs anomaly scores accordingly to determine the abnormal situation, thereby obtaining anomaly detection results.

[0099] In the pre-trained video anomaly detection model based on high- and low-frequency attention and dual dynamic memory network provided by an embodiment of the present invention, a pre-trained deep model feature extractor is used to extract video features containing time and space information in the training set as I3D features of normal videos and I3D features of abnormal videos in the video to be detected; the extracted features are input into a feature learning module FL; for the feature learning module FL, the extracted features are input into a separate attention mechanism for separating high-frequency features and low-frequency features, thereby being divided into two processing paths, one path encodes high-frequency interactions by processing local self-attention of feature maps with relatively high frequencies, and the other path encodes low-frequency interactions by global attention with down-sampling features, thereby improving the more accurate and detailed attention to the behavioral characteristics of normal or abnormal events contained in the video information, so as to better identify abnormal events. For the feature enhancement module FA, the extracted local and global features are input into the normal memory unit and the abnormal memory unit, and the corresponding templates are stored in the dynamically learnable normal and abnormal memory library. The correct template learning is encouraged through query and reading operations. The preset memory loss function is set to limit the normal memory unit to learn normal events and the abnormal memory unit to learn abnormal events, so as to better distinguish normal and abnormal events.

[0100] When testing the trained video anomaly detection model using a test video, only the mean encoder network is used to obtain feature embedding in the video anomaly detection model, and finally, the segment labels are assigned to the frame labels.

[0101] The effect of the video anomaly detection method is verified through specific experiments below.

[0102] The present invention selects two challenging anomaly benchmarks for experiments, and selects the UCF-Crime and XD-Violence datasets as test objects. For example, see the normal and abnormal video frames in the UCF-Crime and XD-Violence datasets. Figure 7a-7f .from Figure 7a-7f It can be seen that Figure 7a , Figure 7b and Figure 7c is the image corresponding to the normal video frame, Figure 7d for Figure 7a Abnormal video frames showing abnormal events in the captured scene, Figure 7e for Figure 7b Abnormal video frames showing abnormal events in the captured scene, Figure 7f for Figure 7c Abnormal video frames where abnormal events occurred in the captured scene. The abnormal events that occurred are indicated by red boxes. Figure 7d The red box in the figure indicates that an attack has occurred. Figure 7e The red box in the image indicates that arson has occurred. Figure 7f The red box in the figure indicates that a traffic accident has occurred, which is an abnormal event. For details, please refer to Table 1 for the comparison of UCF-Crime and XD-Violence datasets.

[0103] Table 1 Comparison of UCF-Crime and XD-Violence datasets

[0104]

[0105] As can be seen from Table 1, the UCF-Crime dataset contains 1900 surveillance videos with a total duration of 128 hours, covering 13 real-world anomalies, including abuse, robbery, explosion, traffic accidents, etc. For the weak supervision setting, the dataset is divided into 1610 training videos and 290 test videos, and only video-level annotations can be used for training, and frame-level annotations can be used for testing. The XD-Violence dataset is the latest and largest multimodal violence dataset, which contains 4754 uncut videos with a total duration of 217 hours. The dataset includes 3954 training videos and 800 test videos from various sources such as surveillance, movies, car cameras, and games. The dataset covers six types of violence, including abuse, car accidents, explosions, fights, riots, and shootings. Most of the videos in this dataset contain artistic expressions such as camera movement and scene switching, which makes video anomaly detection more difficult.

[0106] The embodiment of the present invention uses the I3D deep model pre-trained on the Kinetics-400 dataset as a deep model feature extractor to extract video features from the video to be detected. The same configuration is set for the UCF-Crime and XD-Violence datasets. In the training phase, each video is divided into N = 200 segments, and the learning rate is 0.0001. The batch size is set to 64 and the number of iterations is set to 3000. The video anomaly detection model is trained, and M is set for the UCF-Crime and XD-Violence datasets. a =60 and M n = 60. The hyperparameters λ1, λ2, λ3, and λ4 are set to (0.1, 0.1, 0.001, and 0.0001) respectively to balance the total loss. All operations of topK are equal to Set d=100 in the amplitude distance loss L4.

[0107] When selecting the evaluation metric, for the UCF-Crime dataset, the embodiment of the present invention uses the area under the frame-level receiver operating characteristic curve (AUC) as the evaluation metric. For the XD-Violence dataset, the area under the precision-recall curve, also known as the average precision (AP), is used as the standard evaluation metric. In addition, in order to ensure the reliability of anomaly prediction, the false alarm rate (FAR) with a threshold of 0.5 is used for evaluation. It can be understood that the lower the FAR, the more robust the model is to normal samples.

[0108] Specifically, when evaluating the performance of WS-VAD on the UCF-Crime dataset, the area under the frame-level receiver operating characteristic curve (ROC) (AUC) is used. The performance on the UCF-Crime dataset is shown in Table 2. Frame-level performance AUC comparison table on the UCF-Crime dataset.

[0109] Table 2 Frame-level performance AUC comparison table on UCF-Crime dataset

[0110] Method Feature(RGB) AUC (%) Sultani et al. (2018) I3D 76.21 IBL(2019) C3D 78.66 HL-Net(2020) I3D 82.44 CLAWS(2020) C3D 83.03 MIST(2021) I3D 82.30 RTFM(2021) I3D 84.30 DAR(2022) I3D 85.18 MSL(2022) I3D 85.30 S3R(2022) I3D 85.99 Thakare et al (2023) I3D 83.56 CoMo(2023) I3D 86.10 CR-UNet (2023) I3D 86.24 AFT(2024) I3D 85.07 Ours I3D 87.16

[0111] On the UCF-Crime dataset, the embodiment of the present invention compares the AUC score with the existing state-of-the-art (SOTA) VAD methods. All methods use RGB features, and the method of the embodiment of the present invention is based on 10-crop enhanced I3D features. Compared with other weakly supervised methods, the method of the embodiment of the present invention achieves a SOTA AUC score of 87.16%. Compared with the previous SOTA method CR-UNet (86.24% AUC), the model of the embodiment of the present invention is improved by 0.92%, which is a Transformer-based self-training method. Compared with the MIL-based method RTFM (84.30% AUC), the embodiment of the present invention achieved a significant improvement of 2.86%.

[0112] For the XD-Violence dataset, the model is evaluated using the area (AP) under the frame-level precision-recall curve (PR), and its performance is shown in Table 3, a frame-level performance AP comparison table on the XD-Violence dataset.

[0113] Table 3. Frame-level performance AP comparison on XD-Violence dataset

[0114] Method Feature AP(%) HL-Net(2020) RGB 73.67 CRFD(2021) RGB 75.90 RTFM(2021) RGB 77.81 MSL(2022) RGB 78.28 CU-Net (2023) RGB 78.74 DAR(2022) RGB 78.94 ACF(2022) RGB 80.13 S3R(2022) RGB 80.26 MGFN(2023) RGB 79.19 CoMo(2023) RGB 81.30 Pang et al. (2023) RGB 81.69 AFT(2024) RGB 80.07 Ours RGB 82.70

[0115] The SOTA VAD method in XD-Violence is shown in Table 3. The present invention uses 5-crop I3D RGB features for experiments. Compared with the previous SOTA method Pang et al. (81.69% AP), the AP score of the present invention is 82.70%, which is 1.01% higher. The results of the two datasets show that the model of the present invention is effective in detecting abnormal events with interference.

[0116] To further demonstrate the effectiveness of the video anomaly detection method provided by the embodiment of the present invention, the qualitative results of the anomaly detection performance corresponding to the UCF-Crime and XD-Violence datasets are shown in Figure 2. Figure 8a-8c . The embodiment of the present invention visualizes the anomaly score curve. From the figure, it can be seen that the real value and predicted anomaly score of abnormal videos and normal videos from the UCF-Crime and XD-Violence datasets. The embodiment of the present invention can detect short-term abnormal events and long-term abnormal events in surveillance videos. In addition, the embodiment of the present invention can also detect multiple abnormal events in a video. At the same time, the method also has a high detection accuracy for abnormal events in complex surveillance scenes.

[0117] In order to further evaluate the video anomaly detection method provided by the embodiment of the present invention, the embodiment of the present invention selects to use I3D-RGB features to perform ablation research on the UCF-Crime and XD-Violence datasets.

[0118] Specifically, in order to comprehensively study the impact of different modules on model performance, the embodiment of the present invention shows three combination results of two modules in the performance comparison diagram of the corresponding modules on UCF-Crime and XD-Violence in Table 4, and adopts three key indicators to comprehensively evaluate the proposed method.

[0119] Table 4 Performance comparison of corresponding modules on UCF-Crime and XD-Violence

[0120]

[0121] The metrics used are designed to evaluate the anomaly detection performance of abnormal videos. Meanwhile, the false alarm rate (FAR) is also used to evaluate the detection of normal videos to ensure the reliability of normal video detection. The feature learning module FL, as the backbone of feature learning, shows relatively low performance. However, after adding the feature enhancement module FA, all evaluation metrics show significant improvements. The increase in AUC and AP scores and the decrease in FAR indicate that the dynamic memory unit module is able to effectively store and distinguish abnormal patterns from normal patterns. After the introduction of the two modules, the indicators are further significantly improved. Finally, the complete model achieves the best performance on both datasets, demonstrating its excellent comprehensive detection capabilities.

[0122] In order to verify the effect of different numbers of memory units, the dual memory unit is split into two separate memory units for experiments, as shown in Table 5 Performance comparison of different memory units on UCF-Crime and XD-Violence.

[0123] Table 5 Performance comparison of different memory units on UCF-Crime and XD-Violence

[0124]

[0125] As can be seen from the table, a single abnormal storage unit has difficulty in effectively distinguishing between normal and abnormal in the absence of normal patterns, and therefore exhibits a high false alarm rate (FAR) in normal video detection. This result shows that on weakly supervised datasets, a single abnormal storage unit is not easy to learn and identify abnormal patterns. In contrast, a single normal memory unit can achieve better results in detection because it contains abundant normal data. However, the dual storage unit can provide more comprehensive detection capabilities by storing both normal and abnormal patterns at the same time, thus showing the best performance in all evaluation indicators. This shows that the dual storage unit structure has significant advantages in processing complex video detection tasks, can more accurately distinguish normal and abnormal video patterns, and improves the comprehensive detection performance of the model.

[0126] In order to systematically explore the impact of the number of memory units on model performance, the present invention designs and tests nine different storage cases to find the optimal configuration of the number of memory units. The experimental results are shown in Table 6. A 、M N The results are shown in the comparison table.

[0127] Table 6 Different memory numbers M A 、M N Comparison table of results

[0128]

[0129] It can be seen from the table that in these experiments, M A and M N Represent the number of abnormal memory cells and normal memory cells respectively. Through systematic experimental analysis, it is found that when M A and M N When both are set to 60, the proposed method shows the best performance on both datasets. At this point, the model achieves good results on all evaluation indicators. This configuration not only effectively improves the accuracy of anomaly detection, but also significantly reduces the false alarm rate (FAR), ensuring the reliability of normal video detection. The abnormal memory unit helps the model detect abnormal videos more accurately by effectively recording and identifying abnormal patterns. The normal memory unit enhances the performance of the model in detecting normal videos by storing and identifying normal patterns.

[0130] In order to explore the impact of the four loss terms on the performance of the complete model, the embodiment of the present invention compares the experimental results of 8 different loss term combinations in detail in the result comparison table of different loss functions in Table 7.

[0131] Table 7 Comparison of results of different loss functions

[0132]

[0133] These experiments aim to evaluate the specific contribution and importance of each loss term in the model. From the results of the first three combinations, it can be seen that the L2 loss term is the most effective in distinguishing the distance between normal and abnormal data. Specifically, the L2 loss term significantly improves the accuracy of the model in detecting anomalies, indicating that it plays a key role in enhancing the sensitivity of the model to abnormal behavior. Further analysis of the results of the last five cases shows that the loss terms L2, L3, and L4 are indispensable to the overall performance of the model. The L3 loss term plays an important role in enhancing the model's ability to identify abnormal patterns, while the L4 loss term, although it has a smaller direct impact on detection performance, plays a key role in controlling the boundary between normal and abnormal. This boundary control is crucial for reducing false alarms and improving detection accuracy. The above results show that the combined use of the four loss terms can significantly improve the overall performance of the model. Each loss term enhances the detection ability of the model in different aspects, making the model more accurate and reliable in distinguishing normal and abnormal patterns.

[0134] Based on the above experimental results, it can be proved that the video anomaly detection method provided by the embodiment of the present invention pays more attention to the high-frequency and low-frequency information of the video frame through the feature learning module, and then enhances the features by adopting the feature enhancement module of the dual dynamic memory network to improve the ability of the model to detect abnormal events, thereby achieving robust anomaly detection. The feature learning module based on high frequency and low frequency can effectively capture local and global features in the image. High-frequency features focus on capturing fine details in the image, such as rapidly changing abnormal events, while low-frequency features focus on capturing the overall structure of the image, such as long-standing macroscopic abnormal events. By paying attention to high and low frequency information together, the system's attention to various abnormal events is improved. The dual dynamic memory network is another important component. They store prototypes of normal and abnormal events respectively, and encourage memory units to learn and distinguish normal and abnormal events through query and read operations. This design enables the system to match according to the stored prototype when processing new video features, so as to more accurately determine whether the feature belongs to a normal event or an abnormal event.

[0135] Through the above comparative experiments, the effectiveness and importance of the feature learning module for distinguishing high and low frequencies and the feature enhancement module of the dual dynamic memory unit are verified. The experimental results show that the method provided by the embodiment of the present invention has achieved significant improvements on two public data sets, and the video anomaly detection model has not only achieved a breakthrough in detection accuracy, but also demonstrated strong robustness when processing complex video scenes.

[0136] It should be noted that in the description of the present invention, it should be understood that the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0137] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A video anomaly detection method based on high-low frequency attention and dual dynamic memory network, characterized in that: include: Get the video to be detected; Inputting the video to be detected into a pre-trained video anomaly detection model based on high- and low-frequency attention and dual dynamic memory networks, and outputting anomaly detection results; Wherein, the video anomaly detection model includes: A pre-trained deep model feature extractor is used to extract features from the video to be detected, and obtain the I3D features of the normal video and the I3D features of the abnormal video in the video to be detected; a feature learning module FL is used to process the I3D features of the normal video and the I3D features of the abnormal video based on a separated attention mechanism, and obtain high-frequency features and low-frequency features corresponding to the normal video and high-frequency features and low-frequency features corresponding to the abnormal video; attention calculation is performed on the high-frequency features and low-frequency features corresponding to the normal video, and feature fusion is performed on the calculated results to obtain the normal comprehensive features of the normal video; the high-frequency features corresponding to the abnormal video are processed by the attention calculation module FL, and ... The high-frequency features and low-frequency features are respectively calculated for attention, and the calculated results are subjected to feature fusion to obtain the abnormal comprehensive features of the abnormal video; the feature enhancement module FA is used to use a dual dynamic memory network to input the normal comprehensive features and the abnormal comprehensive features into the normal memory unit and the abnormal memory unit respectively for storage and learning, and obtain normal enhanced comprehensive features and abnormal enhanced comprehensive features respectively; a preset memory loss function is used to constrain the learning of the normal memory unit and the abnormal memory unit; the normal enhanced comprehensive features and the abnormal enhanced comprehensive features are input into a classifier to obtain an abnormal score, thereby obtaining an abnormal detection result.

2. According to claim 1, a video anomaly detection method based on high-low frequency attention and dual dynamic memory network is characterized in that: The deep model feature extractor is obtained by training the I3D model using the Kinetics dataset.

3. According to claim 1, a video anomaly detection method based on high-low frequency attention and dual dynamic memory network is characterized in that: The training process of the video anomaly detection model includes: Using a pre-trained deep model feature extractor, feature extraction is performed on a training data set consisting of videos to be detected, to obtain I3D features of normal videos and I3D features of abnormal videos in the training data set; The feature learning module FL is used to process the I3D features of the normal video and the I3D features of the abnormal video based on the separation attention mechanism to obtain high-frequency features and low-frequency features corresponding to the normal video and high-frequency features and low-frequency features corresponding to the abnormal video; the high-frequency features and low-frequency features corresponding to the normal video are respectively calculated for attention, and the calculated results are subjected to feature fusion to obtain normal comprehensive features of the normal video; the high-frequency features and low-frequency features corresponding to the abnormal video are respectively calculated for attention, and the calculated results are subjected to feature fusion to obtain abnormal comprehensive features of the abnormal video; The normal comprehensive features and the abnormal comprehensive features are input into the normal memory unit and the abnormal memory unit for storage and learning respectively by using the feature enhancement module FA based on the dual dynamic memory network, so as to obtain the normal enhanced comprehensive features and the abnormal enhanced comprehensive features; the learning of the normal memory unit and the abnormal memory unit is constrained by using the preset memory loss function; The normal enhanced comprehensive features and the abnormal enhanced comprehensive features are input into a classifier to obtain an abnormality score, thereby obtaining an abnormality detection result.

4. The video anomaly detection method based on high-low frequency attention and dual dynamic memory network according to claim 3 is characterized in that: The separation attention mechanism includes: N h Self-attention head, high-frequency processing module F-highAttention and low-frequency processing module F-low Attention, fusion module; Among them, the N h In the self-attention heads, (1-α)N h The self-attention heads are connected to the high-frequency processing module F-highAttention, αN h The self-attention heads are connected to the low-frequency processing module F-low Attention; α represents a preset split ratio; The high-frequency processing module F-high Attention and the low-frequency processing module F-low Attention are both connected to the fusion module.

5. The video anomaly detection method based on high-low frequency attention and dual dynamic memory network according to claim 4 is characterized in that: The feature learning module FL processes the I3D features of the normal video and the I3D features of the abnormal video based on the separated attention mechanism to obtain high-frequency features and low-frequency features corresponding to the normal video and high-frequency features and low-frequency features corresponding to the abnormal video, including: Utilizing X h The self-attention heads process the I3D features of the normal video and the I3D features of the abnormal video, obtain the corresponding query matrix Q, key matrix K and value matrix V through linear transformation, and independently calculate the attention score in each self-attention head; Scale and normalize the attention scores corresponding to each self-attention head to obtain the output matrix of each self-attention head; The first formula is used to concatenate the output matrices of all self-attention heads to obtain the high-frequency features and low-frequency features corresponding to normal videos and the high-frequency features and low-frequency features corresponding to abnormal videos.

6. The video anomaly detection method based on high-low frequency attention and dual dynamic memory network according to claim 5 is characterized in that: The first formula is as follows: MSA(X)=concat[SA h (X)]W o ; Among them, MSA(X) represents the result of concatenating the output matrices of all self-attention heads, X represents the input sequence of the self-attention head, S represents the size of the window in the input sequence, and SA h (X) represents the output matrix of the self-attention head, W o Denotes the learnable parameters of the self-attention head.

7. The video anomaly detection method based on high-low frequency attention and dual dynamic memory network according to claim 4 is characterized in that: The attention calculation is performed on the high-frequency features and low-frequency features corresponding to the normal video respectively, and the calculated results are subjected to feature fusion to obtain the normal comprehensive features of the normal video; Attention calculations are performed on the high-frequency features and low-frequency features corresponding to the abnormal video, and the calculated results are subjected to feature fusion to obtain the abnormal comprehensive features of the abnormal video, including: The high-frequency processing module F-high Attention in the feature learning module FL uses convolution operations and high-frequency filters to perform attention calculations on the high-frequency features corresponding to the normal video and the high-frequency features corresponding to the abnormal video, and obtains the high-frequency calculation results corresponding to the normal video and the high-frequency calculation results corresponding to the abnormal video; The low-frequency processing module F-low Attention in the feature learning module FL uses a global average pooling operation and a low-frequency filter to perform attention calculation on the low-frequency features corresponding to the normal video and the low-frequency features corresponding to the abnormal video, so as to obtain the low-frequency calculation results corresponding to the normal video and the low-frequency calculation results corresponding to the abnormal video; The high-frequency calculation results and low-frequency calculation results corresponding to the normal video are feature fused to obtain the normal comprehensive features of the normal video; the high-frequency calculation results and low-frequency calculation results corresponding to the abnormal video are feature fused to obtain the normal comprehensive features of the abnormal video.

8. The video anomaly detection method based on high-low frequency attention and dual dynamic memory network according to claim 3 is characterized in that: The feature enhancement module FA includes: a normal memory unit and an abnormal memory unit; wherein the normal memory unit is used to process and store the features of normal videos, and the abnormal memory unit is used to process and store the features of abnormal videos.

9. The video anomaly detection method based on high-low frequency attention and dual dynamic memory network according to claim 3 is characterized in that: The preset memory loss function is as follows: L=L cls +λ1L1+λ2L2+λ3L3+λ4L4; Among them, L cls represents the BCE loss under video-level label supervision, λ1 represents the dual memory loss hyperparameter, L1 represents the dual memory loss, λ2 represents the triplet loss hyperparameter, L2 represents the triplet loss, λ3 represents the diversified regularization term hyperparameter, L3 represents the diversified regularization term, λ4 represents the magnitude distance loss hyperparameter, and L4 represents the magnitude distance loss.

Images