A video anomaly detection method and device based on spatiotemporal memory network

Abstract

The present invention belongs to the field of computer vision, and specifically relates to a video anomaly detection method and device based on a spatiotemporal memory network. The method of the present invention comprises: in the training phase, inputting video frames of normal behavior into a video anomaly detection model, outputting a multidimensional feature sequence through an encoder in an autoencoder and inputting it into a memory module, the memory module calculating the spatiotemporal relationship information and reading and updating the spatiotemporal relationship information using memory items as carriers, and optimizing the weight balance of the loss function involved in the memory module update; inputting time-series strongly correlated features into a decoder in the autoencoder; in the testing phase, inputting the test video frame into a trained video anomaly detection model and calculating the anomaly score. The present invention utilizes the guidance of the memory module to learn the temporal and global correlations between feature sequences, optimizes the storage and update rules of the memory module, accelerates the convergence of the memory module, and improves the stability of the model and the detection accuracy.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular relates to a video anomaly detection method and device based on a spatiotemporal memory network. Background Art

[0002] In recent years, deep learning has been widely applied in many computer vision fields, such as anomaly detection, object detection, shadow detection, and trajectory prediction. In video anomaly detection, a growing number of methods based on deep neural networks are being applied to real-world tasks. The most popular detection methods are reconstruction or prediction, and the autoencoder (AE) architecture based on a deep convolutional network (CNN) is also the most commonly used network framework.

[0003] In reconstruction-based methods, an AE is used as a generator and trained on a large amount of normal data to learn the underlying data representation of normal samples. During the test phase, generated samples with large reconstruction errors are treated as abnormal data, allowing abnormal behavior to be detected. In prediction-based methods, the reconstruction error of the generated image is defined as the error between the predicted frame and the next frame. Although the prediction method initially involves the temporal relationship between consecutive video frames, it still learns the spatial features of the low-dimensional features of normal data. Learning spatial features alone is difficult to perform the task of abnormal behavior detection, because normal or abnormal behaviors often contain similar spatial features, which allows abnormal samples to be well reconstructed by the generator. Moreover, when the environment of the test sample changes, the spatial features contained in the video frame will change accordingly, and the accuracy of the trained model will decrease accordingly.

[0004] Incorporating methods for extracting temporal features into prediction methods can further improve model performance. Traditional temporal feature extraction methods, such as RNN and LSTM, can learn temporal relationships between corresponding elements within a certain time range and can also be applied to visual tasks, but they lack speed and accuracy. Memory-enhanced autoencoders (MemAEs), as a traditional memory network, use a memory module to store typical data distributions from a large number of normal samples while the network is being trained, thereby guiding feature generation. This memory module uses the characteristics of previously learned video frames from a global perspective to standardize the generation of future frames, but it stores messy and trivial spatial features. Models using MemAEs are unstable and require a long time to train to achieve the desired results. Summary of the Invention

[0005] To solve the above problems, the present invention proposes a video anomaly detection method and device based on a spatiotemporal memory network. The method of the present invention extracts the spatiotemporal information between consecutive video frames from a global perspective and uses a memory module for guidance, thereby improving the speed and accuracy of video anomaly detection.

[0006] The technical solution adopted by the method of the present invention is as follows:

[0007] A video anomaly detection method based on spatiotemporal memory guided network includes a testing phase and a training phase;

[0008] During the training phase, a video anomaly detection model is constructed, which includes an autoencoder and a memory module. Video frames of normal behavior are input into the video anomaly detection model, and a multidimensional feature sequence is output by the encoder in the autoencoder. The multidimensional feature sequence is input into the memory module. The memory module calculates the spatiotemporal relationship information between low-dimensional features in the multidimensional feature sequence. The spatiotemporal relationship information is read and updated using the memory items of the memory module as a carrier, and an uncertainty function is used to optimize the weight balance of the loss function involved in the memory module update. After being guided by the memory module, strong temporal correlation features are obtained, which are input into the decoder in the autoencoder to predict the next video frame and update the video anomaly detection model.

[0009] During the testing phase, the test video frames are input into the trained video anomaly detection model, and the anomaly score is calculated to determine whether the test video contains any anomalies.

[0010] Further as a preferred technical solution of the method of the present invention, the autoencoder includes an encoder and a decoder, the feature dimension of the encoder output and the feature dimension of the input decoder correspond to the dimension of the input video frame and the predicted video frame; the encoding and decoding processes correspond to four layer-by-layer dimensionality reduction and dimensionality increase operations, respectively, and two convolution operations are performed before each dimension change; a jump connection operation is applied between the encoding layer and the decoding layer with the same dimension to compensate for the information loss caused by deep convolution.

[0011] As a further preferred technical solution of the method of the present invention, the process of calculating the spatiotemporal relationship information by the memory module includes:

[0012] At time t, the memory module keeps the multidimensional feature sequence in its original size and expands it into a sequence Seq1 along the channel dimension, and combines Seq1 with the three parameter matrices W q 、W k 、W v Multiply them together to get three eigenvectors q i 、k i 、v i Seq2, where q i is the temporal correlation query item, ki is the temporal correlation matching vector, v i It is the characteristic information passed down from the i-th element in Seq1; the q of the i-th element in Seq1 i Doing a dot product operation with the temporal correlation matching vector of other elements calculates the initial correlation score, as shown in the following formula:

[0013] c ij =q i k j

[0014] Among them, c ij represents the initial correlation score calculated between the i-th element at time t and other elements j, which is the read item of the memory module;

[0015] After the correlation calculation of element i and all other elements is completed, c ij Composing the initial correlation sequence Seq 3,i ; During the training phase, all sequences use the same parameter matrix W q 、W k 、W v Perform parallel computing.

[0016] As a further preferred technical solution of the method of the present invention, the process of reading and updating the spatiotemporal relationship information using the memory items of the memory module as a carrier and obtaining the time series strong correlation features after being guided by the memory module includes:

[0017] The memory module is used to store the initial correlation sequence Seq 3,i Read; item c will be read ij Compare with each stored memory item in the memory module, the comparison step first calculates the read item c ij The Euclidean distance between each stored memory item in the memory module is as follows:

[0018]

[0019] Among them, d l2 represents the Euclidean distance, c m a relevance score stored for the stored memory item;

[0020] When the sequence Seq 3,i After calculating the Euclidean distance between all elements and the storage memory item, the storage memory item with the shortest Euclidean distance is c ij The best matching memory item c m_out , combine the best matching storage memory items into the best matching sequence Store the best match in memory c m_out with c ij Perform the addition operation to obtain the normalized correlation information ci ' j ;

[0021] Represents the normalized correlation information c i ' j The vector is divided by the common dimension d of each element in the sequence Seq1, and a Softmax operation is applied to obtain the final relevance score s of element i in Seq1 to all elements ij , the calculation formula is as follows:

[0022]

[0023] Final relevance score s ij Composition sequence Seq 4,i ; Seq 4,i The elements in Seq2 are the same as the elements in v i One-to-one correspondence, weighted summation, the calculation formula is as follows:

[0024]

[0025] where e i is the spatiotemporal strong correlation feature obtained after the memory module is guided, s ij is the final correlation score, v j is the vector in the multidimensional feature sequence output by the encoder, n is the number of channels of the multidimensional feature sequence output by the encoder; all e i Composed of sequence Seq out , Seq out as input to the decoder.

[0026] As a further preferred technical solution of the method of the present invention, the process of optimizing the weight balance of the loss function involved in the memory module update includes:

[0027] After calculating the Euclidean distance between the read memory item and all stored memory items, the best matching stored memory item c is obtained. m_out Then, minimize c m_out and read item c ij The Euclidean distance between them is used to obtain the temporal feature shrinkage loss L1, as shown in the following formula:

[0028]

[0029] The temporal feature separation loss L2 is used to prevent the second matching storage memory item from being too similar to the best matching storage memory item. The calculation formula of the temporal feature separation loss L2 is as follows:

[0030]

[0031] where c m_outStore the memory item for the best match, c m_sec storing a memory item for the second match;

[0032] A homogeneous uncertainty function L3 is used to optimize the weight between L1 and L2, as shown in the following formula:

[0033]

[0034] Among them, σ1 and σ2 represent learnable parameters;

[0035] Minimize prediction frame With the target frame I t+1 The Euclidean distance between them is used to obtain the prediction loss L pre As shown in the following formula:

[0036]

[0037] According to L3 and L pre Get the target loss function L total , as shown below:

[0038] L total =L pre +λL3

[0039] Where λ is an adjustable parameter.

[0040] As a further preferred technical solution of the method of the present invention, the calculation method of the anomaly score is:

[0041] The peak signal-to-noise ratio calculation formula for measuring the quality of the reconstructed image is as follows:

[0042]

[0043] Where PSNR(·) represents the peak signal-to-noise ratio, I t+1 represents the real video frame at time t+1, represents the predicted video frame at time t+1, express The maximum pixel value of N represents the total number of pixels contained in a video frame image;

[0044] When testing, the trained memory items are kept unchanged. The timing characteristics of abnormal events are different from the timing characteristics of normal events stored in the memory items. The difference is expressed as S t , S t The calculation formula is as follows:

[0045]

[0046] Among them, c m_out represents the best matching storage memory item, cij Represents the read item, m represents the number of read items;

[0047] The peak signal-to-noise ratio and S t After the abnormal score is normalized, the abnormal score S is obtained total The calculation formula for (t) is as follows:

[0048]

[0049] where N(·) represents the anomaly score normalization operation, and w1 and w2 are adjustable parameters.

[0050] The present invention further provides an electronic device, comprising a memory, a processor, and program instructions stored in the memory and executable by the processor, wherein the processor executes the program instructions to implement the steps of the method of the present invention.

[0051] Compared with the prior art, the present invention has the following beneficial effects:

[0052] 1. The method of the present invention uses the guidance of the memory module to learn the temporal and global correlations between feature sequences during the process of generating prediction frames. While extracting spatial features, it combines the temporal correlations of features to fully understand the video from the spatiotemporal dimension, achieving better prediction results.

[0053] 2. The method of the present invention optimizes the storage and update rules of the memory module to represent the potential correlation information between consecutive frames, thereby guiding the network to perform feature learning, accelerating the convergence of the memory module, and improving the stability of the memory module;

[0054] 3. The method of the present invention uses an uncertain function to optimize the weights between multiple losses generated when the memory module is updated, thereby accelerating the convergence speed of the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and detailed embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be derived from these drawings without inventive effort. Among them:

[0056] Figure 1 Schematic diagram of the overall framework of the video anomaly detection method based on spatiotemporal memory network of the present invention;

[0057] Figure 2 A schematic diagram of the structure of an automatic encoder according to an embodiment of the present invention;

[0058] Figure 3 A schematic diagram comparing the storage and update of a traditional memory network and an improved memory module in an embodiment of the present invention;

[0059] Figure 4 Provides a graph of the training results of each generation of the traditional memory network and the improved memory module on three data sets in the embodiment of the present invention;

[0060] Figure 5 The heat map of normal and abnormal events detected using real videos in different scenarios in the embodiments provided by the present invention;

[0061] Figure 6 ROC effect diagrams of MemAE, ST_MemAE w / o BML and ST_MemAE w BML of the embodiments provided by the present invention on three data sets;

[0062] Figure 7 This is a comparison chart of the running speed and detection accuracy of the method of the present invention and other video anomaly detection methods provided by the embodiment of the present invention. DETAILED DESCRIPTION

[0063] The present invention will be further explained below in detail with reference to the accompanying drawings so that those skilled in the art can have a deeper understanding of the present invention and be able to implement it. However, the following reference examples are only used to explain the present invention and are not intended to limit the present invention.

[0064] like Figure 1 As shown, the present invention provides a video anomaly detection method based on a spatiotemporal memory guided network, which includes a testing phase and a training phase;

[0065] During the training phase, a video anomaly detection model is constructed, which includes an autoencoder and a memory module. Video frames of normal behavior are input into the video anomaly detection model, and a multidimensional feature sequence is output by the encoder in the autoencoder. The multidimensional feature sequence is input into the memory module. The memory module calculates the spatiotemporal relationship information between low-dimensional features in the multidimensional feature sequence. The spatiotemporal relationship information is read and updated using the memory items of the memory module as a carrier, and an uncertainty function is used to optimize the weight balance of the loss function involved in the memory module update. After being guided by the memory module, strong temporal correlation features are obtained, which are input into the decoder in the autoencoder to predict the next video frame and update the video anomaly detection model.

[0066] During the testing phase, the test video frames are input into the trained video anomaly detection model, and the anomaly score is calculated to determine whether the test video contains any anomalies.

[0067] The autoencoder consists of an encoder and a decoder. The feature dimensions of the encoder output and the feature dimensions of the decoder input correspond to the dimensions of the input video frame and the predicted video frame. The encoding and decoding processes correspond to four layer-by-layer dimensionality reduction and dimensionality increase operations, respectively. Two convolution operations are performed before each dimensionality change. Skip connection operations are applied between the encoding layer and the decoding layer with the same dimension to compensate for the information loss caused by deep convolution.

[0068] The process of calculating the spatiotemporal relationship information in the memory module includes:

[0069] At time t, the memory module keeps the multidimensional feature sequence in its original size and expands it into a sequence Seq1 along the channel dimension, and combines Seq1 with the three parameter matrices W q 、W k 、W v Multiply them together to get three eigenvectors q i 、k i 、v i Seq2, where q i is the temporal correlation query item, k i is the temporal correlation matching vector, v i It is the characteristic information passed down from the i-th element in Seq1; the q of the i-th element in Seq1 i Doing a dot product operation with the temporal correlation matching vector of other elements calculates the initial correlation score, as shown in the following formula:

[0070] c ij =q i k j

[0071] Among them, c ij represents the initial correlation score calculated between the i-th element at time t and other elements j, which is the read item of the memory module;

[0072] After the correlation calculation of element i and all other elements is completed, c ij Composing the initial correlation sequence Seq 3,i ; During the training phase, all sequences use the same parameter matrix W q 、W k 、W v Perform parallel computing. 3,i c in the sequence ij Represents the dependency between the i-th element in Seq1 and each other element, characterizing the global correlation between multidimensional features.

[0073] The process of reading and updating the spatiotemporal relationship information using the memory items of the memory module as a carrier and obtaining the time series strong correlation features after being guided by the memory module includes:

[0074] The memory module is used to store the initial correlation sequence Seq 3,i Read; item c will be read ij Compare with each stored memory item in the memory module, the comparison step first calculates the read item c ij The Euclidean distance between each stored memory item in the memory module is as follows:

[0075]

[0076] Among them, d l2 represents the Euclidean distance, c m a relevance score stored for the stored memory item;

[0077] When the sequence Seq 3,i After calculating the Euclidean distance between all elements and the storage memory item, the storage memory item with the shortest Euclidean distance is c ij The best matching memory item c m_out , combine the best matching storage memory items into the best matching sequence Store the best match in memory c m_out with c ij Perform the addition operation to obtain the normalized correlation information c i ' j ;

[0078] Represents the normalized correlation information c i ' j The vector is divided by the common dimension d of each element in the sequence Seq1, and a Softmax operation is applied to obtain the final relevance score s of element i in Seq1 to all elements ij , the calculation formula is as follows:

[0079]

[0080] Final relevance score s ij Composition sequence Seq 4,i ; Seq 4,i The elements in Seq2 are the same as the elements in v i One-to-one correspondence, weighted summation, the calculation formula is as follows:

[0081]

[0082] where e i is the spatiotemporal strong correlation feature obtained after the memory module is guided, s ij is the final correlation score, v j is the vector in the multidimensional feature sequence output by the encoder, n is the number of channels of the multidimensional feature sequence output by the encoder; all e iComposed of sequence Seq out , Seq out as input to the decoder.

[0083] The process of optimizing the weight balance of the loss function involved in the memory module update includes:

[0084] After calculating the Euclidean distance between the read memory item and all stored memory items, the best matching stored memory item c is obtained. m_out Then, minimize c m_out and read item c ij The Euclidean distance between them is used to obtain the temporal feature shrinkage loss L1, as shown in the following formula:

[0085]

[0086] The temporal feature separation loss L2 is used to prevent the second matching storage memory item from being too similar to the best matching storage memory item. The calculation formula of the temporal feature separation loss L2 is as follows:

[0087]

[0088] where c m_out Store the memory item for the best match, c m_sec storing a memory item for the second match;

[0089] A homogeneous uncertainty function L3 is used to optimize the weight between L1 and L2, as shown in the following formula:

[0090]

[0091] Among them, σ1 and σ2 represent learnable parameters;

[0092] Minimize prediction frame With the target frame I t+1 The Euclidean distance between them is used to obtain the prediction loss L pre As shown in the following formula:

[0093]

[0094] According to L3 and L pre Get the target loss function L total , as shown below:

[0095] L total =L pre +λL3

[0096] Where λ is an adjustable parameter.

[0097] The anomaly score is calculated as:

[0098] The peak signal-to-noise ratio calculation formula for measuring the quality of the reconstructed image is as follows:

[0099]

[0100] Where PSNR(·) represents the peak signal-to-noise ratio, I t+1 represents the real video frame at time t+1, represents the predicted video frame at time t+1, express The maximum pixel value of N represents the total number of pixels contained in a video frame image;

[0101] When testing, the trained memory items are kept unchanged. The timing characteristics of abnormal events are different from the timing characteristics of normal events stored in the memory items. The difference is expressed as S t , S t The calculation formula is as follows:

[0102]

[0103] Among them, c m_out represents the best matching storage memory item, c ij Represents the read item, m represents the number of read items;

[0104] The peak signal-to-noise ratio and S t After the abnormal score is normalized, the abnormal score S is obtained total The calculation formula for (t) is as follows:

[0105]

[0106] where N(·) represents the anomaly score normalization operation, and w1 and w2 are adjustable parameters.

[0107] The effectiveness of the method of the present invention is verified by a specific experiment. The experimental content includes:

[0108] (1) Prepare the test dataset

[0109] This example uses three test datasets: the UCSD Pedestrian 2 dataset (Ped2), the CUHK Avenue dataset (Avenue), and the ShanghaiTech dataset (ShanghaiTech). The Ped2 dataset contains 16 training videos and 12 test videos, including 12 abnormal events. Ped2's anomalies are all related to unusual traffic behavior events, such as the presence of cars and bicycles, pedestrians passing on skateboards, etc. The Avenue dataset contains 16 training videos and 21 test videos, including 47 abnormal events involving behaviors such as running and throwing objects. Due to the camera shooting angle, the size of pedestrians in this dataset varies with the depth of field. The ShanghaiTech dataset consists of 330 training videos (274K frames) and 107 test videos (42K frames) shot in 13 scenes, containing 130 complex and diverse abnormal events. Due to the switching between different scenes, it is the most challenging anomaly detection dataset available.

[0110] (2) Parameter settings

[0111] The encoder and decoder used in the basic network framework of this embodiment each contain four layers of convolution operations, such as Figure 2 As shown, the image sizes processed by each layer are 256×256, 128×128, 64×64, and 32×32. The input of the network is 5 frames of images, and the first 4 frames of images are used to predict the 5th frame of image. The network is optimized by the Adam optimizer, and the initial learning rate is lr=2e-4. The memory module in this embodiment is configured with 512 storage items, and the size of the storage items is 32×32. The model of this embodiment is trained on the Ped2, Avenue, and ShanghaiTech datasets for 35, 25, and 2 generations, respectively. In the comparative experiment and ablation experiment, 90, 90, and 10 generations are uniformly trained, respectively. A single end-to-end network is implemented using Pytorch, and experiments are conducted on the NVIDIA GeForce RTX 3090 GPU.

[0112] (3) Model framework process

[0113] Existing memory networks require a large number of generations of iterative training on the data set, and the final effect is not stable; complex data sets contain complex scenes, in which the low-dimensional spatial features will change with the training process, while the number of storage items in the memory module is fixed. This is very likely to cause the memory module to fall into an endless cycle of constantly discarding and relearning old scene features during repeated iterations, resulting in the memory module being unable to play a good guiding role. In this embodiment, the memory module is improved. Figure 3This is a comparison chart of the storage and update of the traditional memory module and the improved memory module used in the present invention. The improved memory module used in the present invention ensures that the memory module can be quickly updated to the ideal state and is not disturbed by complex scenes while maintaining the memory module-guided feature learning to improve detection effects. Figure 1 The network framework diagram of the model of the embodiment of the present invention is shown in FIG. The input of the model is continuous video frames, and the output is the predicted frame. i-4 ~I i The encoder E transforms the complete image into a multi-dimensional small-size feature map, which contains the local spatial features of the video frame. Then, the input feature F in Enter the time-space memory guidance model, and after being guided by the memory module, output feature F out The implicit long-sequence temporal features are enhanced from the cluttered spatial features. These enhanced features are finally reconstructed into a complete prediction frame by the encoder module D.

[0114] (4) Model performance comparison

[0115] Performance evaluation and comparison were performed using the Area Under the Receiver Operation Characteristic (ROC) curve (AUC). The AUC is plotted as the True Positive Rate (TPR) and False Positive Rate (FPR) at different thresholds. A higher AUC indicates better accuracy. Compared to existing detection methods, the proposed method achieved comparable results on the Ped2, Avenue, and ShanghaiTech datasets.

[0116] The method of the present invention is compared with the video anomaly detection method using a traditional memory network. Compared with the traditional memory network that only guides spatial features, the improved memory module used in the method of the present invention has a higher AUC value, which increased by 2.9%, 4.4%, and 4.9% on the Ped2, Avenue, and ShanghaiTech datasets, respectively. This is because the improved memory module calculates and stores spatiotemporal correlation information, and guides feature generation based on the connection between consecutive frames or between local features and global features. The advantage of the improved memory module is obvious in the ShanghaiTech dataset. This is because the spatial features stored in the traditional memory network include some fixed scene features, but the scenes in the ShanghaiTech dataset often change. By introducing the guiding role of temporal correlation, the network becomes easier to process complex scenes. Therefore, the improved memory module in the method of the present invention has a more obvious improvement effect in more complex datasets.

[0117] Further analyze the convergence speed of model training and the stability of the model. Figure 4 The training results of the memory module of the present invention and a traditional memory network at various generations on different test datasets are shown. MemAe represents the traditional memory network, and ST_MemAe represents the improved memory module of the present invention. As can be seen from the figure, the traditional memory network exhibits large fluctuations in discounting overall, indicating poor model training stability. On the ShanghaiTech dataset, severe overfitting occurs in the later stages of training. In contrast, the improved memory module of the present invention is very stable in the later generations, and its detection accuracy far exceeds that of the traditional memory network. On the Ped2 dataset, the improved memory module of the present invention reaches its highest value and stabilizes after training for around 35 generations, while the traditional memory network is relatively stable only between 60 and 80 generations. On the Avenue dataset, the improved memory module of the present invention reaches its highest value and stabilizes after training for 25 generations, while the traditional memory network is only slightly stable between 60 and 70 generations. On the ShanghaiTech dataset, the improved memory module of the present invention reaches its highest value after training for the second generation, and subsequent results are very stable compared to the traditional memory network. It can be seen that the improved memory module used in the method of the present invention is more stable, converges faster, and has higher accuracy.

[0118] Heatmaps are used to show the detection results of normal events and abnormal events. Heatmaps show the difference between input frames and model prediction frames. This example lists several heatmaps generated by model testing in different scenarios, as well as the corresponding real videos. Figure 5 As shown in the figure, it can be seen that under normal circumstances, normal events in the video will only produce small color changes, while when abnormal events occur, obvious color changes will appear in the heat map. This proves the sensitivity of the method of the present invention to the occurrence of abnormal events.

[0119] Extensive ablation experiments were conducted to verify the impact of the improved memory module and the optimization of multiple loss functions in the method of the present invention on the model. MemAE is the original memory network, ST_MemAE is the improved memory module, and BML represents the optimization for the multiple loss function problem. Experiments were conducted on three datasets, and the average AUC results after training stabilization are shown in Table 1. The improved memory module improved the accuracy of the original memory network by 3.3%, 2.2%, and 7.0% on the Ped2, Avenue, and ShanghaiTech datasets, respectively. After further optimization of multiple loss functions, the AUC on the three datasets reached 97.0%, 87.8%, and 76.1%.

[0120] Table 1 Ablation analysis of the improved memory network performance

[0121]

[0122] By drawing the Receiver Operating Characteristic curve (ROC), we can more intuitively show the improvement of each component on the model detection effect. ROC is a commonly used evaluation indicator for anomaly detection. The larger the area under the curve, the better the detection effect. Figure 6 As shown in the figure, MemAE represents the original memory network, ST_MemAE w / o BML represents the use of the improved memory module without multi-loss function optimization, and ST_MemAE w / BML represents the use of the improved memory module and multi-loss function optimization. The figure shows that the area under the curve of ST_MemAE w / o BML is larger than that of MemAE in all three datasets. Furthermore, after adding the BML component, the area under the curve of ST_MemAE w / BML is even larger than that of ST_MemAE w / o BML, further improving detection performance, especially on the most complex ShanghaiTech dataset.

[0123] (5) Detection speed

[0124] Frames Per Second (fps) and AUC are used as comprehensive indicators to measure model efficiency. Figure 7 The fps of the model of the method of the present invention and several other video anomaly detection models are shown. It can be seen from the figure that the fps of the model of the present invention is not the highest, but the AUC reaches the highest 76.1, achieving a good balance between speed and accuracy.

[0125] The present invention also proposes an electronic device, which includes a memory, a processor, and program instructions stored in the memory for the processor to run. The processor executes the program instructions to implement the various steps of a video anomaly detection method based on a spatiotemporal memory network proposed by the present invention.

[0126] The program code for implementing the methods of the present application can be written in any combination of one or more programming languages. Such program code can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the program code is executed by the processor or controller, the functions / operations specified in the flow charts and / or block diagrams are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0127] The specific implementation scheme described above further illustrates in detail the purpose, technical solutions and beneficial effects of the present invention. It should be understood that the above is only a specific implementation scheme of the present invention and is not intended to limit the scope of the present invention. Any equivalent changes and modifications made by any technician in this field without departing from the concept and principle of the present invention should fall within the scope of protection of the present invention.

Claims

1. A video anomaly detection method based on spatiotemporal memory guided network, characterized in that: It includes the testing phase and the training phase: During the training phase, a video anomaly detection model is constructed, which includes an autoencoder and a memory module. Video frames of normal behavior are input into the video anomaly detection model, and the encoder in the autoencoder outputs a multidimensional feature sequence. The multidimensional feature sequence is input into the memory module, and the memory module calculates the spatiotemporal relationship information between low-dimensional features in the multidimensional feature sequence. The spatiotemporal relationship information is read and updated using the memory items of the memory module as a carrier, and an uncertainty function is used to optimize the weight balance of the loss function involved in the memory module update. After being guided by the memory module, the time-series strongly correlated features are obtained, which are input into the decoder in the autoencoder to predict the next video frame and update the video anomaly detection model; During the testing phase, the test video frames are input into the trained video anomaly detection model, and the anomaly score is calculated to determine whether the test video has an anomaly. The process of calculating the spatiotemporal relationship information between low-dimensional features in the multi-dimensional feature sequence by the memory module includes: At time t, the memory module keeps the multidimensional feature sequence in its original size and expands it into a sequence Seq1 along the channel dimension, and combines Seq1 with the three parameter matrices W q 、W k 、W v Multiply them together to get three eigenvectors q i 、k i 、v i Seq2, where q i is the temporal correlation query item, k i is the temporal correlation matching vector, v i It is the characteristic information passed down from the i-th element in Seq1; the q of the i-th element in Seq1 i Doing a dot product operation with the temporal correlation matching vector of other elements calculates the initial correlation score, as shown in the following formula: c ij =q i k j Among them, c ij represents the initial correlation score calculated between the i-th element at time t and other elements j, which is the read item of the memory module; When the correlation calculation of element i with all other elements is completed, all c ij Composing the initial correlation sequence Seq 3,i ; During the training phase, all sequences use the same parameter matrix W q 、W k 、W v Perform parallel computing; The process of reading and updating the spatiotemporal relationship information using the memory items of the memory module as a carrier and obtaining the time series strong correlation features after being guided by the memory module includes: The memory module is used to store the initial correlation sequence Seq 3,i Read; item c will be read ij Compare with each stored memory item in the memory module, the comparison step first calculates the read item c ij The Euclidean distance between each stored memory item in the memory module is as follows: Among them, d l2 represents the Euclidean distance, c m a relevance score stored for the stored memory item; When the sequence Seq 3,i After calculating the Euclidean distance between all elements and the storage memory item, the storage memory item with the shortest Euclidean distance is c ij The best matching memory item c m_out , combine the best matching storage memory items into the best matching sequence Store the best match in memory c m_out with c ij Perform the addition operation to obtain the normalized correlation information c′ ij ; Represents the normalized correlation information c′ ij The vector is divided by the common dimension d of each element in the sequence Seq1, and a Softmax operation is applied to obtain the final relevance score s of element i in Seq1 to all elements ij , the calculation formula is as follows: Final relevance score s ij Composition sequence Seq 4,i ; Seq 4,i The elements in Seq2 are the same as the elements in v i One-to-one correspondence, weighted summation, the calculation formula is as follows: where e i is the spatiotemporal strong correlation feature obtained after the memory module is guided, s ij is the final correlation score, v j is the vector in the multidimensional feature sequence output by the encoder, n is the number of channels of the multidimensional feature sequence output by the encoder; all e i Composed of sequence Seq out , Seq out as input to the decoder.

2. The video anomaly detection method based on spatiotemporal memory guided network according to claim 1 is characterized in that: The autoencoder includes an encoder and a decoder, the feature dimensions of the encoder output and the feature dimensions of the decoder input correspond to the dimensions of the input video frame and the predicted video frame; the encoding and decoding processes correspond to four layer-by-layer dimensionality reduction and dimensionality increase operations, respectively, and two convolution operations are performed before each dimensionality change; jump connection operations are applied between the encoding layer and the decoding layer with the same dimension to compensate for the information loss caused by deep convolution.

3. The video anomaly detection method based on spatiotemporal memory guided network according to claim 2 is characterized in that: The process of optimizing the weight balance of the loss function involved in the memory module update includes: After calculating the Euclidean distance between the read memory item and all stored memory items, the best matching stored memory item c is obtained. m_out Then, minimize c m_out and read item c ij The Euclidean distance between them is used to obtain the temporal feature shrinkage loss L1, as shown in the following formula: The temporal feature separation loss L2 is used to prevent the second matching storage memory item from being too similar to the best matching storage memory item. The calculation formula of the temporal feature separation loss L2 is as follows: where c m_out Store the memory item for the best match, c m_sec storing a memory item for the second match; A homogeneous uncertainty function L3 is used to optimize the weight between L1 and L2, as shown in the following formula: Among them, σ1 and σ2 represent learnable parameters; Minimize prediction frame With the target frame I t+1 The Euclidean distance between them is used to obtain the prediction loss L pre As shown in the following formula: According to L3 and L pre Get the target loss function L total , as shown below: L total =L pre +λL3 Where λ is an adjustable parameter.

4. The video anomaly detection method based on spatiotemporal memory guided network according to claim 3 is characterized in that: The calculation method of the anomaly score is: The peak signal-to-noise ratio calculation formula for measuring the quality of the reconstructed image is as follows: Where PSNR(·) represents the peak signal-to-noise ratio, I t+1 represents the real video frame at time t+1, represents the predicted video frame at time t+1, express The maximum pixel value of N represents the total number of pixels contained in a video frame image; When testing, the trained memory items are kept unchanged. The timing characteristics of abnormal events are different from the timing characteristics of normal events stored in the memory items. The difference is expressed as S t , S t The calculation formula is as follows: Among them, c m_out represents the best matching storage memory item, c ij Represents the read item, m represents the number of read items; The peak signal-to-noise ratio and S t After the abnormal score is normalized, the abnormal score S is obtained total The calculation formula for (t) is as follows: where N(·) represents the anomaly score normalization operation, and w1 and w2 are adjustable parameters.

5. An electronic device comprising a memory, a processor, and program instructions stored in the memory and executable by the processor, characterized in that: The processor executes the program instructions to implement the steps of the method according to any one of claims 1 to 4.

Images