Dual dynamic memory network construction method for weakly supervised video anomaly detection

Abstract

The application provides a dual dynamic memory network construction method for weakly supervised video anomaly detection. The method constructs a memory network containing normal and abnormal memory modules, can extract long-term normal and abnormal modes in the video through memory items, and dynamically updates the features through the read-write operation of the memory network. The number of memory items in the memory module is dynamically variable to adapt to different video monitoring scenes. During training, a mode separation loss is designed to increase the discrimination between each memory item, so that each mode has diversity. In online application, since the network can consider long-term video information, it does not need to fuse the features of future frames to realize anomaly detection, and can realize accurate and real-time online detection.

Description

Technical Field

[0001] This invention relates to the field of video anomaly detection, and more specifically, to a method for constructing a dual dynamic memory network for weakly supervised video anomaly detection. Background Technology

[0002] With the rapid development of my country's economy and network information technology, video surveillance systems have become widely used and play a significant role in traffic control, industrial inspection, and public safety. However, most surveillance equipment can only record video footage and cannot effectively analyze it. Detecting abnormal events in the video usually requires manual intervention. Manual processing incurs significant manpower costs, resulting in the underutilization of the rapidly growing amount of surveillance video. Furthermore, the video is often only reviewed afterward, leading to a delay in responding to abnormal events. Therefore, establishing automated and intelligent video surveillance systems can promptly detect anomalies in the video.

[0003] In recent years, deep learning technology has developed rapidly and has been widely applied in the field of video anomaly detection. Early work in video anomaly detection generally employed unsupervised methods, using a training set containing only normal samples to fit the distribution of normal samples and relying on reconstruction errors to determine whether anomalies occurred. However, this approach can have a high false positive rate for normal events not present in the training set. Furthermore, in many real-world scenarios, video backgrounds are highly complex, and the types of video anomalies are diverse; relying solely on reconstruction errors is insufficient to accurately determine whether an anomaly has occurred. Moreover, because autoencoders typically employ deep neural network structures with strong fitting capabilities, they may possess strong reconstruction capabilities for both normal and anomalous data, making it difficult to effectively identify anomalous samples.

[0004] To further address the problem of video anomaly detection in complex scenarios, recent research has focused on weakly supervised video anomaly detection. Unlike unsupervised video anomaly detection, weakly supervised methods can be trained using both normal and anomalous videos. However, the specific normality or abnormality of each frame in an anomalous video is unknown; only video-level labels are available, lacking accurate frame-level labels. Since labeling every frame is unnecessary, significant resources are saved, aligning with real-world needs. Compared to unsupervised methods, weakly supervised video anomaly detection methods can handle more complex and varied video backgrounds, a wider variety of anomalies, and more realistic scenarios.

[0005] To record long-term features of videos, some works employ memory networks for unsupervised video anomaly detection. For example, reference 1 [Gong D, Liu L, Le V, et al. Memorizing normality to detect anomaly: Memory-augmented deep autoencoder for unsupervised anomaly detection[C]. Proceedings of the IEEE / CVF International Conference on Computer Vision, 2019: 1705-1714.] uses memory modules to enhance deep autoencoders, thus avoiding the strong reconstruction capability of autoencoder models for anomalous data. Reference 2 [Park H, Noh J, Ham B. Learning memory-guided normality for anomaly detection[C]. Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2020: 14372-14381.] argues that prototype features are insufficient to represent the multiple modalities of normal data. Therefore, they propose feature compaction loss and feature separation loss to constrain features and memory terms. However, a single memory module is insufficient to record the hidden information contained in a video, and the lack of frame-level labels makes it impossible to accurately record the category information of video frames. Therefore, video anomaly detection methods based on memory networks are difficult to apply to weakly supervised scenarios. Furthermore, the number of memory items in existing memory networks is manually set and kept fixed during training, resulting in limited recorded information and an inability to dynamically adapt to different video anomaly detection scenarios. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the prior art. In a first aspect, it provides a method for constructing a dual dynamic memory network for weakly supervised video anomaly detection, comprising:

[0007] Construct a training dataset where each sample contains multiple non-overlapping video segments obtained from a video segmentation and the corresponding video labels;

[0008] A feature extractor is used to extract features from the video segments of each sample, and a first classifier is used to obtain the anomaly score of each video segment. Based on the anomaly score, confidence samples are sampled to obtain the sample feature set of each sample.

[0009] Construct a dual dynamic memory network, wherein the memory network includes a normal memory module PN and abnormal memory module P A These are used to store memory items for normal videos and memory items for abnormal video samples, respectively.

[0010] The sample feature set of each sample is fed into a dual dynamic memory network, and a read operation is performed based on the dual dynamic memory network. Specifically, the features of the sample feature set for normal video samples are compared with those of the normal memory module P. N The similarity of memory items is used as a weight to weight the memory items to obtain the enhanced features. For the feature set of abnormal video samples, the features are compared with the abnormal memory module P. A The similarity of memory items is used as a weight to weight the memory items to obtain enhanced features; and based on the similarity between the input features and the current memory items, features that the current memory items cannot represent are retained as new memory items. At the same time, existing memory items are combined and updated using features with high similarity. This process is applied to the normal memory module P. N and abnormal memory module P A The update is written, and then the enhanced anomaly score is output using a second classifier based on the enhanced features;

[0011] Construct a loss function, which includes at least the classification losses of the first classifier and the second classifier, and the normal memory module P. N and abnormal memory module P A Modal separation loss of mid-memory items and normal memory module P N and abnormal memory module P A The modality separation loss between the two is used; training is performed with the goal of minimizing the loss function to obtain a well-trained dual dynamic memory network.

[0012] Furthermore, the feature extractor is a pre-trained video feature extractor.

[0013] Furthermore, the normal memory module P N and abnormal memory module P A The memory items stored in the memory are initialized using a Gaussian function.

[0014] Furthermore, for the feature set of normal video samples, the features are compared with those of the normal memory module P. N The similarity of memory items is used as a weight to weight the memory items, resulting in the enhanced features. Represented as:

[0015]

[0016] In the formula, the subscripts i and t represent the t-th segment in the i-th video, and the superscripts h and A represent that the video is a normal video and an abnormal video, respectively; K is the current normal memory module P.N The number of items in memory, It is the current normal memory module P N The k-th memory item in the middle, It is the feature of the t-th segment in the feature set of the i-th normal video sample. For memory items Weighting coefficients;

[0017] The features of the sample feature set of abnormal video samples are based on the features and the abnormal memory module P. A The enhanced feature is obtained by weighting the memory items with the similarity of the memory items, and is represented as:

[0018]

[0019] In the formula, K is the current exception memory module P. A The number of items in memory, Current exception memory module P A The k-th memory item in the middle, It is the feature of the t-th segment in the sample feature set of the i-th abnormal video sample. For memory items The weighting coefficients.

[0020] Furthermore, weighting coefficients It was obtained through a sparsification operation, as follows:

[0021] Calculation features With memory items The similarity is calculated and normalized to obtain the initial weight coefficients.

[0022] Then, set a threshold λ, set the weight coefficients of items with weight coefficients below the threshold to 0, and renormalize to obtain the weight coefficients. The superscript * indicates that the video is a normal video or an abnormal video, respectively.

[0023] Furthermore, the step of obtaining the sample feature set for each sample by sampling confidence samples based on the anomaly score specifically involves selecting the features corresponding to the T video segments with anomaly scores closest to 1 to form the sample feature set.

[0024] Furthermore, based on the similarity between the input features and the current memory item, features that the current memory item cannot represent are retained as new memory items. Simultaneously, existing memory items are combined and updated using features with high similarity. Specifically:

[0025] For the features of the input video segment calculate With memory module P* Similarity of all memory items Take the maximum similarity as maximum similarity Based on the similarity threshold τ, The set U of features less than τ i,l ,Will U is the set of features greater than τ i,h ; set U i,l Features are added as new memory items to memory module P * In the process, the number of memory items is dynamically increased; for currently existing memory items... From set U i,h Selecting and remembering items U is composed of multiple most similar features i,k The set is then weighted based on the weight coefficients of the existing memo items. Update:

[0026]

[0027] In the formula, These are weighting coefficients, expressed as:

[0028]

[0029] The superscript * represents N or A, indicating that the video is normal and abnormal respectively; the subscripts i and t represent the t-th segment in the i-th video; and K is the current memory module P. * The number of items in memory.

[0030] Furthermore, the normal memory module P N and abnormal memory module P A The modal separation loss of the middle memory item is achieved through the normal memory module P. N and abnormal memory module P A The similarity between each memory item and other memory items is calculated internally to obtain the result;

[0031] Normal memory module P N and abnormal memory module P A The modal separation loss between normal and abnormal memory modules is constructed by calculating the similarity between normal and abnormal memory items.

[0032] Furthermore, the normal memory module P N and abnormal memory module P A The modal separation loss of the middle memory term is expressed as:

[0033]

[0034] Normal memory module P N and abnormal memory module P A The modal separation loss between them is expressed as:

[0035]

[0036] Subscripts k and k' represent memory module P * Index of memory items This indicates that the k-th memory term, with the superscript * representing N or A, signifies that the video is a normal video and an abnormal video, respectively; ||*|| represents the L2 norm of the vector.

[0037] Secondly, the present invention also provides a video anomaly detection method, comprising:

[0038] The video to be detected is acquired and segmented into multiple non-overlapping video segments. The video segments are input into a feature extractor to obtain multiple features. The multiple features are input into a dual dynamic memory network trained by the above method. The memory items of the normal memory module and the abnormal memory module are read and weighted to obtain the enhanced features. The enhanced features are then input into a second classifier to obtain the anomaly detection result.

[0039] The beneficial effects of this invention are as follows: This invention proposes a dual dynamic memory network construction method and a video anomaly detection method for weakly supervised video anomaly detection. The method constructs a memory network that simultaneously includes normal and anomaly memory modules, enabling the extraction of long-term normal and anomaly modalities from the video through memory items, and dynamically updating features through read and write operations of the memory network. The number of memory items in the memory modules is dynamically variable to adapt to different video surveillance scenarios. During network training, a modality separation loss is designed to increase the discriminative power between memory items, thus enabling modality diversity. In online applications, because the network can consider long-term video information, anomaly detection can be achieved without fusing features from future frames, enabling accurate and real-time online detection. Comparison with existing methods on three public datasets shows that the proposed method outperforms previous methods and achieves better detection results. Attached Figure Description

[0040] Figure 1 This is a schematic diagram of the overall process of the method of the present invention;

[0041] Figure 2 This is a schematic diagram of a memory network read operation.

[0042] Figure 3 A schematic diagram of the write operation to the memory network; Detailed Implementation

[0043] The present invention will be further described below with reference to embodiments. The description of the embodiments below is only for the purpose of helping to understand the present invention. It should be noted that those skilled in the art can make several improvements and modifications to the present invention without departing from the principle of the present invention, and these improvements and modifications also fall within the protection scope of the claims of the present invention.

[0044] like Figure 1 As shown, this invention addresses the limitations of existing methods in recording long-term normal and abnormal modalities in videos and their inability to be applied online. It provides a weakly supervised dual dynamic memory network construction method and a video anomaly detection method. By designing a memory network containing normal and abnormal memory modules, this network can record long-term normal and abnormal modalities in the video using memory items in the memory modules, and dynamically update the memory items according to different anomaly detection scenarios, thus better adapting to different scenarios. By fusing sample features with event modal features recorded by the memory network, the correlation between sample features and related events can be enhanced, thereby better identifying anomalies. Compared to other video anomaly detection methods that require fusing information from video frames over a future period, this invention's method does not require information from future frames for auxiliary prediction, thus enabling online application.

[0045] The present invention provides a method for constructing a dual dynamic memory network for weakly supervised video anomaly detection, comprising the following steps:

[0046] Step 1: Construct a training dataset. Each sample contains multiple non-overlapping video segments obtained from a video segmentation and their corresponding video labels; details are as follows:

[0047] 1) Collect surveillance video captured by surveillance cameras;

[0048] 2) Based on whether the surveillance video contains abnormal events, classify the video into normal videos and abnormal videos, and construct a training set with M training videos, denoted as M0. It also has the label for each training video, represented as In this embodiment, y is defined i =0 indicates that the video V i It's a normal video, y i =1 indicates that the video V i This is an abnormal video; each video is divided into many non-overlapping segments. In this embodiment, each segment contains 16 video frames, denoted as follows: and Where the subscripts i and t represent the t-th segment in the i-th video, and the superscripts N and A represent that the video is a normal video and an abnormal video, respectively.

[0049] As an optional implementation, a test set can also be built simultaneously for testing performance after training is completed.

[0050] Step 2: Use a feature extractor to extract features from the video segments composed of consecutive video frames of each sample, and use a first classifier to obtain the anomaly score of each video segment. Confidence samples are then sampled based on the anomaly scores.

[0051] As an optional implementation, the feature extractor can be a pre-trained video feature extractor, which can greatly reduce the training time and complexity of this invention. In this embodiment, a pre-trained three-dimensional convolutional neural network I3D is used as the feature extractor to obtain 2048-dimensional features of the video segment, denoted as... and

[0052] The first classifier can be a conventional classification network, such as a classification network composed of fully connected layers and a sigmoid activation function, based on features. and The abnormal score between 0 and 1 is obtained for each small segment in both normal and abnormal videos, and is denoted as . and The closer the anomaly score is to 1, the more likely the video segment is to be anomaly. Based on this, selecting the features corresponding to T video segments with anomaly scores closest to 1 to form a sample feature set can effectively improve training performance. Specifically, the anomaly score in the anomaly videos... Sampling yields T A The feature set whose anomaly score is closest to 1 is used as the confidence positive sample feature set. Based on abnormal scores in normal videos Sampling yields T N Features with anomaly scores close to 1 are considered as a set of negative sample features that are difficult to classify.

[0053] Step 3: Construct a dual dynamic memory network. To record long-term normal and abnormal modes in the video separately, the constructed memory network also includes a normal memory module P. N and abnormal memory module P A Set the initial number K of memory items in the normal memory module and the abnormal memory module, and denote the memory items in the normal and abnormal memory modules as follows: and in Similar to the feature dimensions of video segments, a Gaussian function can generally be used to initialize each memory item.

[0054] Step 4: Feed the sampled confidence samples into the memory network. Perform read and write operations on the memory network to obtain enhanced features and dynamically update the memory items in the memory network. Based on the enhanced features, use a second classifier to obtain enhanced anomaly scores. At the same time, construct a loss function and train the network with the goal of minimizing the loss function to obtain a trained dual dynamic memory network.

[0055] like Figure 2 As shown, the memory network is read to obtain enhanced video segment features. This step is achieved through the following sub-steps:

[0056] 1) The positive sample feature set described in step 1 and negative sample feature set Features of normal and abnormal video segments and The features of the sample feature set for normal video samples are input into the corresponding normal and abnormal memory modules respectively. The features are then compared with those in the normal memory module P. N The similarity of memory items is used as a weight to weight the memory items to obtain the enhanced features. For the feature set of abnormal video samples, the features are compared with the abnormal memory module P. A The similarity of memory items is used as a weight to weight the memory items, resulting in enhanced features; specifically, taking a normal video clip as an example, firstly, the similarity of the memory items is calculated. With the corresponding normal memory module P N Memory items in Similarity:

[0057]

[0058] in Representative sample features With memory items The cosine similarity, where ||*|| represents the L2 norm of the vectors. represent The transpose of .

[0059] 1.1) Regarding and all memory items similarity Performing the Softmax operation yields For memory items Weighting coefficients:

[0060]

[0061] 1.2) Preferably, to avoid modal confusion caused by an excessive number of memory items, a threshold screening method is used to remove samples with excessively low weight coefficients. Specifically, a threshold λ is set, and items with weight coefficients lower than the threshold are set to 0 to obtain new weight coefficients:

[0062]

[0063] By sparsifying the weighting coefficients, interference from too many irrelevant memory terms can be avoided.

[0064] 1.3) Change the new weighting coefficients Renormalization is performed using the Softmax operation shown in formula (2). Representative sample features The weight vector for all memory terms.

[0065] 1.4) The weighting coefficient reflects the degree of matching between sample features and the memory items recorded in the memory module. Since each memory module only records a single modality, by finding the memory items corresponding to the sample features and weighting them together, new sample features are obtained, which can more accurately reflect the modality to which the sample belongs. The formula for calculating the new features is as follows:

[0066]

[0067] 1.5) In order to preserve the original normal segment features The information will With new features The final enhanced features of the normal samples are obtained by splicing them together.

[0068] 1.6) Features of abnormal video segments Similarly, the above steps are used to obtain the final anomalous sample enhancement features.

[0069] Enhance sample features and The enhanced anomaly scores are obtained by inputting them into the second classifier, and are denoted as follows: and This serves as the final anomaly detection result.

[0070] Write to the memory network to obtain the updated memory items, such as Figure 3 As shown, this step involves considering the features of the input video segment. calculate With memory module P * Similarity of all memory items The superscript * represents N or A, indicating that the video is normal or abnormal, respectively. The maximum similarity score is taken as the score. maximum similarity Based on the similarity threshold τ, The set U of features less than τ i,l ,Will U is the set of features greater than τ i,h ; set U i,l Features are added as new memory items to memory module P * In the process, the number of memory items is dynamically increased; for currently existing memory items... From set U i,h Selecting and remembering items U is composed of multiple most similar features i,k The set is then weighted based on the weight coefficients of the existing memo items. Update:

[0071] Taking the update of a memory item in a normal memory module as an example, this step is implemented through the following sub-steps:

[0072] 1) Features of normal video clips Calculate according to formula (1) Similarity to all memory items in a normal memory module Take the maximum similarity as maximum similarity Set a similarity threshold τ to divide the samples into two parts. Let U be the set of features less than τ. i,l ,Will Let U be the set of features greater than τ. i,h .

[0073] 2) U i,l This represents the feature index of all samples with a maximum similarity below a threshold. Since these sample features have low similarity to all memory items, it can be understood that there are currently no memory items that can represent the modality implied by these sample features. Therefore, these sample features are directly added to the memory module as new memory items, dynamically increasing the number of memory items.

[0074] 3) U i,h The index represents the feature indices of all samples with a maximum similarity higher than a threshold, indicating the existence of memory terms that can characterize the modalities of these sample features. Therefore, these sample features are not repeatedly added to the memory modalities but are used for updating the memory terms. This step is implemented through the following sub-steps:

[0075] 3.1) For existing memory items From U i,h Selected sample set U i,k U i,kSamples in With memory items The similarity in the similarity set The largest, that is, these samples believe It is most similar to itself, so U can be used. i,k Update the sample features in the data If no sample exists With memory items The similarity in the similarity set If the largest value is found, then the memory term is found. No updates are made in the current round. First, the weight coefficients of the memory terms and sample features are calculated:

[0076]

[0077] 3.2) The updated memory term can be obtained by weighting and combining the weights based on the weight coefficients. The calculation formula is as follows:

[0078]

[0079] The same steps are used to update the memory items in the abnormal memory module to obtain the updated abnormal memory items.

[0080] The updated memory entries are used in the next iteration to remember the network's reading process.

[0081] Construct a classification loss function. In this invention, the loss function includes at least the classification losses of the first classifier and the second classifier, and the normal memory module P. N and abnormal memory module P A Modal separation loss of middle memory items and normal memory module P N and abnormal memory module P A The modal separation loss between them; specifically, this step is achieved through the following sub-steps:

[0082] 1) Anomaly score obtained in step 1 and The labels for samples in normal videos are set to 0, and the labels for samples in abnormal videos are set to 1. A classification loss is constructed; for example, a binary cross-entropy classification loss function can be used.

[0083]

[0084] 2) Similarly, based on the enhanced anomaly score obtained in step 4 and Constructed binary cross-entropy classification loss function

[0085]

[0086] 3) Adding the two together yields the comprehensive classification loss L. CE :

[0087]

[0088] 4) To enhance the distinguishability between memory items in the memory module, a modality separation loss function is constructed, including:

[0089] 4.1) To ensure the separation of memory items within a memory module as much as possible, the similarity between each memory item and other memory items is calculated within the memory module, minimizing the similarity. This yields the modality separation loss function within the memory module. Any function that can calculate the similarity between each memory item and other memory items is acceptable. For example, the calculation formula is as follows:

[0090]

[0091] 4.2) To separate normal and abnormal memory items as much as possible, the similarity between normal and abnormal memory items is calculated between normal and abnormal memory modules. This similarity is then minimized to obtain the modality separation loss function between memory modules. Similarly, any function that can calculate the similarity between normal and abnormal memory items in normal and abnormal memory modules is acceptable. For example, the calculation formula is as follows:

[0092]

[0093] 4.3) The combined modal separation loss function is the sum of the two:

[0094]

[0095] Where λ intra and λ inter Used to control the ratio of the two.

[0096] 5) Loss function L for overall model training total The calculation formula is:

[0097] L total =λ CE L CE +λ SEP L SEP (13)

[0098] Through the weighting coefficient λ CE and λ SEP Control classification loss function L CEand mode separation loss function L SEP The proportion of.

[0099] 6) Set the sample set size, learning rate, learning rate decay value, and optimizer for each training round, and use the loss function L. total The model is trained to obtain a well-trained dual dynamic memory network.

[0100] This invention also provides a video anomaly detection method. The trained memory network described above can consider long-term video information, thus eliminating the need to fuse features from future frames to achieve anomaly detection, enabling accurate and real-time online detection. Specifically, the detection method is as follows:

[0101] The video to be detected is acquired and segmented into multiple non-overlapping video segments. The video segments are input into a feature extractor to obtain multiple features. These features are then input into a dual dynamic memory network trained using the above method. The memory items of the normal memory module and the abnormal memory module are read and weighted to obtain enhanced features. Finally, these enhanced features are input into a trained second classifier to obtain the anomaly detection result. Specific implementation examples:

[0103] 1. Dataset

[0104] The ShanghaiTech dataset contains video data from multiple perspectives and various scenes, including anomalies caused by sudden movement such as chasing and arguing. The dataset comprises 13 different scenes, each with unique anomalous events. Following standard practice, it is divided into 238 training videos and 199 test videos to suit a weakly supervised setup. The training videos include 175 normal videos and 63 anomalous videos, while the test videos include 155 test videos and 44 anomalous videos.

[0105] The UCF-Crime dataset consists of long stretches of unedited surveillance video, covering 13 real-world crimes. UCF-Crime contains 1900 videos, totaling 128 hours in length. For anomaly detection experiments, the dataset was divided into training and testing sets. The training set contains 1610 videos (800 normal and 810 anomalous); the testing set contains 290 videos (150 normal and 140 anomalous).

[0106] 2. Evaluation Criteria

[0107] The area under the Receiver Operating Characteristic (AUC) curve, abbreviated as AUC-ROC, is used to evaluate the detection performance of the method of this invention. The larger the AUC-ROC value, the better the detection performance.

[0108] 3. Experimental Setup

[0109] Number of abnormal video samples T A =10, Number of normal video samples T N =10, the initial number of normal and abnormal memory items K in the memory network is both set to 10. The weight coefficient threshold λ = 0.02, and the similarity threshold τ = 2. The coefficient λ of the classification loss in the loss function. CE =1, modal separation loss coefficient λ SEP =0.1. The coefficient λ of the intra-modal loss function in the modal separation loss. intra =0.1, intermodal loss function coefficient λ inter =1. The overall model is trained using the Adam optimizer with a learning rate of 1×10⁻⁶. 4 The weight decays to 5×10 -4 The training was performed in 100 rounds with a batch size of 10.

[0110] 4. Experimental Results

[0111] The experimental results were compared with existing methods that can be applied online on the ShangHaiTech dataset and the UCF-Crime dataset, and are shown in Table 1 and Table 2, respectively.

[0112] Table 1 shows the experimental comparison results on the ShanghaiTech dataset.

[0113]

[0114] On the ShangHaiTech dataset, the evaluation results show that the method of this invention achieves a detection accuracy improvement of more than 10% compared to the methods proposed by MLEP and Sultani et al. Furthermore, compared to all methods applicable online, the method of this invention also achieves the best results.

[0115] Table 2 shows the experimental comparison results on the UCF-Crime dataset.

[0116]

[0117] On the UCF-Crime dataset, the results show that the method of this invention outperforms all methods that can be applied online. Compared to the method proposed by Sultani et al., the detection accuracy is improved by 7.66%.

[0118] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A method for constructing a dual dynamic memory network for weakly supervised video anomaly detection, characterized in that, include: Construct a training dataset where each sample contains multiple non-overlapping video segments obtained from a video segmentation and the corresponding video labels; A feature extractor is used to extract features from the video segments of each sample, and a first classifier is used to obtain the anomaly score of each video segment. Based on the anomaly score, confidence samples are sampled to obtain the sample feature set of each sample. Construct a dual dynamic memory network, which includes normal memory modules. and abnormal memory module These are used to store memory items for normal videos and memory items for abnormal video samples, respectively. The sample feature set of each sample is fed into a dual dynamic memory network, and a read operation is performed based on the dual dynamic memory network. Specifically, the features of the sample feature set for normal video samples are compared with those of the normal memory module. The similarity of the memory items is used as a weight to weight the memory items to obtain new features. The features of the sample feature set of normal video samples are concatenated with the corresponding weighted new features to obtain the enhanced features. For the features of the sample feature set of abnormal video samples, the features are combined with the abnormal memory module according to the aforementioned features. The similarity of memory items is used as a weight to weight memory items to obtain new features. The features of the sample feature set of abnormal video samples are concatenated with the corresponding weighted new features to obtain the enhanced features. Based on the similarity between the input features and the current memory items, features that the current memory items cannot represent are retained as new memory items. At the same time, the existing memory items are combined and updated using features with high similarity. This process is applied to normal memory modules. and abnormal memory module The update is written, and then the enhanced anomaly score is output using a second classifier based on the enhanced features; Construct a loss function, which includes at least the classification losses of the first classifier and the second classifier, and the normal memory module. and abnormal memory module Modal separation loss of mid-memory items and normal memory modules and abnormal memory module The modality separation loss between the two is used; training is performed with the goal of minimizing the loss function to obtain a well-trained dual dynamic memory network.

2. The method according to claim 1, characterized in that, The feature extractor is a pre-trained video feature extractor.

3. The method according to claim 1, characterized in that, The normal memory module and abnormal memory module The memory items stored in the memory are initialized using a Gaussian function.

4. The method according to claim 1, characterized in that, The features of the sample feature set for normal video samples are based on the features and the normal memory module. The similarity of memory items is used as a weight to weight the memory items, resulting in the enhanced features. , represented as: In the formula, the subscript , Representing the The first video A segment, superscript This indicates that the video is a normal video; K is the current normal memory module. The number of items in memory, This is the current normal memory module. The k-th memory item in the middle, It is the set of sample features of the i-th normal video sample. Features of each segment For memory items Weighting coefficients; The features of the sample feature set for abnormal video samples are based on the features and the abnormal memory module. The enhanced feature is obtained by weighting the memory items with the similarity of the memory items, and is represented as: In the formula, superscript The video in question is an abnormal video; K is the current abnormal memory module. The number of items in memory, Current Exception Memory Module The k-th memory item in the middle, It is the set of sample features of the i-th abnormal video sample. Features of each segment For memory items The weighting coefficients.

5. The method according to claim 4, characterized in that, Weighting coefficient It was obtained through a sparsification operation, as follows: Calculation features With memory items The similarity is calculated and normalized to obtain the initial weight coefficients. ; Reset the threshold Items with weight coefficients below the threshold are set to 0 and then renormalized to obtain the weight coefficients. The superscript * represents or These represent the video as normal and the video as abnormal, respectively.

6. The method according to claim 1, characterized in that, The step of obtaining the sample feature set for each sample by sampling confidence samples based on the anomaly score is as follows: Select the features corresponding to the T video segments with anomaly scores closest to 1 to form the sample feature set.

7. The method according to claim 1, characterized in that, The similarity between the input features and the current memory item is used to retain features that the current memory item cannot represent as new memory items. Simultaneously, existing memory items are updated by combining features with high similarity. Specifically: For the features of the input video segment ,calculate With memory module Similarity of all memory items Take the maximum similarity as maximum similarity Based on similarity threshold ,Will Less than Feature composition set ,Will Greater than Feature composition set ; set Features are added to the memory module as new memory items. In the process, the number of memory items is dynamically increased; for currently existing memory items... From the set Selecting and remembering items Composed of the most similar multiple features The set is then weighted based on the weight coefficients for the existing memory items. Update: In the formula, These are weighting coefficients, expressed as: The superscript * represents or These represent whether the video is normal or abnormal, respectively. , Representing the The first video There are several segments, where K is the current memory module. The number of items in memory.

8. The method according to claim 1, characterized in that, The normal memory module and abnormal memory module Modal separation loss of middle memory items through normal memory modules and abnormal memory module The similarity between each memory item and other memory items is calculated internally to obtain the result; Normal memory module and abnormal memory module The modal separation loss between normal and abnormal memory modules is constructed by calculating the similarity between normal and abnormal memory items.

9. The method according to claim 1, characterized in that, The normal memory module and abnormal memory module The modal separation loss of the middle memory term is expressed as: + Normal memory module and abnormal memory module The modal separation loss between them is expressed as: The subscripts k and k' represent memory modules. Index of memory items This indicates that the k-th memory item, with the superscript *, is... or These represent whether the video is normal or abnormal, respectively. The second norm of a vector.

10. A video anomaly detection method, characterized in that, include: The video to be detected is acquired and segmented into multiple non-overlapping video segments. The video segments are input into a feature extractor to obtain multiple features. The multiple features are input into a dual dynamic memory network trained by the method described in any one of claims 1-9. The memory items of the normal memory module and the abnormal memory module are read and weighted to obtain enhanced features. The enhanced features are then input into a second classifier to obtain the anomaly detection result.

Images